2024-03-14T12:35:45.903 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-14T12:35:45.903 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807 --name yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/orig.config.yaml 2024-03-14T12:35:45.959 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-14T12:35:46.050 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807 branch: wip-yuri-testing-2024-03-12-1240-reef description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7598807' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2024-03-12-1240-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2024-03-12-1240-reef sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8503 sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri-testing-2024-03-12-1240-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiGeaTlEBDf3fT4y2Tk4oW9BNNEBI8QQBc7u0GkFK/HgXXXMYiW2qApndruZ1euBW5ZTI8V5MAjiFnTvyj9wFw= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwDu9CRd7MFONPpr3DW2WyFZg0XARx+QOWfcPT3N/MZZCnEq1K3ArSnz/UQqirSlx11E4o71e2gumdqoWLGqZQ= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-13_19:31:43 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-14T12:35:46.050 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa; will attempt to use it 2024-03-14T12:35:46.051 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks 2024-03-14T12:35:46.051 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-14T12:35:46.053 INFO:teuthology.task.internal:Checking packages... 2024-03-14T12:35:46.076 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2f465319e3c83d48f999325459bcdfd0346449c0' 2024-03-14T12:35:46.076 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:35:46.076 INFO:teuthology.packaging:ref: None 2024-03-14T12:35:46.076 INFO:teuthology.packaging:tag: None 2024-03-14T12:35:46.076 INFO:teuthology.packaging:branch: wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T12:35:46.076 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:35:46.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T12:35:46.297 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-825.g2f465319 2024-03-14T12:35:46.299 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-14T12:35:46.306 INFO:teuthology.task.internal:no buildpackages task found 2024-03-14T12:35:46.306 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-14T12:35:46.333 INFO:teuthology.task.internal:Saving configuration 2024-03-14T12:35:46.353 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-14T12:35:46.366 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T12:35:46.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 12:30:11.944335', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiGeaTlEBDf3fT4y2Tk4oW9BNNEBI8QQBc7u0GkFK/HgXXXMYiW2qApndruZ1euBW5ZTI8V5MAjiFnTvyj9wFw='} 2024-03-14T12:35:46.416 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 12:30:11.942737', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwDu9CRd7MFONPpr3DW2WyFZg0XARx+QOWfcPT3N/MZZCnEq1K3ArSnz/UQqirSlx11E4o71e2gumdqoWLGqZQ='} 2024-03-14T12:35:46.416 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-14T12:35:46.422 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-14T12:35:46.422 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-14T12:35:46.422 INFO:teuthology.run_tasks:Running task console_log... 2024-03-14T12:35:46.518 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7c2913b160>, signals=[15]) 2024-03-14T12:35:46.518 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-14T12:35:46.524 INFO:teuthology.task.internal:Opening connections... 2024-03-14T12:35:46.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:35:46.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:35:46.602 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-03-14T12:35:46.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:35:46.675 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-14T12:35:46.681 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-03-14T12:35:46.703 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-03-14T12:35:46.703 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-14T12:35:46.765 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-14T12:35:46.766 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-14T12:35:46.766 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-03-14T12:35:46.794 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-03-14T12:35:46.817 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-03-14T12:35:46.817 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-03-14T12:35:46.877 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-03-14T12:35:46.877 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2024-03-14T12:35:46.877 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-03-14T12:35:46.877 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-14T12:35:46.878 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-14T12:35:46.878 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-03-14T12:35:46.901 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-14T12:35:46.911 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-14T12:35:46.942 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-14T12:35:46.943 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-03-14T12:35:46.945 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-03-14T12:35:46.965 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-14T12:35:46.976 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-14T12:35:46.976 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-03-14T12:35:47.006 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-03-14T12:35:47.035 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-14T12:35:47.093 INFO:teuthology.run_tasks:Running task kernel... 2024-03-14T12:35:47.108 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-14T12:35:47.108 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-14T12:35:47.109 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-14T12:35:47.109 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-14T12:35:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:35:47.130 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-14T12:35:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:35:47.131 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-14T12:35:47.192 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-489.el8.x86_64 2024-03-14T12:35:47.192 INFO:teuthology.task.kernel:Running kernel on smithi111: 4.18.0-489.el8.x86_64 2024-03-14T12:35:47.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-03-14T12:35:47.193 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-489.el8.x86_64 2024-03-14T12:35:47.194 INFO:teuthology.task.kernel:Running kernel on smithi137: 4.18.0-489.el8.x86_64 2024-03-14T12:35:47.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-03-14T12:35:50.354 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 46 MB/s | 28 MB 00:00 2024-03-14T12:35:50.384 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 49 MB/s | 28 MB 00:00 2024-03-14T12:35:56.064 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-03-14T12:35:56.125 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-03-14T12:35:57.992 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 69 kB/s | 18 kB 00:00 2024-03-14T12:35:58.082 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-03-14T12:35:58.493 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.2 kB 00:00 2024-03-14T12:35:58.565 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 7.2 kB 00:00 2024-03-14T12:35:58.753 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-03-14T12:35:58.792 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-14T12:35:59.909 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-03-14T12:36:02.839 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 16 MB 00:03 2024-03-14T12:36:03.726 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 305 kB/s | 24 kB 00:00 2024-03-14T12:36:06.622 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 427 kB/s | 24 kB 00:00 2024-03-14T12:36:06.723 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-14T12:36:06.723 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-14T12:36:06.824 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 4.18.0-544.el8 baseos 10 M 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 4.18.0-544.el8 baseos 44 M 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 4.18.0-544.el8 baseos 36 M 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout:Install 3 Packages 2024-03-14T12:36:06.825 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:06.826 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 90 M 2024-03-14T12:36:06.826 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 97 M 2024-03-14T12:36:06.826 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:36:07.802 INFO:teuthology.orchestra.run.smithi111.stdout:(1/3): kernel-4.18.0-544.el8.x86_64.rpm 11 MB/s | 10 MB 00:00 2024-03-14T12:36:08.769 INFO:teuthology.orchestra.run.smithi111.stdout:(2/3): kernel-modules-4.18.0-544.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-03-14T12:36:09.270 INFO:teuthology.orchestra.run.smithi111.stdout:(3/3): kernel-core-4.18.0-544.el8.x86_64.rpm 18 MB/s | 44 MB 00:02 2024-03-14T12:36:09.270 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:36:09.270 INFO:teuthology.orchestra.run.smithi111.stdout:Total 37 MB/s | 90 MB 00:02 2024-03-14T12:36:09.619 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-14T12:36:09.619 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-14T12:36:09.718 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 4.18.0-544.el8 baseos 10 M 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 4.18.0-544.el8 baseos 44 M 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 4.18.0-544.el8 baseos 36 M 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:09.719 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout:Install 3 Packages 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 90 M 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 97 M 2024-03-14T12:36:09.720 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:36:09.950 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:36:10.349 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:36:10.349 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:36:10.507 INFO:teuthology.orchestra.run.smithi137.stdout:(1/3): kernel-4.18.0-544.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2024-03-14T12:36:11.523 INFO:teuthology.orchestra.run.smithi137.stdout:(2/3): kernel-modules-4.18.0-544.el8.x86_64.rpm 21 MB/s | 36 MB 00:01 2024-03-14T12:36:11.593 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:36:11.611 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:36:12.015 INFO:teuthology.orchestra.run.smithi137.stdout:(3/3): kernel-core-4.18.0-544.el8.x86_64.rpm 19 MB/s | 44 MB 00:02 2024-03-14T12:36:12.015 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:36:12.015 INFO:teuthology.orchestra.run.smithi137.stdout:Total 39 MB/s | 90 MB 00:02 2024-03-14T12:36:12.703 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:36:13.128 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:36:13.128 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:36:14.365 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:36:14.396 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:36:14.401 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:36:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:17.220 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:36:17.692 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:18.423 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:18.835 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:21.608 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:22.272 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:25.049 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:25.334 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:44.458 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:44.481 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:44.826 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:44.826 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:44.826 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:45.296 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:45.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:45.297 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:36:45.297 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-4.18.0-544.el8.x86_64 kernel-core-4.18.0-544.el8.x86_64 2024-03-14T12:36:45.297 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-4.18.0-544.el8.x86_64 2024-03-14T12:36:45.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:45.297 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:36:45.443 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-14T12:36:45.659 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-14T12:36:45.659 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-14T12:36:45.659 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-544.el8.x86_64 2024-03-14T12:36:45.659 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-14T12:36:45.659 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-14T12:36:45.659 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-14T12:36:45.660 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-03-14T12:36:45.660 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-03-14T12:36:45.869 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-14T12:36:45.869 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-14T12:36:45.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.02 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 129.el8 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Group : System Environment/Base 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 9265586 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-03-14T12:36:45.903 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Relocations : (not relocatable) 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : CentOS BuildSystem 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-03-14T12:36:45.904 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:36:45.905 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-03-14T12:36:45.905 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-14T12:36:45.905 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 4.18.0-544.el8.x86_64 2024-03-14T12:36:45.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-14T12:36:47.310 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:47.334 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:47.540 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-03-14T12:36:47.685 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-4.18.0-544.el8.x86_64 1/3 2024-03-14T12:36:47.686 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-4.18.0-544.el8.x86_64 2/3 2024-03-14T12:36:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-4.18.0-544.el8.x86_64 3/3 2024-03-14T12:36:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:48.156 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:36:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-4.18.0-544.el8.x86_64 kernel-core-4.18.0-544.el8.x86_64 2024-03-14T12:36:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-4.18.0-544.el8.x86_64 2024-03-14T12:36:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:48.157 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:36:48.344 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-14T12:36:48.588 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-14T12:36:48.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-14T12:36:48.589 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-544.el8.x86_64 2024-03-14T12:36:48.589 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-14T12:36:48.589 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-14T12:36:48.589 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-14T12:36:48.589 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-03-14T12:36:48.589 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-03-14T12:36:48.801 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-4.18.0-544.el8.x86_64 2024-03-14T12:36:48.801 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-544.el8.x86_64 2024-03-14T12:36:48.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.02 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 129.el8 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Group : System Environment/Base 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 9265586 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-14T12:36:48.836 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Relocations : (not relocatable) 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : CentOS BuildSystem 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:36:48.837 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-03-14T12:36:48.838 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-14T12:36:48.838 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 4.18.0-544.el8.x86_64 2024-03-14T12:36:48.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-14T12:36:49.861 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-03-14T12:36:49.862 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-03-14T12:36:49.892 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.nxFFtU1re0 2024-03-14T12:36:49.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nxFFtU1re0 2024-03-14T12:36:49.966 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.nxFFtU1re0 2024-03-14T12:36:50.116 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.nxFFtU1re0 is 6KB 2024-03-14T12:36:50.164 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.nxFFtU1re0 2024-03-14T12:36:50.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-03-14T12:36:50.254 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-14T12:36:50.254 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-14T12:36:50.254 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-14T12:36:50.254 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64.conf 2024-03-14T12:36:50.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64 2024-03-14T12:36:50.407 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-03-14T12:36:50.703 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-03-14T12:36:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-03-14T12:36:52.936 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-03-14T12:36:53.019 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.kpdcf8BrhB 2024-03-14T12:36:53.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kpdcf8BrhB 2024-03-14T12:36:53.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.kpdcf8BrhB 2024-03-14T12:36:53.236 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.kpdcf8BrhB is 6KB 2024-03-14T12:36:53.283 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.kpdcf8BrhB 2024-03-14T12:36:53.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-03-14T12:36:53.377 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-14T12:36:53.378 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-14T12:36:53.378 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-14T12:36:53.378 INFO:teuthology.orchestra.run.smithi137.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64.conf 2024-03-14T12:36:53.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-544.el8.x86_64 2024-03-14T12:36:53.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-03-14T12:37:20.435 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-14T12:37:20.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:37:20.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:37:23.570 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-14T12:37:23.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-14T12:37:23.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:37:38.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-14T12:37:47.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:37:47.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:37:51.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-14T12:38:03.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:38:03.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:38:06.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-14T12:38:21.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:38:21.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:38:23.574 DEBUG:teuthology.orchestra.remote:timed out 2024-03-14T12:38:32.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-14T12:38:32.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:38:33.023 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-14T12:38:33.356 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-14T12:38:33.356 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-544.el8.x86_64"... 2024-03-14T12:38:33.356 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-03-14T12:38:33.373 INFO:teuthology.orchestra.run.smithi137.stdout:4.18.0-544.el8.x86_64 2024-03-14T12:38:33.373 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-544.el8.x86_64 vs 4.18.0-544.el8.x86_64 2024-03-14T12:38:33.373 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-14T12:38:33.373 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-14T12:38:34.375 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-14T12:38:34.375 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-14T12:38:34.462 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-03-14T12:38:34.474 DEBUG:teuthology.parallel:result is None 2024-03-14T12:38:39.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-03-14T12:38:57.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:38:57.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:38:58.017 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-14T12:38:58.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:38:58.299 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-544.el8.x86_64"... 2024-03-14T12:38:58.300 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-03-14T12:38:58.359 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-544.el8.x86_64 2024-03-14T12:38:58.359 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-544.el8.x86_64 vs 4.18.0-544.el8.x86_64 2024-03-14T12:38:58.359 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-14T12:38:58.359 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-14T12:38:59.360 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-14T12:38:59.361 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-14T12:38:59.433 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-03-14T12:38:59.445 DEBUG:teuthology.parallel:result is None 2024-03-14T12:38:59.446 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-14T12:38:59.453 INFO:teuthology.task.internal:Creating test directory... 2024-03-14T12:38:59.453 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-14T12:38:59.456 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-14T12:38:59.480 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-14T12:38:59.487 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-14T12:38:59.535 INFO:teuthology.task.internal:Creating archive directory... 2024-03-14T12:38:59.535 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-14T12:38:59.537 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-14T12:38:59.583 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-14T12:38:59.591 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-14T12:38:59.591 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-14T12:38:59.617 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-14T12:38:59.651 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T12:38:59.662 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T12:38:59.662 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T12:38:59.673 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-14T12:38:59.675 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-14T12:38:59.683 INFO:teuthology.task.internal:Configuring sudo... 2024-03-14T12:38:59.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-14T12:38:59.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-14T12:38:59.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-14T12:38:59.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-14T12:38:59.745 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-14T12:38:59.776 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-14T12:38:59.795 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T12:38:59.883 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T12:38:59.933 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T12:39:00.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T12:39:00.061 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:39:00.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-14T12:39:00.127 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T12:39:00.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-14T12:39:00.226 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T12:39:00.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-14T12:39:00.358 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:39:00.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-14T12:39:00.427 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-03-14T12:39:00.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-03-14T12:39:00.488 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T12:39:00.521 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T12:39:01.005 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-14T12:39:01.014 INFO:teuthology.task.internal:Starting timer... 2024-03-14T12:39:01.014 INFO:teuthology.run_tasks:Running task pcp... 2024-03-14T12:39:01.065 INFO:teuthology.run_tasks:Running task selinux... 2024-03-14T12:39:01.113 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-14T12:39:01.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-03-14T12:39:01.209 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-03-14T12:39:01.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-03-14T12:39:01.294 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-03-14T12:39:01.296 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-14T12:39:01.296 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-03-14T12:39:01.361 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-03-14T12:39:01.362 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-03-14T12:39:01.388 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-03-14T12:39:01.389 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-03-14T12:39:01.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T12:39:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:39:01.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T12:39:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:39:01.458 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-14T12:39:01.458 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-03-14T12:39:01.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-03-14T12:39:01.525 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-14T12:39:01.536 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-14T12:39:01.541 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-14T12:39:01.654 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-14T12:39:01.670 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-14T12:39:01.671 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-14T12:45:18.785 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-03-14T12:45:18.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:45:18.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:45:18.863 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-03-14T12:45:18.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-03-14T12:45:18.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-14T12:45:18.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-14T12:45:19.004 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-03-14T12:45:19.073 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-03-14T12:45:19.073 INFO:teuthology.run_tasks:Running task clock... 2024-03-14T12:45:19.084 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-14T12:45:19.085 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-14T12:45:19.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T12:45:19.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-14T12:45:19.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T12:45:19.118 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-14T12:45:19.132 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-14T12:45:19.144 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-14T12:45:19.158 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-14T12:45:19.201 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-03-14T12:45:19.212 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-03-14T12:45:19.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-03-14T12:45:19.225 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-14T12:45:19.233 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-03-14T12:45:19.239 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-14T12:45:19.246 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-14T12:45:19.259 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-14T12:45:19.303 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-03-14T12:45:19.306 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T12:45:19.306 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-14T12:45:19.306 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.307 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.307 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.307 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.335 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.339 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T12:45:19.340 INFO:teuthology.run_tasks:Running task pexec... 2024-03-14T12:45:19.349 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-14T12:45:19.349 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:45:19.349 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-14T12:45:19.350 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2024-03-14T12:45:19.350 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-14T12:45:20.135 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:34 ago on Thu 14 Mar 2024 12:43:45 PM UTC. 2024-03-14T12:45:20.168 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:35 ago on Thu 14 Mar 2024 12:43:44 PM UTC. 2024-03-14T12:45:22.169 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:Resetting modules: 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:22.170 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:22.266 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:Resetting modules: 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:22.267 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:22.418 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:45:22.517 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:45:23.242 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:38 ago on Thu 14 Mar 2024 12:43:45 PM UTC. 2024-03-14T12:45:23.382 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:39 ago on Thu 14 Mar 2024 12:43:44 PM UTC. 2024-03-14T12:45:25.315 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:45:25.351 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:45:25.351 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:25.351 INFO:teuthology.orchestra.run.smithi111.stdout:Installing group/module packages: 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-03-14T12:45:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout:Installing module profiles: 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools/common 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools rhel8 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-14T12:45:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:25.354 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 27 M 2024-03-14T12:45:25.354 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 91 M 2024-03-14T12:45:25.354 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:45:25.515 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:45:25.550 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:25.550 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:45:25.550 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:25.550 INFO:teuthology.orchestra.run.smithi137.stdout:Installing group/module packages: 2024-03-14T12:45:25.550 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-14T12:45:25.551 INFO:teuthology.orchestra.run.smithi137.stdout:Installing module profiles: 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools/common 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling module streams: 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout: container-tools rhel8 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-03-14T12:45:25.552 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:25.553 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 27 M 2024-03-14T12:45:25.553 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 91 M 2024-03-14T12:45:25.553 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:45:26.600 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 597 kB/s | 689 kB 00:01 2024-03-14T12:45:26.725 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 303 kB/s | 38 kB 00:00 2024-03-14T12:45:26.742 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 633 kB/s | 689 kB 00:01 2024-03-14T12:45:26.834 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 413 kB/s | 38 kB 00:00 2024-03-14T12:45:26.900 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 708 kB/s | 1.0 MB 00:01 2024-03-14T12:45:26.968 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 785 kB/s | 1.0 MB 00:01 2024-03-14T12:45:27.125 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 644 kB/s | 258 kB 00:00 2024-03-14T12:45:27.127 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 883 kB/s | 258 kB 00:00 2024-03-14T12:45:27.418 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 619 kB/s | 180 kB 00:00 2024-03-14T12:45:27.425 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 601 kB/s | 180 kB 00:00 2024-03-14T12:45:27.543 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2024-03-14T12:45:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 198 kB/s | 25 kB 00:00 2024-03-14T12:45:27.726 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 512 kB/s | 94 kB 00:00 2024-03-14T12:45:27.727 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 512 kB/s | 94 kB 00:00 2024-03-14T12:45:29.610 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 1.5 MB/s | 4.1 MB 00:02 2024-03-14T12:45:30.411 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 1.2 MB/s | 4.1 MB 00:03 2024-03-14T12:45:30.943 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 1.9 MB/s | 2.5 MB 00:01 2024-03-14T12:45:31.152 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 253 kB/s | 53 kB 00:00 2024-03-14T12:45:31.277 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 328 kB/s | 41 kB 00:00 2024-03-14T12:45:31.644 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 1.6 MB/s | 9.6 MB 00:05 2024-03-14T12:45:31.820 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 301 kB/s | 53 kB 00:00 2024-03-14T12:45:31.928 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 378 kB/s | 41 kB 00:00 2024-03-14T12:45:32.174 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 1.4 MB/s | 2.5 MB 00:01 2024-03-14T12:45:32.495 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.8 MB/s | 8.8 MB 00:04 2024-03-14T12:45:32.496 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:45:32.496 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.0 MB/s | 27 MB 00:06 2024-03-14T12:45:32.569 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.8 MB/s | 8.8 MB 00:04 2024-03-14T12:45:32.766 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:45:32.787 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:45:32.787 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:45:33.077 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:45:33.080 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:45:33.528 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): buildah-1.33.6-2.module_el8+976+113607 1.2 MB/s | 9.6 MB 00:08 2024-03-14T12:45:33.528 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:45:33.528 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.4 MB/s | 27 MB 00:08 2024-03-14T12:45:33.799 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:45:33.820 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:45:33.820 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:45:33.871 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:45:34.077 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:45:34.080 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:45:34.749 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:45:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-14T12:45:34.995 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-14T12:45:35.158 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-14T12:45:35.351 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-14T12:45:35.602 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-14T12:45:35.725 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-14T12:45:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-14T12:45:36.097 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-14T12:45:36.171 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-14T12:45:36.295 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-14T12:45:36.551 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-14T12:45:37.114 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-14T12:45:37.191 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-14T12:45:37.415 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-14T12:45:37.626 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-14T12:45:38.094 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-14T12:45:38.095 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-14T12:45:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-14T12:45:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-14T12:45:38.574 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-14T12:45:38.658 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-14T12:45:38.848 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-14T12:45:39.305 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-14T12:45:39.306 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-14T12:45:39.306 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-14T12:45:39.306 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-14T12:45:39.306 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-14T12:45:39.306 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-14T12:45:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-14T12:45:39.562 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-14T12:45:39.803 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-14T12:45:39.847 INFO:teuthology.orchestra.run.smithi137.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-14T12:45:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-14T12:45:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-14T12:45:39.848 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:45:39.848 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:45:40.004 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-14T12:45:40.152 DEBUG:teuthology.parallel:result is None 2024-03-14T12:45:40.405 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-14T12:45:40.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-14T12:45:40.920 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-14T12:45:40.921 INFO:teuthology.orchestra.run.smithi111.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-14T12:45:40.922 INFO:teuthology.orchestra.run.smithi111.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-14T12:45:40.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:45:40.922 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:45:41.185 DEBUG:teuthology.parallel:result is None 2024-03-14T12:45:41.185 INFO:teuthology.run_tasks:Running task install... 2024-03-14T12:45:41.195 DEBUG:teuthology.task.install:project ceph 2024-03-14T12:45:41.195 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0'}} 2024-03-14T12:45:41.195 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0'} 2024-03-14T12:45:41.196 INFO:teuthology.task.install:Using flavor: default 2024-03-14T12:45:41.203 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-14T12:45:41.203 INFO:teuthology.task.install:extra packages: [] 2024-03-14T12:45:41.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'tag': None, 'wait_for_package': False} 2024-03-14T12:45:41.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:45:41.204 INFO:teuthology.packaging:ref: None 2024-03-14T12:45:41.204 INFO:teuthology.packaging:tag: None 2024-03-14T12:45:41.204 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:45:41.204 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:45:41.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-14T12:45:41.207 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0', 'tag': None, 'wait_for_package': False} 2024-03-14T12:45:41.207 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:45:41.207 INFO:teuthology.packaging:ref: None 2024-03-14T12:45:41.207 INFO:teuthology.packaging:tag: None 2024-03-14T12:45:41.207 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:45:41.207 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:45:41.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-14T12:45:41.471 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/ 2024-03-14T12:45:41.471 INFO:teuthology.task.install.rpm:Package version is 17.2.7-646.gb92acd1f 2024-03-14T12:45:41.681 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/ 2024-03-14T12:45:41.682 INFO:teuthology.task.install.rpm:Package version is 17.2.7-646.gb92acd1f 2024-03-14T12:45:41.744 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-14T12:45:41.744 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:45:41.745 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-14T12:45:41.774 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-14T12:45:41.774 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:45:41.774 INFO:teuthology.packaging:ref: None 2024-03-14T12:45:41.775 INFO:teuthology.packaging:tag: None 2024-03-14T12:45:41.775 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:45:41.775 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:45:41.775 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-14T12:45:41.843 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-14T12:45:41.920 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-14T12:45:41.960 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/b92acd1f9b519af8166ff5b5bae92bf72f1fd797/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-14T12:45:41.960 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:45:41.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-14T12:45:41.987 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-03-14T12:45:41.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-03-14T12:45:41.993 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-14T12:45:41.993 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:45:41.993 INFO:teuthology.packaging:ref: None 2024-03-14T12:45:41.993 INFO:teuthology.packaging:tag: None 2024-03-14T12:45:41.993 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:45:41.993 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:45:41.993 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-14T12:45:42.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-14T12:45:42.145 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-14T12:45:42.213 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-03-14T12:45:42.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-03-14T12:45:42.254 INFO:teuthology.orchestra.run.smithi111.stdout:76 files removed 2024-03-14T12:45:42.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-03-14T12:45:42.476 INFO:teuthology.orchestra.run.smithi137.stdout:76 files removed 2024-03-14T12:45:42.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-03-14T12:45:43.221 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 146 kB/s | 78 kB 00:00 2024-03-14T12:45:43.329 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 187 kB/s | 78 kB 00:00 2024-03-14T12:45:43.614 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 26 kB/s | 9.1 kB 00:00 2024-03-14T12:45:43.705 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2024-03-14T12:45:43.998 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2024-03-14T12:45:44.080 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2024-03-14T12:45:58.088 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 2.0 MB/s | 28 MB 00:14 2024-03-14T12:46:01.170 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - AppStream 1.6 MB/s | 28 MB 00:17 2024-03-14T12:46:05.047 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 5.3 MB/s | 10 MB 00:01 2024-03-14T12:46:06.890 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 74 kB/s | 18 kB 00:00 2024-03-14T12:46:07.265 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 27 kB/s | 7.2 kB 00:00 2024-03-14T12:46:07.596 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - BaseOS 7.0 MB/s | 10 MB 00:01 2024-03-14T12:46:09.438 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras 77 kB/s | 18 kB 00:00 2024-03-14T12:46:10.281 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - Extras common packages 9.7 kB/s | 7.2 kB 00:00 2024-03-14T12:46:28.399 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - PowerTools 239 kB/s | 4.9 MB 00:21 2024-03-14T12:46:29.499 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-03-14T12:46:31.142 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-03-14T12:46:34.925 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 534 kB/s | 24 kB 00:00 2024-03-14T12:46:35.464 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 8 - PowerTools 200 kB/s | 4.9 MB 00:25 2024-03-14T12:46:36.555 INFO:teuthology.orchestra.run.smithi137.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-03-14T12:46:38.471 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:46:38.474 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:46:38.474 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:46:38.474 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:46:38.474 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:46:38.474 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 17 M 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.5 M 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.6 M 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 5.2 M 2024-03-14T12:46:38.475 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 23 M 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 24 k 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 797 k 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-14T12:46:38.476 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 507 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 5.6 M 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 45 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 116 k 2024-03-14T12:46:38.477 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 190 k 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 370 k 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 357 k 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 107 k 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:46:38.478 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:46:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Install 21 Packages 2024-03-14T12:46:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-03-14T12:46:38.479 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:46:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 64 M 2024-03-14T12:46:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:46:39.851 INFO:teuthology.orchestra.run.smithi111.stdout:(1/23): ceph-base-17.2.7-646.gb92acd1f.el8.x86_ 7.8 MB/s | 5.2 MB 00:00 2024-03-14T12:46:39.918 INFO:teuthology.orchestra.run.smithi111.stdout:(2/23): ceph-selinux-17.2.7-646.gb92acd1f.el8.x 365 kB/s | 24 kB 00:00 2024-03-14T12:46:40.043 INFO:teuthology.orchestra.run.smithi111.stdout:(3/23): libcephfs2-17.2.7-646.gb92acd1f.el8.x86 6.2 MB/s | 797 kB 00:00 2024-03-14T12:46:40.127 INFO:teuthology.orchestra.run.smithi111.stdout:(4/23): libradosstriper1-17.2.7-646.gb92acd1f.e 5.9 MB/s | 507 kB 00:00 2024-03-14T12:46:40.434 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2024-03-14T12:46:40.802 INFO:teuthology.orchestra.run.smithi111.stdout:(5/23): ceph-radosgw-17.2.7-646.gb92acd1f.el8.x 10 MB/s | 17 MB 00:01 2024-03-14T12:46:40.886 INFO:teuthology.orchestra.run.smithi111.stdout:(6/23): librgw2-17.2.7-646.gb92acd1f.el8.x86_64 7.3 MB/s | 5.6 MB 00:00 2024-03-14T12:46:40.911 INFO:teuthology.orchestra.run.smithi111.stdout:(7/23): python3-ceph-argparse-17.2.7-646.gb92ac 413 kB/s | 45 kB 00:00 2024-03-14T12:46:40.936 INFO:teuthology.orchestra.run.smithi111.stdout:(8/23): python3-ceph-common-17.2.7-646.gb92acd1 2.3 MB/s | 116 kB 00:00 2024-03-14T12:46:40.970 INFO:teuthology.orchestra.run.smithi111.stdout:(9/23): python3-cephfs-17.2.7-646.gb92acd1f.el8 3.2 MB/s | 190 kB 00:00 2024-03-14T12:46:40.995 INFO:teuthology.orchestra.run.smithi111.stdout:(10/23): python3-rados-17.2.7-646.gb92acd1f.el8 6.2 MB/s | 370 kB 00:00 2024-03-14T12:46:41.029 INFO:teuthology.orchestra.run.smithi111.stdout:(11/23): python3-rbd-17.2.7-646.gb92acd1f.el8.x 6.0 MB/s | 357 kB 00:00 2024-03-14T12:46:41.055 INFO:teuthology.orchestra.run.smithi111.stdout:(12/23): python3-rgw-17.2.7-646.gb92acd1f.el8.x 1.8 MB/s | 107 kB 00:00 2024-03-14T12:46:41.780 INFO:teuthology.orchestra.run.smithi111.stdout:(13/23): ceph-common-17.2.7-646.gb92acd1f.el8.x 9.1 MB/s | 23 MB 00:02 2024-03-14T12:46:41.805 INFO:teuthology.orchestra.run.smithi111.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 346 kB/s | 259 kB 00:00 2024-03-14T12:46:41.830 INFO:teuthology.orchestra.run.smithi111.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 845 kB/s | 677 kB 00:00 2024-03-14T12:46:41.998 INFO:teuthology.orchestra.run.smithi111.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 245 kB/s | 47 kB 00:00 2024-03-14T12:46:42.023 INFO:teuthology.orchestra.run.smithi111.stdout:(17/23): python3-prettytable-0.7.2-14.el8.noarc 181 kB/s | 44 kB 00:00 2024-03-14T12:46:42.073 INFO:teuthology.orchestra.run.smithi111.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-03-14T12:46:42.174 INFO:teuthology.orchestra.run.smithi111.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 335 kB/s | 59 kB 00:00 2024-03-14T12:46:42.316 INFO:teuthology.orchestra.run.smithi111.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 7.2 MB/s | 1.7 MB 00:00 2024-03-14T12:46:42.491 INFO:teuthology.orchestra.run.smithi111.stdout:(21/23): librados2-17.2.7-646.gb92acd1f.el8.x86 11 MB/s | 3.5 MB 00:00 2024-03-14T12:46:42.866 INFO:teuthology.orchestra.run.smithi111.stdout:(22/23): librbd1-17.2.7-646.gb92acd1f.el8.x86_6 6.5 MB/s | 3.6 MB 00:00 2024-03-14T12:46:43.208 INFO:teuthology.orchestra.run.smithi111.stdout:(23/23): libunwind-1.3.1-3.el8.x86_64.rpm 63 kB/s | 75 kB 00:01 2024-03-14T12:46:43.210 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:46:43.210 INFO:teuthology.orchestra.run.smithi111.stdout:Total 13 MB/s | 64 MB 00:04 2024-03-14T12:46:43.251 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:46:43.295 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:46:43.295 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:46:43.653 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:46:43.656 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:46:44.173 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 542 kB/s | 24 kB 00:00 2024-03-14T12:46:44.682 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:46:44.864 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-14T12:46:45.037 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-14T12:46:45.412 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-14T12:46:45.570 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:46:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:46:45.886 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 4/25 2024-03-14T12:46:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:46:46.488 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:46:46.695 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:46:46.903 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:46:47.096 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-14T12:46:47.296 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-14T12:46:47.775 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:46:47.777 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:46:47.777 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:46:47.777 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 17 M 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.5 M 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.6 M 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 5.2 M 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 23 M 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 24 k 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 797 k 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-14T12:46:47.778 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 507 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 5.6 M 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 45 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 116 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 190 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 370 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 357 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 107 k 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:46:47.779 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:46:47.780 INFO:teuthology.orchestra.run.smithi137.stdout:Install 21 Packages 2024-03-14T12:46:47.780 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-03-14T12:46:47.780 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:46:47.780 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 64 M 2024-03-14T12:46:47.780 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:46:47.949 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-14T12:46:48.120 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:46:48.192 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:46:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x 11/25 2024-03-14T12:46:48.525 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 12/25 2024-03-14T12:46:48.738 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 13/25 2024-03-14T12:46:48.939 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 14/25 2024-03-14T12:46:48.948 INFO:teuthology.orchestra.run.smithi137.stdout:(1/23): ceph-base-17.2.7-646.gb92acd1f.el8.x86_ 7.1 MB/s | 5.2 MB 00:00 2024-03-14T12:46:49.024 INFO:teuthology.orchestra.run.smithi137.stdout:(2/23): ceph-selinux-17.2.7-646.gb92acd1f.el8.x 324 kB/s | 24 kB 00:00 2024-03-14T12:46:49.069 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 15/25 2024-03-14T12:46:49.148 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 15/25 2024-03-14T12:46:49.166 INFO:teuthology.orchestra.run.smithi137.stdout:(3/23): libcephfs2-17.2.7-646.gb92acd1f.el8.x86 5.5 MB/s | 797 kB 00:00 2024-03-14T12:46:49.258 INFO:teuthology.orchestra.run.smithi137.stdout:(4/23): libradosstriper1-17.2.7-646.gb92acd1f.e 5.4 MB/s | 507 kB 00:00 2024-03-14T12:46:49.388 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-14T12:46:49.561 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-14T12:46:49.731 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-14T12:46:49.917 INFO:teuthology.orchestra.run.smithi137.stdout:(5/23): librgw2-17.2.7-646.gb92acd1f.el8.x86_64 8.5 MB/s | 5.6 MB 00:00 2024-03-14T12:46:49.953 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86 19/25 2024-03-14T12:46:50.075 INFO:teuthology.orchestra.run.smithi137.stdout:(6/23): ceph-radosgw-17.2.7-646.gb92acd1f.el8.x 9.0 MB/s | 17 MB 00:01 2024-03-14T12:46:50.100 INFO:teuthology.orchestra.run.smithi137.stdout:(7/23): python3-ceph-argparse-17.2.7-646.gb92ac 244 kB/s | 45 kB 00:00 2024-03-14T12:46:50.134 INFO:teuthology.orchestra.run.smithi137.stdout:(8/23): python3-ceph-common-17.2.7-646.gb92acd1 1.9 MB/s | 116 kB 00:00 2024-03-14T12:46:50.159 INFO:teuthology.orchestra.run.smithi137.stdout:(9/23): python3-cephfs-17.2.7-646.gb92acd1f.el8 3.2 MB/s | 190 kB 00:00 2024-03-14T12:46:50.201 INFO:teuthology.orchestra.run.smithi137.stdout:(10/23): python3-rados-17.2.7-646.gb92acd1f.el8 5.4 MB/s | 370 kB 00:00 2024-03-14T12:46:50.235 INFO:teuthology.orchestra.run.smithi137.stdout:(11/23): python3-rbd-17.2.7-646.gb92acd1f.el8.x 4.6 MB/s | 357 kB 00:00 2024-03-14T12:46:50.260 INFO:teuthology.orchestra.run.smithi137.stdout:(12/23): python3-rgw-17.2.7-646.gb92acd1f.el8.x 1.8 MB/s | 107 kB 00:00 2024-03-14T12:46:50.594 INFO:teuthology.orchestra.run.smithi137.stdout:(13/23): ceph-common-17.2.7-646.gb92acd1f.el8.x 9.9 MB/s | 23 MB 00:02 2024-03-14T12:46:50.719 INFO:teuthology.orchestra.run.smithi137.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 566 kB/s | 259 kB 00:00 2024-03-14T12:46:50.819 INFO:teuthology.orchestra.run.smithi137.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 195 kB/s | 44 kB 00:00 2024-03-14T12:46:50.895 INFO:teuthology.orchestra.run.smithi137.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 269 kB/s | 47 kB 00:00 2024-03-14T12:46:50.953 INFO:teuthology.orchestra.run.smithi137.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 943 kB/s | 677 kB 00:00 2024-03-14T12:46:51.245 INFO:teuthology.orchestra.run.smithi137.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 168 kB/s | 59 kB 00:00 2024-03-14T12:46:51.304 INFO:teuthology.orchestra.run.smithi137.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 214 kB/s | 75 kB 00:00 2024-03-14T12:46:51.354 INFO:teuthology.orchestra.run.smithi137.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 573 kB/s | 306 kB 00:00 2024-03-14T12:46:51.613 INFO:teuthology.orchestra.run.smithi137.stdout:(21/23): librados2-17.2.7-646.gb92acd1f.el8.x86 11 MB/s | 3.5 MB 00:00 2024-03-14T12:46:51.730 INFO:teuthology.orchestra.run.smithi137.stdout:(22/23): librbd1-17.2.7-646.gb92acd1f.el8.x86_6 9.6 MB/s | 3.6 MB 00:00 2024-03-14T12:46:52.155 INFO:teuthology.orchestra.run.smithi137.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 1.9 MB/s | 1.7 MB 00:00 2024-03-14T12:46:52.158 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:46:52.158 INFO:teuthology.orchestra.run.smithi137.stdout:Total 15 MB/s | 64 MB 00:04 2024-03-14T12:46:52.201 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:46:52.247 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:46:52.247 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:46:52.626 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:46:52.629 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:46:52.645 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:46:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:46:53.464 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:46:53.660 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:46:53.747 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 21/25 2024-03-14T12:46:53.853 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 21/25 2024-03-14T12:46:53.885 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-14T12:46:54.037 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-14T12:46:54.085 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:46:54.419 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-14T12:46:54.628 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:46:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:46:54.936 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 4/25 2024-03-14T12:46:55.087 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:46:55.485 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:46:55.653 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:46:55.867 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:46:56.077 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-14T12:46:56.278 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-14T12:46:56.931 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-14T12:46:57.119 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:46:57.172 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:46:57.330 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x 11/25 2024-03-14T12:46:57.497 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 12/25 2024-03-14T12:46:57.687 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 13/25 2024-03-14T12:46:57.889 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 14/25 2024-03-14T12:46:58.036 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 15/25 2024-03-14T12:46:58.103 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 15/25 2024-03-14T12:46:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-14T12:46:58.469 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-14T12:46:58.652 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-14T12:46:58.886 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86 19/25 2024-03-14T12:47:01.499 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:47:01.719 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:47:02.290 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 20/25 2024-03-14T12:47:02.538 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 21/25 2024-03-14T12:47:02.639 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 21/25 2024-03-14T12:47:02.794 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:47:13.556 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-03-14T12:47:13.557 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:15.327 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 23/25 2024-03-14T12:47:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 23/25 2024-03-14T12:47:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-14T12:47:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-14T12:47:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:16.300 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:16.300 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 1/25 2024-03-14T12:47:16.300 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2/25 2024-03-14T12:47:16.300 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:47:16.300 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 4/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 7/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x 8/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86 9/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 11/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 12/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 13/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-14T12:47:16.301 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-14T12:47:16.302 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 24/25 2024-03-14T12:47:17.155 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:17.155 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:17.155 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-03-14T12:47:17.155 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-14T12:47:17.156 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:17.157 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:47:17.418 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-03-14T12:47:18.181 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:44 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:47:19.779 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 52 M 2024-03-14T12:47:19.780 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 179 k 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-14T12:47:19.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:19.782 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 53 M 2024-03-14T12:47:19.782 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 234 M 2024-03-14T12:47:19.782 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:47:20.207 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): libcephsqlite-17.2.7-646.gb92acd1f.el8.x 944 kB/s | 179 kB 00:00 2024-03-14T12:47:20.557 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 376 kB/s | 203 kB 00:00 2024-03-14T12:47:20.799 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 318 kB/s | 188 kB 00:00 2024-03-14T12:47:20.899 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 945 kB/s | 323 kB 00:00 2024-03-14T12:47:21.041 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 292 kB/s | 71 kB 00:00 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /run 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-03-14T12:47:21.963 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /dev 2024-03-14T12:47:21.964 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:22.533 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-17.2.7-646.gb92acd1f.el8.x86_6 21 MB/s | 52 MB 00:02 2024-03-14T12:47:22.535 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:22.535 INFO:teuthology.orchestra.run.smithi111.stdout:Total 19 MB/s | 53 MB 00:02 2024-03-14T12:47:22.576 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:47:22.598 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:47:22.598 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:47:22.886 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:47:22.889 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:47:23.358 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:47:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-14T12:47:23.685 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 23/25 2024-03-14T12:47:23.759 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 23/25 2024-03-14T12:47:23.759 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-14T12:47:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-14T12:47:23.897 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-14T12:47:23.967 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-14T12:47:23.967 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:23.979 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-14T12:47:24.157 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-14T12:47:24.322 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 5/6 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 1/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 3/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 4/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 5/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 6/25 2024-03-14T12:47:24.676 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 7/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x 8/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86 9/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 10/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 11/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 12/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 13/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-14T12:47:24.677 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-14T12:47:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 22/25 2024-03-14T12:47:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-14T12:47:24.678 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 24/25 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-14T12:47:25.569 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-14T12:47:25.570 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:25.571 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:47:25.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-03-14T12:47:26.660 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:42 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:47:28.252 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 52 M 2024-03-14T12:47:28.253 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 179 k 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-03-14T12:47:28.254 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:28.255 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 53 M 2024-03-14T12:47:28.255 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 234 M 2024-03-14T12:47:28.255 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:47:28.909 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 359 kB/s | 203 kB 00:00 2024-03-14T12:47:29.059 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): libcephsqlite-17.2.7-646.gb92acd1f.el8.x 250 kB/s | 179 kB 00:00 2024-03-14T12:47:29.218 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 610 kB/s | 188 kB 00:00 2024-03-14T12:47:29.291 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 5/6 2024-03-14T12:47:29.326 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 653 kB/s | 71 kB 00:00 2024-03-14T12:47:29.553 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 6/6 2024-03-14T12:47:29.735 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 478 kB/s | 323 kB 00:00 2024-03-14T12:47:29.847 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 6/6 2024-03-14T12:47:29.847 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 1/6 2024-03-14T12:47:29.847 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 2/6 2024-03-14T12:47:29.847 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-14T12:47:29.848 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-14T12:47:29.848 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-14T12:47:30.304 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-14T12:47:30.304 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: jq-1.6-8.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:30.305 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:47:30.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-03-14T12:47:30.802 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-test-17.2.7-646.gb92acd1f.el8.x86_6 21 MB/s | 52 MB 00:02 2024-03-14T12:47:30.804 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:30.804 INFO:teuthology.orchestra.run.smithi137.stdout:Total 21 MB/s | 53 MB 00:02 2024-03-14T12:47:30.846 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:47:30.866 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:47:30.866 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:47:31.152 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:47:31.155 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:47:31.253 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:57 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:47:31.626 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:47:31.842 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-14T12:47:32.025 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-14T12:47:32.139 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-14T12:47:32.223 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-14T12:47:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-14T12:47:32.588 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 5/6 2024-03-14T12:47:32.825 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:47:32.828 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 6.4 k 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 2.3 M 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 1.6 M 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 239 k 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.9 M 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 17 M 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-14T12:47:32.829 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-14T12:47:32.830 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout:Install 32 Packages 2024-03-14T12:47:32.831 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:32.834 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 28 M 2024-03-14T12:47:32.834 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 110 M 2024-03-14T12:47:32.834 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:47:33.395 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-17.2.7-646.gb92acd1f.el8.x86_64.rp 54 kB/s | 6.4 kB 00:00 2024-03-14T12:47:33.629 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-mds-17.2.7-646.gb92acd1f.el8.x86_6 6.6 MB/s | 2.3 MB 00:00 2024-03-14T12:47:33.921 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): ceph-mon-17.2.7-646.gb92acd1f.el8.x86_6 7.5 MB/s | 3.9 MB 00:00 2024-03-14T12:47:33.988 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): ceph-mgr-modules-core-17.2.7-646.gb92ac 3.5 MB/s | 239 kB 00:00 2024-03-14T12:47:34.130 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): ceph-mgr-17.2.7-646.gb92acd1f.el8.x86_6 1.8 MB/s | 1.6 MB 00:00 2024-03-14T12:47:34.530 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): ceph-osd-17.2.7-646.gb92acd1f.el8.x86_6 19 MB/s | 17 MB 00:00 2024-03-14T12:47:34.555 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 91 kB/s | 39 kB 00:00 2024-03-14T12:47:34.581 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 265 kB/s | 157 kB 00:00 2024-03-14T12:47:34.773 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 427 kB/s | 103 kB 00:00 2024-03-14T12:47:34.889 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 171 kB/s | 57 kB 00:00 2024-03-14T12:47:35.132 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 831 kB/s | 457 kB 00:00 2024-03-14T12:47:35.407 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 397 kB/s | 109 kB 00:00 2024-03-14T12:47:35.490 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 347 kB/s | 249 kB 00:00 2024-03-14T12:47:35.807 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 109 kB/s | 44 kB 00:00 2024-03-14T12:47:36.032 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 345 kB/s | 185 kB 00:00 2024-03-14T12:47:36.083 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 627 kB/s | 173 kB 00:00 2024-03-14T12:47:36.209 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2024-03-14T12:47:36.258 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-03-14T12:47:36.325 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 152 kB/s | 18 kB 00:00 2024-03-14T12:47:36.367 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2024-03-14T12:47:36.409 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): python3-cryptography-3.2.1-7.el8.x86_6 422 kB/s | 641 kB 00:01 2024-03-14T12:47:36.434 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 538 kB/s | 59 kB 00:00 2024-03-14T12:47:36.502 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-03-14T12:47:36.527 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-03-14T12:47:36.602 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 251 kB/s | 24 kB 00:00 2024-03-14T12:47:36.636 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 269 kB/s | 29 kB 00:00 2024-03-14T12:47:36.703 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-03-14T12:47:36.753 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): python3-portend-2.6-1.el8.noarch.rpm 45 kB/s | 16 kB 00:00 2024-03-14T12:47:36.812 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2024-03-14T12:47:36.845 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.7 MB/s | 251 kB 00:00 2024-03-14T12:47:36.929 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2024-03-14T12:47:36.996 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 352 kB/s | 85 kB 00:00 2024-03-14T12:47:37.000 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:37.000 INFO:teuthology.orchestra.run.smithi111.stdout:Total 6.8 MB/s | 28 MB 00:04 2024-03-14T12:47:37.079 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:47:37.116 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:47:37.116 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:47:37.410 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:47:37.413 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:47:37.538 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 5/6 2024-03-14T12:47:37.795 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 6/6 2024-03-14T12:47:38.080 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 6/6 2024-03-14T12:47:38.080 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 1/6 2024-03-14T12:47:38.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 2/6 2024-03-14T12:47:38.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-14T12:47:38.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-14T12:47:38.081 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-14T12:47:38.493 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-14T12:47:38.493 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:38.493 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:47:38.493 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-8.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:38.494 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:47:38.542 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:47:38.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-03-14T12:47:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-14T12:47:39.020 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-14T12:47:39.246 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-14T12:47:39.476 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-14T12:47:39.477 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:55 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:47:39.674 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-14T12:47:39.863 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-14T12:47:40.031 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-14T12:47:40.208 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-14T12:47:40.426 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-14T12:47:40.629 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-14T12:47:40.859 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-14T12:47:41.067 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 6.4 k 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 2.3 M 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 1.6 M 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 239 k 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.9 M 2024-03-14T12:47:41.071 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 17 M 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-14T12:47:41.072 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:41.073 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Install 32 Packages 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 28 M 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 110 M 2024-03-14T12:47:41.074 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:47:41.107 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-14T12:47:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-14T12:47:41.731 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-14T12:47:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-14T12:47:41.978 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-17.2.7-646.gb92acd1f.el8.x86_64.rp 49 kB/s | 6.4 kB 00:00 2024-03-14T12:47:42.220 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-14T12:47:42.303 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-mon-17.2.7-646.gb92acd1f.el8.x86_6 12 MB/s | 3.9 MB 00:00 2024-03-14T12:47:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-14T12:47:42.746 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-14T12:47:42.912 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): ceph-mgr-17.2.7-646.gb92acd1f.el8.x86_6 1.5 MB/s | 1.6 MB 00:01 2024-03-14T12:47:42.991 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-14T12:47:43.138 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-14T12:47:43.229 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): ceph-osd-17.2.7-646.gb92acd1f.el8.x86_6 18 MB/s | 17 MB 00:00 2024-03-14T12:47:43.254 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): ceph-mgr-modules-core-17.2.7-646.gb92ac 698 kB/s | 239 kB 00:00 2024-03-14T12:47:43.304 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): ceph-mds-17.2.7-646.gb92acd1f.el8.x86_6 1.6 MB/s | 2.3 MB 00:01 2024-03-14T12:47:43.307 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-14T12:47:43.533 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-14T12:47:43.596 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 113 kB/s | 39 kB 00:00 2024-03-14T12:47:43.747 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 303 kB/s | 157 kB 00:00 2024-03-14T12:47:43.776 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-14T12:47:43.780 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 310 kB/s | 57 kB 00:00 2024-03-14T12:47:43.805 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 206 kB/s | 103 kB 00:00 2024-03-14T12:47:43.995 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-14T12:47:44.181 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2024-03-14T12:47:44.206 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-03-14T12:47:44.289 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-14T12:47:44.373 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 420 kB/s | 249 kB 00:00 2024-03-14T12:47:44.565 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 284 kB/s | 109 kB 00:00 2024-03-14T12:47:44.582 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-14T12:47:44.649 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 98 kB/s | 44 kB 00:00 2024-03-14T12:47:44.936 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.n 27/32 2024-03-14T12:47:45.007 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 292 kB/s | 185 kB 00:00 2024-03-14T12:47:45.116 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 824 kB/s | 384 kB 00:00 2024-03-14T12:47:45.133 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-03-14T12:47:45.181 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 28/32 2024-03-14T12:47:45.200 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 272 kB/s | 173 kB 00:00 2024-03-14T12:47:45.242 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2024-03-14T12:47:45.284 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 323 kB/s | 49 kB 00:00 2024-03-14T12:47:45.359 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 369 kB/s | 59 kB 00:00 2024-03-14T12:47:45.426 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2024-03-14T12:47:45.451 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-03-14T12:47:45.493 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-03-14T12:47:45.585 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 150 kB/s | 24 kB 00:00 2024-03-14T12:47:45.610 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 184 kB/s | 29 kB 00:00 2024-03-14T12:47:45.636 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 193 kB/s | 27 kB 00:00 2024-03-14T12:47:45.802 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-03-14T12:47:45.828 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 445 kB/s | 85 kB 00:00 2024-03-14T12:47:45.878 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 873 kB/s | 255 kB 00:00 2024-03-14T12:47:45.928 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-03-14T12:47:45.930 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:45.930 INFO:teuthology.orchestra.run.smithi137.stdout:Total 5.9 MB/s | 28 MB 00:04 2024-03-14T12:47:46.010 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:47:46.047 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:47:46.047 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:47:46.350 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:47:46.353 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:47:46.661 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 28/32 2024-03-14T12:47:46.880 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 29/32 2024-03-14T12:47:47.281 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 29/32 2024-03-14T12:47:47.450 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:47:47.485 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 30/32 2024-03-14T12:47:47.666 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-14T12:47:47.702 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 30/32 2024-03-14T12:47:47.886 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-14T12:47:47.911 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 31/32 2024-03-14T12:47:47.928 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 31/32 2024-03-14T12:47:48.060 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 32/32 2024-03-14T12:47:48.111 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-14T12:47:48.376 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-14T12:47:48.473 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 32/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 1/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 2/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 3/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 4/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 5/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.n 6/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-14T12:47:48.474 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-14T12:47:48.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-14T12:47:48.476 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-14T12:47:48.476 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-14T12:47:48.476 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-14T12:47:48.476 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-14T12:47:48.476 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-14T12:47:48.557 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-14T12:47:48.739 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-14T12:47:48.906 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-14T12:47:49.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-14T12:47:49.285 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-14T12:47:49.480 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-14T12:47:49.709 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:49.954 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-14T12:47:49.955 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:49.956 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:47:49.975 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-14T12:47:50.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-03-14T12:47:50.341 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-14T12:47:50.580 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-14T12:47:50.760 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-14T12:47:50.944 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:16 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:47:51.034 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-14T12:47:51.228 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-14T12:47:51.555 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-14T12:47:51.791 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-14T12:47:51.946 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-14T12:47:52.116 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-14T12:47:52.333 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-14T12:47:52.479 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:47:52.525 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:47:52.526 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:47:52.527 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:47:52.554 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-14T12:47:52.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-03-14T12:47:52.753 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-14T12:47:53.062 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-14T12:47:53.373 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-14T12:47:53.388 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:19 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:47:53.735 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.n 27/32 2024-03-14T12:47:53.930 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 28/32 2024-03-14T12:47:54.983 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 81 k 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:54.984 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 81 k 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 370 k 2024-03-14T12:47:54.985 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:47:55.144 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-17.2.7-646.gb92acd1f.el8.noarch.rpm 513 kB/s | 81 kB 00:00 2024-03-14T12:47:55.144 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:55.145 INFO:teuthology.orchestra.run.smithi111.stdout:Total 509 kB/s | 81 kB 00:00 2024-03-14T12:47:55.145 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:47:55.153 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:47:55.153 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:47:55.154 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:47:55.157 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:47:55.198 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:47:55.414 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 28/32 2024-03-14T12:47:55.613 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 29/32 2024-03-14T12:47:55.710 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:47:55.900 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:47:56.004 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 29/32 2024-03-14T12:47:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:47:56.185 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 30/32 2024-03-14T12:47:56.400 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 30/32 2024-03-14T12:47:56.475 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:47:56.476 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:56.476 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:47:56.476 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:47:56.476 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:56.476 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:47:56.619 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 31/32 2024-03-14T12:47:56.636 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 31/32 2024-03-14T12:47:56.663 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-03-14T12:47:57.329 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 32/32 2024-03-14T12:47:57.419 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:23 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:47:57.743 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 32/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 1/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 2/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 3/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 4/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 5/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.n 6/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-14T12:47:57.744 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-14T12:47:57.745 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-14T12:47:57.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-14T12:47:57.746 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-14T12:47:58.559 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-14T12:47:58.560 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-2.6-1.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-14T12:47:58.561 INFO:teuthology.orchestra.run.smithi137.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:47:58.562 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:47:58.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-03-14T12:47:59.011 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:47:59.011 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:59.011 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-14T12:47:59.011 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:59.011 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 162 k 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:47:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 162 k 2024-03-14T12:47:59.013 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 494 k 2024-03-14T12:47:59.013 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:47:59.207 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-17.2.7-646.gb92acd1 834 kB/s | 162 kB 00:00 2024-03-14T12:47:59.207 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:47:59.208 INFO:teuthology.orchestra.run.smithi111.stdout:Total 829 kB/s | 162 kB 00:00 2024-03-14T12:47:59.208 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:47:59.218 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:47:59.218 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:47:59.266 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:47:59.269 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:47:59.432 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:47:59.618 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:15 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:01.161 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:48:01.208 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:01.210 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:48:01.210 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:01.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-03-14T12:48:01.922 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:02.077 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:18 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:02.305 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:02.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-03-14T12:48:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:29 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:03.670 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:03.671 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:03.671 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:48:03.671 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 81 k 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 81 k 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 370 k 2024-03-14T12:48:03.672 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:04.137 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-17.2.7-646.gb92acd1f.el8.noarch.rpm 175 kB/s | 81 kB 00:00 2024-03-14T12:48:04.138 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:04.138 INFO:teuthology.orchestra.run.smithi137.stdout:Total 175 kB/s | 81 kB 00:00 2024-03-14T12:48:04.138 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:04.146 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:04.146 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:04.148 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:04.151 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:04.193 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:04.647 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:48:04.807 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:48:04.849 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:48:04.855 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:48:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:04.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-03-14T12:48:05.074 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/1 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:05.453 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:05.692 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:31 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:05.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-03-14T12:48:06.501 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:22 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:07.287 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 1.7 M 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 23 k 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 14 k 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-14T12:48:07.288 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.0 M 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 8.0 M 2024-03-14T12:48:07.289 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:48:08.077 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:08.077 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:08.077 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 162 k 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 162 k 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 494 k 2024-03-14T12:48:08.078 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:08.168 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): ceph-prometheus-alerts-17.2.7-646.gb92ac 95 kB/s | 14 kB 00:00 2024-03-14T12:48:08.369 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-grafana-dashboards-17.2.7-646.gb92a 66 kB/s | 23 kB 00:00 2024-03-14T12:48:08.386 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 196 kB/s | 43 kB 00:00 2024-03-14T12:48:08.561 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2024-03-14T12:48:08.628 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 813 kB/s | 196 kB 00:00 2024-03-14T12:48:09.045 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-mgr-dashboard-17.2.7-646.gb92acd1f. 1.6 MB/s | 1.7 MB 00:01 2024-03-14T12:48:09.048 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:09.049 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-03-14T12:48:09.061 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:48:09.071 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:48:09.071 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:48:09.134 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-17.2.7-646.gb92acd1 153 kB/s | 162 kB 00:01 2024-03-14T12:48:09.135 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:09.135 INFO:teuthology.orchestra.run.smithi137.stdout:Total 153 kB/s | 162 kB 00:01 2024-03-14T12:48:09.136 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:09.148 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:09.148 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:09.157 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:48:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:48:09.200 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:09.203 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:09.457 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:48:09.459 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:09.671 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-14T12:48:09.949 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-14T12:48:10.056 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-14T12:48:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.no 4/6 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.e 1/1 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:10.425 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:10.553 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.n 5/6 2024-03-14T12:48:10.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-03-14T12:48:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 6/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 6/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.n 1/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 2/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.no 3/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-14T12:48:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-14T12:48:11.423 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:27 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:11.449 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-14T12:48:11.450 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-14T12:48:11.450 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-14T12:48:11.450 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:11.450 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:11.573 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-14T12:48:12.323 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:38 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:12.982 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:48:13.030 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:13.032 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:48:13.032 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:13.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-03-14T12:48:13.892 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:29 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:13.906 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:13.907 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-14T12:48:13.907 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 7.4 M 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:13.908 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout:Install 8 Packages 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 36 M 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 210 M 2024-03-14T12:48:13.909 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:48:15.494 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 1.7 M 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 23 k 2024-03-14T12:48:15.495 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 14 k 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-03-14T12:48:15.496 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:15.497 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.0 M 2024-03-14T12:48:15.497 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 8.0 M 2024-03-14T12:48:15.497 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:16.245 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): ceph-prometheus-alerts-17.2.7-646.gb92ac 34 kB/s | 14 kB 00:00 2024-03-14T12:48:16.261 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): ceph-grafana-dashboards-17.2.7-646.gb92a 52 kB/s | 23 kB 00:00 2024-03-14T12:48:16.453 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2024-03-14T12:48:16.470 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-03-14T12:48:16.660 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 954 kB/s | 196 kB 00:00 2024-03-14T12:48:16.743 INFO:teuthology.orchestra.run.smithi111.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 1.8 MB/s | 4.7 MB 00:02 2024-03-14T12:48:17.420 INFO:teuthology.orchestra.run.smithi111.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.9 MB 00:03 2024-03-14T12:48:17.904 INFO:teuthology.orchestra.run.smithi111.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 466 kB/s | 225 kB 00:00 2024-03-14T12:48:18.321 INFO:teuthology.orchestra.run.smithi111.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.4 MB/s | 3.7 MB 00:01 2024-03-14T12:48:18.521 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): ceph-mgr-dashboard-17.2.7-646.gb92acd1f. 642 kB/s | 1.7 MB 00:02 2024-03-14T12:48:18.523 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:18.524 INFO:teuthology.orchestra.run.smithi137.stdout:Total 676 kB/s | 2.0 MB 00:03 2024-03-14T12:48:18.535 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:18.544 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:18.545 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:18.629 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:18.632 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:18.890 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:19.079 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-03-14T12:48:19.357 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-03-14T12:48:19.554 INFO:teuthology.orchestra.run.smithi111.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 523 kB/s | 645 kB 00:01 2024-03-14T12:48:19.567 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-03-14T12:48:19.670 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.no 4/6 2024-03-14T12:48:19.938 INFO:teuthology.orchestra.run.smithi111.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 450 kB/s | 172 kB 00:00 2024-03-14T12:48:20.210 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.n 5/6 2024-03-14T12:48:20.425 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 6/6 2024-03-14T12:48:20.696 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 6/6 2024-03-14T12:48:20.697 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.n 1/6 2024-03-14T12:48:20.697 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 2/6 2024-03-14T12:48:20.697 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.no 3/6 2024-03-14T12:48:20.697 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-03-14T12:48:20.697 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:21.101 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:21.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-14T12:48:21.646 INFO:teuthology.orchestra.run.smithi111.stdout:(7/8): ceph-mgr-diskprediction-local-17.2.7-646 998 kB/s | 7.4 MB 00:07 2024-03-14T12:48:21.997 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:23.413 INFO:teuthology.orchestra.run.smithi111.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.6 MB/s | 14 MB 00:05 2024-03-14T12:48:23.415 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:23.415 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.8 MB/s | 36 MB 00:09 2024-03-14T12:48:23.592 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 7.4 M 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-03-14T12:48:23.594 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:Install 8 Packages 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 36 M 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 210 M 2024-03-14T12:48:23.595 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:48:23.666 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:48:23.666 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:48:24.071 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:48:24.074 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:48:24.660 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:48:24.830 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-14T12:48:24.954 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-14T12:48:25.104 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-14T12:48:25.704 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-14T12:48:25.854 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:26.337 INFO:teuthology.orchestra.run.smithi137.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.0 MB/s | 4.9 MB 00:02 2024-03-14T12:48:26.472 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:26.480 INFO:teuthology.orchestra.run.smithi137.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 1.8 MB/s | 4.7 MB 00:02 2024-03-14T12:48:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-14T12:48:26.680 INFO:teuthology.orchestra.run.smithi137.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2024-03-14T12:48:27.338 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-14T12:48:27.698 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-14T12:48:27.921 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-14T12:48:28.547 INFO:teuthology.orchestra.run.smithi137.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.7 MB/s | 3.7 MB 00:02 2024-03-14T12:48:29.572 INFO:teuthology.orchestra.run.smithi137.stdout:(5/8): ceph-mgr-diskprediction-local-17.2.7-646 1.3 MB/s | 7.4 MB 00:05 2024-03-14T12:48:29.670 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-14T12:48:30.047 INFO:teuthology.orchestra.run.smithi137.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 363 kB/s | 172 kB 00:00 2024-03-14T12:48:30.656 INFO:teuthology.orchestra.run.smithi137.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 306 kB/s | 645 kB 00:02 2024-03-14T12:48:30.873 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-14T12:48:31.087 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 8/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 8/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 1/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-14T12:48:31.387 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-14T12:48:31.406 INFO:teuthology.orchestra.run.smithi137.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 3.0 MB/s | 14 MB 00:04 2024-03-14T12:48:31.406 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:31.407 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.6 MB/s | 36 MB 00:07 2024-03-14T12:48:31.638 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:31.656 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:31.657 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:31.789 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-14T12:48:31.789 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:31.789 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-14T12:48:31.790 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-14T12:48:31.791 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-14T12:48:31.791 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-14T12:48:31.791 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:31.791 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:31.893 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-03-14T12:48:32.055 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:32.058 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:32.604 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:32.643 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:58 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:32.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-14T12:48:32.841 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-03-14T12:48:32.979 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-14T12:48:33.572 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-03-14T12:48:33.720 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:34.244 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 47 k 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:48:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:34.247 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.9 M 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 25 M 2024-03-14T12:48:34.248 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:48:34.322 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:34.445 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-14T12:48:35.117 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-03-14T12:48:35.155 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 91 kB/s | 27 kB 00:00 2024-03-14T12:48:35.185 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 62 kB/s | 20 kB 00:00 2024-03-14T12:48:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-pyasn1-modules-0.3.7-6.el8.noar 440 kB/s | 110 kB 00:00 2024-03-14T12:48:35.448 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-03-14T12:48:35.486 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 383 kB/s | 126 kB 00:00 2024-03-14T12:48:35.511 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): ceph-mgr-rook-17.2.7-646.gb92acd1f.el8. 73 kB/s | 47 kB 00:00 2024-03-14T12:48:35.670 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-14T12:48:35.761 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 155 kB/s | 43 kB 00:00 2024-03-14T12:48:35.895 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 86 kB/s | 33 kB 00:00 2024-03-14T12:48:36.087 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 50 kB/s | 16 kB 00:00 2024-03-14T12:48:36.112 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 375 kB/s | 81 kB 00:00 2024-03-14T12:48:36.137 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 221 kB/s | 155 kB 00:00 2024-03-14T12:48:36.238 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 513 kB/s | 64 kB 00:00 2024-03-14T12:48:36.555 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 147 kB/s | 61 kB 00:00 2024-03-14T12:48:36.597 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2024-03-14T12:48:36.599 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:36.600 INFO:teuthology.orchestra.run.smithi111.stdout:Total 840 kB/s | 1.9 MB 00:02 2024-03-14T12:48:36.656 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:48:36.672 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:48:36.672 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:48:36.761 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:48:36.764 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:48:37.312 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:48:37.420 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-03-14T12:48:37.547 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-14T12:48:37.735 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-14T12:48:37.988 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-14T12:48:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-14T12:48:38.433 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-14T12:48:38.656 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-14T12:48:38.657 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-03-14T12:48:38.828 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 8/8 2024-03-14T12:48:38.931 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-14T12:48:39.143 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 8/8 2024-03-14T12:48:39.143 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f 1/8 2024-03-14T12:48:39.143 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-03-14T12:48:39.143 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-03-14T12:48:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-03-14T12:48:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-03-14T12:48:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-03-14T12:48:39.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-03-14T12:48:39.181 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.15-4.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-03-14T12:48:39.556 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:39.557 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:39.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-03-14T12:48:40.210 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-14T12:48:40.457 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:56 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:40.492 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-14T12:48:40.690 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-14T12:48:40.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-14T12:48:41.002 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 13/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 13/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 1/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-14T12:48:41.315 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-14T12:48:41.316 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-14T12:48:41.764 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-14T12:48:41.765 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-14T12:48:41.766 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-14T12:48:41.766 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:41.766 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:41.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-03-14T12:48:42.047 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:42.048 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:42.048 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 47 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-03-14T12:48:42.049 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-03-14T12:48:42.050 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:42.051 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 1.9 M 2024-03-14T12:48:42.051 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 25 M 2024-03-14T12:48:42.051 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:42.651 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:08 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:42.670 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 80 kB/s | 20 kB 00:00 2024-03-14T12:48:42.695 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 97 kB/s | 27 kB 00:00 2024-03-14T12:48:42.937 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 474 kB/s | 126 kB 00:00 2024-03-14T12:48:42.962 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 412 kB/s | 110 kB 00:00 2024-03-14T12:48:43.138 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): ceph-mgr-rook-17.2.7-646.gb92acd1f.el8. 66 kB/s | 47 kB 00:00 2024-03-14T12:48:43.163 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 213 kB/s | 43 kB 00:00 2024-03-14T12:48:43.297 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 122 kB/s | 16 kB 00:00 2024-03-14T12:48:43.314 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 188 kB/s | 33 kB 00:00 2024-03-14T12:48:43.414 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 695 kB/s | 81 kB 00:00 2024-03-14T12:48:43.481 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 285 kB/s | 155 kB 00:00 2024-03-14T12:48:43.506 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 698 kB/s | 64 kB 00:00 2024-03-14T12:48:43.590 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 4.2 MB/s | 1.2 MB 00:00 2024-03-14T12:48:43.640 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 386 kB/s | 61 kB 00:00 2024-03-14T12:48:43.640 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:43.641 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2024-03-14T12:48:43.698 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:43.714 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:43.714 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:43.801 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:43.804 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:44.252 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 121 k 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:Install 5 Packages 2024-03-14T12:48:44.254 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 6.0 M 2024-03-14T12:48:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-03-14T12:48:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:48:44.347 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:44.562 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-03-14T12:48:44.743 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-03-14T12:48:44.987 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-03-14T12:48:45.078 INFO:teuthology.orchestra.run.smithi111.stdout:(1/5): ceph-mgr-cephadm-17.2.7-646.gb92acd1f.el 206 kB/s | 121 kB 00:00 2024-03-14T12:48:45.200 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-03-14T12:48:45.304 INFO:teuthology.orchestra.run.smithi111.stdout:(2/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 709 kB/s | 578 kB 00:00 2024-03-14T12:48:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-03-14T12:48:45.496 INFO:teuthology.orchestra.run.smithi111.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.1 MB/s | 476 kB 00:00 2024-03-14T12:48:45.563 INFO:teuthology.orchestra.run.smithi111.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 233 kB/s | 60 kB 00:00 2024-03-14T12:48:45.630 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-03-14T12:48:45.937 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-03-14T12:48:46.147 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-03-14T12:48:47.155 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-03-14T12:48:47.454 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-03-14T12:48:47.675 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-03-14T12:48:47.868 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-03-14T12:48:48.018 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 13/13 2024-03-14T12:48:48.054 INFO:teuthology.orchestra.run.smithi111.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 1.3 MB/s | 4.8 MB 00:03 2024-03-14T12:48:48.057 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:48.058 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.6 MB/s | 6.0 MB 00:03 2024-03-14T12:48:48.110 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:48:48.122 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:48:48.123 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:48:48.223 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:48:48.228 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:48:48.303 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 13/13 2024-03-14T12:48:48.303 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 1/13 2024-03-14T12:48:48.303 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-03-14T12:48:48.304 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-03-14T12:48:48.504 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-14T12:48:48.754 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:48.755 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:48.814 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-14T12:48:48.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-03-14T12:48:49.631 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:05 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:49.820 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-14T12:48:50.145 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-14T12:48:50.436 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-14T12:48:50.617 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 5/5 2024-03-14T12:48:50.892 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 5/5 2024-03-14T12:48:50.893 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/5 2024-03-14T12:48:50.893 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-14T12:48:50.893 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-14T12:48:50.893 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-14T12:48:51.221 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:17.2.7-646.gb92acd1f.el8 ceph-noarch 121 k 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-03-14T12:48:51.222 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:Install 5 Packages 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 6.0 M 2024-03-14T12:48:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-03-14T12:48:51.224 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:51.278 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:51.382 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-03-14T12:48:52.143 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:18 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:52.381 INFO:teuthology.orchestra.run.smithi137.stdout:(1/5): python3-jinja2-2.10.1-4.el8.noarch.rpm 787 kB/s | 578 kB 00:00 2024-03-14T12:48:52.438 INFO:teuthology.orchestra.run.smithi137.stdout:(2/5): ceph-mgr-cephadm-17.2.7-646.gb92acd1f.el 153 kB/s | 121 kB 00:00 2024-03-14T12:48:52.781 INFO:teuthology.orchestra.run.smithi137.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 176 kB/s | 60 kB 00:00 2024-03-14T12:48:52.981 INFO:teuthology.orchestra.run.smithi137.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 792 kB/s | 476 kB 00:00 2024-03-14T12:48:53.721 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:53.722 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:53.722 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-03-14T12:48:53.722 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:53.722 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 876 k 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-03-14T12:48:53.723 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:53.724 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 959 k 2024-03-14T12:48:53.724 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.1 M 2024-03-14T12:48:53.724 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:48:54.578 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 108 kB/s | 83 kB 00:00 2024-03-14T12:48:55.065 INFO:teuthology.orchestra.run.smithi137.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 1.4 MB/s | 4.8 MB 00:03 2024-03-14T12:48:55.066 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:55.066 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.6 MB/s | 6.0 MB 00:03 2024-03-14T12:48:55.119 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:48:55.130 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:48:55.130 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:48:55.230 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:48:55.233 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:48:55.500 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:48:55.670 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-17.2.7-646.gb92acd1f.el8.x86_6 471 kB/s | 876 kB 00:01 2024-03-14T12:48:55.671 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:48:55.671 INFO:teuthology.orchestra.run.smithi111.stdout:Total 492 kB/s | 959 kB 00:01 2024-03-14T12:48:55.683 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:48:55.705 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:48:55.705 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:48:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2024-03-14T12:48:55.796 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:48:55.799 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:48:55.964 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:48:56.250 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-14T12:48:56.474 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2/2 2024-03-14T12:48:56.730 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2024-03-14T12:48:56.854 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2/2 2024-03-14T12:48:56.854 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/2 2024-03-14T12:48:57.078 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2024-03-14T12:48:57.235 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-14T12:48:57.235 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:57.235 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:48:57.235 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-14T12:48:57.236 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:57.236 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:48:57.466 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-03-14T12:48:57.661 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 4/5 2024-03-14T12:48:57.850 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 5/5 2024-03-14T12:48:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 5/5 2024-03-14T12:48:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 1/5 2024-03-14T12:48:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2024-03-14T12:48:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/5 2024-03-14T12:48:58.144 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2024-03-14T12:48:58.234 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:24 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:17.2.7-646.gb92acd1f.el8.noarch 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-14T12:48:58.520 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-14T12:48:58.521 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-03-14T12:48:58.521 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-14T12:48:58.521 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:48:58.521 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:48:58.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-03-14T12:48:59.424 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:15 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:48:59.812 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:48:59.812 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 130 k 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 130 k 2024-03-14T12:48:59.813 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 464 k 2024-03-14T12:48:59.814 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:49:00.611 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-17.2.7-646.gb92acd1f.el8.x86_64. 163 kB/s | 130 kB 00:00 2024-03-14T12:49:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:Total 162 kB/s | 130 kB 00:00 2024-03-14T12:49:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:49:00.618 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:49:00.618 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:49:00.670 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:49:00.673 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:49:00.794 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:49:01.026 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:01.026 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:01.026 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-03-14T12:49:01.026 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 876 k 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-03-14T12:49:01.027 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:01.028 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 959 k 2024-03-14T12:49:01.028 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 3.1 M 2024-03-14T12:49:01.028 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:01.070 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:01.376 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:01.455 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 246 kB/s | 83 kB 00:00 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:01.792 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:01.988 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-03-14T12:49:02.754 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:28 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:02.981 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-17.2.7-646.gb92acd1f.el8.x86_6 470 kB/s | 876 kB 00:01 2024-03-14T12:49:02.981 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:02.981 INFO:teuthology.orchestra.run.smithi137.stdout:Total 491 kB/s | 959 kB 00:01 2024-03-14T12:49:02.994 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:03.016 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:03.016 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:03.110 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:03.113 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:03.276 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:03.549 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-14T12:49:03.748 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2/2 2024-03-14T12:49:04.123 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2/2 2024-03-14T12:49:04.123 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/2 2024-03-14T12:49:04.295 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:04.344 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:04.345 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:04.345 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:04.433 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-03-14T12:49:04.543 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-14T12:49:04.543 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:04.543 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:04.543 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-14T12:49:04.544 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:04.544 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:04.757 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-03-14T12:49:05.189 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:31 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:05.528 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:21 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:06.781 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:06.782 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:06.782 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 25 k 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 25 k 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 109 k 2024-03-14T12:49:06.783 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 130 k 2024-03-14T12:49:07.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 130 k 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 464 k 2024-03-14T12:49:07.124 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:07.373 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-17.2.7-646.gb92acd1f.el8.x86_64 42 kB/s | 25 kB 00:00 2024-03-14T12:49:07.373 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:07.374 INFO:teuthology.orchestra.run.smithi111.stdout:Total 42 kB/s | 25 kB 00:00 2024-03-14T12:49:07.374 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:49:07.378 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:49:07.378 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:49:07.380 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:49:07.383 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:49:07.432 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:49:07.622 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:07.874 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-17.2.7-646.gb92acd1f.el8.x86_64. 173 kB/s | 130 kB 00:00 2024-03-14T12:49:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:Total 172 kB/s | 130 kB 00:00 2024-03-14T12:49:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:07.936 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:07.939 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:08.215 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:08.216 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:08.216 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:49:08.216 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:08.216 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:08.216 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:08.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-03-14T12:49:09.080 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:35 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:10.231 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:10.538 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:10.614 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:10.661 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:10.662 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:10.662 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:10.876 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:10.916 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:11.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-03-14T12:49:11.623 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:37 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:11.949 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:27 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:13.166 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:13.213 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:13.214 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:13.214 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:13.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-03-14T12:49:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:13.548 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:13.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-03-14T12:49:14.059 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:40 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:14.427 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:30 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:15.596 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:15.643 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:15.644 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:15.644 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:15.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-03-14T12:49:16.016 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:16.016 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 25 k 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:49:16.017 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:16.018 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 25 k 2024-03-14T12:49:16.018 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 109 k 2024-03-14T12:49:16.018 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:16.383 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-17.2.7-646.gb92acd1f.el8.x86_64 68 kB/s | 25 kB 00:00 2024-03-14T12:49:16.383 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:16.383 INFO:teuthology.orchestra.run.smithi137.stdout:Total 68 kB/s | 25 kB 00:00 2024-03-14T12:49:16.384 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:16.387 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:16.387 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:16.389 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:16.392 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:16.443 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:16.493 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:42 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:16.638 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:16.913 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:17.258 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:17.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-03-14T12:49:18.029 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:18.076 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:18.077 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:18.077 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:18.139 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:34 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:18.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-03-14T12:49:18.941 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:44 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:19.686 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:19.733 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:19.734 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:19.734 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:19.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-03-14T12:49:20.483 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:20.530 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:20.531 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:20.531 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:20.606 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:36 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:20.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-03-14T12:49:21.381 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:47 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:22.154 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:22.201 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:22.202 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:22.202 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:22.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-03-14T12:49:22.942 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:22.988 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:22.989 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-03-14T12:49:22.989 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:23.073 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:39 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:23.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-03-14T12:49:23.840 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:49 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:24.629 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:24.676 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:24.677 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:24.677 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:24.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-03-14T12:49:25.434 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 91 k 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:25.435 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 91 k 2024-03-14T12:49:25.436 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 268 k 2024-03-14T12:49:25.436 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:49:25.550 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:41 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:25.954 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-17.2.7-646.gb92acd1f.el8.x86_64.rpm 175 kB/s | 91 kB 00:00 2024-03-14T12:49:25.954 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:25.954 INFO:teuthology.orchestra.run.smithi111.stdout:Total 175 kB/s | 91 kB 00:00 2024-03-14T12:49:25.955 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:49:25.964 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:49:25.964 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:49:26.015 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:49:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:49:26.123 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:49:26.344 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:26.629 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:27.086 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:27.134 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:27.134 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:27.134 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:27.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-03-14T12:49:27.262 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:27.262 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:27.262 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:49:27.263 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:27.263 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:27.263 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:27.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-03-14T12:49:28.022 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:44 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:28.206 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:54 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:29.546 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:29.594 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:29.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-03-14T12:49:29.809 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.5 M 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:49:29.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:29.811 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.5 M 2024-03-14T12:49:29.811 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 16 M 2024-03-14T12:49:29.811 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:49:30.469 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:46 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:32.006 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:17.2.7-646.gb92acd1f.el8.x86_64 is already installed. 2024-03-14T12:49:32.054 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:32.055 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-03-14T12:49:32.055 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:32.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-03-14T12:49:32.240 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-17.2.7-646.gb92acd1f.el8.x86_64.rpm 1.4 MB/s | 3.5 MB 00:02 2024-03-14T12:49:32.241 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:32.241 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 3.5 MB 00:02 2024-03-14T12:49:32.242 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:49:32.254 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:49:32.254 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:49:32.322 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:49:32.325 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:49:32.763 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:49:32.928 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:48 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:32.994 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:33.408 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:33.794 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:34.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-03-14T12:49:34.527 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:34.527 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 91 k 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:49:34.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:34.529 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 91 k 2024-03-14T12:49:34.529 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 268 k 2024-03-14T12:49:34.529 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:34.803 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:03:00 ago on Thu 14 Mar 2024 12:46:34 PM UTC. 2024-03-14T12:49:34.922 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-17.2.7-646.gb92acd1f.el8.x86_64.rpm 231 kB/s | 91 kB 00:00 2024-03-14T12:49:34.922 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:34.922 INFO:teuthology.orchestra.run.smithi137.stdout:Total 230 kB/s | 91 kB 00:00 2024-03-14T12:49:34.923 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:34.932 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:34.932 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:34.980 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:34.983 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:35.082 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:35.292 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:35.591 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:35.979 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:35.979 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:35.979 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:35.979 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:35.979 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:35.980 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:36.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-03-14T12:49:36.391 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-03-14T12:49:36.391 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-03-14T12:49:36.391 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 175 k 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:36.392 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 175 k 2024-03-14T12:49:36.393 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 556 k 2024-03-14T12:49:36.393 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-03-14T12:49:36.626 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-17.2.7-646.gb92acd1f.el8.x86_64.rpm 750 kB/s | 175 kB 00:00 2024-03-14T12:49:36.627 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:36.627 INFO:teuthology.orchestra.run.smithi111.stdout:Total 746 kB/s | 175 kB 00:00 2024-03-14T12:49:36.627 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-03-14T12:49:36.637 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-03-14T12:49:36.637 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-03-14T12:49:36.687 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-03-14T12:49:36.690 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-03-14T12:49:36.804 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-03-14T12:49:36.970 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:02:52 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:37.017 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:37.311 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:37.718 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:37.719 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-03-14T12:49:37.719 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:37.719 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:49:37.719 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-03-14T12:49:37.923 DEBUG:teuthology.parallel:result is None 2024-03-14T12:49:38.566 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:38.567 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:38.567 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-03-14T12:49:38.567 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:38.567 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 3.5 M 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.5 M 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 16 M 2024-03-14T12:49:38.568 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:43.907 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-17.2.7-646.gb92acd1f.el8.x86_64.rpm 663 kB/s | 3.5 MB 00:05 2024-03-14T12:49:43.908 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:43.908 INFO:teuthology.orchestra.run.smithi137.stdout:Total 663 kB/s | 3.5 MB 00:05 2024-03-14T12:49:43.909 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:43.921 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:43.921 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:43.987 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:43.990 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:44.431 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:44.600 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:45.017 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:45.427 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:45.427 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:45.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-03-14T12:49:46.417 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:03:02 ago on Thu 14 Mar 2024 12:46:44 PM UTC. 2024-03-14T12:49:48.006 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-03-14T12:49:48.006 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:48.006 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:17.2.7-646.gb92acd1f.el8 ceph 175 k 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-03-14T12:49:48.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:48.008 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 175 k 2024-03-14T12:49:48.008 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 556 k 2024-03-14T12:49:48.008 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-03-14T12:49:48.426 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-17.2.7-646.gb92acd1f.el8.x86_64.rpm 418 kB/s | 175 kB 00:00 2024-03-14T12:49:48.427 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:49:48.427 INFO:teuthology.orchestra.run.smithi137.stdout:Total 416 kB/s | 175 kB 00:00 2024-03-14T12:49:48.427 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-03-14T12:49:48.437 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-03-14T12:49:48.437 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-03-14T12:49:48.486 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-03-14T12:49:48.489 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-03-14T12:49:48.606 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-03-14T12:49:48.798 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:49.091 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:49.437 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 1/1 2024-03-14T12:49:49.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:49.438 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-03-14T12:49:49.438 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:17.2.7-646.gb92acd1f.el8.x86_64 2024-03-14T12:49:49.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:49:49.438 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-03-14T12:49:49.686 DEBUG:teuthology.parallel:result is None 2024-03-14T12:49:49.686 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:49:49.687 INFO:teuthology.packaging:ref: None 2024-03-14T12:49:49.687 INFO:teuthology.packaging:tag: None 2024-03-14T12:49:49.687 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:49:49.687 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:49:49.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-14T12:49:50.060 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-14T12:49:50.089 INFO:teuthology.orchestra.run.smithi111.stdout:17.2.7-646.gb92acd1f.el8 2024-03-14T12:49:50.089 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-646.gb92acd1f.el8 2024-03-14T12:49:50.090 INFO:teuthology.task.install:The correct ceph version 17.2.7-646.gb92acd1f is installed. 2024-03-14T12:49:50.095 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-14T12:49:50.095 INFO:teuthology.packaging:ref: None 2024-03-14T12:49:50.095 INFO:teuthology.packaging:tag: None 2024-03-14T12:49:50.095 INFO:teuthology.packaging:branch: quincy 2024-03-14T12:49:50.095 INFO:teuthology.packaging:sha1: 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:49:50.095 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-14T12:49:50.523 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-14T12:49:50.547 INFO:teuthology.orchestra.run.smithi137.stdout:17.2.7-646.gb92acd1f.el8 2024-03-14T12:49:50.547 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-646.gb92acd1f.el8 2024-03-14T12:49:50.547 INFO:teuthology.task.install:The correct ceph version 17.2.7-646.gb92acd1f is installed. 2024-03-14T12:49:50.549 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-14T12:49:50.549 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:49:50.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-14T12:49:50.583 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:49:50.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-14T12:49:50.615 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-14T12:49:50.615 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:49:50.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-03-14T12:49:50.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-14T12:49:50.724 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:49:50.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-03-14T12:49:50.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-14T12:49:50.819 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-14T12:49:50.820 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:49:50.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-14T12:49:50.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-14T12:49:50.919 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:49:50.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-14T12:49:50.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-14T12:49:51.014 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-14T12:49:51.015 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:49:51.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-03-14T12:49:51.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-14T12:49:51.113 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:49:51.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-03-14T12:49:51.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-14T12:49:51.206 INFO:teuthology.run_tasks:Running task print... 2024-03-14T12:49:51.216 INFO:teuthology.task.print:**** done install task... 2024-03-14T12:49:51.216 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-14T12:49:51.316 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '2f465319e3c83d48f999325459bcdfd0346449c0'} 2024-03-14T12:49:51.317 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-14T12:49:51.317 INFO:tasks.cephadm:Cluster fsid is 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:49:51.317 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-14T12:49:51.317 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-14T12:49:51.317 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi111': '172.21.15.111', 'mon.smithi137': '172.21.15.137'} 2024-03-14T12:49:51.317 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-14T12:49:51.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-03-14T12:49:51.352 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-03-14T12:49:51.380 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-14T12:49:51.380 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.667 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-rw-r--. 1 ubuntu ubuntu 373007 Mar 14 12:49 /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.667 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.869 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-rw-r--. 1 ubuntu ubuntu 373007 Mar 14 12:49 /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.869 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.892 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-14T12:49:51.920 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-14T12:49:51.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-14T12:49:51.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-14T12:49:53.086 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-14T12:49:53.112 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-14T12:50:19.657 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)", 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "a07fb176315d8840db15ba3ad0dc4eddc3a7cee2fa138062a96eea1508036767", 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:4c3f2234bee3fc97201a33f93a732b47a63f50905ac157bc7d3c4032ad9d5ab9" 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-03-14T12:50:19.658 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-03-14T12:50:22.766 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T12:50:22.766 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)", 2024-03-14T12:50:22.766 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "a07fb176315d8840db15ba3ad0dc4eddc3a7cee2fa138062a96eea1508036767", 2024-03-14T12:50:22.767 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-03-14T12:50:22.767 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:4c3f2234bee3fc97201a33f93a732b47a63f50905ac157bc7d3c4032ad9d5ab9" 2024-03-14T12:50:22.767 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-03-14T12:50:22.767 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T12:50:22.786 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-03-14T12:50:22.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-03-14T12:50:22.854 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-03-14T12:50:22.883 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-03-14T12:50:22.926 INFO:tasks.cephadm:Writing seed config... 2024-03-14T12:50:22.927 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-14T12:50:22.927 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-14T12:50:22.927 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-14T12:50:22.928 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-14T12:50:22.929 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-14T12:50:22.930 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-14T12:50:22.931 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:50:22.931 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-14T12:50:22.948 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 591fecbc-e201-11ee-95c9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-14T12:50:22.949 DEBUG:teuthology.orchestra.run.smithi111:mon.smithi111> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service 2024-03-14T12:50:22.991 INFO:tasks.cephadm:Bootstrapping... 2024-03-14T12:50:22.991 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 591fecbc-e201-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.111 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-14T12:50:23.017 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:-- Logs begin at Thu 2024-03-14 12:38:31 UTC. -- 2024-03-14T12:50:23.190 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-03-14T12:50:23.190 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '591fecbc-e201-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.111', '--skip-admin-label'] 2024-03-14T12:50:23.226 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-14T12:50:23.227 INFO:teuthology.orchestra.run.smithi111.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-14T12:50:23.227 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying podman|docker is present... 2024-03-14T12:50:23.262 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-14T12:50:23.262 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying lvm2 is present... 2024-03-14T12:50:23.262 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying time synchronization is in place... 2024-03-14T12:50:23.269 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-14T12:50:23.269 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-14T12:50:23.275 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-14T12:50:23.275 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:50:23.281 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2024-03-14T12:50:23.287 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2024-03-14T12:50:23.287 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2024-03-14T12:50:23.287 INFO:teuthology.orchestra.run.smithi111.stdout:Repeating the final host check... 2024-03-14T12:50:23.315 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-14T12:50:23.316 INFO:teuthology.orchestra.run.smithi111.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-14T12:50:23.316 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl is present 2024-03-14T12:50:23.316 INFO:teuthology.orchestra.run.smithi111.stdout:lvcreate is present 2024-03-14T12:50:23.321 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-14T12:50:23.322 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-14T12:50:23.327 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-14T12:50:23.327 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:50:23.333 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout enabled 2024-03-14T12:50:23.339 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout active 2024-03-14T12:50:23.339 INFO:teuthology.orchestra.run.smithi111.stdout:Unit chronyd.service is enabled and running 2024-03-14T12:50:23.339 INFO:teuthology.orchestra.run.smithi111.stdout:Host looks OK 2024-03-14T12:50:23.339 INFO:teuthology.orchestra.run.smithi111.stdout:Cluster fsid: 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:23.340 INFO:teuthology.orchestra.run.smithi111.stdout:Acquiring lock 140464532396520 on /run/cephadm/591fecbc-e201-11ee-95c9-87774f69a715.lock 2024-03-14T12:50:23.340 INFO:teuthology.orchestra.run.smithi111.stdout:Lock 140464532396520 acquired on /run/cephadm/591fecbc-e201-11ee-95c9-87774f69a715.lock 2024-03-14T12:50:23.340 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 3300 ... 2024-03-14T12:50:23.341 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying IP 172.21.15.111 port 6789 ... 2024-03-14T12:50:23.341 INFO:teuthology.orchestra.run.smithi111.stdout:Base mon IP(s) is [172.21.15.111:3300, 172.21.15.111:6789], mon addrv is [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-03-14T12:50:23.343 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.111 metric 100 2024-03-14T12:50:23.343 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.111 metric 100 2024-03-14T12:50:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-14T12:50:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-14T12:50:23.346 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1755sec hoplimit 64 pref medium 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2024-03-14T12:50:23.349 INFO:teuthology.orchestra.run.smithi111.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-14T12:50:23.350 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2024-03-14T12:50:23.350 INFO:teuthology.orchestra.run.smithi111.stdout:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2024-03-14T12:50:23.351 INFO:teuthology.orchestra.run.smithi111.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-14T12:50:23.351 INFO:teuthology.orchestra.run.smithi111.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-14T12:50:23.351 INFO:teuthology.orchestra.run.smithi111.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-14T12:50:23.783 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stdout a07fb176315d8840db15ba3ad0dc4eddc3a7cee2fa138062a96eea1508036767 2024-03-14T12:50:23.783 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-14T12:50:23.783 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Getting image source signatures 2024-03-14T12:50:23.783 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:9391575bb439fb790c83dd7c0657bf3ffc1728f54e0a8652f24a85e513441929 2024-03-14T12:50:23.783 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-14T12:50:23.784 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Copying config sha256:a07fb176315d8840db15ba3ad0dc4eddc3a7cee2fa138062a96eea1508036767 2024-03-14T12:50:23.784 INFO:teuthology.orchestra.run.smithi111.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-14T12:50:29.115 INFO:teuthology.orchestra.run.smithi111.stdout:ceph: stdout ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable) 2024-03-14T12:50:29.115 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph version: ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable) 2024-03-14T12:50:29.115 INFO:teuthology.orchestra.run.smithi111.stdout:Extracting ceph user uid/gid from container image... 2024-03-14T12:50:33.338 INFO:teuthology.orchestra.run.smithi111.stdout:stat: stdout 167 167 2024-03-14T12:50:33.338 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial keys... 2024-03-14T12:50:34.533 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQCZ8vJlnZFmLRAAhtcIL4RLG2BZOJ4EgMi68g== 2024-03-14T12:50:35.708 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQCa8vJltbQbOhAAWr7qH573qax5aUNdsm6frA== 2024-03-14T12:50:36.847 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph-authtool: stdout AQCc8vJl8FaSBxAAY6Kq1jvA68knnaLEQ+3+tQ== 2024-03-14T12:50:36.848 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial monmap... 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool for smithi111 [v2:172.21.15.111:3300,v1:172.21.15.111:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = octopus 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: set fsid to 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:50:37.833 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mon... 2024-03-14T12:50:38.922 INFO:teuthology.orchestra.run.smithi111.stdout:create mon.smithi111 on 2024-03-14T12:50:39.027 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-14T12:50:39.120 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-14T12:50:39.210 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target. 2024-03-14T12:50:39.210 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target. 2024-03-14T12:50:39.330 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111 2024-03-14T12:50:39.330 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service: Unit ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service not loaded. 2024-03-14T12:50:39.420 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target.wants/ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service → /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715@.service. 2024-03-14T12:50:40.016 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T12:50:40.016 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-03-14T12:50:40.023 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T12:50:40.023 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:50:40.024 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-03-14T12:50:40.024 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2024-03-14T12:50:40.024 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon to start... 2024-03-14T12:50:40.024 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mon... 2024-03-14T12:50:40.248 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 bash[39167]: 0266623a6b5d4853e145708d9f81e5a5ebbaf4d15de57dc1962592a09efc84a8 2024-03-14T12:50:40.248 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 systemd[1]: Started Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T12:50:41.087 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout cluster: 2024-03-14T12:50:41.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout id: 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:41.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-14T12:50:41.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:41.093 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout services: 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi111 (age 0.650648s) 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout data: 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout pgs: 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.538+0000 7f57b8657700 1 Processor -- start 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.538+0000 7f57b8657700 1 -- start start 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.538+0000 7f57b8657700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.538+0000 7f57b8657700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0102ed0 con 0x7f57b0102580 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b63f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b63f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33104/0 (socket says 172.21.15.111:33104) 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b63f3700 1 -- 172.21.15.111:0/1133303727 learned_addr learned my addr 172.21.15.111:0/1133303727 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b63f3700 1 -- 172.21.15.111:0/1133303727 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0103010 con 0x7f57b0102580 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b63f3700 1 --2- 172.21.15.111:0/1133303727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0098e0 tx=0x7f57ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f159a030a43016 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b53f1700 1 -- 172.21.15.111:0/1133303727 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac00b860 con 0x7f57b0102580 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b53f1700 1 -- 172.21.15.111:0/1133303727 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac00b9c0 con 0x7f57b0102580 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.539+0000 7f57b53f1700 1 -- 172.21.15.111:0/1133303727 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac00bc90 con 0x7f57b0102580 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 -- 172.21.15.111:0/1133303727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 msgr2=0x7f57b0102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 --2- 172.21.15.111:0/1133303727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0098e0 tx=0x7f57ac009250 comp rx=0 tx=0).stop 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 -- 172.21.15.111:0/1133303727 shutdown_connections 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 --2- 172.21.15.111:0/1133303727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0102990 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:41.095 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 -- 172.21.15.111:0/1133303727 >> 172.21.15.111:0/1133303727 conn(0x7f57b00fdd00 msgr2=0x7f57b0100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 -- 172.21.15.111:0/1133303727 shutdown_connections 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.540+0000 7f57b8657700 1 -- 172.21.15.111:0/1133303727 wait complete. 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b8657700 1 Processor -- start 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b8657700 1 -- start start 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b8657700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b8657700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0111110 con 0x7f57b0102580 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b63f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b63f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33120/0 (socket says 172.21.15.111:33120) 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b63f3700 1 -- 172.21.15.111:0/2590278608 learned_addr learned my addr 172.21.15.111:0/2590278608 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b63f3700 1 -- 172.21.15.111:0/2590278608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac0094f0 con 0x7f57b0102580 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b63f3700 1 --2- 172.21.15.111:0/2590278608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f57ac003080 tx=0x7f57ac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac01c5c0 con 0x7f57b0102580 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac01c720 con 0x7f57b0102580 2024-03-14T12:50:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.541+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b0111310 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.542+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ac01ca30 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.542+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b0111e80 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.542+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f57ac01a450 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.542+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f57ac02e790 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.542+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b005f5f0 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.545+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f57ac01a890 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.613+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f57b0112890 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.613+0000 7f57a6ffd700 1 -- 172.21.15.111:0/2590278608 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f57ac01aa30 con 0x7f57b0102580 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 msgr2=0x7f57b0110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 --2- 172.21.15.111:0/2590278608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f57ac003080 tx=0x7f57ac016000 comp rx=0 tx=0).stop 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 shutdown_connections 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 --2- 172.21.15.111:0/2590278608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b0102580 0x7f57b0110bd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 >> 172.21.15.111:0/2590278608 conn(0x7f57b00fdd00 msgr2=0x7f57b0104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:41.097 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 shutdown_connections 2024-03-14T12:50:41.098 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:40.614+0000 7f57b8657700 1 -- 172.21.15.111:0/2590278608 wait complete. 2024-03-14T12:50:41.098 INFO:teuthology.orchestra.run.smithi111.stdout:mon is available 2024-03-14T12:50:41.098 INFO:teuthology.orchestra.run.smithi111.stdout:Assimilating anything we can from ceph.conf... 2024-03-14T12:50:41.336 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-03-14T12:50:41.337 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T12:50:41.337 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: fsmap 2024-03-14T12:50:41.337 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: osdmap e1: 0 total, 0 up, 0 in 2024-03-14T12:50:41.337 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: mgrmap e1: no daemons active 2024-03-14T12:50:41.337 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:40 smithi111 ceph-mon[39205]: from='client.? 172.21.15.111:0/2590278608' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-14T12:50:42.332 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939b3d0700 1 Processor -- start 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939b3d0700 1 -- start start 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939b3d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939b3d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394105730 con 0x7f9394076c40 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939916c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939916c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33136/0 (socket says 172.21.15.111:33136) 2024-03-14T12:50:42.333 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939916c700 1 -- 172.21.15.111:0/425376771 learned_addr learned my addr 172.21.15.111:0/425376771 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.617+0000 7f939916c700 1 -- 172.21.15.111:0/425376771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394105870 con 0x7f9394076c40 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939916c700 1 --2- 172.21.15.111:0/425376771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f93900098e0 tx=0x7f9390009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42bf905952798af server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f938bfff700 1 -- 172.21.15.111:0/425376771 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939000b860 con 0x7f9394076c40 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f938bfff700 1 -- 172.21.15.111:0/425376771 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939000b9c0 con 0x7f9394076c40 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f938bfff700 1 -- 172.21.15.111:0/425376771 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939000bc90 con 0x7f9394076c40 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 -- 172.21.15.111:0/425376771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 msgr2=0x7f93941051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 --2- 172.21.15.111:0/425376771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f93900098e0 tx=0x7f9390009250 comp rx=0 tx=0).stop 2024-03-14T12:50:42.334 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 -- 172.21.15.111:0/425376771 shutdown_connections 2024-03-14T12:50:42.337 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 --2- 172.21.15.111:0/425376771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f93941051f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:42.337 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 -- 172.21.15.111:0/425376771 >> 172.21.15.111:0/425376771 conn(0x7f93940fcc60 msgr2=0x7f93940ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:42.337 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 -- 172.21.15.111:0/425376771 shutdown_connections 2024-03-14T12:50:42.337 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.618+0000 7f939b3d0700 1 -- 172.21.15.111:0/425376771 wait complete. 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939b3d0700 1 Processor -- start 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939b3d0700 1 -- start start 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939b3d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939b3d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9394103080 con 0x7f9394076c40 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939916c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939916c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33150/0 (socket says 172.21.15.111:33150) 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939916c700 1 -- 172.21.15.111:0/2754939513 learned_addr learned my addr 172.21.15.111:0/2754939513 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939916c700 1 -- 172.21.15.111:0/2754939513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93900094f0 con 0x7f9394076c40 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.619+0000 7f939916c700 1 --2- 172.21.15.111:0/2754939513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9390009290 tx=0x7f9390016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939001c5c0 con 0x7f9394076c40 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939001c740 con 0x7f9394076c40 2024-03-14T12:50:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f939001ca50 con 0x7f9394076c40 2024-03-14T12:50:42.339 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:41 smithi111 ceph-mon[39205]: from='client.? 172.21.15.111:0/2754939513' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-14T12:50:42.339 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:41 smithi111 ceph-mon[39205]: from='client.? 172.21.15.111:0/2754939513' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-14T12:50:42.340 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9394101150 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9394101640 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f939001a450 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.620+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f939002e790 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.621+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939404bed0 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.623+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f939001ce30 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.690+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f939410a320 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.693+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f939002e9f0 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.694+0000 7f9389ffb700 1 -- 172.21.15.111:0/2754939513 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f9390024630 con 0x7f9394076c40 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 msgr2=0x7f9394102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 --2- 172.21.15.111:0/2754939513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9390009290 tx=0x7f9390016000 comp rx=0 tx=0).stop 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 shutdown_connections 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 --2- 172.21.15.111:0/2754939513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9394076c40 0x7f9394102b40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 >> 172.21.15.111:0/2754939513 conn(0x7f93940fcc60 msgr2=0x7f93940fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:42.341 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 shutdown_connections 2024-03-14T12:50:42.342 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:41.695+0000 7f939b3d0700 1 -- 172.21.15.111:0/2754939513 wait complete. 2024-03-14T12:50:42.342 INFO:teuthology.orchestra.run.smithi111.stdout:Generating new minimal ceph.conf... 2024-03-14T12:50:43.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:42 smithi111 ceph-mon[39205]: from='client.? 172.21.15.111:0/1468674826' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:50:43.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.859+0000 7f0489872700 1 Processor -- start 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.859+0000 7f0489872700 1 -- start start 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.859+0000 7f0489872700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.859+0000 7f0489872700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0484103380 con 0x7f0484102a30 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0482ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0482ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33154/0 (socket says 172.21.15.111:33154) 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0482ffd700 1 -- 172.21.15.111:0/4058701831 learned_addr learned my addr 172.21.15.111:0/4058701831 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0482ffd700 1 -- 172.21.15.111:0/4058701831 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04841034c0 con 0x7f0484102a30 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0482ffd700 1 --2- 172.21.15.111:0/4058701831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e50 tx=0x7f0474009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e152056a1adf36 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0481ffb700 1 -- 172.21.15.111:0/4058701831 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04740092e0 con 0x7f0484102a30 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0481ffb700 1 -- 172.21.15.111:0/4058701831 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f047400be90 con 0x7f0484102a30 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.860+0000 7f0481ffb700 1 -- 172.21.15.111:0/4058701831 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0474007400 con 0x7f0484102a30 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 -- 172.21.15.111:0/4058701831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 msgr2=0x7f0484102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:43.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 --2- 172.21.15.111:0/4058701831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0474009e50 tx=0x7f0474009840 comp rx=0 tx=0).stop 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 -- 172.21.15.111:0/4058701831 shutdown_connections 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 --2- 172.21.15.111:0/4058701831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484102e40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 -- 172.21.15.111:0/4058701831 >> 172.21.15.111:0/4058701831 conn(0x7f04840fc400 msgr2=0x7f04840fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 -- 172.21.15.111:0/4058701831 shutdown_connections 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.861+0000 7f0489872700 1 -- 172.21.15.111:0/4058701831 wait complete. 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 Processor -- start 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 -- start start 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0484114140 con 0x7f0484102a30 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0482ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0482ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33156/0 (socket says 172.21.15.111:33156) 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0482ffd700 1 -- 172.21.15.111:0/1468674826 learned_addr learned my addr 172.21.15.111:0/1468674826 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:43.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0482ffd700 1 -- 172.21.15.111:0/1468674826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0474009ae0 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0482ffd700 1 --2- 172.21.15.111:0/1468674826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0474012b30 tx=0x7f0474009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0474007670 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0474007c40 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484114280 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047401ca20 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.862+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0484114770 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.863+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0474019070 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.863+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0474025cd0 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.863+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048404bed0 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.866+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f047402a060 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.932+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f04841022c0 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.932+0000 7f047b7fe700 1 -- 172.21.15.111:0/1468674826 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f0474020070 con 0x7f0484102a30 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 msgr2=0x7f0484101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:43.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 --2- 172.21.15.111:0/1468674826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0474012b30 tx=0x7f0474009840 comp rx=0 tx=0).stop 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 shutdown_connections 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 --2- 172.21.15.111:0/1468674826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0484102a30 0x7f0484101270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 >> 172.21.15.111:0/1468674826 conn(0x7f04840fc400 msgr2=0x7f04840fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 shutdown_connections 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:42.933+0000 7f0489872700 1 -- 172.21.15.111:0/1468674826 wait complete. 2024-03-14T12:50:43.450 INFO:teuthology.orchestra.run.smithi111.stdout:Restarting the monitor... 2024-03-14T12:50:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:43 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T12:50:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:43 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[39182]: 2024-03-14T12:50:43.672+0000 7f61348cf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T12:50:43.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:43 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[39182]: 2024-03-14T12:50:43.672+0000 7f61348cf700 -1 mon.smithi111@0(leader) e1 *** Got Signal Terminated *** 2024-03-14T12:50:44.410 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:44 smithi111 bash[39530]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111 2024-03-14T12:50:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:44 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service: Succeeded. 2024-03-14T12:50:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:44 smithi111 systemd[1]: Stopped Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T12:50:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:44 smithi111 systemd[1]: Starting Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T12:50:45.204 INFO:teuthology.orchestra.run.smithi111.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: set uid:gid to 167:167 (ceph:ceph) 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable), process ceph-mon, pid 7 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: pidfile_write: ignore empty --pid-file 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: load: jerasure load: lrc 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: RocksDB version: 6.15.5 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Compile date Mar 14 2024 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: DB SUMMARY 2024-03-14T12:50:45.416 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: DB Session ID: HOWYCF107OCJ2WKMJOFS 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: CURRENT file: CURRENT 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: IDENTITY file: IDENTITY 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 1, files: 000008.sst 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 000010.log size: 96760 ; 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.error_if_exists: 0 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.create_if_missing: 0 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.paranoid_checks: 1 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.env: 0x555821f22a80 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.fs: Posix File System 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.info_log: 0x555822c77c00 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_file_opening_threads: 16 2024-03-14T12:50:45.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.statistics: (nil) 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.use_fsync: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_log_file_size: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.keep_log_file_num: 1000 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.recycle_log_file_num: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_fallocate: 1 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_mmap_reads: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_mmap_writes: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.use_direct_reads: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.create_missing_column_families: 0 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.db_log_dir: 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi111/store.db 2024-03-14T12:50:45.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.advise_random_on_open: 1 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.db_write_buffer_size: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.write_buffer_manager: 0x5558239eca50 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.rate_limiter: (nil) 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-14T12:50:45.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.wal_recovery_mode: 2 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.enable_thread_tracking: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.enable_pipelined_write: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.unordered_write: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.row_cache: None 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.wal_filter: None 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_ingest_behind: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.preserve_deletes: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.two_write_queues: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T12:50:45.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.atomic_flush: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.log_readahead_size: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_background_jobs: 2 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_background_compactions: -1 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_subcompactions: 1 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T12:50:45.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_open_files: -1 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_background_flushes: -1 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Compression algorithms supported: 2024-03-14T12:50:45.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kZSTD supported: 0 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kXpressCompression supported: 0 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kZlibCompression supported: 1 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kSnappyCompression supported: 1 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kLZ4Compression supported: 1 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: kBZip2Compression supported: 0 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.merge_operator: 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_filter: None 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T12:50:45.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555822cccf58) 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_type: 0 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_index_type: 0 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_shortening: 1 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: hash_index_allow_collision: 1 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: checksum: 1 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: no_block_cache: 0 2024-03-14T12:50:45.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache: 0x555822cff090 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_options: 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: capacity : 536870912 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_shard_bits : 4 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: strict_capacity_limit : 0 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_compressed: (nil) 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: persistent_cache: (nil) 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size: 4096 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size_deviation: 10 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_restart_interval: 16 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_block_restart_interval: 1 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_block_size: 4096 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: partition_filters: 0 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: use_delta_encoding: 1 2024-03-14T12:50:45.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: whole_key_filtering: 1 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: verify_compression: 0 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: format_version: 4 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_index_compression: 1 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_align: 0 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression: NoCompression 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.num_levels: 7 2024-03-14T12:50:45.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T12:50:45.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T12:50:45.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.arena_block_size: 4194304 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T12:50:45.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.table_properties_collectors: 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.inplace_update_support: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.bloom_locality: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.max_successive_merges: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T12:50:45.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.ttl: 2592000 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.enable_blob_files: false 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.min_blob_size: 0 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710420645148441, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710420645149171, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92129, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90224, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82963, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710420645, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1f5f6b4a-8e77-42f1-9bab-b608194c8bf2", "db_session_id": "HOWYCF107OCJ2WKMJOFS"}} 2024-03-14T12:50:45.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710420645149490, "job": 1, "event": "recovery_finished"} 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi111/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x555822c93180 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: DB pointer 0x555822d68000 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** DB Stats ** 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T12:50:45.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T12:50:45.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 28.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 28.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 186.5 0.00 0.00 1 0.000 0 0 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 28.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T12:50:45.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: starting mon.smithi111 rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi111 fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???) e1 preinit fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).mds e1 new map 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).mds e1 print_map 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: e1 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: legacy client fscid: -1 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: No filesystems configured 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-14T12:50:45.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 bash[39650]: 90683c439e1ee119edfe07a20d26463dbe8344889373becf40b490ff9b90622b 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 systemd[1]: Started Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: fsmap 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: osdmap e1: 0 total, 0 up, 0 in 2024-03-14T12:50:45.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:45 smithi111 ceph-mon[39688]: mgrmap e1: no daemons active 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.700+0000 7f3aafa89700 1 Processor -- start 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.700+0000 7f3aafa89700 1 -- start start 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aafa89700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aafa89700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8102ec0 con 0x7f3aa8102570 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aad825700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aad825700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33160/0 (socket says 172.21.15.111:33160) 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aad825700 1 -- 172.21.15.111:0/191763058 learned_addr learned my addr 172.21.15.111:0/191763058 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aad825700 1 -- 172.21.15.111:0/191763058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8103000 con 0x7f3aa8102570 2024-03-14T12:50:46.377 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aad825700 1 --2- 172.21.15.111:0/191763058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a0c87c71135b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.701+0000 7f3aac823700 1 -- 172.21.15.111:0/191763058 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a9800b860 con 0x7f3aa8102570 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aac823700 1 -- 172.21.15.111:0/191763058 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3a9800be30 con 0x7f3aa8102570 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aac823700 1 -- 172.21.15.111:0/191763058 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a98010380 con 0x7f3aa8102570 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 -- 172.21.15.111:0/191763058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 msgr2=0x7f3aa8102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 --2- 172.21.15.111:0/191763058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a98009250 comp rx=0 tx=0).stop 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 -- 172.21.15.111:0/191763058 shutdown_connections 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 --2- 172.21.15.111:0/191763058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa8102980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 -- 172.21.15.111:0/191763058 >> 172.21.15.111:0/191763058 conn(0x7f3aa80fdcd0 msgr2=0x7f3aa8100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 -- 172.21.15.111:0/191763058 shutdown_connections 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.702+0000 7f3aafa89700 1 -- 172.21.15.111:0/191763058 wait complete. 2024-03-14T12:50:46.378 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aafa89700 1 Processor -- start 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aafa89700 1 -- start start 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aafa89700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aafa89700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa8111900 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aad825700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aad825700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33162/0 (socket says 172.21.15.111:33162) 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.703+0000 7f3aad825700 1 -- 172.21.15.111:0/960286469 learned_addr learned my addr 172.21.15.111:0/960286469 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3aad825700 1 -- 172.21.15.111:0/960286469 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a980094f0 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3aad825700 1 --2- 172.21.15.111:0/960286469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a98010960 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3a98024440 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa8111b00 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a9801a5f0 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa8112670 con 0x7f3aa8102570 2024-03-14T12:50:46.379 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3a9801aaa0 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.704+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3a9802da40 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.705+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa805f5f0 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.707+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3a98024950 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.772+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f3aa8104380 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.774+0000 7f3a9e7fc700 1 -- 172.21.15.111:0/960286469 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3a980245b0 con 0x7f3aa8102570 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 msgr2=0x7f3aa81113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 --2- 172.21.15.111:0/960286469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3a98000c00 tx=0x7f3a98009250 comp rx=0 tx=0).stop 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 shutdown_connections 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 --2- 172.21.15.111:0/960286469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3aa8102570 0x7f3aa81113c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 >> 172.21.15.111:0/960286469 conn(0x7f3aa80fdcd0 msgr2=0x7f3aa80fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 shutdown_connections 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:45.775+0000 7f3aafa89700 1 -- 172.21.15.111:0/960286469 wait complete. 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-14T12:50:46.380 INFO:teuthology.orchestra.run.smithi111.stdout:Creating mgr... 2024-03-14T12:50:46.381 INFO:teuthology.orchestra.run.smithi111.stdout:Verifying port 9283 ... 2024-03-14T12:50:46.507 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-591fecbc-e201-11ee-95c9-87774f69a715@mgr.smithi111.atysez 2024-03-14T12:50:46.507 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Failed to reset failed state of unit ceph-591fecbc-e201-11ee-95c9-87774f69a715@mgr.smithi111.atysez.service: Unit ceph-591fecbc-e201-11ee-95c9-87774f69a715@mgr.smithi111.atysez.service not loaded. 2024-03-14T12:50:46.596 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715.target.wants/ceph-591fecbc-e201-11ee-95c9-87774f69a715@mgr.smithi111.atysez.service → /etc/systemd/system/ceph-591fecbc-e201-11ee-95c9-87774f69a715@.service. 2024-03-14T12:50:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:46 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/960286469' entity='client.admin' 2024-03-14T12:50:47.261 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T12:50:47.261 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-03-14T12:50:47.269 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T12:50:47.269 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:50:47.269 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-03-14T12:50:47.269 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to enable service . firewalld.service is not available 2024-03-14T12:50:47.277 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T12:50:47.277 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr to start... 2024-03-14T12:50:47.285 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr... 2024-03-14T12:50:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:47 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1240754546' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T12:50:48.433 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "591fecbc-e201-11ee-95c9-87774f69a715", 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "smithi111" 2024-03-14T12:50:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T12:50:48.435 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T12:50:48.443 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T12:50:48.444 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T12:50:39.965456+0000", 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T12:50:48.445 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.796+0000 7fdfb5f8b700 1 Processor -- start 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.796+0000 7fdfb5f8b700 1 -- start start 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfb5f8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfb5f8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01033c0 con 0x7fdfb0102a70 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfaf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfaf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33186/0 (socket says 172.21.15.111:33186) 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfaf7fe700 1 -- 172.21.15.111:0/4015940589 learned_addr learned my addr 172.21.15.111:0/4015940589 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfaf7fe700 1 -- 172.21.15.111:0/4015940589 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0103500 con 0x7fdfb0102a70 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfaf7fe700 1 --2- 172.21.15.111:0/4015940589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009e50 tx=0x7fdfa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3b49007d8015fc server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.797+0000 7fdfae7fc700 1 -- 172.21.15.111:0/4015940589 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa80092e0 con 0x7fdfb0102a70 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfae7fc700 1 -- 172.21.15.111:0/4015940589 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdfa800be90 con 0x7fdfb0102a70 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfae7fc700 1 -- 172.21.15.111:0/4015940589 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa8007400 con 0x7fdfb0102a70 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/4015940589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 msgr2=0x7fdfb0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:48.446 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 --2- 172.21.15.111:0/4015940589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8009e50 tx=0x7fdfa8009840 comp rx=0 tx=0).stop 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/4015940589 shutdown_connections 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 --2- 172.21.15.111:0/4015940589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb0102e80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/4015940589 >> 172.21.15.111:0/4015940589 conn(0x7fdfb00fc400 msgr2=0x7fdfb00fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/4015940589 shutdown_connections 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.798+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/4015940589 wait complete. 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfb5f8b700 1 Processor -- start 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfb5f8b700 1 -- start start 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfb5f8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfb5f8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb0113af0 con 0x7fdfb0102a70 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfaf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfaf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33198/0 (socket says 172.21.15.111:33198) 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.799+0000 7fdfaf7fe700 1 -- 172.21.15.111:0/1240754546 learned_addr learned my addr 172.21.15.111:0/1240754546 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdfaf7fe700 1 -- 172.21.15.111:0/1240754546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa8009ae0 con 0x7fdfb0102a70 2024-03-14T12:50:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdfaf7fe700 1 --2- 172.21.15.111:0/1240754546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000f80 tx=0x7fdfa8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa801c620 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb0113cf0 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdfa801cbf0 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa8007b40 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb0114860 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdfa8019070 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.800+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdfa8025d70 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.801+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb005f5f0 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.803+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fdfa802a060 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.874+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdfb004bed0 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.875+0000 7fdf97fff700 1 -- 172.21.15.111:0/1240754546 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fdfa8020070 con 0x7fdfb0102a70 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 msgr2=0x7fdfb01135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 --2- 172.21.15.111:0/1240754546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8000f80 tx=0x7fdfa8009840 comp rx=0 tx=0).stop 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 shutdown_connections 2024-03-14T12:50:48.448 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 --2- 172.21.15.111:0/1240754546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb0102a70 0x7fdfb01135b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 >> 172.21.15.111:0/1240754546 conn(0x7fdfb00fc400 msgr2=0x7fdfb00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 shutdown_connections 2024-03-14T12:50:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:47.876+0000 7fdfb5f8b700 1 -- 172.21.15.111:0/1240754546 wait complete. 2024-03-14T12:50:48.449 INFO:teuthology.orchestra.run.smithi111.stdout:mgr not available, waiting (1/15)... 2024-03-14T12:50:51.342 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:51 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1946345877' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T12:50:51.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:51.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:50:51.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "591fecbc-e201-11ee-95c9-87774f69a715", 2024-03-14T12:50:51.789 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "smithi111" 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T12:50:51.790 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T12:50:51.791 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T12:50:51.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T12:50:51.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T12:50:51.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T12:50:51.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T12:50:51.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T12:50:39.965456+0000", 2024-03-14T12:50:51.801 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09d4834700 1 Processor -- start 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09d4834700 1 -- start start 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09d4834700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09d4834700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0103040 con 0x7f09d0076be0 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09ce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09ce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35486/0 (socket says 172.21.15.111:35486) 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09ce59c700 1 -- 172.21.15.111:0/2573314106 learned_addr learned my addr 172.21.15.111:0/2573314106 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.090+0000 7f09ce59c700 1 -- 172.21.15.111:0/2573314106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d0107b90 con 0x7f09d0076be0 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09ce59c700 1 --2- 172.21.15.111:0/2573314106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f09c00098e0 tx=0x7f09c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2674aacad1543ac server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:51.802 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09cd59a700 1 -- 172.21.15.111:0/2573314106 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c000b860 con 0x7f09d0076be0 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09cd59a700 1 -- 172.21.15.111:0/2573314106 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f09c000be30 con 0x7f09d0076be0 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09cd59a700 1 -- 172.21.15.111:0/2573314106 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c0010380 con 0x7f09d0076be0 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09d4834700 1 -- 172.21.15.111:0/2573314106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 msgr2=0x7f09d0107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09d4834700 1 --2- 172.21.15.111:0/2573314106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f09c00098e0 tx=0x7f09c0009250 comp rx=0 tx=0).stop 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.091+0000 7f09d4834700 1 -- 172.21.15.111:0/2573314106 shutdown_connections 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 --2- 172.21.15.111:0/2573314106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0107650 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 -- 172.21.15.111:0/2573314106 >> 172.21.15.111:0/2573314106 conn(0x7f09d00fcc60 msgr2=0x7f09d00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 -- 172.21.15.111:0/2573314106 shutdown_connections 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 -- 172.21.15.111:0/2573314106 wait complete. 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 Processor -- start 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 -- start start 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09d4834700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0113aa0 con 0x7f09d0076be0 2024-03-14T12:50:51.803 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09ce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09ce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35490/0 (socket says 172.21.15.111:35490) 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.092+0000 7f09ce59c700 1 -- 172.21.15.111:0/1946345877 learned_addr learned my addr 172.21.15.111:0/1946345877 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09ce59c700 1 -- 172.21.15.111:0/1946345877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c00094f0 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09ce59c700 1 --2- 172.21.15.111:0/1946345877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f09c0006b00 tx=0x7f09c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c0010a80 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d0113ca0 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f09c001a430 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c00245b0 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d0114810 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f09c0024ab0 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.093+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f09c002eb00 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.094+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d005f5f0 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.096+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f09c002ed60 con 0x7f09d0076be0 2024-03-14T12:50:51.804 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.167+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f09d01087f0 con 0x7f09d0076be0 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.168+0000 7f09beffd700 1 -- 172.21.15.111:0/1946345877 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f09c0010be0 con 0x7f09d0076be0 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 msgr2=0x7f09d0113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 --2- 172.21.15.111:0/1946345877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f09c0006b00 tx=0x7f09c0009250 comp rx=0 tx=0).stop 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 shutdown_connections 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 --2- 172.21.15.111:0/1946345877 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d0076be0 0x7f09d0113560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 >> 172.21.15.111:0/1946345877 conn(0x7f09d00fcc60 msgr2=0x7f09d00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 shutdown_connections 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:51.169+0000 7f09d4834700 1 -- 172.21.15.111:0/1946345877 wait complete. 2024-03-14T12:50:51.805 INFO:teuthology.orchestra.run.smithi111.stdout:mgr not available, waiting (2/15)... 2024-03-14T12:50:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: Activating manager daemon smithi111.atysez 2024-03-14T12:50:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: mgrmap e2: smithi111.atysez(active, starting, since 0.002895s) 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: Manager daemon smithi111.atysez is now available 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' 2024-03-14T12:50:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:53 smithi111 ceph-mon[39688]: from='mgr.14100 172.21.15.111:0/2876383542' entity='mgr.smithi111.atysez' 2024-03-14T12:50:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:55 smithi111 ceph-mon[39688]: mgrmap e3: smithi111.atysez(active, since 1.00725s) 2024-03-14T12:50:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:57 smithi111 ceph-mon[39688]: mgrmap e4: smithi111.atysez(active, since 3s) 2024-03-14T12:50:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:57 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3374733211' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsid": "591fecbc-e201-11ee-95c9-87774f69a715", 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "health": { 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-14T12:50:57.687 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 0 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "smithi111" 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.688 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-14T12:50:57.689 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-14T12:50:57.690 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "restful" 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ], 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "modified": "2024-03-14T12:50:39.965456+0000", 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout }, 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.732+0000 7f978d236700 1 Processor -- start 2024-03-14T12:50:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.732+0000 7f978d236700 1 -- start start 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.732+0000 7f978d236700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.732+0000 7f978d236700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9788079140 con 0x7f9788078d30 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9786d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9786d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35572/0 (socket says 172.21.15.111:35572) 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9786d9d700 1 -- 172.21.15.111:0/2122850333 learned_addr learned my addr 172.21.15.111:0/2122850333 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9786d9d700 1 -- 172.21.15.111:0/2122850333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9788077780 con 0x7f9788078d30 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9786d9d700 1 --2- 172.21.15.111:0/2122850333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97700098e0 tx=0x7f9770009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af67bceac9c7d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9785d9b700 1 -- 172.21.15.111:0/2122850333 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f977000b860 con 0x7f9788078d30 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9785d9b700 1 -- 172.21.15.111:0/2122850333 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f977000be30 con 0x7f9788078d30 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.733+0000 7f9785d9b700 1 -- 172.21.15.111:0/2122850333 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9770010380 con 0x7f9788078d30 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- 172.21.15.111:0/2122850333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 msgr2=0x7f9788077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 --2- 172.21.15.111:0/2122850333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97700098e0 tx=0x7f9770009250 comp rx=0 tx=0).stop 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- 172.21.15.111:0/2122850333 shutdown_connections 2024-03-14T12:50:57.692 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 --2- 172.21.15.111:0/2122850333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f9788077240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- 172.21.15.111:0/2122850333 >> 172.21.15.111:0/2122850333 conn(0x7f97880fdd00 msgr2=0x7f9788100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- 172.21.15.111:0/2122850333 shutdown_connections 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- 172.21.15.111:0/2122850333 wait complete. 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 Processor -- start 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.734+0000 7f978d236700 1 -- start start 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f978d236700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f978d236700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9788113af0 con 0x7f9788078d30 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f9786d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f9786d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35580/0 (socket says 172.21.15.111:35580) 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f9786d9d700 1 -- 172.21.15.111:0/3374733211 learned_addr learned my addr 172.21.15.111:0/3374733211 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f9786d9d700 1 -- 172.21.15.111:0/3374733211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700094f0 con 0x7f9788078d30 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f9786d9d700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9770006fd0 tx=0x7f9770009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:57.693 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9770010750 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9770010d20 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f977001ada0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9788113cf0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.735+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9788114860 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.736+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f97700193c0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.736+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978805f5f0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.736+0000 7f977f7fe700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 0x7f9774038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.736+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9770049db0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.736+0000 7f978659c700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 0x7f9774038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.737+0000 7f978659c700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 0x7f9774038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9778009990 tx=0x7f9778006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.741+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f9770022e40 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.999+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f978804bed0 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:56.999+0000 7f977f7fe700 1 -- 172.21.15.111:0/3374733211 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f9770022e40 con 0x7f9788078d30 2024-03-14T12:50:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 msgr2=0x7f9774038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 0x7f9774038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9778009990 tx=0x7f9778006e40 comp rx=0 tx=0).stop 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 msgr2=0x7f97881135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9770006fd0 tx=0x7f9770009250 comp rx=0 tx=0).stop 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 shutdown_connections 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f9774036790 0x7f9774038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 --2- 172.21.15.111:0/3374733211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9788078d30 0x7f97881135b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 >> 172.21.15.111:0/3374733211 conn(0x7f97880fdd00 msgr2=0x7f97880fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 shutdown_connections 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:57.001+0000 7f978d236700 1 -- 172.21.15.111:0/3374733211 wait complete. 2024-03-14T12:50:57.695 INFO:teuthology.orchestra.run.smithi111.stdout:mgr is available 2024-03-14T12:50:58.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:58 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/132213117' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-14T12:50:58.989 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:58.989 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [global] 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout fsid = 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout [osd] 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-14T12:50:58.990 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.202+0000 7fa077f2d700 1 Processor -- start 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.202+0000 7fa077f2d700 1 -- start start 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.202+0000 7fa077f2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.202+0000 7fa077f2d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070101e30 con 0x7fa0701014e0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa075cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa075cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35586/0 (socket says 172.21.15.111:35586) 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa075cc9700 1 -- 172.21.15.111:0/3905383147 learned_addr learned my addr 172.21.15.111:0/3905383147 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa075cc9700 1 -- 172.21.15.111:0/3905383147 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070101f70 con 0x7fa0701014e0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa075cc9700 1 --2- 172.21.15.111:0/3905383147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7723d1e2c4c39712 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa074cc7700 1 -- 172.21.15.111:0/3905383147 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c00b860 con 0x7fa0701014e0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa074cc7700 1 -- 172.21.15.111:0/3905383147 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa06c00be30 con 0x7fa0701014e0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.203+0000 7fa074cc7700 1 -- 172.21.15.111:0/3905383147 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c010380 con 0x7fa0701014e0 2024-03-14T12:50:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- 172.21.15.111:0/3905383147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 msgr2=0x7fa0701018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 --2- 172.21.15.111:0/3905383147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009250 comp rx=0 tx=0).stop 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- 172.21.15.111:0/3905383147 shutdown_connections 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 --2- 172.21.15.111:0/3905383147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa0701018f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- 172.21.15.111:0/3905383147 >> 172.21.15.111:0/3905383147 conn(0x7fa0700fcc60 msgr2=0x7fa0700ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- 172.21.15.111:0/3905383147 shutdown_connections 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- 172.21.15.111:0/3905383147 wait complete. 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 Processor -- start 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- start start 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.204+0000 7fa077f2d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0701140a0 con 0x7fa0701014e0 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa075cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa075cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35598/0 (socket says 172.21.15.111:35598) 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa075cc9700 1 -- 172.21.15.111:0/132213117 learned_addr learned my addr 172.21.15.111:0/132213117 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:50:58.992 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa075cc9700 1 -- 172.21.15.111:0/132213117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0094f0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa075cc9700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006b00 tx=0x7fa06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c0109b0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa06c024440 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0701141e0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c01a5f0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.205+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa070114670 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.206+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fa06c01a860 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.206+0000 7fa0667fc700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 0x7fa05c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.206+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07004bed0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.206+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa06c04a8f0 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.206+0000 7fa0754c8700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 0x7fa05c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.207+0000 7fa0754c8700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 0x7fa05c038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa060009990 tx=0x7fa060006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.211+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fa06c022340 con 0x7fa0701014e0 2024-03-14T12:50:58.993 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.391+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa0701032a0 con 0x7fa0701014e0 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.392+0000 7fa0667fc700 1 -- 172.21.15.111:0/132213117 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fa06c00fdc0 con 0x7fa0701014e0 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 msgr2=0x7fa05c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 0x7fa05c038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa060009990 tx=0x7fa060006e40 comp rx=0 tx=0).stop 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 msgr2=0x7fa070076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006b00 tx=0x7fa06c009250 comp rx=0 tx=0).stop 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 shutdown_connections 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fa05c036790 0x7fa05c038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 --2- 172.21.15.111:0/132213117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0701014e0 0x7fa070076b30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 >> 172.21.15.111:0/132213117 conn(0x7fa0700fcc60 msgr2=0x7fa0700fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 shutdown_connections 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:58.394+0000 7fa077f2d700 1 -- 172.21.15.111:0/132213117 wait complete. 2024-03-14T12:50:58.994 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling cephadm module... 2024-03-14T12:51:00.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:50:59 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2959044234' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-14T12:51:00.453 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b9845b700 1 Processor -- start 2024-03-14T12:51:00.453 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b9845b700 1 -- start start 2024-03-14T12:51:00.453 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b9845b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:00.453 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b9845b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90105630 con 0x7f2b90076d00 2024-03-14T12:51:00.453 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b961f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b961f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35612/0 (socket says 172.21.15.111:35612) 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b961f7700 1 -- 172.21.15.111:0/1598130371 learned_addr learned my addr 172.21.15.111:0/1598130371 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.517+0000 7f2b961f7700 1 -- 172.21.15.111:0/1598130371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b90105770 con 0x7f2b90076d00 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b961f7700 1 --2- 172.21.15.111:0/1598130371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098e0 tx=0x7f2b8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70224fd3f8fd6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b951f5700 1 -- 172.21.15.111:0/1598130371 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b8c00b860 con 0x7f2b90076d00 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b951f5700 1 -- 172.21.15.111:0/1598130371 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2b8c00be30 con 0x7f2b90076d00 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b951f5700 1 -- 172.21.15.111:0/1598130371 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b8c010380 con 0x7f2b90076d00 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 -- 172.21.15.111:0/1598130371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 msgr2=0x7f2b901050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 --2- 172.21.15.111:0/1598130371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098e0 tx=0x7f2b8c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 -- 172.21.15.111:0/1598130371 shutdown_connections 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 --2- 172.21.15.111:0/1598130371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90076d00 0x7f2b901050f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 -- 172.21.15.111:0/1598130371 >> 172.21.15.111:0/1598130371 conn(0x7f2b900fcb20 msgr2=0x7f2b900fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 -- 172.21.15.111:0/1598130371 shutdown_connections 2024-03-14T12:51:00.454 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.518+0000 7f2b9845b700 1 -- 172.21.15.111:0/1598130371 wait complete. 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b9845b700 1 Processor -- start 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b9845b700 1 -- start start 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b9845b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b9845b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90111cb0 con 0x7f2b90111360 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b961f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b961f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35624/0 (socket says 172.21.15.111:35624) 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b961f7700 1 -- 172.21.15.111:0/2959044234 learned_addr learned my addr 172.21.15.111:0/2959044234 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.519+0000 7f2b961f7700 1 -- 172.21.15.111:0/2959044234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c0094f0 con 0x7f2b90111360 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b961f7700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c006b00 tx=0x7f2b8c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b8c010a80 con 0x7f2b90111360 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2b8c01a430 con 0x7f2b90111360 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b90111eb0 con 0x7f2b90111360 2024-03-14T12:51:00.455 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b8c0245b0 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.520+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b90112890 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f2b8c0247e0 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9004bed0 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b86ffd700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 0x7f2b7c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2b8c04ad70 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b959f6700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 0x7f2b7c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.521+0000 7f2b959f6700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 0x7f2b7c038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009990 tx=0x7f2b80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.526+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f2b8c02c8f0 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.759+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2b90112ad0 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.812+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2b8c02e960 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.813+0000 7f2b86ffd700 1 -- 172.21.15.111:0/2959044234 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2b8c019d40 con 0x7f2b90111360 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 msgr2=0x7f2b7c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:00.456 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 0x7f2b7c038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009990 tx=0x7f2b80006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 msgr2=0x7f2b90111770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c006b00 tx=0x7f2b8c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 shutdown_connections 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2b7c036790 0x7f2b7c038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 --2- 172.21.15.111:0/2959044234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b90111360 0x7f2b90111770 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 >> 172.21.15.111:0/2959044234 conn(0x7f2b900fcb20 msgr2=0x7f2b900fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 shutdown_connections 2024-03-14T12:51:00.457 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:50:59.816+0000 7f2b9845b700 1 -- 172.21.15.111:0/2959044234 wait complete. 2024-03-14T12:51:01.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:00 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2959044234' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-14T12:51:01.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:00 smithi111 ceph-mon[39688]: mgrmap e5: smithi111.atysez(active, since 6s) 2024-03-14T12:51:01.852 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:51:01.852 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-14T12:51:01.852 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "smithi111.atysez", 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.007+0000 7fd5af587700 1 Processor -- start 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5af587700 1 -- start start 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5af587700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5af587700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8102ed0 con 0x7fd5a8102580 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5ad323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5ad323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52580/0 (socket says 172.21.15.111:52580) 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5ad323700 1 -- 172.21.15.111:0/730645376 learned_addr learned my addr 172.21.15.111:0/730645376 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5ad323700 1 -- 172.21.15.111:0/730645376 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8103010 con 0x7fd5a8102580 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.008+0000 7fd5ad323700 1 --2- 172.21.15.111:0/730645376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9e291feac4a7d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.009+0000 7fd59ffff700 1 -- 172.21.15.111:0/730645376 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a400b860 con 0x7fd5a8102580 2024-03-14T12:51:01.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.009+0000 7fd59ffff700 1 -- 172.21.15.111:0/730645376 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd5a400be30 con 0x7fd5a8102580 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.009+0000 7fd59ffff700 1 -- 172.21.15.111:0/730645376 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a4010380 con 0x7fd5a8102580 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.009+0000 7fd5af587700 1 -- 172.21.15.111:0/730645376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 msgr2=0x7fd5a8102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.009+0000 7fd5af587700 1 --2- 172.21.15.111:0/730645376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009250 comp rx=0 tx=0).stop 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- 172.21.15.111:0/730645376 shutdown_connections 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 --2- 172.21.15.111:0/730645376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8102990 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- 172.21.15.111:0/730645376 >> 172.21.15.111:0/730645376 conn(0x7fd5a80fdd00 msgr2=0x7fd5a8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- 172.21.15.111:0/730645376 shutdown_connections 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- 172.21.15.111:0/730645376 wait complete. 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 Processor -- start 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- start start 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5af587700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8111880 con 0x7fd5a8102580 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5ad323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5ad323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52586/0 (socket says 172.21.15.111:52586) 2024-03-14T12:51:01.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.010+0000 7fd5ad323700 1 -- 172.21.15.111:0/2119965527 learned_addr learned my addr 172.21.15.111:0/2119965527 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd5ad323700 1 -- 172.21.15.111:0/2119965527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40094f0 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd5ad323700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a4010a80 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd5a401a430 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a8111a80 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a40245b0 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.011+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a81125f0 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd5a40247f0 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd59dffb700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 0x7fd594038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd5a404ad80 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a805f5f0 con 0x7fd5a8102580 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd5acb22700 1 -- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 msgr2=0x7fd594038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.012+0000 7fd5acb22700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 0x7fd594038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:51:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.017+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd5a4018350 con 0x7fd5a8102580 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.213+0000 7fd5acb22700 1 -- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 msgr2=0x7fd594038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.213+0000 7fd5acb22700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 0x7fd594038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.259+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd5a8112930 con 0x7fd5a8102580 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.259+0000 7fd59dffb700 1 -- 172.21.15.111:0/2119965527 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd5a4023b80 con 0x7fd5a8102580 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 msgr2=0x7fd594038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 0x7fd594038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 msgr2=0x7fd5a8111340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4000c00 tx=0x7fd5a4009250 comp rx=0 tx=0).stop 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 shutdown_connections 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7fd594036790 0x7fd594038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 --2- 172.21.15.111:0/2119965527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5a8102580 0x7fd5a8111340 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.261+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 >> 172.21.15.111:0/2119965527 conn(0x7fd5a80fdd00 msgr2=0x7fd5a80fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.262+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 shutdown_connections 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:01.262+0000 7fd5af587700 1 -- 172.21.15.111:0/2119965527 wait complete. 2024-03-14T12:51:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2024-03-14T12:51:01.857 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 5... 2024-03-14T12:51:02.071 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:01 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2119965527' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-14T12:51:06.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: Active manager daemon smithi111.atysez restarted 2024-03-14T12:51:06.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: Activating manager daemon smithi111.atysez 2024-03-14T12:51:06.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: osdmap e2: 0 total, 0 up, 0 in 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: mgrmap e6: smithi111.atysez(active, starting, since 0.00334136s) 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: Manager daemon smithi111.atysez is now available 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:06.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:06 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: Found migration_current of "None". Setting to last migration. 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T12:51:07.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:07 smithi111 ceph-mon[39688]: mgrmap e7: smithi111.atysez(active, since 1.00543s) 2024-03-14T12:51:07.794 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.694+0000 7f286ade9700 1 Processor -- start 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f286ade9700 1 -- start start 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f286ade9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f286ade9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864079160 con 0x7f2864078d50 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f2868b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f2868b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52600/0 (socket says 172.21.15.111:52600) 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f2868b85700 1 -- 172.21.15.111:0/1436559929 learned_addr learned my addr 172.21.15.111:0/1436559929 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f2868b85700 1 -- 172.21.15.111:0/1436559929 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28640777a0 con 0x7f2864078d50 2024-03-14T12:51:07.795 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.695+0000 7f2868b85700 1 --2- 172.21.15.111:0/1436559929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28580098e0 tx=0x7f2858009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3368587b41afced server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f28637fe700 1 -- 172.21.15.111:0/1436559929 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f285800b860 con 0x7f2864078d50 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f28637fe700 1 -- 172.21.15.111:0/1436559929 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f285800be30 con 0x7f2864078d50 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f28637fe700 1 -- 172.21.15.111:0/1436559929 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2858010380 con 0x7f2864078d50 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 -- 172.21.15.111:0/1436559929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 msgr2=0x7f2864077260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 --2- 172.21.15.111:0/1436559929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28580098e0 tx=0x7f2858009250 comp rx=0 tx=0).stop 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 -- 172.21.15.111:0/1436559929 shutdown_connections 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 --2- 172.21.15.111:0/1436559929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f2864077260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 -- 172.21.15.111:0/1436559929 >> 172.21.15.111:0/1436559929 conn(0x7f28640fdd00 msgr2=0x7f2864100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 -- 172.21.15.111:0/1436559929 shutdown_connections 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.696+0000 7f286ade9700 1 -- 172.21.15.111:0/1436559929 wait complete. 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f286ade9700 1 Processor -- start 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f286ade9700 1 -- start start 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f286ade9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f286ade9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2864113ae0 con 0x7f2864078d50 2024-03-14T12:51:07.796 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f2868b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f2868b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52614/0 (socket says 172.21.15.111:52614) 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.697+0000 7f2868b85700 1 -- 172.21.15.111:0/914923697 learned_addr learned my addr 172.21.15.111:0/914923697 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f2868b85700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580094f0 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f2868b85700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f285800b440 tx=0x7f2858009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28580107a0 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2864113ce0 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2858010d70 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f285801adc0 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.698+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2864114850 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f28580193c0 con 0x7f2864078d50 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f28617fa700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2854039330 con 0x7f2854036770 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f2863fff700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f285804ae50 con 0x7f2864078d50 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.699+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.900+0000 7f2863fff700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:02.900+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:03.301+0000 7f2863fff700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:03.301+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:04.102+0000 7f2863fff700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:04.102+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:05.703+0000 7f2863fff700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/337686092 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:05.703+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:06.186+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f2858019710 con 0x7f2864078d50 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:06.186+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 msgr2=0x7f2854038c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:06.186+0000 7f28617fa700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.187+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f2858019a90 con 0x7f2864078d50 2024-03-14T12:51:07.798 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.187+0000 7f28617fa700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 0x7f285403c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.187+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2854039330 con 0x7f2854039ff0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.188+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 0x7f285403c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.188+0000 7f2863fff700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 0x7f285403c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f28500039d0 tx=0x7f2850009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.189+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f2854039330 con 0x7f2854039ff0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2864114170 con 0x7f2854039ff0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f28617fa700 1 -- 172.21.15.111:0/914923697 <== mgr.14118 v2:172.21.15.111:6800/119405983 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2864114170 con 0x7f2854039ff0 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 msgr2=0x7f285403c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f286ade9700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 0x7f285403c3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f28500039d0 tx=0x7f2850009350 comp rx=0 tx=0).stop 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 msgr2=0x7f28641135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.195+0000 7f286ade9700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f285800b440 tx=0x7f2858009250 comp rx=0 tx=0).stop 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 shutdown_connections 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f2854039ff0 0x7f285403c3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2864078d50 0x7f28641135a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 --2- 172.21.15.111:0/914923697 >> [v2:172.21.15.111:6800/337686092,v1:172.21.15.111:6801/337686092] conn(0x7f2854036770 0x7f2854038c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:07.799 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 >> 172.21.15.111:0/914923697 conn(0x7f28640fdd00 msgr2=0x7f28640fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:07.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 shutdown_connections 2024-03-14T12:51:07.800 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:07.196+0000 7f286ade9700 1 -- 172.21.15.111:0/914923697 wait complete. 2024-03-14T12:51:07.800 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 5 is available 2024-03-14T12:51:07.800 INFO:teuthology.orchestra.run.smithi111.stdout:Setting orchestrator backend to cephadm... 2024-03-14T12:51:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-14T12:51:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:07] ENGINE Bus STARTING 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:07] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:07] ENGINE Bus STARTED 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:08 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.315+0000 7f64f5a5e700 1 Processor -- start 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.315+0000 7f64f5a5e700 1 -- start start 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.315+0000 7f64f5a5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.315+0000 7f64f5a5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0101dd0 con 0x7f64f0101480 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52668/0 (socket says 172.21.15.111:52668) 2024-03-14T12:51:09.158 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64eeffd700 1 -- 172.21.15.111:0/294633746 learned_addr learned my addr 172.21.15.111:0/294633746 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64eeffd700 1 -- 172.21.15.111:0/294633746 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f0101f10 con 0x7f64f0101480 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64eeffd700 1 --2- 172.21.15.111:0/294633746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f64e40098e0 tx=0x7f64e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db275c53fda31d91 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64edffb700 1 -- 172.21.15.111:0/294633746 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e400b860 con 0x7f64f0101480 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64edffb700 1 -- 172.21.15.111:0/294633746 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f64e400be30 con 0x7f64f0101480 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.316+0000 7f64edffb700 1 -- 172.21.15.111:0/294633746 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e4010380 con 0x7f64f0101480 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 -- 172.21.15.111:0/294633746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 msgr2=0x7f64f0101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/294633746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f64e40098e0 tx=0x7f64e4009250 comp rx=0 tx=0).stop 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 -- 172.21.15.111:0/294633746 shutdown_connections 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/294633746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f0101890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 -- 172.21.15.111:0/294633746 >> 172.21.15.111:0/294633746 conn(0x7f64f00fcc00 msgr2=0x7f64f00ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 -- 172.21.15.111:0/294633746 shutdown_connections 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.317+0000 7f64f5a5e700 1 -- 172.21.15.111:0/294633746 wait complete. 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64f5a5e700 1 Processor -- start 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64f5a5e700 1 -- start start 2024-03-14T12:51:09.159 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64f5a5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64f5a5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0111900 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52678/0 (socket says 172.21.15.111:52678) 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64eeffd700 1 -- 172.21.15.111:0/2190017484 learned_addr learned my addr 172.21.15.111:0/2190017484 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64eeffd700 1 -- 172.21.15.111:0/2190017484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40094f0 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64eeffd700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f64e4003080 tx=0x7f64e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e40109b0 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f64e401a430 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0111b00 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.318+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e40245b0 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f01125d0 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f64e40247f0 con 0x7f64f0101480 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64d77fe700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 0x7f64d8038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:09.160 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f64e404a5b0 con 0x7f64f0101480 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f005f5f0 con 0x7f64f0101480 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.319+0000 7f64ee7fc700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 0x7f64d8038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.320+0000 7f64ee7fc700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 0x7f64d8038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009990 tx=0x7f64e0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.324+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f64e4018350 con 0x7f64f0101480 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.520+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f64f01130f0 con 0x7f64d8036660 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.528+0000 7f64d77fe700 1 -- 172.21.15.111:0/2190017484 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f64f01130f0 con 0x7f64d8036660 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 msgr2=0x7f64d8038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 0x7f64d8038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009990 tx=0x7f64e0006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 msgr2=0x7f64f01113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f64e4003080 tx=0x7f64e4009250 comp rx=0 tx=0).stop 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 shutdown_connections 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f64d8036660 0x7f64d8038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 --2- 172.21.15.111:0/2190017484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64f0101480 0x7f64f01113c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.530+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 >> 172.21.15.111:0/2190017484 conn(0x7f64f00fcc00 msgr2=0x7f64f00fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:09.161 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.531+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 shutdown_connections 2024-03-14T12:51:09.162 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:08.531+0000 7f64f5a5e700 1 -- 172.21.15.111:0/2190017484 wait complete. 2024-03-14T12:51:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:09 smithi111 ceph-mon[39688]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:09 smithi111 ceph-mon[39688]: mgrmap e8: smithi111.atysez(active, since 2s) 2024-03-14T12:51:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:09 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:10.482 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-14T12:51:10.482 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.654+0000 7f69107d5700 1 Processor -- start 2024-03-14T12:51:10.482 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.654+0000 7f69107d5700 1 -- start start 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f69107d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f69107d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69081023c0 con 0x7f6908101a70 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f690e571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f690e571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52682/0 (socket says 172.21.15.111:52682) 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f690e571700 1 -- 172.21.15.111:0/2568690991 learned_addr learned my addr 172.21.15.111:0/2568690991 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f690e571700 1 -- 172.21.15.111:0/2568690991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908102500 con 0x7f6908101a70 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.655+0000 7f690e571700 1 --2- 172.21.15.111:0/2568690991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68f80098e0 tx=0x7f68f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe04f37df63f3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f690d56f700 1 -- 172.21.15.111:0/2568690991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f800b860 con 0x7f6908101a70 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f690d56f700 1 -- 172.21.15.111:0/2568690991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f68f800be30 con 0x7f6908101a70 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f690d56f700 1 -- 172.21.15.111:0/2568690991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f8010380 con 0x7f6908101a70 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 -- 172.21.15.111:0/2568690991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 msgr2=0x7f6908101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 --2- 172.21.15.111:0/2568690991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f68f80098e0 tx=0x7f68f8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 -- 172.21.15.111:0/2568690991 shutdown_connections 2024-03-14T12:51:10.483 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 --2- 172.21.15.111:0/2568690991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908101e80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 -- 172.21.15.111:0/2568690991 >> 172.21.15.111:0/2568690991 conn(0x7f69080fc400 msgr2=0x7f69080fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 -- 172.21.15.111:0/2568690991 shutdown_connections 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.656+0000 7f69107d5700 1 -- 172.21.15.111:0/2568690991 wait complete. 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f69107d5700 1 Processor -- start 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f69107d5700 1 -- start start 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f69107d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f69107d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908110790 con 0x7f6908101a70 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f690e571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f690e571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52692/0 (socket says 172.21.15.111:52692) 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.657+0000 7f690e571700 1 -- 172.21.15.111:0/1207692521 learned_addr learned my addr 172.21.15.111:0/1207692521 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f690e571700 1 -- 172.21.15.111:0/1207692521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f80094f0 con 0x7f6908101a70 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f690e571700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68f8006d40 tx=0x7f68f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f80109b0 con 0x7f6908101a70 2024-03-14T12:51:10.484 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f68f8024440 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68f801a5f0 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6908110990 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.658+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6908111500 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f68f801a820 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f68feffd700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 0x7f68f4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f68f804a5c0 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f690dd70700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 0x7f68f4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690805f5f0 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.659+0000 7f690dd70700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 0x7f68f4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6904009990 tx=0x7f6904006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.664+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f68f801aad0 con 0x7f6908101a70 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.845+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f69080fd5f0 con 0x7f68f40367e0 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.846+0000 7f68feffd700 1 -- 172.21.15.111:0/1207692521 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f69080fd5f0 con 0x7f68f40367e0 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 msgr2=0x7f68f4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:10.485 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 0x7f68f4038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6904009990 tx=0x7f6904006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 msgr2=0x7f6908110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f68f8006d40 tx=0x7f68f8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 shutdown_connections 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f68f40367e0 0x7f68f4038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 --2- 172.21.15.111:0/1207692521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6908101a70 0x7f6908110250 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 >> 172.21.15.111:0/1207692521 conn(0x7f69080fc400 msgr2=0x7f69080fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 shutdown_connections 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:09.848+0000 7f69107d5700 1 -- 172.21.15.111:0/1207692521 wait complete. 2024-03-14T12:51:10.486 INFO:teuthology.orchestra.run.smithi111.stdout:Generating ssh key... 2024-03-14T12:51:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:10 smithi111 ceph-mon[39688]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.026+0000 7f7f128fa700 1 Processor -- start 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f128fa700 1 -- start start 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f128fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f128fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c1056f0 con 0x7f7f0c076c80 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38346/0 (socket says 172.21.15.111:38346) 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0bfff700 1 -- 172.21.15.111:0/986343593 learned_addr learned my addr 172.21.15.111:0/986343593 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0bfff700 1 -- 172.21.15.111:0/986343593 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c105830 con 0x7f7f0c076c80 2024-03-14T12:51:12.004 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0bfff700 1 --2- 172.21.15.111:0/986343593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0098e0 tx=0x7f7efc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef53d02f11d93de server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0affd700 1 -- 172.21.15.111:0/986343593 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7efc00b860 con 0x7f7f0c076c80 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0affd700 1 -- 172.21.15.111:0/986343593 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7efc00be30 con 0x7f7f0c076c80 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.027+0000 7f7f0affd700 1 -- 172.21.15.111:0/986343593 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7efc010380 con 0x7f7f0c076c80 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 -- 172.21.15.111:0/986343593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 msgr2=0x7f7f0c1051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 --2- 172.21.15.111:0/986343593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0098e0 tx=0x7f7efc009250 comp rx=0 tx=0).stop 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 -- 172.21.15.111:0/986343593 shutdown_connections 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 --2- 172.21.15.111:0/986343593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c1051b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 -- 172.21.15.111:0/986343593 >> 172.21.15.111:0/986343593 conn(0x7f7f0c0fcc60 msgr2=0x7f7f0c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 -- 172.21.15.111:0/986343593 shutdown_connections 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.028+0000 7f7f128fa700 1 -- 172.21.15.111:0/986343593 wait complete. 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f128fa700 1 Processor -- start 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f128fa700 1 -- start start 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f128fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f128fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c1118b0 con 0x7f7f0c076c80 2024-03-14T12:51:12.005 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f0bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38348/0 (socket says 172.21.15.111:38348) 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f0bfff700 1 -- 172.21.15.111:0/3876846247 learned_addr learned my addr 172.21.15.111:0/3876846247 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f0bfff700 1 -- 172.21.15.111:0/3876846247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0094f0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.029+0000 7f7f0bfff700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7efc006d40 tx=0x7f7efc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7efc0109b0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7efc024440 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c111ab0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7efc01a5f0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c112620 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7efc01a870 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.030+0000 7f7f08ff9700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 0x7f7ef8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.031+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7efc04a6e0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.031+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c05f5f0 con 0x7f7f0c076c80 2024-03-14T12:51:12.006 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.031+0000 7f7f0b7fe700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 0x7f7ef8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.031+0000 7f7f0b7fe700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 0x7f7ef8038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009990 tx=0x7f7ef4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.035+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f7efc018d90 con 0x7f7f0c076c80 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.218+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f0c0fde50 con 0x7f7ef8036790 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.481+0000 7f7f08ff9700 1 -- 172.21.15.111:0/3876846247 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7f0c0fde50 con 0x7f7ef8036790 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 msgr2=0x7f7ef8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 0x7f7ef8038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009990 tx=0x7f7ef4006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 msgr2=0x7f7f0c111370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7efc006d40 tx=0x7f7efc009250 comp rx=0 tx=0).stop 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 shutdown_connections 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f7ef8036790 0x7f7ef8038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:12.007 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 --2- 172.21.15.111:0/3876846247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f0c076c80 0x7f7f0c111370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:12.008 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 >> 172.21.15.111:0/3876846247 conn(0x7f7f0c0fcc60 msgr2=0x7f7f0c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:12.008 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 shutdown_connections 2024-03-14T12:51:12.008 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:11.483+0000 7f7f128fa700 1 -- 172.21.15.111:0/3876846247 wait complete. 2024-03-14T12:51:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:12 smithi111 ceph-mon[39688]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:12 smithi111 ceph-mon[39688]: Generating ssh key... 2024-03-14T12:51:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:12 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:12.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:12 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxVTN0xCw/SDwds4enQ5odcJW32BZ0gR2vdKkyE9NitFE5LrorQU5lWGft3J4E/D2FmQqRlNlRQs2yy9bsP7jUQX/A10oor3sxbd3gz40UcGoW24cLA9rr8o0KymJhkB4nXkOVpC4suZXhb/eH02li+Qkbb0Qch7/IWgKNrCZoLC9AkGCtPKD/ru+TXXV8KgpxknIfSTQTDOtWeVhhBSVF2a5djzEJsMIudqP8JAGdYqFPT003/wWvbdxqUKQKQrLthMqcWA8kMahk79QtG+PbR/ODTBqvoB/NN7p4Do91nMbe3p9CxEhPufyLrn60dw1gmh+L/Ktk+XBGdKh2xlOvBhMX3xBEQn9qVMNreMtXF5+ov4TBonChO3BXxrTAOrIZUS9D+DMixJyc6OKch6475WSnD7TBrL6FKFhE7iVDlD2GgTEsqCuAzbIebnhYeQK3eCwsNT7vyQfOkj4A5S6QdQJBKUNY0pf3WvOT0hNASC/btvWZDowTRvpcSWxfeU= ceph-591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a2b5d700 1 Processor -- start 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a2b5d700 1 -- start start 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a2b5d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a2b5d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c101e30 con 0x7f299c1014e0 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38356/0 (socket says 172.21.15.111:38356) 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.535+0000 7f29a08f9700 1 -- 172.21.15.111:0/2223022723 learned_addr learned my addr 172.21.15.111:0/2223022723 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f29a08f9700 1 -- 172.21.15.111:0/2223022723 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c101f70 con 0x7f299c1014e0 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f29a08f9700 1 --2- 172.21.15.111:0/2223022723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29900098e0 tx=0x7f2990009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85e68437aa767e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f299b7fe700 1 -- 172.21.15.111:0/2223022723 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299000b860 con 0x7f299c1014e0 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f299b7fe700 1 -- 172.21.15.111:0/2223022723 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f299000be30 con 0x7f299c1014e0 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f299b7fe700 1 -- 172.21.15.111:0/2223022723 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2990010380 con 0x7f299c1014e0 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f29a2b5d700 1 -- 172.21.15.111:0/2223022723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 msgr2=0x7f299c1018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.536+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/2223022723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f29900098e0 tx=0x7f2990009250 comp rx=0 tx=0).stop 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- 172.21.15.111:0/2223022723 shutdown_connections 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/2223022723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c1018f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- 172.21.15.111:0/2223022723 >> 172.21.15.111:0/2223022723 conn(0x7f299c0fcc60 msgr2=0x7f299c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- 172.21.15.111:0/2223022723 shutdown_connections 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- 172.21.15.111:0/2223022723 wait complete. 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 Processor -- start 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- start start 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a2b5d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c1140a0 con 0x7f299c1014e0 2024-03-14T12:51:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.537+0000 7f29a08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38364/0 (socket says 172.21.15.111:38364) 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a08f9700 1 -- 172.21.15.111:0/1127891203 learned_addr learned my addr 172.21.15.111:0/1127891203 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a08f9700 1 -- 172.21.15.111:0/1127891203 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29900094f0 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a08f9700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2990006b00 tx=0x7f2990009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29900109b0 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2990024440 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c1141e0 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299001a5f0 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.538+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c114670 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f299001a860 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f29997fa700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 0x7f298c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c04bed0 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f299004a940 con 0x7f299c1014e0 2024-03-14T12:51:13.247 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f299bfff700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 0x7f298c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.539+0000 7f299bfff700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 0x7f298c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2988009990 tx=0x7f2988006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.544+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f299001ab60 con 0x7f299c1014e0 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.727+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f299c0fde50 con 0x7f298c0367e0 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.728+0000 7f29997fa700 1 -- 172.21.15.111:0/1127891203 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f299c0fde50 con 0x7f298c0367e0 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 msgr2=0x7f298c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 0x7f298c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2988009990 tx=0x7f2988006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 msgr2=0x7f299c076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2990006b00 tx=0x7f2990009250 comp rx=0 tx=0).stop 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 shutdown_connections 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f298c0367e0 0x7f298c038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 --2- 172.21.15.111:0/1127891203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f299c1014e0 0x7f299c076b30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 >> 172.21.15.111:0/1127891203 conn(0x7f299c0fcc60 msgr2=0x7f299c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 shutdown_connections 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:12.730+0000 7f29a2b5d700 1 -- 172.21.15.111:0/1127891203 wait complete. 2024-03-14T12:51:13.248 INFO:teuthology.orchestra.run.smithi111.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-14T12:51:13.249 INFO:teuthology.orchestra.run.smithi111.stdout:Adding key to root@localhost authorized_keys... 2024-03-14T12:51:13.249 INFO:teuthology.orchestra.run.smithi111.stdout:Adding host smithi111... 2024-03-14T12:51:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:13 smithi111 ceph-mon[39688]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:14 smithi111 ceph-mon[39688]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:14 smithi111 ceph-mon[39688]: Deploying cephadm binary to smithi111 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Added host 'smithi111' with addr '172.21.15.111' 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b392b700 1 Processor -- start 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b392b700 1 -- start start 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b392b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b392b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac079180 con 0x7fb4ac078d70 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b16c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:16.295 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b16c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38372/0 (socket says 172.21.15.111:38372) 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b16c7700 1 -- 172.21.15.111:0/43373615 learned_addr learned my addr 172.21.15.111:0/43373615 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.790+0000 7fb4b16c7700 1 -- 172.21.15.111:0/43373615 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac0777c0 con 0x7fb4ac078d70 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b16c7700 1 --2- 172.21.15.111:0/43373615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80098e0 tx=0x7fb4a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d09dc8b70cd9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4a3fff700 1 -- 172.21.15.111:0/43373615 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a800b860 con 0x7fb4ac078d70 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4a3fff700 1 -- 172.21.15.111:0/43373615 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb4a800be30 con 0x7fb4ac078d70 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4a3fff700 1 -- 172.21.15.111:0/43373615 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a8010380 con 0x7fb4ac078d70 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 -- 172.21.15.111:0/43373615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 msgr2=0x7fb4ac077280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 --2- 172.21.15.111:0/43373615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80098e0 tx=0x7fb4a8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 -- 172.21.15.111:0/43373615 shutdown_connections 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 --2- 172.21.15.111:0/43373615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac077280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 -- 172.21.15.111:0/43373615 >> 172.21.15.111:0/43373615 conn(0x7fb4ac0fdcd0 msgr2=0x7fb4ac100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 -- 172.21.15.111:0/43373615 shutdown_connections 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.791+0000 7fb4b392b700 1 -- 172.21.15.111:0/43373615 wait complete. 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b392b700 1 Processor -- start 2024-03-14T12:51:16.296 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b392b700 1 -- start start 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b392b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b392b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac113b40 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b16c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b16c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38378/0 (socket says 172.21.15.111:38378) 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b16c7700 1 -- 172.21.15.111:0/3824206893 learned_addr learned my addr 172.21.15.111:0/3824206893 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.792+0000 7fb4b16c7700 1 -- 172.21.15.111:0/3824206893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80094f0 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4b16c7700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8006d40 tx=0x7fb4a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a80109b0 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb4a801a430 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac113d40 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a80245b0 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.793+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac1148b0 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb4a802d3f0 con 0x7fb4ac078d70 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4a1ffb700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 0x7fb498038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:16.297 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb4a804a4f0 con 0x7fb4ac078d70 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4b0ec6700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 0x7fb498038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac05f5f0 con 0x7fb4ac078d70 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.794+0000 7fb4b0ec6700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 0x7fb498038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009990 tx=0x7fb49c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.799+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb4a802b8e0 con 0x7fb4ac078d70 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:13.985+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4ac0ff090 con 0x7fb4980367e0 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.666+0000 7fb4a1ffb700 1 -- 172.21.15.111:0/3824206893 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb4ac0ff090 con 0x7fb4980367e0 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 msgr2=0x7fb498038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 0x7fb498038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009990 tx=0x7fb49c006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 msgr2=0x7fb4ac113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8006d40 tx=0x7fb4a8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 shutdown_connections 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb4980367e0 0x7fb498038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 --2- 172.21.15.111:0/3824206893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4ac078d70 0x7fb4ac113600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:16.299 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 >> 172.21.15.111:0/3824206893 conn(0x7fb4ac0fdcd0 msgr2=0x7fb4ac0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:16.299 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 shutdown_connections 2024-03-14T12:51:16.299 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:15.669+0000 7fb4b392b700 1 -- 172.21.15.111:0/3824206893 wait complete. 2024-03-14T12:51:16.299 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying mon service with default placement... 2024-03-14T12:51:17.104 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:16 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:17.104 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:16 smithi111 ceph-mon[39688]: Added host smithi111 2024-03-14T12:51:17.104 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:16 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:18.319 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:18 smithi111 ceph-mon[39688]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:18.319 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:18 smithi111 ceph-mon[39688]: Saving service mon spec with placement count:5 2024-03-14T12:51:18.319 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:18 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:18.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-14T12:51:18.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6db9db700 1 Processor -- start 2024-03-14T12:51:18.629 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6db9db700 1 -- start start 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6db9db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6db9db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d4101e30 con 0x7fe6d41014e0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6d9777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6d9777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38388/0 (socket says 172.21.15.111:38388) 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.849+0000 7fe6d9777700 1 -- 172.21.15.111:0/1981686444 learned_addr learned my addr 172.21.15.111:0/1981686444 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6d9777700 1 -- 172.21.15.111:0/1981686444 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d4101f70 con 0x7fe6d41014e0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6d9777700 1 --2- 172.21.15.111:0/1981686444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000bb80 tx=0x7fe6d000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cf54d95e5d8d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6cbfff700 1 -- 172.21.15.111:0/1981686444 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d000dd00 con 0x7fe6d41014e0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6cbfff700 1 -- 172.21.15.111:0/1981686444 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe6d000de60 con 0x7fe6d41014e0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6cbfff700 1 -- 172.21.15.111:0/1981686444 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d0018840 con 0x7fe6d41014e0 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 -- 172.21.15.111:0/1981686444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 msgr2=0x7fe6d41018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1981686444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000bb80 tx=0x7fe6d000b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 -- 172.21.15.111:0/1981686444 shutdown_connections 2024-03-14T12:51:18.630 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1981686444 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d41018f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 -- 172.21.15.111:0/1981686444 >> 172.21.15.111:0/1981686444 conn(0x7fe6d40fcc60 msgr2=0x7fe6d40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 -- 172.21.15.111:0/1981686444 shutdown_connections 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.850+0000 7fe6db9db700 1 -- 172.21.15.111:0/1981686444 wait complete. 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6db9db700 1 Processor -- start 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6db9db700 1 -- start start 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6db9db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6db9db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d410f620 con 0x7fe6d41014e0 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6d9777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6d9777700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38396/0 (socket says 172.21.15.111:38396) 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6d9777700 1 -- 172.21.15.111:0/1495703318 learned_addr learned my addr 172.21.15.111:0/1495703318 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.851+0000 7fe6d9777700 1 -- 172.21.15.111:0/1495703318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d000b790 con 0x7fe6d41014e0 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6d9777700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0006d40 tx=0x7fe6d000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d0010a00 con 0x7fe6d41014e0 2024-03-14T12:51:18.631 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe6d001d050 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d410f820 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6d0022590 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.852+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d4110390 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe6d0022740 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d405f5f0 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6c9ffb700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe6d0014070 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6d8f76700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.853+0000 7fe6d8f76700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 0x7fe6c0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:16.857+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fe6d000e360 con 0x7fe6d41014e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.068+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6d4110740 con 0x7fe6c00367e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.074+0000 7fe6c9ffb700 1 -- 172.21.15.111:0/1495703318 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe6d4110740 con 0x7fe6c00367e0 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 msgr2=0x7fe6c0038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:18.632 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 0x7fe6c0038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009990 tx=0x7fe6c4006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 msgr2=0x7fe6d410f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0006d40 tx=0x7fe6d000b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 shutdown_connections 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe6c00367e0 0x7fe6c0038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 --2- 172.21.15.111:0/1495703318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6d41014e0 0x7fe6d410f0e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 >> 172.21.15.111:0/1495703318 conn(0x7fe6d40fcc60 msgr2=0x7fe6d4105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 shutdown_connections 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:17.076+0000 7fe6db9db700 1 -- 172.21.15.111:0/1495703318 wait complete. 2024-03-14T12:51:18.633 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying mgr service with default placement... 2024-03-14T12:51:19.547 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:19 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:19.547 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:19 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:20.070 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f4760dd6700 1 Processor -- start 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f4760dd6700 1 -- start start 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f4760dd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f4760dd6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c102ed0 con 0x7f475c102580 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f475a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f475a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38412/0 (socket says 172.21.15.111:38412) 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.196+0000 7f475a59c700 1 -- 172.21.15.111:0/2038112668 learned_addr learned my addr 172.21.15.111:0/2038112668 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.197+0000 7f475a59c700 1 -- 172.21.15.111:0/2038112668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c103010 con 0x7f475c102580 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.197+0000 7f475a59c700 1 --2- 172.21.15.111:0/2038112668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f474c0098e0 tx=0x7f474c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250adc86c3fd3bac server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.197+0000 7f475959a700 1 -- 172.21.15.111:0/2038112668 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c00b860 con 0x7f475c102580 2024-03-14T12:51:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.197+0000 7f475959a700 1 -- 172.21.15.111:0/2038112668 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f474c00be70 con 0x7f475c102580 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.197+0000 7f475959a700 1 -- 172.21.15.111:0/2038112668 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c010380 con 0x7f475c102580 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 -- 172.21.15.111:0/2038112668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 msgr2=0x7f475c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 --2- 172.21.15.111:0/2038112668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f474c0098e0 tx=0x7f474c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 -- 172.21.15.111:0/2038112668 shutdown_connections 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 --2- 172.21.15.111:0/2038112668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c102990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 -- 172.21.15.111:0/2038112668 >> 172.21.15.111:0/2038112668 conn(0x7f475c0fdd00 msgr2=0x7f475c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 -- 172.21.15.111:0/2038112668 shutdown_connections 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.198+0000 7f4760dd6700 1 -- 172.21.15.111:0/2038112668 wait complete. 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f4760dd6700 1 Processor -- start 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f4760dd6700 1 -- start start 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f4760dd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f4760dd6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475c1118c0 con 0x7f475c102580 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f475a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f475a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38420/0 (socket says 172.21.15.111:38420) 2024-03-14T12:51:20.072 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.199+0000 7f475a59c700 1 -- 172.21.15.111:0/3951024073 learned_addr learned my addr 172.21.15.111:0/3951024073 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f475a59c700 1 -- 172.21.15.111:0/3951024073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c0094f0 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f475a59c700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f474c011ca0 tx=0x7f474c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c010850 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f474c010e60 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475c111ac0 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474c01ae20 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.200+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475c112630 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.201+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f474c02d3f0 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.201+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475c05f5f0 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.201+0000 7f474affd700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 0x7f4744038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.201+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f474c04a490 con 0x7f475c102580 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.201+0000 7f4759d9b700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 0x7f4744038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.202+0000 7f4759d9b700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 0x7f4744038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4750009990 tx=0x7f4750006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:20.073 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.206+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f474c02b8e0 con 0x7f475c102580 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.418+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f475c0fef40 con 0x7f4744036790 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.425+0000 7f474affd700 1 -- 172.21.15.111:0/3951024073 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f475c0fef40 con 0x7f4744036790 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 msgr2=0x7f4744038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 0x7f4744038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4750009990 tx=0x7f4750006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 msgr2=0x7f475c111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f474c011ca0 tx=0x7f474c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 shutdown_connections 2024-03-14T12:51:20.074 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f4744036790 0x7f4744038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:20.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 --2- 172.21.15.111:0/3951024073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f475c102580 0x7f475c111380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:20.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 >> 172.21.15.111:0/3951024073 conn(0x7f475c0fdd00 msgr2=0x7f475c0fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:20.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 shutdown_connections 2024-03-14T12:51:20.075 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:19.427+0000 7f4760dd6700 1 -- 172.21.15.111:0/3951024073 wait complete. 2024-03-14T12:51:20.075 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying crash service with default placement... 2024-03-14T12:51:20.854 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:20 smithi111 ceph-mon[39688]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:20.854 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:20 smithi111 ceph-mon[39688]: Saving service mgr spec with placement count:2 2024-03-14T12:51:20.854 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:20 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:21.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-14T12:51:21.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f4801197700 1 Processor -- start 2024-03-14T12:51:21.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f4801197700 1 -- start start 2024-03-14T12:51:21.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f4801197700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:21.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f4801197700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc1012d0 con 0x7f47fc100980 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36330/0 (socket says 172.21.15.111:36330) 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.609+0000 7f47fad9d700 1 -- 172.21.15.111:0/3463492046 learned_addr learned my addr 172.21.15.111:0/3463492046 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.610+0000 7f47fad9d700 1 -- 172.21.15.111:0/3463492046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc101410 con 0x7f47fc100980 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.610+0000 7f47fad9d700 1 --2- 172.21.15.111:0/3463492046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47f00098e0 tx=0x7f47f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f50cc94662e627 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.610+0000 7f47f9d9b700 1 -- 172.21.15.111:0/3463492046 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f000b860 con 0x7f47fc100980 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.610+0000 7f47f9d9b700 1 -- 172.21.15.111:0/3463492046 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f000be70 con 0x7f47fc100980 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.610+0000 7f47f9d9b700 1 -- 172.21.15.111:0/3463492046 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0010380 con 0x7f47fc100980 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- 172.21.15.111:0/3463492046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 msgr2=0x7f47fc100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 --2- 172.21.15.111:0/3463492046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f47f00098e0 tx=0x7f47f0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- 172.21.15.111:0/3463492046 shutdown_connections 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 --2- 172.21.15.111:0/3463492046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc100d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- 172.21.15.111:0/3463492046 >> 172.21.15.111:0/3463492046 conn(0x7f47fc0fc400 msgr2=0x7f47fc0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- 172.21.15.111:0/3463492046 shutdown_connections 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- 172.21.15.111:0/3463492046 wait complete. 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 Processor -- start 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- start start 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.611+0000 7f4801197700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc111a90 con 0x7f47fc100980 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47fad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36340/0 (socket says 172.21.15.111:36340) 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47fad9d700 1 -- 172.21.15.111:0/1630563579 learned_addr learned my addr 172.21.15.111:0/1630563579 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47fad9d700 1 -- 172.21.15.111:0/1630563579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00094f0 con 0x7f47fc100980 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47fad9d700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47f0011ca0 tx=0x7f47f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0010850 con 0x7f47fc100980 2024-03-14T12:51:21.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f0010e60 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f001ae20 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc111c90 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.612+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc112800 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f47f002d3f0 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47fc05f5f0 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f47e37fe700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 0x7f47e4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f47f004a560 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f47fa59c700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 0x7f47e4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.613+0000 7f47fa59c700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 0x7f47e4038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009990 tx=0x7f47ec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.618+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f47f002b2f0 con 0x7f47fc100980 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.817+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f47fc0fd590 con 0x7f47e4036790 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.823+0000 7f47e37fe700 1 -- 172.21.15.111:0/1630563579 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f47fc0fd590 con 0x7f47e4036790 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 msgr2=0x7f47e4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:21.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 0x7f47e4038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009990 tx=0x7f47ec006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 msgr2=0x7f47fc111550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47f0011ca0 tx=0x7f47f0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 shutdown_connections 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f47e4036790 0x7f47e4038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.825+0000 7f4801197700 1 --2- 172.21.15.111:0/1630563579 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47fc100980 0x7f47fc111550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.826+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 >> 172.21.15.111:0/1630563579 conn(0x7f47fc0fc400 msgr2=0x7f47fc0fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.826+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 shutdown_connections 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:20.826+0000 7f4801197700 1 -- 172.21.15.111:0/1630563579 wait complete. 2024-03-14T12:51:21.682 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying prometheus service with default placement... 2024-03-14T12:51:21.953 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:21 smithi111 ceph-mon[39688]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:21.953 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:21 smithi111 ceph-mon[39688]: Saving service crash spec with placement * 2024-03-14T12:51:21.953 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:21 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:21.953 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:21 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:21.953 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:21 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:23.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-14T12:51:23.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.255+0000 7efc05775700 1 Processor -- start 2024-03-14T12:51:23.515 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.255+0000 7efc05775700 1 -- start start 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.255+0000 7efc05775700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.255+0000 7efc05775700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00102fb0 con 0x7efc00102660 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.255+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36342/0 (socket says 172.21.15.111:36342) 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfeffd700 1 -- 172.21.15.111:0/2971653118 learned_addr learned my addr 172.21.15.111:0/2971653118 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfeffd700 1 -- 172.21.15.111:0/2971653118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc001030f0 con 0x7efc00102660 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfeffd700 1 --2- 172.21.15.111:0/2971653118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098e0 tx=0x7efbf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed6dcad7d0a4c97 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfdffb700 1 -- 172.21.15.111:0/2971653118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf000b860 con 0x7efc00102660 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfdffb700 1 -- 172.21.15.111:0/2971653118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf000be70 con 0x7efc00102660 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.256+0000 7efbfdffb700 1 -- 172.21.15.111:0/2971653118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0010380 con 0x7efc00102660 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 -- 172.21.15.111:0/2971653118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 msgr2=0x7efc00102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 --2- 172.21.15.111:0/2971653118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efbf00098e0 tx=0x7efbf0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 -- 172.21.15.111:0/2971653118 shutdown_connections 2024-03-14T12:51:23.516 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 --2- 172.21.15.111:0/2971653118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00102a70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 -- 172.21.15.111:0/2971653118 >> 172.21.15.111:0/2971653118 conn(0x7efc000fdde0 msgr2=0x7efc001001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 -- 172.21.15.111:0/2971653118 shutdown_connections 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.257+0000 7efc05775700 1 -- 172.21.15.111:0/2971653118 wait complete. 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efc05775700 1 Processor -- start 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efc05775700 1 -- start start 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efc05775700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efc05775700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc00111960 con 0x7efc00102660 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efbfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36346/0 (socket says 172.21.15.111:36346) 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.258+0000 7efbfeffd700 1 -- 172.21.15.111:0/499526945 learned_addr learned my addr 172.21.15.111:0/499526945 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efbfeffd700 1 -- 172.21.15.111:0/499526945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf00094f0 con 0x7efc00102660 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efbfeffd700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efbf0011ca0 tx=0x7efbf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0010850 con 0x7efc00102660 2024-03-14T12:51:23.517 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf0010e60 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc00111b60 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf001ae70 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.259+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc001126d0 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.260+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0005f5f0 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.261+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7efbf00109b0 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.261+0000 7efbe77fe700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 0x7efbe8038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.261+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efbf004ac00 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.261+0000 7efbfe7fc700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 0x7efbe8038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.262+0000 7efbfe7fc700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 0x7efbe8038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efbf4009990 tx=0x7efbf4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.268+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7efbf0010c60 con 0x7efc00102660 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.466+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7efc000ff190 con 0x7efbe8036790 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.471+0000 7efbe77fe700 1 -- 172.21.15.111:0/499526945 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7efc000ff190 con 0x7efbe8036790 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 msgr2=0x7efbe8038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:23.518 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 0x7efbe8038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efbf4009990 tx=0x7efbf4006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 msgr2=0x7efc00111420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efbf0011ca0 tx=0x7efbf0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 shutdown_connections 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe8036790 0x7efbe8038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 --2- 172.21.15.111:0/499526945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc00102660 0x7efc00111420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 >> 172.21.15.111:0/499526945 conn(0x7efc000fdde0 msgr2=0x7efc000fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 shutdown_connections 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:22.473+0000 7efc05775700 1 -- 172.21.15.111:0/499526945 wait complete. 2024-03-14T12:51:23.519 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying grafana service with default placement... 2024-03-14T12:51:23.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:23 smithi111 ceph-mon[39688]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:23.721 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:23 smithi111 ceph-mon[39688]: Saving service prometheus spec with placement count:1 2024-03-14T12:51:23.721 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:23 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:25.031 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3559592700 1 Processor -- start 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3559592700 1 -- start start 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3559592700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3559592700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3554079130 con 0x7f3554078d20 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36358/0 (socket says 172.21.15.111:36358) 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3552ffd700 1 -- 172.21.15.111:0/1225956477 learned_addr learned my addr 172.21.15.111:0/1225956477 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3552ffd700 1 -- 172.21.15.111:0/1225956477 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35540777c0 con 0x7f3554078d20 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.229+0000 7f3552ffd700 1 --2- 172.21.15.111:0/1225956477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f35440098e0 tx=0x7f3544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efce55416748f18d server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3551ffb700 1 -- 172.21.15.111:0/1225956477 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f354400b860 con 0x7f3554078d20 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3551ffb700 1 -- 172.21.15.111:0/1225956477 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f354400be70 con 0x7f3554078d20 2024-03-14T12:51:25.032 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3551ffb700 1 -- 172.21.15.111:0/1225956477 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3544010380 con 0x7f3554078d20 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 -- 172.21.15.111:0/1225956477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 msgr2=0x7f3554077280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 --2- 172.21.15.111:0/1225956477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f35440098e0 tx=0x7f3544009250 comp rx=0 tx=0).stop 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 -- 172.21.15.111:0/1225956477 shutdown_connections 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 --2- 172.21.15.111:0/1225956477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f3554077280 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 -- 172.21.15.111:0/1225956477 >> 172.21.15.111:0/1225956477 conn(0x7f35540fdce0 msgr2=0x7f3554100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 -- 172.21.15.111:0/1225956477 shutdown_connections 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.230+0000 7f3559592700 1 -- 172.21.15.111:0/1225956477 wait complete. 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3559592700 1 Processor -- start 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3559592700 1 -- start start 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3559592700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3559592700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3554113b10 con 0x7f3554078d20 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3552ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36374/0 (socket says 172.21.15.111:36374) 2024-03-14T12:51:25.033 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3552ffd700 1 -- 172.21.15.111:0/2244265419 learned_addr learned my addr 172.21.15.111:0/2244265419 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.231+0000 7f3552ffd700 1 -- 172.21.15.111:0/2244265419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35440094f0 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f3552ffd700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3544006d40 tx=0x7f3544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3544010960 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f354401a430 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3554113d10 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35440245b0 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.232+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3554114880 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f354402d3f0 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f353b7fe700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 0x7f353c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f354404a440 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f35527fc700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 0x7f353c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355405f5f0 con 0x7f3554078d20 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.233+0000 7f35527fc700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 0x7f353c038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3548009990 tx=0x7f3548006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:25.034 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.238+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f354402b8e0 con 0x7f3554078d20 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.431+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f35540fefa0 con 0x7f353c036790 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.435+0000 7f353b7fe700 1 -- 172.21.15.111:0/2244265419 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f35540fefa0 con 0x7f353c036790 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 msgr2=0x7f353c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 0x7f353c038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3548009990 tx=0x7f3548006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 msgr2=0x7f35541135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:25.035 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3544006d40 tx=0x7f3544009250 comp rx=0 tx=0).stop 2024-03-14T12:51:25.037 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 shutdown_connections 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f353c036790 0x7f353c038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 --2- 172.21.15.111:0/2244265419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3554078d20 0x7f35541135d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 >> 172.21.15.111:0/2244265419 conn(0x7f35540fdce0 msgr2=0x7f35540fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 shutdown_connections 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:24.437+0000 7f3559592700 1 -- 172.21.15.111:0/2244265419 wait complete. 2024-03-14T12:51:25.038 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying node-exporter service with default placement... 2024-03-14T12:51:25.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:25 smithi111 ceph-mon[39688]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:25.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:25 smithi111 ceph-mon[39688]: Saving service grafana spec with placement count:1 2024-03-14T12:51:25.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:25 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:26.371 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-14T12:51:26.371 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.523+0000 7fe137dda700 1 Processor -- start 2024-03-14T12:51:26.371 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe137dda700 1 -- start start 2024-03-14T12:51:26.371 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe137dda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe137dda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe130101e30 con 0x7fe1301014e0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe135b76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe135b76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36380/0 (socket says 172.21.15.111:36380) 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe135b76700 1 -- 172.21.15.111:0/2089131991 learned_addr learned my addr 172.21.15.111:0/2089131991 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe135b76700 1 -- 172.21.15.111:0/2089131991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe130101f70 con 0x7fe1301014e0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.524+0000 7fe135b76700 1 --2- 172.21.15.111:0/2089131991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe12000bb80 tx=0x7fe12000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcac5d1bfd7b2db server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe134b74700 1 -- 172.21.15.111:0/2089131991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe12000dd00 con 0x7fe1301014e0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe134b74700 1 -- 172.21.15.111:0/2089131991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe12000de60 con 0x7fe1301014e0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe134b74700 1 -- 172.21.15.111:0/2089131991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1200188d0 con 0x7fe1301014e0 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 -- 172.21.15.111:0/2089131991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 msgr2=0x7fe1301018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 --2- 172.21.15.111:0/2089131991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe12000bb80 tx=0x7fe12000b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 -- 172.21.15.111:0/2089131991 shutdown_connections 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 --2- 172.21.15.111:0/2089131991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe1301018f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 -- 172.21.15.111:0/2089131991 >> 172.21.15.111:0/2089131991 conn(0x7fe1300fcc60 msgr2=0x7fe1300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 -- 172.21.15.111:0/2089131991 shutdown_connections 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.525+0000 7fe137dda700 1 -- 172.21.15.111:0/2089131991 wait complete. 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe137dda700 1 Processor -- start 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe137dda700 1 -- start start 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe137dda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe137dda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13010f660 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe135b76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe135b76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36382/0 (socket says 172.21.15.111:36382) 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe135b76700 1 -- 172.21.15.111:0/3985402625 learned_addr learned my addr 172.21.15.111:0/3985402625 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.526+0000 7fe135b76700 1 -- 172.21.15.111:0/3985402625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12000b790 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe135b76700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe120018630 tx=0x7fe12000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe120010bd0 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13010f860 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe12001d050 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe120022590 con 0x7fe1301014e0 2024-03-14T12:51:26.373 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1301103d0 con 0x7fe1301014e0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.527+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe120022850 con 0x7fe1301014e0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.528+0000 7fe1267fc700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 0x7fe11c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.528+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe120014070 con 0x7fe1301014e0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.528+0000 7fe135375700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 0x7fe11c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.528+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe13005f5f0 con 0x7fe1301014e0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.528+0000 7fe135375700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 0x7fe11c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009990 tx=0x7fe12c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.533+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fe120022b50 con 0x7fe1301014e0 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.724+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1301106d0 con 0x7fe11c036790 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.728+0000 7fe1267fc700 1 -- 172.21.15.111:0/3985402625 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe1301106d0 con 0x7fe11c036790 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 msgr2=0x7fe11c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 0x7fe11c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009990 tx=0x7fe12c006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 msgr2=0x7fe13010f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:26.374 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe120018630 tx=0x7fe12000b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 shutdown_connections 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fe11c036790 0x7fe11c038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 --2- 172.21.15.111:0/3985402625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1301014e0 0x7fe13010f120 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 >> 172.21.15.111:0/3985402625 conn(0x7fe1300fcc60 msgr2=0x7fe130105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 shutdown_connections 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:25.730+0000 7fe137dda700 1 -- 172.21.15.111:0/3985402625 wait complete. 2024-03-14T12:51:26.375 INFO:teuthology.orchestra.run.smithi111.stdout:Deploying alertmanager service with default placement... 2024-03-14T12:51:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:26 smithi111 ceph-mon[39688]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:26 smithi111 ceph-mon[39688]: Saving service node-exporter spec with placement * 2024-03-14T12:51:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:26 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbfae86700 1 Processor -- start 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbfae86700 1 -- start start 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbfae86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbfae86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf4105730 con 0x7efbf4076c40 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbf8c22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbf8c22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36390/0 (socket says 172.21.15.111:36390) 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbf8c22700 1 -- 172.21.15.111:0/2718724323 learned_addr learned my addr 172.21.15.111:0/2718724323 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.918+0000 7efbf8c22700 1 -- 172.21.15.111:0/2718724323 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4105870 con 0x7efbf4076c40 2024-03-14T12:51:27.768 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbf8c22700 1 --2- 172.21.15.111:0/2718724323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efbe80098e0 tx=0x7efbe8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0b59057fd8ecda server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbf37fe700 1 -- 172.21.15.111:0/2718724323 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe800b860 con 0x7efbf4076c40 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbf37fe700 1 -- 172.21.15.111:0/2718724323 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbe800be70 con 0x7efbf4076c40 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbf37fe700 1 -- 172.21.15.111:0/2718724323 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe8010380 con 0x7efbf4076c40 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 -- 172.21.15.111:0/2718724323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 msgr2=0x7efbf41051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 --2- 172.21.15.111:0/2718724323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efbe80098e0 tx=0x7efbe8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 -- 172.21.15.111:0/2718724323 shutdown_connections 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 --2- 172.21.15.111:0/2718724323 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf41051f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 -- 172.21.15.111:0/2718724323 >> 172.21.15.111:0/2718724323 conn(0x7efbf40fcc60 msgr2=0x7efbf40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 -- 172.21.15.111:0/2718724323 shutdown_connections 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.919+0000 7efbfae86700 1 -- 172.21.15.111:0/2718724323 wait complete. 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbfae86700 1 Processor -- start 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbfae86700 1 -- start start 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbfae86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbfae86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf4103080 con 0x7efbf4076c40 2024-03-14T12:51:27.769 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbf8c22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbf8c22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36394/0 (socket says 172.21.15.111:36394) 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.920+0000 7efbf8c22700 1 -- 172.21.15.111:0/3138362835 learned_addr learned my addr 172.21.15.111:0/3138362835 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbf8c22700 1 -- 172.21.15.111:0/3138362835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbe80094f0 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbf8c22700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbe80094c0 tx=0x7efbe8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe8010850 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbe8010e60 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf4101150 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbe801a430 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.921+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf4101640 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.922+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7efbe801a6b0 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.922+0000 7efbf17fa700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 0x7efbe4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.922+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf404bed0 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.922+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efbe804a7b0 con 0x7efbf4076c40 2024-03-14T12:51:27.770 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.922+0000 7efbf3fff700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 0x7efbe4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.923+0000 7efbf3fff700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 0x7efbe4038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009990 tx=0x7efbe0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:26.927+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7efbe8018db0 con 0x7efbf4076c40 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.120+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7efbf4101db0 con 0x7efbe4036790 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.125+0000 7efbf17fa700 1 -- 172.21.15.111:0/3138362835 <== mgr.14118 v2:172.21.15.111:6800/119405983 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7efbf4101db0 con 0x7efbe4036790 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.127+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 msgr2=0x7efbe4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.127+0000 7efbfae86700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 0x7efbe4038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efbe0009990 tx=0x7efbe0006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.127+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 msgr2=0x7efbf4102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.127+0000 7efbfae86700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efbe80094c0 tx=0x7efbe8009250 comp rx=0 tx=0).stop 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 shutdown_connections 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7efbe4036790 0x7efbe4038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 --2- 172.21.15.111:0/3138362835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efbf4076c40 0x7efbf4102b40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 >> 172.21.15.111:0/3138362835 conn(0x7efbf40fcc60 msgr2=0x7efbf40fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 shutdown_connections 2024-03-14T12:51:27.771 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:27.128+0000 7efbfae86700 1 -- 172.21.15.111:0/3138362835 wait complete. 2024-03-14T12:51:27.994 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:27 smithi111 ceph-mon[39688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:51:27.994 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:27 smithi111 ceph-mon[39688]: from='mgr.14118 172.21.15.111:0/196741891' entity='mgr.smithi111.atysez' 2024-03-14T12:51:28.810 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:28 smithi111 ceph-mon[39688]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:28.810 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:28 smithi111 ceph-mon[39688]: Saving service alertmanager spec with placement count:1 2024-03-14T12:51:28.810 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:28 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1391533278' entity='client.admin' 2024-03-14T12:51:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.277+0000 7fc4384d1700 1 Processor -- start 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.277+0000 7fc4384d1700 1 -- start start 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.277+0000 7fc4384d1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.277+0000 7fc4384d1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430101e30 con 0x7fc4301014e0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43626d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43626d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36396/0 (socket says 172.21.15.111:36396) 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43626d700 1 -- 172.21.15.111:0/1023952866 learned_addr learned my addr 172.21.15.111:0/1023952866 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43626d700 1 -- 172.21.15.111:0/1023952866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430101f70 con 0x7fc4301014e0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43626d700 1 --2- 172.21.15.111:0/1023952866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc4200098e0 tx=0x7fc420009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e23706f98c8936a server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43526b700 1 -- 172.21.15.111:0/1023952866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc42000b860 con 0x7fc4301014e0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43526b700 1 -- 172.21.15.111:0/1023952866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc42000be70 con 0x7fc4301014e0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.278+0000 7fc43526b700 1 -- 172.21.15.111:0/1023952866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420010380 con 0x7fc4301014e0 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- 172.21.15.111:0/1023952866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 msgr2=0x7fc4301018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:29.678 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1023952866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc4200098e0 tx=0x7fc420009250 comp rx=0 tx=0).stop 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- 172.21.15.111:0/1023952866 shutdown_connections 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1023952866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc4301018f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- 172.21.15.111:0/1023952866 >> 172.21.15.111:0/1023952866 conn(0x7fc4300fcc60 msgr2=0x7fc4300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- 172.21.15.111:0/1023952866 shutdown_connections 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- 172.21.15.111:0/1023952866 wait complete. 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 Processor -- start 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.279+0000 7fc4384d1700 1 -- start start 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc4384d1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc4384d1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4301107a0 con 0x7fc4301014e0 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc43626d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc43626d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36406/0 (socket says 172.21.15.111:36406) 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc43626d700 1 -- 172.21.15.111:0/1391533278 learned_addr learned my addr 172.21.15.111:0/1391533278 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc43626d700 1 -- 172.21.15.111:0/1391533278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4200094f0 con 0x7fc4301014e0 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc43626d700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c010 tx=0x7fc420009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:29.679 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420010960 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc42001a430 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4301109a0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420019580 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.280+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc430111510 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.281+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc4200197b0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.281+0000 7fc426ffd700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 0x7fc41c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.281+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43005f5f0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.281+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc42004a6c0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.281+0000 7fc435a6c700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 0x7fc41c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.282+0000 7fc435a6c700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 0x7fc41c038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009990 tx=0x7fc42c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.286+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fc42002b4e0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.466+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc43004bed0 con 0x7fc4301014e0 2024-03-14T12:51:29.680 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.469+0000 7fc426ffd700 1 -- 172.21.15.111:0/1391533278 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc420049020 con 0x7fc4301014e0 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 msgr2=0x7fc41c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 0x7fc41c038c90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009990 tx=0x7fc42c006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 msgr2=0x7fc430110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c010 tx=0x7fc420009250 comp rx=0 tx=0).stop 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 shutdown_connections 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fc41c0367e0 0x7fc41c038c90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 --2- 172.21.15.111:0/1391533278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4301014e0 0x7fc430110260 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 >> 172.21.15.111:0/1391533278 conn(0x7fc4300fcc60 msgr2=0x7fc4300fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 shutdown_connections 2024-03-14T12:51:29.681 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:28.471+0000 7fc4384d1700 1 -- 172.21.15.111:0/1391533278 wait complete. 2024-03-14T12:51:29.947 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:29 smithi111 ceph-mon[39688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:51:31.050 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d80c71700 1 Processor -- start 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d80c71700 1 -- start start 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d80c71700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d80c71700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c102ed0 con 0x7f5d7c102580 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36418/0 (socket says 172.21.15.111:36418) 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.285+0000 7f5d7a59c700 1 -- 172.21.15.111:0/1836516047 learned_addr learned my addr 172.21.15.111:0/1836516047 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d7a59c700 1 -- 172.21.15.111:0/1836516047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c103010 con 0x7f5d7c102580 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d7a59c700 1 --2- 172.21.15.111:0/1836516047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0098e0 tx=0x7f5d6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577b5961e8665ca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d7959a700 1 -- 172.21.15.111:0/1836516047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c00b860 con 0x7f5d7c102580 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d7959a700 1 -- 172.21.15.111:0/1836516047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6c00be70 con 0x7f5d7c102580 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d7959a700 1 -- 172.21.15.111:0/1836516047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c010380 con 0x7f5d7c102580 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d80c71700 1 -- 172.21.15.111:0/1836516047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 msgr2=0x7f5d7c102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1836516047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0098e0 tx=0x7f5d6c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:31.051 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d80c71700 1 -- 172.21.15.111:0/1836516047 shutdown_connections 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1836516047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c102990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.286+0000 7f5d80c71700 1 -- 172.21.15.111:0/1836516047 >> 172.21.15.111:0/1836516047 conn(0x7f5d7c0fdd00 msgr2=0x7f5d7c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 -- 172.21.15.111:0/1836516047 shutdown_connections 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 -- 172.21.15.111:0/1836516047 wait complete. 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 Processor -- start 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 -- start start 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d80c71700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c111940 con 0x7f5d7c102580 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d7a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36424/0 (socket says 172.21.15.111:36424) 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.287+0000 7f5d7a59c700 1 -- 172.21.15.111:0/1627305242 learned_addr learned my addr 172.21.15.111:0/1627305242 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d7a59c700 1 -- 172.21.15.111:0/1627305242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6c0094f0 con 0x7f5d7c102580 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d7a59c700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c011ca0 tx=0x7f5d6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:31.052 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c010850 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d6c010e60 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c111b40 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d6c01ae70 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.288+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c1126b0 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.289+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5d6c0244a0 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.289+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c05f5f0 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.289+0000 7f5d6affd700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 0x7f5d64038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.289+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5d6c04a720 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.289+0000 7f5d79d9b700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 0x7f5d64038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.290+0000 7f5d79d9b700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 0x7f5d64038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009990 tx=0x7f5d70006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.294+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f5d6c02cb40 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.476+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f5d7c04bed0 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.482+0000 7f5d6affd700 1 -- 172.21.15.111:0/1627305242 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f5d6c023370 con 0x7f5d7c102580 2024-03-14T12:51:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 msgr2=0x7f5d64038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 0x7f5d64038c90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009990 tx=0x7f5d70006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 msgr2=0x7f5d7c111400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c011ca0 tx=0x7f5d6c009250 comp rx=0 tx=0).stop 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 shutdown_connections 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f5d640367e0 0x7f5d64038c90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 --2- 172.21.15.111:0/1627305242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d7c102580 0x7f5d7c111400 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 >> 172.21.15.111:0/1627305242 conn(0x7f5d7c0fdd00 msgr2=0x7f5d7c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 shutdown_connections 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:30.484+0000 7f5d80c71700 1 -- 172.21.15.111:0/1627305242 wait complete. 2024-03-14T12:51:31.054 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling the dashboard module... 2024-03-14T12:51:31.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:31 smithi111 ceph-mon[39688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:51:31.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:31 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1627305242' entity='client.admin' 2024-03-14T12:51:32.843 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:32 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3649664049' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3ea5c2b700 1 Processor -- start 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3ea5c2b700 1 -- start start 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3ea5c2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3ea5c2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0102ef0 con 0x7f3ea01025a0 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36054/0 (socket says 172.21.15.111:36054) 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9f7fe700 1 -- 172.21.15.111:0/2926731888 learned_addr learned my addr 172.21.15.111:0/2926731888 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9f7fe700 1 -- 172.21.15.111:0/2926731888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0103030 con 0x7f3ea01025a0 2024-03-14T12:51:33.116 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9f7fe700 1 --2- 172.21.15.111:0/2926731888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e940098e0 tx=0x7f3e94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9815b6c012cd7b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9e7fc700 1 -- 172.21.15.111:0/2926731888 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e9400b860 con 0x7f3ea01025a0 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9e7fc700 1 -- 172.21.15.111:0/2926731888 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e9400be70 con 0x7f3ea01025a0 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.595+0000 7f3e9e7fc700 1 -- 172.21.15.111:0/2926731888 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e94010380 con 0x7f3ea01025a0 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/2926731888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 msgr2=0x7f3ea01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/2926731888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e940098e0 tx=0x7f3e94009250 comp rx=0 tx=0).stop 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/2926731888 shutdown_connections 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/2926731888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea01029b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/2926731888 >> 172.21.15.111:0/2926731888 conn(0x7f3ea00fdd20 msgr2=0x7f3ea0100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/2926731888 shutdown_connections 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.596+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/2926731888 wait complete. 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3ea5c2b700 1 Processor -- start 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3ea5c2b700 1 -- start start 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3ea5c2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3ea5c2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea01118d0 con 0x7f3ea01025a0 2024-03-14T12:51:33.117 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:33.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36066/0 (socket says 172.21.15.111:36066) 2024-03-14T12:51:33.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e9f7fe700 1 -- 172.21.15.111:0/3649664049 learned_addr learned my addr 172.21.15.111:0/3649664049 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:33.118 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e9f7fe700 1 -- 172.21.15.111:0/3649664049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940094f0 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e9f7fe700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e94011ca0 tx=0x7f3e94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.597+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e94010850 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.598+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea0111ad0 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.598+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e94010e60 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.598+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e9401ae70 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.598+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea0111ee0 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.599+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea004bed0 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.599+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3e940109b0 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.599+0000 7f3e87fff700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 0x7f3e8803d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.599+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3e9404ac10 con 0x7f3ea01025a0 2024-03-14T12:51:33.119 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.599+0000 7f3e9effd700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 0x7f3e8803d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.600+0000 7f3e9effd700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 0x7f3e8803d050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009990 tx=0x7f3e90006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.603+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f3e94010c60 con 0x7f3ea01025a0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:31.835+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3ea0112640 con 0x7f3ea01025a0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.493+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f3e94049db0 con 0x7f3ea01025a0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.494+0000 7f3e87fff700 1 -- 172.21.15.111:0/3649664049 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3e94006d70 con 0x7f3ea01025a0 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 msgr2=0x7f3e8803d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 0x7f3e8803d050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009990 tx=0x7f3e90006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 msgr2=0x7f3ea0111390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e94011ca0 tx=0x7f3e94009250 comp rx=0 tx=0).stop 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 shutdown_connections 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7f3e8803aba0 0x7f3e8803d050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 --2- 172.21.15.111:0/3649664049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea01025a0 0x7f3ea0111390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 >> 172.21.15.111:0/3649664049 conn(0x7f3ea00fdd20 msgr2=0x7f3ea00fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:33.120 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 shutdown_connections 2024-03-14T12:51:33.121 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:32.497+0000 7f3ea5c2b700 1 -- 172.21.15.111:0/3649664049 wait complete. 2024-03-14T12:51:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:33 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3649664049' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-14T12:51:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:33 smithi111 ceph-mon[39688]: mgrmap e9: smithi111.atysez(active, since 26s) 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "active_name": "smithi111.atysez", 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.627+0000 7fb1f35f2700 1 Processor -- start 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.627+0000 7fb1f35f2700 1 -- start start 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.627+0000 7fb1f35f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.627+0000 7fb1f35f2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec101e30 con 0x7fb1ec1014e0 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1f138e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:34.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1f138e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36072/0 (socket says 172.21.15.111:36072) 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1f138e700 1 -- 172.21.15.111:0/882107192 learned_addr learned my addr 172.21.15.111:0/882107192 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1f138e700 1 -- 172.21.15.111:0/882107192 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec101f70 con 0x7fb1ec1014e0 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1f138e700 1 --2- 172.21.15.111:0/882107192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800bb80 tx=0x7fb1e800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f239365b4b735ac server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1e3fff700 1 -- 172.21.15.111:0/882107192 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e800dd00 con 0x7fb1ec1014e0 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1e3fff700 1 -- 172.21.15.111:0/882107192 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1e800de60 con 0x7fb1ec1014e0 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.628+0000 7fb1e3fff700 1 -- 172.21.15.111:0/882107192 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e80188d0 con 0x7fb1ec1014e0 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- 172.21.15.111:0/882107192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 msgr2=0x7fb1ec1018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/882107192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800bb80 tx=0x7fb1e800b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- 172.21.15.111:0/882107192 shutdown_connections 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/882107192 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec1018f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- 172.21.15.111:0/882107192 >> 172.21.15.111:0/882107192 conn(0x7fb1ec0fcc60 msgr2=0x7fb1ec0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- 172.21.15.111:0/882107192 shutdown_connections 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- 172.21.15.111:0/882107192 wait complete. 2024-03-14T12:51:34.596 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 Processor -- start 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.629+0000 7fb1f35f2700 1 -- start start 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f35f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f35f2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec10f660 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f138e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f138e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36078/0 (socket says 172.21.15.111:36078) 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f138e700 1 -- 172.21.15.111:0/1572986205 learned_addr learned my addr 172.21.15.111:0/1572986205 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f138e700 1 -- 172.21.15.111:0/1572986205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e800b790 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f138e700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8018630 tx=0x7fb1e800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8010b90 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec10f860 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1e801d050 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8022590 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.630+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec1103d0 con 0x7fb1ec1014e0 2024-03-14T12:51:34.597 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.631+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb1e8022850 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.631+0000 7fb1e1ffb700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 0x7fb1d8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.631+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb1e8014070 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.632+0000 7fb1f0b8d700 1 -- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 msgr2=0x7fb1d8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.632+0000 7fb1f0b8d700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 0x7fb1d8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.632+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec05f5f0 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.640+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb1e800e360 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.833+0000 7fb1f0b8d700 1 -- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 msgr2=0x7fb1d8038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.833+0000 7fb1f0b8d700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 0x7fb1d8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.882+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb1ec110e10 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.882+0000 7fb1e1ffb700 1 -- 172.21.15.111:0/1572986205 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb1e80202d0 con 0x7fb1ec1014e0 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 msgr2=0x7fb1d8038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 0x7fb1d8038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.598 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 msgr2=0x7fb1ec10f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8018630 tx=0x7fb1e800b4f0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 shutdown_connections 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fb1d80367e0 0x7fb1d8038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 --2- 172.21.15.111:0/1572986205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1ec1014e0 0x7fb1ec10f120 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 >> 172.21.15.111:0/1572986205 conn(0x7fb1ec0fcc60 msgr2=0x7fb1ec105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 shutdown_connections 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:33.884+0000 7fb1f35f2700 1 -- 172.21.15.111:0/1572986205 wait complete. 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for the mgr to restart... 2024-03-14T12:51:34.599 INFO:teuthology.orchestra.run.smithi111.stdout:Waiting for mgr epoch 9... 2024-03-14T12:51:34.824 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:34 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1572986205' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: Active manager daemon smithi111.atysez restarted 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: Activating manager daemon smithi111.atysez 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: osdmap e3: 0 total, 0 up, 0 in 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: mgrmap e10: smithi111.atysez(active, starting, since 0.00365516s) 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: Manager daemon smithi111.atysez is now available 2024-03-14T12:51:39.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:39.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:39 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:51:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:40 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T12:51:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:40 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T12:51:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:40 smithi111 ceph-mon[39688]: mgrmap e11: smithi111.atysez(active, since 1.0057s) 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout { 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout } 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.145+0000 7fadc09e6700 1 Processor -- start 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.145+0000 7fadc09e6700 1 -- start start 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadc09e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadc09e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc1033c0 con 0x7fadbc102a70 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36084/0 (socket says 172.21.15.111:36084) 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadba59c700 1 -- 172.21.15.111:0/3474917092 learned_addr learned my addr 172.21.15.111:0/3474917092 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadba59c700 1 -- 172.21.15.111:0/3474917092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadbc103500 con 0x7fadbc102a70 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadba59c700 1 --2- 172.21.15.111:0/3474917092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fadb00098e0 tx=0x7fadb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139e28d99a9f7125 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadb959a700 1 -- 172.21.15.111:0/3474917092 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb000b860 con 0x7fadbc102a70 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadb959a700 1 -- 172.21.15.111:0/3474917092 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadb000be70 con 0x7fadbc102a70 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.146+0000 7fadb959a700 1 -- 172.21.15.111:0/3474917092 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb0010380 con 0x7fadbc102a70 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 -- 172.21.15.111:0/3474917092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 msgr2=0x7fadbc102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3474917092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fadb00098e0 tx=0x7fadb0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 -- 172.21.15.111:0/3474917092 shutdown_connections 2024-03-14T12:51:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3474917092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc102e80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 -- 172.21.15.111:0/3474917092 >> 172.21.15.111:0/3474917092 conn(0x7fadbc0fc400 msgr2=0x7fadbc0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 -- 172.21.15.111:0/3474917092 shutdown_connections 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.147+0000 7fadc09e6700 1 -- 172.21.15.111:0/3474917092 wait complete. 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadc09e6700 1 Processor -- start 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadc09e6700 1 -- start start 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadc09e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadc09e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadbc10f690 con 0x7fadbc102a70 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36086/0 (socket says 172.21.15.111:36086) 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadba59c700 1 -- 172.21.15.111:0/3939395658 learned_addr learned my addr 172.21.15.111:0/3939395658 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadba59c700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb00094f0 con 0x7fadbc102a70 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadba59c700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006d40 tx=0x7fadb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:40.696 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb0010960 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.148+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadb001a430 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.149+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadbc10f890 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.149+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb00245b0 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.149+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadbc110400 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.149+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fadb002d3f0 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.150+0000 7fadaaffd700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.150+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fada403b640 con 0x7fada4038a80 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.150+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fadb004a2a0 con 0x7fadbc102a70 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.150+0000 7fadb9d9b700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:40.697 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.150+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.351+0000 7fadb9d9b700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.351+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.752+0000 7fadb9d9b700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:35.752+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:36.553+0000 7fadb9d9b700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:36.553+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:38.153+0000 7fadb9d9b700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/119405983 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:38.153+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:38.988+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fadb002d740 con 0x7fadbc102a70 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:38.988+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 msgr2=0x7fada403af30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:51:40.698 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:38.988+0000 7fadaaffd700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.990+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fadb001a9f0 con 0x7fadbc102a70 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.990+0000 7fadaaffd700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 0x7fada403e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.990+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fada403b640 con 0x7fada403c530 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.990+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 0x7fada403e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.991+0000 7fadb9d9b700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 0x7fada403e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fadac0039d0 tx=0x7fadac009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.991+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fada403b640 con 0x7fada403c530 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.997+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fadbc10fd20 con 0x7fada403c530 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.997+0000 7fadaaffd700 1 -- 172.21.15.111:0/3939395658 <== mgr.14160 v2:172.21.15.111:6800/4041156471 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fadbc10fd20 con 0x7fada403c530 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 msgr2=0x7fada403e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 0x7fada403e910 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fadac0039d0 tx=0x7fadac009350 comp rx=0 tx=0).stop 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 msgr2=0x7fadbc10f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fadb0006d40 tx=0x7fadb0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:40.699 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 shutdown_connections 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fada403c530 0x7fada403e910 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fadbc102a70 0x7fadbc10f150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 --2- 172.21.15.111:0/3939395658 >> [v2:172.21.15.111:6800/119405983,v1:172.21.15.111:6801/119405983] conn(0x7fada4038a80 0x7fada403af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 >> 172.21.15.111:0/3939395658 conn(0x7fadbc0fc400 msgr2=0x7fadbc1059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 shutdown_connections 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:39.998+0000 7fadc09e6700 1 -- 172.21.15.111:0/3939395658 wait complete. 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:mgr epoch 9 is available 2024-03-14T12:51:40.700 INFO:teuthology.orchestra.run.smithi111.stdout:Generating a dashboard self-signed certificate... 2024-03-14T12:51:41.579 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:41 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:40] ENGINE Bus STARTING 2024-03-14T12:51:41.579 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:41 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:40] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T12:51:41.580 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:41 smithi111 ceph-mon[39688]: [14/Mar/2024:12:51:40] ENGINE Bus STARTED 2024-03-14T12:51:41.580 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:41 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:41.580 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:41 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e18eb8700 1 Processor -- start 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e18eb8700 1 -- start start 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e18eb8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e18eb8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e14105770 con 0x7f6e14076c00 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e1259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e1259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34786/0 (socket says 172.21.15.111:34786) 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e1259c700 1 -- 172.21.15.111:0/1467005676 learned_addr learned my addr 172.21.15.111:0/1467005676 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.252+0000 7f6e1259c700 1 -- 172.21.15.111:0/1467005676 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e141058b0 con 0x7f6e14076c00 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e1259c700 1 --2- 172.21.15.111:0/1467005676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e040098e0 tx=0x7f6e04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4658c38a41b972a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:42.207 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e1159a700 1 -- 172.21.15.111:0/1467005676 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e0400b860 con 0x7f6e14076c00 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e1159a700 1 -- 172.21.15.111:0/1467005676 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e0400be70 con 0x7f6e14076c00 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e1159a700 1 -- 172.21.15.111:0/1467005676 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e04010380 con 0x7f6e14076c00 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e18eb8700 1 -- 172.21.15.111:0/1467005676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 msgr2=0x7f6e14105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/1467005676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6e040098e0 tx=0x7f6e04009250 comp rx=0 tx=0).stop 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e18eb8700 1 -- 172.21.15.111:0/1467005676 shutdown_connections 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/1467005676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e14105230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.253+0000 7f6e18eb8700 1 -- 172.21.15.111:0/1467005676 >> 172.21.15.111:0/1467005676 conn(0x7f6e140fcc60 msgr2=0x7f6e140ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 -- 172.21.15.111:0/1467005676 shutdown_connections 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 -- 172.21.15.111:0/1467005676 wait complete. 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 Processor -- start 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 -- start start 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:42.208 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.254+0000 7f6e18eb8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e141118f0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e1259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e1259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34794/0 (socket says 172.21.15.111:34794) 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e1259c700 1 -- 172.21.15.111:0/4074738590 learned_addr learned my addr 172.21.15.111:0/4074738590 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e1259c700 1 -- 172.21.15.111:0/4074738590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040094f0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e1259c700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006d40 tx=0x7f6e04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e04010960 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e0401a430 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e14111af0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e040245b0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.255+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14112660 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.256+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1405f5f0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.256+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f6e0401a9f0 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.257+0000 7f6e02ffd700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 0x7f6dfc038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.257+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6e04053840 con 0x7f6e14076c00 2024-03-14T12:51:42.209 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.257+0000 7f6e11d9b700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 0x7f6dfc038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.257+0000 7f6e11d9b700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 0x7f6dfc038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009990 tx=0x7f6e08006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.261+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f6e04050060 con 0x7f6e14076c00 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.477+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e140fdeb0 con 0x7f6dfc0366b0 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.501+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f6e040193c0 con 0x7f6e14076c00 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.552+0000 7f6e02ffd700 1 -- 172.21.15.111:0/4074738590 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f6e140fdeb0 con 0x7f6dfc0366b0 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 msgr2=0x7f6dfc038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 0x7f6dfc038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009990 tx=0x7f6e08006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 msgr2=0x7f6e141113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006d40 tx=0x7f6e04009250 comp rx=0 tx=0).stop 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 shutdown_connections 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6dfc0366b0 0x7f6dfc038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 --2- 172.21.15.111:0/4074738590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e14076c00 0x7f6e141113b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 >> 172.21.15.111:0/4074738590 conn(0x7f6e140fcc60 msgr2=0x7f6e140fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:42.210 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 shutdown_connections 2024-03-14T12:51:42.211 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:41.554+0000 7f6e18eb8700 1 -- 172.21.15.111:0/4074738590 wait complete. 2024-03-14T12:51:42.211 INFO:teuthology.orchestra.run.smithi111.stdout:Creating initial admin user... 2024-03-14T12:51:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:42 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:42 smithi111 ceph-mon[39688]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:42 smithi111 ceph-mon[39688]: mgrmap e12: smithi111.atysez(active, since 2s) 2024-03-14T12:51:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:42 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:42 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$CLVDSSxAXMoQX9G1QyWfIuqcFiBR2KTknIRY4/EXRxVemYEa3UHc6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710420703, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.722+0000 7f0ac1c41700 1 Processor -- start 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.722+0000 7f0ac1c41700 1 -- start start 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0ac1c41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0ac1c41700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc104530 con 0x7f0abc103be0 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:44.251 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34802/0 (socket says 172.21.15.111:34802) 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abb7fe700 1 -- 172.21.15.111:0/2652440022 learned_addr learned my addr 172.21.15.111:0/2652440022 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abb7fe700 1 -- 172.21.15.111:0/2652440022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc104670 con 0x7f0abc103be0 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abb7fe700 1 --2- 172.21.15.111:0/2652440022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0098e0 tx=0x7f0aac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7f0d2e167b1666 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abaffd700 1 -- 172.21.15.111:0/2652440022 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac00b860 con 0x7f0abc103be0 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abaffd700 1 -- 172.21.15.111:0/2652440022 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac00be70 con 0x7f0abc103be0 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.723+0000 7f0abaffd700 1 -- 172.21.15.111:0/2652440022 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac010380 con 0x7f0abc103be0 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 -- 172.21.15.111:0/2652440022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 msgr2=0x7f0abc103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/2652440022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0098e0 tx=0x7f0aac009250 comp rx=0 tx=0).stop 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 -- 172.21.15.111:0/2652440022 shutdown_connections 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/2652440022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc103ff0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 -- 172.21.15.111:0/2652440022 >> 172.21.15.111:0/2652440022 conn(0x7f0abc0fc400 msgr2=0x7f0abc0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 -- 172.21.15.111:0/2652440022 shutdown_connections 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.724+0000 7f0ac1c41700 1 -- 172.21.15.111:0/2652440022 wait complete. 2024-03-14T12:51:44.252 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0ac1c41700 1 Processor -- start 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0ac1c41700 1 -- start start 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0ac1c41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0ac1c41700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc111840 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0abb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34806/0 (socket says 172.21.15.111:34806) 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0abb7fe700 1 -- 172.21.15.111:0/1965163101 learned_addr learned my addr 172.21.15.111:0/1965163101 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.725+0000 7f0abb7fe700 1 -- 172.21.15.111:0/1965163101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac0094f0 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0abb7fe700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0aac011ca0 tx=0x7f0aac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac010850 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac010e60 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc111a40 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac01ae20 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc1125b0 con 0x7f0abc103be0 2024-03-14T12:51:44.253 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.726+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f0aac02d3f0 con 0x7f0abc103be0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.727+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc05f5f0 con 0x7f0abc103be0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.727+0000 7f0ab8ff9700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 0x7f0aa8038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.727+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0aac04a710 con 0x7f0abc103be0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.727+0000 7f0ab3fff700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 0x7f0aa8038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.727+0000 7f0ab3fff700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 0x7f0aa8038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009990 tx=0x7f0aa4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.731+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f0aac02b4d0 con 0x7f0abc103be0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:42.942+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0abc05f410 con 0x7f0aa80367e0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.165+0000 7f0ab8ff9700 1 -- 172.21.15.111:0/1965163101 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f0abc05f410 con 0x7f0aa80367e0 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 msgr2=0x7f0aa8038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 0x7f0aa8038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009990 tx=0x7f0aa4006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 msgr2=0x7f0abc111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:44.254 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0aac011ca0 tx=0x7f0aac009250 comp rx=0 tx=0).stop 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 shutdown_connections 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0aa80367e0 0x7f0aa8038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 --2- 172.21.15.111:0/1965163101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0abc103be0 0x7f0abc111300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 >> 172.21.15.111:0/1965163101 conn(0x7f0abc0fc400 msgr2=0x7f0abc0fce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 shutdown_connections 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:43.167+0000 7f0ac1c41700 1 -- 172.21.15.111:0/1965163101 wait complete. 2024-03-14T12:51:44.255 INFO:teuthology.orchestra.run.smithi111.stdout:Fetching dashboard port number... 2024-03-14T12:51:44.469 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:44 smithi111 ceph-mon[39688]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:44.469 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:44 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:45.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:45 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3566104148' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stdout 8443 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f676452b700 1 Processor -- start 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f676452b700 1 -- start start 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f676452b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f676452b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c102fc0 con 0x7f675c102670 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f67622c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f67622c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34814/0 (socket says 172.21.15.111:34814) 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.930+0000 7f67622c7700 1 -- 172.21.15.111:0/2052900768 learned_addr learned my addr 172.21.15.111:0/2052900768 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f67622c7700 1 -- 172.21.15.111:0/2052900768 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c103100 con 0x7f675c102670 2024-03-14T12:51:45.853 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f67622c7700 1 --2- 172.21.15.111:0/2052900768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f67580098e0 tx=0x7f6758009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424a08b60bdcf134 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f67612c5700 1 -- 172.21.15.111:0/2052900768 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f675800b860 con 0x7f675c102670 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f67612c5700 1 -- 172.21.15.111:0/2052900768 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f675800be70 con 0x7f675c102670 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f67612c5700 1 -- 172.21.15.111:0/2052900768 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6758010380 con 0x7f675c102670 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f676452b700 1 -- 172.21.15.111:0/2052900768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 msgr2=0x7f675c102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.931+0000 7f676452b700 1 --2- 172.21.15.111:0/2052900768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f67580098e0 tx=0x7f6758009250 comp rx=0 tx=0).stop 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- 172.21.15.111:0/2052900768 shutdown_connections 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 --2- 172.21.15.111:0/2052900768 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c102a80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- 172.21.15.111:0/2052900768 >> 172.21.15.111:0/2052900768 conn(0x7f675c0fdd40 msgr2=0x7f675c100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- 172.21.15.111:0/2052900768 shutdown_connections 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- 172.21.15.111:0/2052900768 wait complete. 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 Processor -- start 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- start start 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:45.854 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.932+0000 7f676452b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c1118f0 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f67622c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f67622c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34820/0 (socket says 172.21.15.111:34820) 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f67622c7700 1 -- 172.21.15.111:0/3566104148 learned_addr learned my addr 172.21.15.111:0/3566104148 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f67622c7700 1 -- 172.21.15.111:0/3566104148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67580094f0 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f67622c7700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6758011ca0 tx=0x7f6758009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6758010850 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6758010e60 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f675801ae70 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c111af0 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.933+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c112660 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.934+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f67580109b0 con 0x7f675c102670 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.934+0000 7f6752ffd700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 0x7f6748038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:45.855 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.934+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f675804a710 con 0x7f675c102670 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.934+0000 7f6761ac6700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 0x7f6748038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.934+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c05f5f0 con 0x7f675c102670 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.935+0000 7f6761ac6700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 0x7f6748038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f674c009990 tx=0x7f674c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:44.939+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f67580189a0 con 0x7f675c102670 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.125+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f675c04bed0 con 0x7f675c102670 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.125+0000 7f6752ffd700 1 -- 172.21.15.111:0/3566104148 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f6758019e40 con 0x7f675c102670 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 msgr2=0x7f6748038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 0x7f6748038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f674c009990 tx=0x7f674c006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 msgr2=0x7f675c1113b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6758011ca0 tx=0x7f6758009250 comp rx=0 tx=0).stop 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 shutdown_connections 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f67480367e0 0x7f6748038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 --2- 172.21.15.111:0/3566104148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f675c102670 0x7f675c1113b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:45.856 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 >> 172.21.15.111:0/3566104148 conn(0x7f675c0fdd40 msgr2=0x7f675c0fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:45.857 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 shutdown_connections 2024-03-14T12:51:45.857 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:45.127+0000 7f676452b700 1 -- 172.21.15.111:0/3566104148 wait complete. 2024-03-14T12:51:45.861 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-14T12:51:45.861 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout disabled 2024-03-14T12:51:45.869 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-14T12:51:45.869 INFO:teuthology.orchestra.run.smithi111.stdout:systemctl: stdout inactive 2024-03-14T12:51:45.869 INFO:teuthology.orchestra.run.smithi111.stdout:firewalld.service is not enabled 2024-03-14T12:51:45.869 INFO:teuthology.orchestra.run.smithi111.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout:Ceph Dashboard is now available at: 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout: URL: https://smithi111.front.sepia.ceph.com:8443/ 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout: User: admin 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout: Password: x4e2d9q96m 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:45.872 INFO:teuthology.orchestra.run.smithi111.stdout:Saving cluster configuration to /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config directory 2024-03-14T12:51:45.873 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling autotune for osd_memory_target 2024-03-14T12:51:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:46 smithi111 ceph-mon[39688]: mgrmap e13: smithi111.atysez(active, since 6s) 2024-03-14T12:51:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.426+0000 7f0bd765f700 1 Processor -- start 2024-03-14T12:51:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.426+0000 7f0bd765f700 1 -- start start 2024-03-14T12:51:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.426+0000 7f0bd765f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.426+0000 7f0bd765f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0101d30 con 0x7f0bd01013e0 2024-03-14T12:51:47.133 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bd53fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bd53fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34836/0 (socket says 172.21.15.111:34836) 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bd53fb700 1 -- 172.21.15.111:0/1958645336 learned_addr learned my addr 172.21.15.111:0/1958645336 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bd53fb700 1 -- 172.21.15.111:0/1958645336 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0101e70 con 0x7f0bd01013e0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bd53fb700 1 --2- 172.21.15.111:0/1958645336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00098e0 tx=0x7f0bc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df2f6979c6c8751 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bc7fff700 1 -- 172.21.15.111:0/1958645336 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc000b860 con 0x7f0bd01013e0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bc7fff700 1 -- 172.21.15.111:0/1958645336 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bc000be70 con 0x7f0bd01013e0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.427+0000 7f0bc7fff700 1 -- 172.21.15.111:0/1958645336 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc0010380 con 0x7f0bd01013e0 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- 172.21.15.111:0/1958645336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 msgr2=0x7f0bd01017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1958645336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0bc00098e0 tx=0x7f0bc0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- 172.21.15.111:0/1958645336 shutdown_connections 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1958645336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd01017f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- 172.21.15.111:0/1958645336 >> 172.21.15.111:0/1958645336 conn(0x7f0bd00fcb20 msgr2=0x7f0bd00fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- 172.21.15.111:0/1958645336 shutdown_connections 2024-03-14T12:51:47.134 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- 172.21.15.111:0/1958645336 wait complete. 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 Processor -- start 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.428+0000 7f0bd765f700 1 -- start start 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd765f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd765f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0111eb0 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd53fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd53fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34840/0 (socket says 172.21.15.111:34840) 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd53fb700 1 -- 172.21.15.111:0/1327200496 learned_addr learned my addr 172.21.15.111:0/1327200496 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd53fb700 1 -- 172.21.15.111:0/1327200496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00094f0 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd53fb700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000c010 tx=0x7f0bc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc0010960 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bc001a430 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd01164c0 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc0019580 con 0x7f0bd01013e0 2024-03-14T12:51:47.135 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.429+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd0116890 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.430+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f0bc00197b0 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.430+0000 7f0bc5ffb700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 0x7f0bbc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.430+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0bc004a740 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.430+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd004bed0 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.430+0000 7f0bd4bfa700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 0x7f0bbc038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.431+0000 7f0bd4bfa700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 0x7f0bbc038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009990 tx=0x7f0bcc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.435+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f0bc0019b50 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.617+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0bd0116a80 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.618+0000 7f0bc5ffb700 1 -- 172.21.15.111:0/1327200496 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0bc0029e60 con 0x7f0bd01013e0 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 msgr2=0x7f0bbc038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 0x7f0bbc038d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc009990 tx=0x7f0bcc006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 msgr2=0x7f0bd0076b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000c010 tx=0x7f0bc0009250 comp rx=0 tx=0).stop 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 shutdown_connections 2024-03-14T12:51:47.136 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f0bbc0368d0 0x7f0bbc038d80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 --2- 172.21.15.111:0/1327200496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd01013e0 0x7f0bd0076b30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 >> 172.21.15.111:0/1327200496 conn(0x7f0bd00fcb20 msgr2=0x7f0bd00fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 shutdown_connections 2024-03-14T12:51:47.137 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:46.620+0000 7f0bd765f700 1 -- 172.21.15.111:0/1327200496 wait complete. 2024-03-14T12:51:48.590 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca94d3a700 1 Processor -- start 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca94d3a700 1 -- start start 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca94d3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca94d3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca90101e00 con 0x7fca901014b0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca8e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca8e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34852/0 (socket says 172.21.15.111:34852) 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.694+0000 7fca8e59c700 1 -- 172.21.15.111:0/3989631054 learned_addr learned my addr 172.21.15.111:0/3989631054 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca8e59c700 1 -- 172.21.15.111:0/3989631054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90101f40 con 0x7fca901014b0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca8e59c700 1 --2- 172.21.15.111:0/3989631054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fca800098e0 tx=0x7fca80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dfc5c056e2bb49 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca8d59a700 1 -- 172.21.15.111:0/3989631054 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca8000b860 con 0x7fca901014b0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca8d59a700 1 -- 172.21.15.111:0/3989631054 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca8000be70 con 0x7fca901014b0 2024-03-14T12:51:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca8d59a700 1 -- 172.21.15.111:0/3989631054 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca80010380 con 0x7fca901014b0 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca94d3a700 1 -- 172.21.15.111:0/3989631054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 msgr2=0x7fca901018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca94d3a700 1 --2- 172.21.15.111:0/3989631054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fca800098e0 tx=0x7fca80009250 comp rx=0 tx=0).stop 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca94d3a700 1 -- 172.21.15.111:0/3989631054 shutdown_connections 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca94d3a700 1 --2- 172.21.15.111:0/3989631054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca901018c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.695+0000 7fca94d3a700 1 -- 172.21.15.111:0/3989631054 >> 172.21.15.111:0/3989631054 conn(0x7fca900fcc30 msgr2=0x7fca900ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 -- 172.21.15.111:0/3989631054 shutdown_connections 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 -- 172.21.15.111:0/3989631054 wait complete. 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 Processor -- start 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 -- start start 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.696+0000 7fca94d3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca90114090 con 0x7fca901014b0 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca8e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca8e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34854/0 (socket says 172.21.15.111:34854) 2024-03-14T12:51:48.592 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca8e59c700 1 -- 172.21.15.111:0/4095884517 learned_addr learned my addr 172.21.15.111:0/4095884517 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca8e59c700 1 -- 172.21.15.111:0/4095884517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800094f0 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca8e59c700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca80011ca0 tx=0x7fca80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca80010850 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca80010e60 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca901141d0 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca8001ae20 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.697+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca90114660 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.698+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fca800109b0 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.698+0000 7fca7effd700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 0x7fca78038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.698+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9004bed0 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.698+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fca8004a630 con 0x7fca901014b0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.698+0000 7fca8dd9b700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 0x7fca78038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:48.593 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.699+0000 7fca8dd9b700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 0x7fca78038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca84009990 tx=0x7fca84006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.703+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fca800166e0 con 0x7fca901014b0 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.987+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fca90103240 con 0x7fca901014b0 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.990+0000 7fca7effd700 1 -- 172.21.15.111:0/4095884517 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fca800166e0 con 0x7fca901014b0 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 msgr2=0x7fca78038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 0x7fca78038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca84009990 tx=0x7fca84006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 msgr2=0x7fca90076b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca80011ca0 tx=0x7fca80009250 comp rx=0 tx=0).stop 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 shutdown_connections 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fca78036880 0x7fca78038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 --2- 172.21.15.111:0/4095884517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca901014b0 0x7fca90076b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 >> 172.21.15.111:0/4095884517 conn(0x7fca900fcc30 msgr2=0x7fca900fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 shutdown_connections 2024-03-14T12:51:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr 2024-03-14T12:51:47.992+0000 7fca94d3a700 1 -- 172.21.15.111:0/4095884517 wait complete. 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout:Or, if you are only running a single cluster on this host: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: ceph telemetry on 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout:For more information see: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.595 INFO:teuthology.orchestra.run.smithi111.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-14T12:51:48.596 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:51:48.596 INFO:teuthology.orchestra.run.smithi111.stdout:Bootstrap complete. 2024-03-14T12:51:48.621 INFO:tasks.cephadm:Fetching config... 2024-03-14T12:51:48.622 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:51:48.622 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-14T12:51:48.640 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-14T12:51:48.640 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:51:48.640 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-14T12:51:48.699 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-14T12:51:48.699 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:51:48.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/keyring of=/dev/stdout 2024-03-14T12:51:48.766 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-14T12:51:48.767 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:51:48.767 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-14T12:51:48.824 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-14T12:51:48.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxVTN0xCw/SDwds4enQ5odcJW32BZ0gR2vdKkyE9NitFE5LrorQU5lWGft3J4E/D2FmQqRlNlRQs2yy9bsP7jUQX/A10oor3sxbd3gz40UcGoW24cLA9rr8o0KymJhkB4nXkOVpC4suZXhb/eH02li+Qkbb0Qch7/IWgKNrCZoLC9AkGCtPKD/ru+TXXV8KgpxknIfSTQTDOtWeVhhBSVF2a5djzEJsMIudqP8JAGdYqFPT003/wWvbdxqUKQKQrLthMqcWA8kMahk79QtG+PbR/ODTBqvoB/NN7p4Do91nMbe3p9CxEhPufyLrn60dw1gmh+L/Ktk+XBGdKh2xlOvBhMX3xBEQn9qVMNreMtXF5+ov4TBonChO3BXxrTAOrIZUS9D+DMixJyc6OKch6475WSnD7TBrL6FKFhE7iVDlD2GgTEsqCuAzbIebnhYeQK3eCwsNT7vyQfOkj4A5S6QdQJBKUNY0pf3WvOT0hNASC/btvWZDowTRvpcSWxfeU= ceph-591fecbc-e201-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-14T12:51:48.901 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxVTN0xCw/SDwds4enQ5odcJW32BZ0gR2vdKkyE9NitFE5LrorQU5lWGft3J4E/D2FmQqRlNlRQs2yy9bsP7jUQX/A10oor3sxbd3gz40UcGoW24cLA9rr8o0KymJhkB4nXkOVpC4suZXhb/eH02li+Qkbb0Qch7/IWgKNrCZoLC9AkGCtPKD/ru+TXXV8KgpxknIfSTQTDOtWeVhhBSVF2a5djzEJsMIudqP8JAGdYqFPT003/wWvbdxqUKQKQrLthMqcWA8kMahk79QtG+PbR/ODTBqvoB/NN7p4Do91nMbe3p9CxEhPufyLrn60dw1gmh+L/Ktk+XBGdKh2xlOvBhMX3xBEQn9qVMNreMtXF5+ov4TBonChO3BXxrTAOrIZUS9D+DMixJyc6OKch6475WSnD7TBrL6FKFhE7iVDlD2GgTEsqCuAzbIebnhYeQK3eCwsNT7vyQfOkj4A5S6QdQJBKUNY0pf3WvOT0hNASC/btvWZDowTRvpcSWxfeU= ceph-591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:51:48.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxVTN0xCw/SDwds4enQ5odcJW32BZ0gR2vdKkyE9NitFE5LrorQU5lWGft3J4E/D2FmQqRlNlRQs2yy9bsP7jUQX/A10oor3sxbd3gz40UcGoW24cLA9rr8o0KymJhkB4nXkOVpC4suZXhb/eH02li+Qkbb0Qch7/IWgKNrCZoLC9AkGCtPKD/ru+TXXV8KgpxknIfSTQTDOtWeVhhBSVF2a5djzEJsMIudqP8JAGdYqFPT003/wWvbdxqUKQKQrLthMqcWA8kMahk79QtG+PbR/ODTBqvoB/NN7p4Do91nMbe3p9CxEhPufyLrn60dw1gmh+L/Ktk+XBGdKh2xlOvBhMX3xBEQn9qVMNreMtXF5+ov4TBonChO3BXxrTAOrIZUS9D+DMixJyc6OKch6475WSnD7TBrL6FKFhE7iVDlD2GgTEsqCuAzbIebnhYeQK3eCwsNT7vyQfOkj4A5S6QdQJBKUNY0pf3WvOT0hNASC/btvWZDowTRvpcSWxfeU= ceph-591fecbc-e201-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-14T12:51:48.960 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOxVTN0xCw/SDwds4enQ5odcJW32BZ0gR2vdKkyE9NitFE5LrorQU5lWGft3J4E/D2FmQqRlNlRQs2yy9bsP7jUQX/A10oor3sxbd3gz40UcGoW24cLA9rr8o0KymJhkB4nXkOVpC4suZXhb/eH02li+Qkbb0Qch7/IWgKNrCZoLC9AkGCtPKD/ru+TXXV8KgpxknIfSTQTDOtWeVhhBSVF2a5djzEJsMIudqP8JAGdYqFPT003/wWvbdxqUKQKQrLthMqcWA8kMahk79QtG+PbR/ODTBqvoB/NN7p4Do91nMbe3p9CxEhPufyLrn60dw1gmh+L/Ktk+XBGdKh2xlOvBhMX3xBEQn9qVMNreMtXF5+ov4TBonChO3BXxrTAOrIZUS9D+DMixJyc6OKch6475WSnD7TBrL6FKFhE7iVDlD2GgTEsqCuAzbIebnhYeQK3eCwsNT7vyQfOkj4A5S6QdQJBKUNY0pf3WvOT0hNASC/btvWZDowTRvpcSWxfeU= ceph-591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:51:48.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-14T12:51:49.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:48 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/4095884517' entity='client.admin' 2024-03-14T12:51:49.238 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:51:51.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 -- 172.21.15.111:0/3503328349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 msgr2=0x7fe8dc0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3503328349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc0fd460 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc0048c0 tx=0x7fe8cc009e30 comp rx=0 tx=0).stop 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 -- 172.21.15.111:0/3503328349 shutdown_connections 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3503328349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc0fd460 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 -- 172.21.15.111:0/3503328349 >> 172.21.15.111:0/3503328349 conn(0x7fe8dc0f70d0 msgr2=0x7fe8dc0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 -- 172.21.15.111:0/3503328349 shutdown_connections 2024-03-14T12:51:51.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.680+0000 7fe8e4306700 1 -- 172.21.15.111:0/3503328349 wait complete. 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 Processor -- start 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 -- start start 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc10fcb0 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e20a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e20a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44656/0 (socket says 172.21.15.111:44656) 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e20a2700 1 -- 172.21.15.111:0/3694079969 learned_addr learned my addr 172.21.15.111:0/3694079969 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e20a2700 1 -- 172.21.15.111:0/3694079969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc005040 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e20a2700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009300 tx=0x7fe8cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc009600 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8cc009760 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc01c5b0 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc10fdf0 con 0x7fe8dc0fd050 2024-03-14T12:51:51.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.681+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc1102e0 con 0x7fe8dc0fd050 2024-03-14T12:51:51.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.682+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe8cc01c820 con 0x7fe8dc0fd050 2024-03-14T12:51:51.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.682+0000 7fe8d2ffd700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 0x7fe8c8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:51.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.682+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc106060 con 0x7fe8dc0fd050 2024-03-14T12:51:51.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.682+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe8cc049b20 con 0x7fe8dc0fd050 2024-03-14T12:51:51.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.682+0000 7fe8e18a1700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 0x7fe8c8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:51.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.683+0000 7fe8e18a1700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 0x7fe8c8038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009990 tx=0x7fe8d8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:51.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.687+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fe8cc011b60 con 0x7fe8dc0fd050 2024-03-14T12:51:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.871+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe8dc101d30 con 0x7fe8dc0fd050 2024-03-14T12:51:51.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.874+0000 7fe8d2ffd700 1 -- 172.21.15.111:0/3694079969 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe8cc013420 con 0x7fe8dc0fd050 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 msgr2=0x7fe8c8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 0x7fe8c8038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009990 tx=0x7fe8d8006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 msgr2=0x7fe8dc065bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009300 tx=0x7fe8cc009e30 comp rx=0 tx=0).stop 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 shutdown_connections 2024-03-14T12:51:51.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe8c80368d0 0x7fe8c8038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:51.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 --2- 172.21.15.111:0/3694079969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8dc0fd050 0x7fe8dc065bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:51.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 >> 172.21.15.111:0/3694079969 conn(0x7fe8dc0f70d0 msgr2=0x7fe8dc0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:51.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 shutdown_connections 2024-03-14T12:51:51.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:51.876+0000 7fe8e4306700 1 -- 172.21.15.111:0/3694079969 wait complete. 2024-03-14T12:51:52.766 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-14T12:51:52.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-14T12:51:53.039 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:51:53.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:52 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3694079969' entity='client.admin' 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:51:55.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:54 smithi111 ceph-mon[39688]: Deploying daemon crash.smithi111 on smithi111 2024-03-14T12:51:55.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/3566119911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 msgr2=0x7ff1e40ffa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/3566119911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e40ffa90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0048c0 tx=0x7ff1dc009e30 comp rx=0 tx=0).stop 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/3566119911 shutdown_connections 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/3566119911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e40ffa90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/3566119911 >> 172.21.15.111:0/3566119911 conn(0x7ff1e40f70d0 msgr2=0x7ff1e40f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/3566119911 shutdown_connections 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.426+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/3566119911 wait complete. 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e9c3a700 1 Processor -- start 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e9c3a700 1 -- start start 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e9c3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e9c3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e410f720 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44672/0 (socket says 172.21.15.111:44672) 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e37fe700 1 -- 172.21.15.111:0/55017838 learned_addr learned my addr 172.21.15.111:0/55017838 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:55.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.427+0000 7ff1e37fe700 1 -- 172.21.15.111:0/55017838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc005040 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1e37fe700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b3b0 tx=0x7ff1dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1dc0095d0 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1dc009730 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e410f920 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1dc01c5b0 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.428+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e4110490 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7ff1dc01c830 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e4102e20 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1dbfff700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 0x7ff1d0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff1dc0499f0 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1e2ffd700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 0x7ff1d0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.429+0000 7ff1e2ffd700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 0x7ff1d0038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009990 tx=0x7ff1cc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.434+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7ff1dc023300 con 0x7ff1e40fd6b0 2024-03-14T12:51:55.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.625+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1e40fba20 con 0x7ff1d0036880 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.631+0000 7ff1dbfff700 1 -- 172.21.15.111:0/55017838 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff1e40fba20 con 0x7ff1d0036880 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.632+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 msgr2=0x7ff1d0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.632+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 0x7ff1d0038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009990 tx=0x7ff1cc006e40 comp rx=0 tx=0).stop 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.632+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 msgr2=0x7ff1e410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.632+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b3b0 tx=0x7ff1dc009e30 comp rx=0 tx=0).stop 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 shutdown_connections 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff1d0036880 0x7ff1d0038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 --2- 172.21.15.111:0/55017838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1e40fd6b0 0x7ff1e410f1e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 >> 172.21.15.111:0/55017838 conn(0x7ff1e40f70d0 msgr2=0x7ff1e40f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 shutdown_connections 2024-03-14T12:51:55.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:55.633+0000 7ff1e9c3a700 1 -- 172.21.15.111:0/55017838 wait complete. 2024-03-14T12:51:56.296 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi137 2024-03-14T12:51:56.296 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:51:56.296 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.conf 2024-03-14T12:51:56.318 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:51:56.318 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T12:51:56.379 INFO:tasks.cephadm:Adding host smithi137 to orchestrator... 2024-03-14T12:51:56.379 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch host add smithi137 2024-03-14T12:51:56.823 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:51:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:56 smithi111 ceph-mon[39688]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:51:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:56 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 -- 172.21.15.111:0/3549588126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 msgr2=0x7f14380fb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 --2- 172.21.15.111:0/3549588126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f14380fb2c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14280048c0 tx=0x7f1428009e30 comp rx=0 tx=0).stop 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 -- 172.21.15.111:0/3549588126 shutdown_connections 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 --2- 172.21.15.111:0/3549588126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f14380fb2c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 -- 172.21.15.111:0/3549588126 >> 172.21.15.111:0/3549588126 conn(0x7f14380f6850 msgr2=0x7f14380f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 -- 172.21.15.111:0/3549588126 shutdown_connections 2024-03-14T12:51:58.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.918+0000 7f143d969700 1 -- 172.21.15.111:0/3549588126 wait complete. 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f143d969700 1 Processor -- start 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f143d969700 1 -- start start 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f143d969700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f143d969700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143810d4f0 con 0x7f14380faeb0 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f1436ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f1436ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44704/0 (socket says 172.21.15.111:44704) 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f1436ffd700 1 -- 172.21.15.111:0/203331391 learned_addr learned my addr 172.21.15.111:0/203331391 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f1436ffd700 1 -- 172.21.15.111:0/203331391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428005040 con 0x7f14380faeb0 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f1436ffd700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1428000c00 tx=0x7f1428009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:58.920 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1428009620 con 0x7f14380faeb0 2024-03-14T12:51:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143810d6f0 con 0x7f14380faeb0 2024-03-14T12:51:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1428009780 con 0x7f14380faeb0 2024-03-14T12:51:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.919+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f142801c5b0 con 0x7f14380faeb0 2024-03-14T12:51:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.920+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143810e260 con 0x7f14380faeb0 2024-03-14T12:51:58.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.921+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1438102e50 con 0x7f14380faeb0 2024-03-14T12:51:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.921+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1428013a60 con 0x7f14380faeb0 2024-03-14T12:51:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.921+0000 7f141f7fe700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 0x7f1420038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:51:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.921+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f142804a580 con 0x7f14380faeb0 2024-03-14T12:51:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.921+0000 7f14367fc700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 0x7f1420038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:51:58.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.922+0000 7f14367fc700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 0x7f1420038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f142c009990 tx=0x7f142c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:51:58.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:58.925+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f142801bb80 con 0x7f14380faeb0 2024-03-14T12:51:59.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:51:59.139+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f143810e4a0 con 0x7f1420036880 2024-03-14T12:51:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:51:59.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-14T12:51:59.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:51:59 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:00.820 INFO:teuthology.orchestra.run.smithi111.stdout:Added host 'smithi137' with addr '172.21.15.137' 2024-03-14T12:52:00.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.820+0000 7f141f7fe700 1 -- 172.21.15.111:0/203331391 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f143810e4a0 con 0x7f1420036880 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 msgr2=0x7f1420038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 0x7f1420038d30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f142c009990 tx=0x7f142c006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 msgr2=0x7f143810cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1428000c00 tx=0x7f1428009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 shutdown_connections 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f1420036880 0x7f1420038d30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 --2- 172.21.15.111:0/203331391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14380faeb0 0x7f143810cfb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 >> 172.21.15.111:0/203331391 conn(0x7f14380f6850 msgr2=0x7f1438103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 shutdown_connections 2024-03-14T12:52:00.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:00.822+0000 7f143d969700 1 -- 172.21.15.111:0/203331391 wait complete. 2024-03-14T12:52:00.849 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:00 smithi111 ceph-mon[39688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:00.850 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:00 smithi111 ceph-mon[39688]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:52:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:01 smithi111 ceph-mon[39688]: Deploying cephadm binary to smithi137 2024-03-14T12:52:01.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:01 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:02.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-14T12:52:02.433 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:52:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:02 smithi111 ceph-mon[39688]: Added host smithi137 2024-03-14T12:52:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:02 smithi111 ceph-mon[39688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:04.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/563132288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 msgr2=0x7f3cb80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:04.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/563132288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80fd460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ca80048c0 tx=0x7f3ca8009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:04.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/563132288 shutdown_connections 2024-03-14T12:52:04.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/563132288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80fd460 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/563132288 >> 172.21.15.111:0/563132288 conn(0x7f3cb80f70d0 msgr2=0x7f3cb80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/563132288 shutdown_connections 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.598+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/563132288 wait complete. 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cbe6a4700 1 Processor -- start 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cbe6a4700 1 -- start start 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cbe6a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cbe6a4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80f95a0 con 0x7f3cb80fd050 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54988/0 (socket says 172.21.15.111:54988) 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb7fff700 1 -- 172.21.15.111:0/383864060 learned_addr learned my addr 172.21.15.111:0/383864060 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb7fff700 1 -- 172.21.15.111:0/383864060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca8005040 con 0x7f3cb80fd050 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb7fff700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8000c00 tx=0x7f3ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.599+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca8009620 con 0x7f3cb80fd050 2024-03-14T12:52:04.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80f7670 con 0x7f3cb80fd050 2024-03-14T12:52:04.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ca8009780 con 0x7f3cb80fd050 2024-03-14T12:52:04.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ca801c5b0 con 0x7f3cb80fd050 2024-03-14T12:52:04.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb80f7b60 con 0x7f3cb80fd050 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f3ca801c7f0 con 0x7f3cb80fd050 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cb4ff9700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 0x7f3ca0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.600+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3ca8049b00 con 0x7f3cb80fd050 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.601+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb805f5f0 con 0x7f3cb80fd050 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.601+0000 7f3cb77fe700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 0x7f3ca0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:04.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.601+0000 7f3cb77fe700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 0x7f3ca0038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009990 tx=0x7f3cac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:04.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.605+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f3ca8011dd0 con 0x7f3cb80fd050 2024-03-14T12:52:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.834+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cb8065780 con 0x7f3ca0036880 2024-03-14T12:52:04.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.835+0000 7f3cb4ff9700 1 -- 172.21.15.111:0/383864060 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f3cb8065780 con 0x7f3ca0036880 2024-03-14T12:52:04.835 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:52:04.835 INFO:teuthology.orchestra.run.smithi111.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}] 2024-03-14T12:52:04.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 msgr2=0x7f3ca0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 0x7f3ca0038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3cac009990 tx=0x7f3cac006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 msgr2=0x7f3cb80f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8000c00 tx=0x7f3ca8009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 shutdown_connections 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f3ca0036880 0x7f3ca0038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 --2- 172.21.15.111:0/383864060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cb80fd050 0x7f3cb80f9060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 >> 172.21.15.111:0/383864060 conn(0x7f3cb80f70d0 msgr2=0x7f3cb8065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 shutdown_connections 2024-03-14T12:52:04.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:04.837+0000 7f3cbe6a4700 1 -- 172.21.15.111:0/383864060 wait complete. 2024-03-14T12:52:04.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:04 smithi111 ceph-mon[39688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:04.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:04 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:04.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:04 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:04.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:04 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:04.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:04 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:05.667 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-14T12:52:05.667 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-14T12:52:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:05 smithi111 ceph-mon[39688]: Deploying daemon alertmanager.smithi111 on smithi111 2024-03-14T12:52:05.934 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:52:06.554 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:06 smithi111 ceph-mon[39688]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T12:52:06.554 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:06 smithi111 ceph-mon[39688]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:08.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 -- 172.21.15.111:0/4121878025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 msgr2=0x7f79600ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:08.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 --2- 172.21.15.111:0/4121878025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f79600ff720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7950009c60 tx=0x7f7950004d10 comp rx=0 tx=0).stop 2024-03-14T12:52:08.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 -- 172.21.15.111:0/4121878025 shutdown_connections 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 --2- 172.21.15.111:0/4121878025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f79600ff720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 -- 172.21.15.111:0/4121878025 >> 172.21.15.111:0/4121878025 conn(0x7f79600f70d0 msgr2=0x7f79600f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 -- 172.21.15.111:0/4121878025 shutdown_connections 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 -- 172.21.15.111:0/4121878025 wait complete. 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.342+0000 7f7965e92700 1 Processor -- start 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7965e92700 1 -- start start 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7965e92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7965e92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796010d5a0 con 0x7f7960065d90 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f795f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f795f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54998/0 (socket says 172.21.15.111:54998) 2024-03-14T12:52:08.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f795f7fe700 1 -- 172.21.15.111:0/3902178971 learned_addr learned my addr 172.21.15.111:0/3902178971 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f795f7fe700 1 -- 172.21.15.111:0/3902178971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7950009940 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f795f7fe700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7950009490 tx=0x7f7950004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f795000be40 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f795001d8f0 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.343+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796010d7a0 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.344+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7950012890 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.344+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796010e310 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.344+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f7950019040 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.344+0000 7f7957fff700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 0x7f794c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.344+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7960102e20 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.345+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f795004b1a0 con 0x7f7960065d90 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.345+0000 7f795effd700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 0x7f794c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.345+0000 7f795effd700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 0x7f794c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:08.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.349+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f795002e940 con 0x7f7960065d90 2024-03-14T12:52:08.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:08.553+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f796010ec70 con 0x7f7960065d90 2024-03-14T12:52:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:08 smithi111 ceph-mon[39688]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:09.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.513+0000 7f7957fff700 1 -- 172.21.15.111:0/3902178971 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f795002c480 con 0x7f7960065d90 2024-03-14T12:52:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 msgr2=0x7f794c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:09.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 0x7f794c038d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 msgr2=0x7f796010d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7950009490 tx=0x7f7950004d10 comp rx=0 tx=0).stop 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 shutdown_connections 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f794c036880 0x7f794c038d30 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7948009990 tx=0x7f7948006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 --2- 172.21.15.111:0/3902178971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7960065d90 0x7f796010d060 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.516+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 >> 172.21.15.111:0/3902178971 conn(0x7f79600f70d0 msgr2=0x7f79600f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:09.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.517+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 shutdown_connections 2024-03-14T12:52:09.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:52:09.520+0000 7f7965e92700 1 -- 172.21.15.111:0/3902178971 wait complete. 2024-03-14T12:52:09.520 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2024-03-14T12:52:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:09 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3902178971' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-14T12:52:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:09 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:10.838 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:10 smithi111 ceph-mon[39688]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:10.838 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3902178971' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-14T12:52:10.839 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:10 smithi111 ceph-mon[39688]: osdmap e4: 0 total, 0 up, 0 in 2024-03-14T12:52:10.861 INFO:tasks.cephadm:Adding mon.smithi111 on smithi111 2024-03-14T12:52:10.861 INFO:tasks.cephadm:Adding mon.smithi137 on smithi137 2024-03-14T12:52:10.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch apply mon '2;smithi111:172.21.15.111=smithi111;smithi137:172.21.15.137=smithi137' 2024-03-14T12:52:11.154 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:11.216 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:12.799 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:12 smithi111 ceph-mon[39688]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- 172.21.15.137:0/2440713879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 msgr2=0x7f89b00fe580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 --2- 172.21.15.137:0/2440713879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fe580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f89a00048c0 tx=0x7f89a0009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- 172.21.15.137:0/2440713879 shutdown_connections 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 --2- 172.21.15.137:0/2440713879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fe580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- 172.21.15.137:0/2440713879 >> 172.21.15.137:0/2440713879 conn(0x7f89b00f6850 msgr2=0x7f89b00f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- 172.21.15.137:0/2440713879 shutdown_connections 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- 172.21.15.137:0/2440713879 wait complete. 2024-03-14T12:52:12.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 Processor -- start 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- start start 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.919+0000 7f89b8627700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b00fddc0 con 0x7f89b00fe170 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b63c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b63c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:42828/0 (socket says 172.21.15.137:42828) 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b63c3700 1 -- 172.21.15.137:0/3518538337 learned_addr learned my addr 172.21.15.137:0/3518538337 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b63c3700 1 -- 172.21.15.137:0/3518538337 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0005040 con 0x7f89b00fe170 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b63c3700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c00 tx=0x7f89a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:12.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a0009620 con 0x7f89b00fe170 2024-03-14T12:52:12.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b00fbe90 con 0x7f89b00fe170 2024-03-14T12:52:12.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89a0009780 con 0x7f89b00fe170 2024-03-14T12:52:12.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89a001c5b0 con 0x7f89b00fe170 2024-03-14T12:52:12.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.920+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b00fc380 con 0x7f89b00fe170 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.921+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f89a0013420 con 0x7f89b00fe170 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.922+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b005f5f0 con 0x7f89b00fe170 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.922+0000 7f89a6ffd700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 0x7f899c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.922+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f89a0049b80 con 0x7f89b00fe170 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.922+0000 7f89b5bc2700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 0x7f899c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:12.924 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.922+0000 7f89b5bc2700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 0x7f899c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009990 tx=0x7f89ac006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:12.928 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:12.928+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f89a004be60 con 0x7f89b00fe170 2024-03-14T12:52:13.118 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.117+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 --> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}) v1 -- 0x7f89b0106cb0 con 0x7f899c036880 2024-03-14T12:52:13.124 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.124+0000 7f89a6ffd700 1 -- 172.21.15.137:0/3518538337 <== mgr.14160 v2:172.21.15.111:6800/4041156471 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f89b0106cb0 con 0x7f899c036880 2024-03-14T12:52:13.125 INFO:teuthology.orchestra.run.smithi137.stdout:Scheduled mon update... 2024-03-14T12:52:13.126 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 msgr2=0x7f899c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:13.126 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 0x7f899c038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f89ac009990 tx=0x7f89ac006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:13.126 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 msgr2=0x7f89b00fd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c00 tx=0x7f89a0009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 shutdown_connections 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f899c036880 0x7f899c038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 --2- 172.21.15.137:0/3518538337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89b00fe170 0x7f89b00fd880 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 >> 172.21.15.137:0/3518538337 conn(0x7f89b00f6850 msgr2=0x7f89b00f7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.125+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 shutdown_connections 2024-03-14T12:52:13.127 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:13.126+0000 7f89b8627700 1 -- 172.21.15.137:0/3518538337 wait complete. 2024-03-14T12:52:13.787 DEBUG:teuthology.orchestra.run.smithi137:mon.smithi137> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi137.service 2024-03-14T12:52:13.790 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:13.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:13.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:-- Logs begin at Thu 2024-03-14 12:38:18 UTC. -- 2024-03-14T12:52:14.096 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:14.154 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:14.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:14 smithi111 ceph-mon[39688]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:14.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:14 smithi111 ceph-mon[39688]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi137:172.21.15.137=smithi137", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:52:14.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:14 smithi111 ceph-mon[39688]: Saving service mon spec with placement smithi111:172.21.15.111=smithi111;smithi137:172.21.15.137=smithi137;count:2 2024-03-14T12:52:14.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:14 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.833+0000 7f060598d700 1 -- 172.21.15.137:0/2614769480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 msgr2=0x7f06000fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.833+0000 7f060598d700 1 --2- 172.21.15.137:0/2614769480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f06000fbd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05e80048c0 tx=0x7f05e8009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.833+0000 7f060598d700 1 -- 172.21.15.137:0/2614769480 shutdown_connections 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.833+0000 7f060598d700 1 --2- 172.21.15.137:0/2614769480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f06000fbd70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.833+0000 7f060598d700 1 -- 172.21.15.137:0/2614769480 >> 172.21.15.137:0/2614769480 conn(0x7f06000f70c0 msgr2=0x7f06000f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 -- 172.21.15.137:0/2614769480 shutdown_connections 2024-03-14T12:52:15.834 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 -- 172.21.15.137:0/2614769480 wait complete. 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 Processor -- start 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 -- start start 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.834+0000 7f060598d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060010fcd0 con 0x7f06000fb960 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:42850/0 (socket says 172.21.15.137:42850) 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05feffd700 1 -- 172.21.15.137:0/2713817572 learned_addr learned my addr 172.21.15.137:0/2713817572 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05feffd700 1 -- 172.21.15.137:0/2713817572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e8005040 con 0x7f06000fb960 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05feffd700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009300 tx=0x7f05e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:15.836 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05e8009600 con 0x7f06000fb960 2024-03-14T12:52:15.837 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060010fe10 con 0x7f06000fb960 2024-03-14T12:52:15.837 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05e8009760 con 0x7f06000fb960 2024-03-14T12:52:15.837 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05e801c5b0 con 0x7f06000fb960 2024-03-14T12:52:15.837 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.835+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0600110300 con 0x7f06000fb960 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.836+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f05e8013420 con 0x7f06000fb960 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.836+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0600106060 con 0x7f06000fb960 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.837+0000 7f05f77fe700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 0x7f05ec038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.837+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f05e8049a70 con 0x7f06000fb960 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.837+0000 7f05fe7fc700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 0x7f05ec038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.837+0000 7f05fe7fc700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 0x7f05ec038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009990 tx=0x7f05f0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:15.841 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:15.842+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f05e8013960 con 0x7f06000fb960 2024-03-14T12:52:16.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.098+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f060005f5f0 con 0x7f06000fb960 2024-03-14T12:52:16.099 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.099+0000 7f05f77fe700 1 -- 172.21.15.137:0/2713817572 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f05e8011370 con 0x7f06000fb960 2024-03-14T12:52:16.099 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:16.099 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:16.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 msgr2=0x7f05ec038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:16.101 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 0x7f05ec038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009990 tx=0x7f05f0006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 msgr2=0x7f0600066880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009300 tx=0x7f05e8009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 shutdown_connections 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f05ec036880 0x7f05ec038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 --2- 172.21.15.137:0/2713817572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06000fb960 0x7f0600066880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 >> 172.21.15.137:0/2713817572 conn(0x7f06000f70c0 msgr2=0x7f06000f79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 shutdown_connections 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:16.101+0000 7f060598d700 1 -- 172.21.15.137:0/2713817572 wait complete. 2024-03-14T12:52:16.102 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: Deploying daemon grafana.smithi111 on smithi111 2024-03-14T12:52:16.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:15 smithi111 ceph-mon[39688]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:16 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2713817572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:17.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:17.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:17.901 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:17.959 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:18.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:17 smithi111 ceph-mon[39688]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 -- 172.21.15.137:0/657649459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 msgr2=0x7f2f6c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/657649459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c0fd460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0048c0 tx=0x7f2f5c009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 -- 172.21.15.137:0/657649459 shutdown_connections 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/657649459 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c0fd460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 -- 172.21.15.137:0/657649459 >> 172.21.15.137:0/657649459 conn(0x7f2f6c0f70d0 msgr2=0x7f2f6c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 -- 172.21.15.137:0/657649459 shutdown_connections 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.670+0000 7f2f71dd2700 1 -- 172.21.15.137:0/657649459 wait complete. 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.671+0000 7f2f71dd2700 1 Processor -- start 2024-03-14T12:52:19.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.671+0000 7f2f71dd2700 1 -- start start 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.671+0000 7f2f71dd2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.671+0000 7f2f71dd2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c10c460 con 0x7f2f6c0fd050 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f6b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:41258/0 (socket says 172.21.15.137:41258) 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f6b7fe700 1 -- 172.21.15.137:0/1724496107 learned_addr learned my addr 172.21.15.137:0/1724496107 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f6b7fe700 1 -- 172.21.15.137:0/1724496107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c005040 con 0x7f2f6c0fd050 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f6b7fe700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009300 tx=0x7f2f5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c009810 con 0x7f2f6c0fd050 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f5c01e8d0 con 0x7f2f6c0fd050 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c013740 con 0x7f2f6c0fd050 2024-03-14T12:52:19.674 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c10c660 con 0x7f2f6c0fd050 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.672+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c10d0f0 con 0x7f2f6c0fd050 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.673+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f2f5c0139b0 con 0x7f2f6c0fd050 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.673+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c101d30 con 0x7f2f6c0fd050 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.673+0000 7f2f53fff700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 0x7f2f54038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.673+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2f5c04c7a0 con 0x7f2f6c0fd050 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.673+0000 7f2f6affd700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 0x7f2f54038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:19.675 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.674+0000 7f2f6affd700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 0x7f2f54038d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009990 tx=0x7f2f60006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:19.678 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.678+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f2f5c011780 con 0x7f2f6c0fd050 2024-03-14T12:52:19.932 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.931+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f6c05f5f0 con 0x7f2f6c0fd050 2024-03-14T12:52:19.933 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.932+0000 7f2f53fff700 1 -- 172.21.15.137:0/1724496107 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2f5c026360 con 0x7f2f6c0fd050 2024-03-14T12:52:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:19.934 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:19.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 msgr2=0x7f2f54038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:19.935 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 0x7f2f54038d80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009990 tx=0x7f2f60006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 msgr2=0x7f2f6c10bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009300 tx=0x7f2f5c009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 shutdown_connections 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f2f540368d0 0x7f2f54038d80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 --2- 172.21.15.137:0/1724496107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f6c0fd050 0x7f2f6c10bf20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:19.936 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 >> 172.21.15.137:0/1724496107 conn(0x7f2f6c0f70d0 msgr2=0x7f2f6c0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:19.937 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 shutdown_connections 2024-03-14T12:52:19.937 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:19.934+0000 7f2f71dd2700 1 -- 172.21.15.137:0/1724496107 wait complete. 2024-03-14T12:52:19.937 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:20.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:20 smithi111 ceph-mon[39688]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:20.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:20 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:20.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:20 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1724496107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:21.505 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:21.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:21.795 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:21.858 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:22 smithi111 ceph-mon[39688]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 -- 172.21.15.137:0/872001442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 msgr2=0x7f14e40fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 --2- 172.21.15.137:0/872001442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e40fce60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f14d40048c0 tx=0x7f14d4009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 -- 172.21.15.137:0/872001442 shutdown_connections 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 --2- 172.21.15.137:0/872001442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e40fce60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 -- 172.21.15.137:0/872001442 >> 172.21.15.137:0/872001442 conn(0x7f14e40f8190 msgr2=0x7f14e40fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 -- 172.21.15.137:0/872001442 shutdown_connections 2024-03-14T12:52:23.385 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.385+0000 7f14e9985700 1 -- 172.21.15.137:0/872001442 wait complete. 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e9985700 1 Processor -- start 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e9985700 1 -- start start 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e9985700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e9985700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e410d4e0 con 0x7f14e40fca50 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:41288/0 (socket says 172.21.15.137:41288) 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2679428930 learned_addr learned my addr 172.21.15.137:0/2679428930 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.386+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2679428930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4005040 con 0x7f14e40fca50 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14e2ffd700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009300 tx=0x7f14d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:23.388 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d4009600 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e410d6e0 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d4009760 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d401c5b0 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.387+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e410e250 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.388+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e4102e20 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.388+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f14d4013420 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.388+0000 7f14cb7fe700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 0x7f14cc038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.388+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f14d4049a70 con 0x7f14e40fca50 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.388+0000 7f14e27fc700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 0x7f14cc038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:23.389 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.389+0000 7f14e27fc700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 0x7f14cc038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009990 tx=0x7f14d8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:23.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.393+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f14d4013960 con 0x7f14e40fca50 2024-03-14T12:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.644+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e410ec60 con 0x7f14e40fca50 2024-03-14T12:52:23.644 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.644+0000 7f14cb7fe700 1 -- 172.21.15.137:0/2679428930 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f14d4011370 con 0x7f14e40fca50 2024-03-14T12:52:23.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:23.645 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 msgr2=0x7f14cc038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 0x7f14cc038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009990 tx=0x7f14d8006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 msgr2=0x7f14e410cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009300 tx=0x7f14d4009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:23.646 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 shutdown_connections 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14cc036880 0x7f14cc038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 --2- 172.21.15.137:0/2679428930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14e40fca50 0x7f14e410cfa0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 >> 172.21.15.137:0/2679428930 conn(0x7f14e40f8190 msgr2=0x7f14e40f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 shutdown_connections 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:23.646+0000 7f14e9985700 1 -- 172.21.15.137:0/2679428930 wait complete. 2024-03-14T12:52:23.647 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:24.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:24 smithi111 ceph-mon[39688]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:24.561 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:24 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2679428930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:25.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:25.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:25.593 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:25.657 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:26.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:26 smithi111 ceph-mon[39688]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:27.172 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 -- 172.21.15.137:0/4117760913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 msgr2=0x7f9e000fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:27.172 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 --2- 172.21.15.137:0/4117760913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e000fd460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9df40048c0 tx=0x7f9df4009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 -- 172.21.15.137:0/4117760913 shutdown_connections 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 --2- 172.21.15.137:0/4117760913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e000fd460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 -- 172.21.15.137:0/4117760913 >> 172.21.15.137:0/4117760913 conn(0x7f9e000f70d0 msgr2=0x7f9e000f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 -- 172.21.15.137:0/4117760913 shutdown_connections 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.172+0000 7f9e08780700 1 -- 172.21.15.137:0/4117760913 wait complete. 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e08780700 1 Processor -- start 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e08780700 1 -- start start 2024-03-14T12:52:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e08780700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:27.174 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e08780700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0010fcb0 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e0651c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e0651c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:41306/0 (socket says 172.21.15.137:41306) 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e0651c700 1 -- 172.21.15.137:0/234994318 learned_addr learned my addr 172.21.15.137:0/234994318 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e0651c700 1 -- 172.21.15.137:0/234994318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4005040 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.173+0000 7f9e0651c700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009300 tx=0x7f9df4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.175+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df4009600 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.175+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9df4009760 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.175+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9df401c5b0 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.175+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0010fdf0 con 0x7f9e000fd050 2024-03-14T12:52:27.175 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.175+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e001102e0 con 0x7f9e000fd050 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f9df401c820 con 0x7f9e000fd050 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e00106060 con 0x7f9e000fd050 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9df2ffd700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 0x7f9dec038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9df4049b10 con 0x7f9e000fd050 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9e05d1b700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 0x7f9dec038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:27.177 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.176+0000 7f9e05d1b700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 0x7f9dec038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009990 tx=0x7f9dfc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:27.181 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.181+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f9df4011dd0 con 0x7f9e000fd050 2024-03-14T12:52:27.432 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.432+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e00101d30 con 0x7f9e000fd050 2024-03-14T12:52:27.433 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.433+0000 7f9df2ffd700 1 -- 172.21.15.137:0/234994318 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9df4024360 con 0x7f9e000fd050 2024-03-14T12:52:27.433 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:27.433 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 msgr2=0x7f9dec038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 0x7f9dec038d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009990 tx=0x7f9dfc006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 msgr2=0x7f9e00065bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009300 tx=0x7f9df4009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 shutdown_connections 2024-03-14T12:52:27.435 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f9dec0368d0 0x7f9dec038d80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:27.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 --2- 172.21.15.137:0/234994318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e000fd050 0x7f9e00065bf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:27.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 >> 172.21.15.137:0/234994318 conn(0x7f9e000f70d0 msgr2=0x7f9e000f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:27.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 shutdown_connections 2024-03-14T12:52:27.436 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:27.435+0000 7f9e08780700 1 -- 172.21.15.137:0/234994318 wait complete. 2024-03-14T12:52:27.436 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:28 smithi111 ceph-mon[39688]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:28 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/234994318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:29.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:29.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:29.294 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:29.354 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:30 smithi111 ceph-mon[39688]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:31.048 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 -- 172.21.15.137:0/2152136700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 msgr2=0x7f8e9c0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 --2- 172.21.15.137:0/2152136700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c0fce60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8e980048c0 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 -- 172.21.15.137:0/2152136700 shutdown_connections 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 --2- 172.21.15.137:0/2152136700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c0fce60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 -- 172.21.15.137:0/2152136700 >> 172.21.15.137:0/2152136700 conn(0x7f8e9c0f81d0 msgr2=0x7f8e9c0fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 -- 172.21.15.137:0/2152136700 shutdown_connections 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.048+0000 7f8ea4483700 1 -- 172.21.15.137:0/2152136700 wait complete. 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea4483700 1 Processor -- start 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea4483700 1 -- start start 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea4483700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:31.049 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea4483700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9c10d4c0 con 0x7f8e9c0fca50 2024-03-14T12:52:31.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea221f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:31.051 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea221f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:51326/0 (socket says 172.21.15.137:51326) 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea221f700 1 -- 172.21.15.137:0/739678416 learned_addr learned my addr 172.21.15.137:0/739678416 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.049+0000 7f8ea221f700 1 -- 172.21.15.137:0/739678416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98005040 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8ea221f700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009300 tx=0x7f8e98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e98009600 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e9c10d6c0 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e98009760 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e9801c5b0 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.050+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e9c10e230 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f8e9801c870 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8e92ffd700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 0x7f8e88038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9c102e20 con 0x7f8e9c0fca50 2024-03-14T12:52:31.052 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8e98049b40 con 0x7f8e9c0fca50 2024-03-14T12:52:31.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8ea1a1e700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 0x7f8e88038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:31.053 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.051+0000 7f8ea1a1e700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 0x7f8e88038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009990 tx=0x7f8e8c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:31.055 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.056+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8e98048070 con 0x7f8e9c0fca50 2024-03-14T12:52:31.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.310+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e9c04bed0 con 0x7f8e9c0fca50 2024-03-14T12:52:31.311 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.311+0000 7f8e92ffd700 1 -- 172.21.15.137:0/739678416 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8e98011dd0 con 0x7f8e9c0fca50 2024-03-14T12:52:31.311 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:31.311 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 msgr2=0x7f8e88038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 0x7f8e88038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009990 tx=0x7f8e8c006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 msgr2=0x7f8e9c10cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009300 tx=0x7f8e98009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 shutdown_connections 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8e88036880 0x7f8e88038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 --2- 172.21.15.137:0/739678416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e9c0fca50 0x7f8e9c10cf80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 >> 172.21.15.137:0/739678416 conn(0x7f8e9c0f81d0 msgr2=0x7f8e9c0f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 shutdown_connections 2024-03-14T12:52:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:31.313+0000 7f8ea4483700 1 -- 172.21.15.137:0/739678416 wait complete. 2024-03-14T12:52:31.314 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:32.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:32 smithi111 ceph-mon[39688]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:32.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:32 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/739678416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:32.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:32.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:33.267 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:33.331 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:34.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:34 smithi111 ceph-mon[39688]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:34.919 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 -- 172.21.15.137:0/3066298873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 msgr2=0x7fe4380fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 --2- 172.21.15.137:0/3066298873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe4380fce70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe4280048c0 tx=0x7fe428009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 -- 172.21.15.137:0/3066298873 shutdown_connections 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 --2- 172.21.15.137:0/3066298873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe4380fce70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 -- 172.21.15.137:0/3066298873 >> 172.21.15.137:0/3066298873 conn(0x7fe4380f81c0 msgr2=0x7fe4380fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 -- 172.21.15.137:0/3066298873 shutdown_connections 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 -- 172.21.15.137:0/3066298873 wait complete. 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.919+0000 7fe43e689700 1 Processor -- start 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe43e689700 1 -- start start 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe43e689700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:34.920 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe43e689700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43810d530 con 0x7fe4380fca60 2024-03-14T12:52:34.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe437fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:34.921 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe437fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:51348/0 (socket says 172.21.15.137:51348) 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe437fff700 1 -- 172.21.15.137:0/3463067152 learned_addr learned my addr 172.21.15.137:0/3463067152 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.920+0000 7fe437fff700 1 -- 172.21.15.137:0/3463067152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428005040 con 0x7fe4380fca60 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe437fff700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe428000c00 tx=0x7fe428009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe428009620 con 0x7fe4380fca60 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43810d730 con 0x7fe4380fca60 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe428009780 con 0x7fe4380fca60 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe42801c5b0 con 0x7fe4380fca60 2024-03-14T12:52:34.922 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.921+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43810e2a0 con 0x7fe4380fca60 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.922+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fe428013a60 con 0x7fe4380fca60 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.922+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe438102e30 con 0x7fe4380fca60 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.922+0000 7fe434ff9700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 0x7fe420038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.922+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe428049a80 con 0x7fe4380fca60 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.922+0000 7fe4377fe700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 0x7fe420038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:34.923 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.923+0000 7fe4377fe700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 0x7fe420038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009990 tx=0x7fe42c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:34.927 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:34.927+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fe428025400 con 0x7fe4380fca60 2024-03-14T12:52:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.180+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe43810ecb0 con 0x7fe4380fca60 2024-03-14T12:52:35.180 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.180+0000 7fe434ff9700 1 -- 172.21.15.137:0/3463067152 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fe428011b60 con 0x7fe4380fca60 2024-03-14T12:52:35.180 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:35.180 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:35.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 msgr2=0x7fe420038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:35.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 0x7fe420038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009990 tx=0x7fe42c006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:35.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 msgr2=0x7fe43810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:35.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe428000c00 tx=0x7fe428009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:35.182 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 shutdown_connections 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fe420036880 0x7fe420038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 --2- 172.21.15.137:0/3463067152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4380fca60 0x7fe43810cff0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 >> 172.21.15.137:0/3463067152 conn(0x7fe4380f81c0 msgr2=0x7fe4380f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 shutdown_connections 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:35.182+0000 7fe43e689700 1 -- 172.21.15.137:0/3463067152 wait complete. 2024-03-14T12:52:35.183 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:36 smithi111 ceph-mon[39688]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:36 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3463067152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:36.815 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:36.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:37.101 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:37.164 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:38 smithi111 ceph-mon[39688]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 -- 172.21.15.137:0/4016717219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 msgr2=0x7fbca80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/4016717219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80fd460 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbc980048c0 tx=0x7fbc98009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 -- 172.21.15.137:0/4016717219 shutdown_connections 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/4016717219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80fd460 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 -- 172.21.15.137:0/4016717219 >> 172.21.15.137:0/4016717219 conn(0x7fbca80f70d0 msgr2=0x7fbca80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 -- 172.21.15.137:0/4016717219 shutdown_connections 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.794+0000 7fbcac8ac700 1 -- 172.21.15.137:0/4016717219 wait complete. 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbcac8ac700 1 Processor -- start 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbcac8ac700 1 -- start start 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbcac8ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbcac8ac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80f95a0 con 0x7fbca80fd050 2024-03-14T12:52:38.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbca659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbca659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:51784/0 (socket says 172.21.15.137:51784) 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbca659c700 1 -- 172.21.15.137:0/251629895 learned_addr learned my addr 172.21.15.137:0/251629895 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.795+0000 7fbca659c700 1 -- 172.21.15.137:0/251629895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98005040 con 0x7fbca80fd050 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbca659c700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009300 tx=0x7fbc98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc98009600 con 0x7fbca80fd050 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca80f7670 con 0x7fbca80fd050 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc98009760 con 0x7fbca80fd050 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc9801c5b0 con 0x7fbca80fd050 2024-03-14T12:52:38.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.796+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca80f7b60 con 0x7fbca80fd050 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.797+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca805f5f0 con 0x7fbca80fd050 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.797+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fbc98013960 con 0x7fbca80fd050 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.797+0000 7fbc9effd700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 0x7fbc94038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.797+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbc98049a90 con 0x7fbca80fd050 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.797+0000 7fbca5d9b700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 0x7fbc94038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.798+0000 7fbca5d9b700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 0x7fbc94038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009990 tx=0x7fbc90006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:38.802 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:38.802+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fbc980098d0 con 0x7fbca80fd050 2024-03-14T12:52:39.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.059+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbca80f7e80 con 0x7fbca80fd050 2024-03-14T12:52:39.059 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.059+0000 7fbc9effd700 1 -- 172.21.15.137:0/251629895 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fbc98011370 con 0x7fbca80fd050 2024-03-14T12:52:39.059 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:39.059 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:39.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 msgr2=0x7fbc94038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:39.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 0x7fbc94038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009990 tx=0x7fbc90006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:39.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 msgr2=0x7fbca80f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:39.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009300 tx=0x7fbc98009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:39.061 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 shutdown_connections 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fbc94036880 0x7fbc94038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 --2- 172.21.15.137:0/251629895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbca80fd050 0x7fbca80f9060 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 >> 172.21.15.137:0/251629895 conn(0x7fbca80f70d0 msgr2=0x7fbca8065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 shutdown_connections 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:39.061+0000 7fbcac8ac700 1 -- 172.21.15.137:0/251629895 wait complete. 2024-03-14T12:52:39.062 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:39 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/251629895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:40 smithi111 ceph-mon[39688]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:40.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:40.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:40.992 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:41.057 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:42 smithi111 ceph-mon[39688]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 -- 172.21.15.137:0/1551759649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 msgr2=0x7efe2c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 --2- 172.21.15.137:0/1551759649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c0fd460 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efe140048c0 tx=0x7efe14009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 -- 172.21.15.137:0/1551759649 shutdown_connections 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 --2- 172.21.15.137:0/1551759649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c0fd460 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 -- 172.21.15.137:0/1551759649 >> 172.21.15.137:0/1551759649 conn(0x7efe2c0f70d0 msgr2=0x7efe2c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:42.683 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 -- 172.21.15.137:0/1551759649 shutdown_connections 2024-03-14T12:52:42.684 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.683+0000 7efe314c6700 1 -- 172.21.15.137:0/1551759649 wait complete. 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe314c6700 1 Processor -- start 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe314c6700 1 -- start start 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe314c6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe314c6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c10d4f0 con 0x7efe2c0fd050 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe2affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe2affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:51802/0 (socket says 172.21.15.137:51802) 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe2affd700 1 -- 172.21.15.137:0/2138557526 learned_addr learned my addr 172.21.15.137:0/2138557526 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.684+0000 7efe2affd700 1 -- 172.21.15.137:0/2138557526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe14005040 con 0x7efe2c0fd050 2024-03-14T12:52:42.685 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe2affd700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe14000c00 tx=0x7efe14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:42.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe14009620 con 0x7efe2c0fd050 2024-03-14T12:52:42.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c10d6f0 con 0x7efe2c0fd050 2024-03-14T12:52:42.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe14009780 con 0x7efe2c0fd050 2024-03-14T12:52:42.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe1401c5b0 con 0x7efe2c0fd050 2024-03-14T12:52:42.686 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.685+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c10e260 con 0x7efe2c0fd050 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7efe1401c7a0 con 0x7efe2c0fd050 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c101d30 con 0x7efe2c0fd050 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe237fe700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 0x7efe18038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efe14049b40 con 0x7efe2c0fd050 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe2a7fc700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 0x7efe18038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:42.687 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.686+0000 7efe2a7fc700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 0x7efe18038d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009990 tx=0x7efe1c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:42.691 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.691+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7efe14011dd0 con 0x7efe2c0fd050 2024-03-14T12:52:42.946 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.946+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe2c10e4a0 con 0x7efe2c0fd050 2024-03-14T12:52:42.947 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.947+0000 7efe237fe700 1 -- 172.21.15.137:0/2138557526 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7efe14011370 con 0x7efe2c0fd050 2024-03-14T12:52:42.947 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:42.947 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 msgr2=0x7efe18038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 0x7efe18038d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efe1c009990 tx=0x7efe1c006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 msgr2=0x7efe2c10cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efe14000c00 tx=0x7efe14009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 shutdown_connections 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7efe180368d0 0x7efe18038d80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 --2- 172.21.15.137:0/2138557526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0fd050 0x7efe2c10cfb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 >> 172.21.15.137:0/2138557526 conn(0x7efe2c0f70d0 msgr2=0x7efe2c065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 shutdown_connections 2024-03-14T12:52:42.949 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:42.949+0000 7efe314c6700 1 -- 172.21.15.137:0/2138557526 wait complete. 2024-03-14T12:52:42.950 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:43 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2138557526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:44.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:44.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:44 smithi111 ceph-mon[39688]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:44.888 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:44.957 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 -- 172.21.15.137:0/2603950380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 msgr2=0x7fb3500fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 --2- 172.21.15.137:0/2603950380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb3500fd460 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb3400048c0 tx=0x7fb340009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 -- 172.21.15.137:0/2603950380 shutdown_connections 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 --2- 172.21.15.137:0/2603950380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb3500fd460 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 -- 172.21.15.137:0/2603950380 >> 172.21.15.137:0/2603950380 conn(0x7fb3500f70d0 msgr2=0x7fb3500f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 -- 172.21.15.137:0/2603950380 shutdown_connections 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.611+0000 7fb354e52700 1 -- 172.21.15.137:0/2603950380 wait complete. 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb354e52700 1 Processor -- start 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb354e52700 1 -- start start 2024-03-14T12:52:46.613 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb354e52700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb354e52700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35010d510 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb34e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb34e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:51828/0 (socket says 172.21.15.137:51828) 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb34e59c700 1 -- 172.21.15.137:0/1177080172 learned_addr learned my addr 172.21.15.137:0/1177080172 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.612+0000 7fb34e59c700 1 -- 172.21.15.137:0/1177080172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340005040 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb34e59c700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb340009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb340009620 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb340009780 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35010d710 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34001c5b0 con 0x7fb3500fd050 2024-03-14T12:52:46.614 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.613+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35010e280 con 0x7fb3500fd050 2024-03-14T12:52:46.615 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fb340013420 con 0x7fb3500fd050 2024-03-14T12:52:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb350101d30 con 0x7fb3500fd050 2024-03-14T12:52:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb346ffd700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 0x7fb33c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb340049b20 con 0x7fb3500fd050 2024-03-14T12:52:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb34dd9b700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 0x7fb33c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:46.616 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.614+0000 7fb34dd9b700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 0x7fb33c038d80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb338009990 tx=0x7fb338006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:46.619 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.619+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb340025400 con 0x7fb3500fd050 2024-03-14T12:52:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:46 smithi111 ceph-mon[39688]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:46.879 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.878+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb35010e580 con 0x7fb3500fd050 2024-03-14T12:52:46.879 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.879+0000 7fb346ffd700 1 -- 172.21.15.137:0/1177080172 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb340025400 con 0x7fb3500fd050 2024-03-14T12:52:46.879 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:46.879 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 msgr2=0x7fb33c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 0x7fb33c038d80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb338009990 tx=0x7fb338006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 msgr2=0x7fb35010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb340000c00 tx=0x7fb340009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 shutdown_connections 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7fb33c0368d0 0x7fb33c038d80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:46.881 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 --2- 172.21.15.137:0/1177080172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3500fd050 0x7fb35010cfd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:46.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 >> 172.21.15.137:0/1177080172 conn(0x7fb3500f70d0 msgr2=0x7fb350065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:46.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 shutdown_connections 2024-03-14T12:52:46.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:46.881+0000 7fb354e52700 1 -- 172.21.15.137:0/1177080172 wait complete. 2024-03-14T12:52:46.882 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:47 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1177080172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:48.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:48.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:48.595 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:48 smithi111 ceph-mon[39688]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:48.692 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:48.753 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:50.324 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 -- 172.21.15.137:0/130759037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 msgr2=0x7f8a3c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:50.324 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 --2- 172.21.15.137:0/130759037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c0fd460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8a300048c0 tx=0x7f8a30009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:50.324 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 -- 172.21.15.137:0/130759037 shutdown_connections 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 --2- 172.21.15.137:0/130759037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c0fd460 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 -- 172.21.15.137:0/130759037 >> 172.21.15.137:0/130759037 conn(0x7f8a3c0f70d0 msgr2=0x7f8a3c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 -- 172.21.15.137:0/130759037 shutdown_connections 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 -- 172.21.15.137:0/130759037 wait complete. 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.324+0000 7f8a42e06700 1 Processor -- start 2024-03-14T12:52:50.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a42e06700 1 -- start start 2024-03-14T12:52:50.326 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a42e06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a42e06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c10d500 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a40ba2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a40ba2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:55702/0 (socket says 172.21.15.137:55702) 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a40ba2700 1 -- 172.21.15.137:0/3182247298 learned_addr learned my addr 172.21.15.137:0/3182247298 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.325+0000 7f8a40ba2700 1 -- 172.21.15.137:0/3182247298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a30005040 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a40ba2700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009300 tx=0x7f8a30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a30009600 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c10d700 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a30009760 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a3001c5b0 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.326+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c10e270 con 0x7f8a3c0fd050 2024-03-14T12:52:50.327 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.327+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f8a3001c820 con 0x7f8a3c0fd050 2024-03-14T12:52:50.328 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.327+0000 7f8a397fa700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 0x7f8a2c038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:50.328 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.327+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8a30049a50 con 0x7f8a3c0fd050 2024-03-14T12:52:50.328 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.327+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a3c101d30 con 0x7f8a3c0fd050 2024-03-14T12:52:50.328 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.327+0000 7f8a3bfff700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 0x7f8a2c038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:50.328 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.328+0000 7f8a3bfff700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 0x7f8a2c038d80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009990 tx=0x7f8a28006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:50.332 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.332+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8a30023300 con 0x7f8a3c0fd050 2024-03-14T12:52:50.449 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:50 smithi111 ceph-mon[39688]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:50.587 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.587+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a3c10e4b0 con 0x7f8a3c0fd050 2024-03-14T12:52:50.588 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.588+0000 7f8a397fa700 1 -- 172.21.15.137:0/3182247298 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8a30011b60 con 0x7f8a3c0fd050 2024-03-14T12:52:50.588 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:50.588 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 msgr2=0x7f8a2c038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 0x7f8a2c038d80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009990 tx=0x7f8a28006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 msgr2=0x7f8a3c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009300 tx=0x7f8a30009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 shutdown_connections 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f8a2c0368d0 0x7f8a2c038d80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 --2- 172.21.15.137:0/3182247298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a3c0fd050 0x7f8a3c10cfc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 >> 172.21.15.137:0/3182247298 conn(0x7f8a3c0f70d0 msgr2=0x7f8a3c065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 shutdown_connections 2024-03-14T12:52:50.590 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:50.590+0000 7f8a42e06700 1 -- 172.21.15.137:0/3182247298 wait complete. 2024-03-14T12:52:50.591 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:51.445 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:51 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3182247298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:52.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:52.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:52.517 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:52.579 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:52.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:52 smithi111 ceph-mon[39688]: Deploying daemon prometheus.smithi111 on smithi111 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 -- 172.21.15.137:0/1648809171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 msgr2=0x7f166c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 --2- 172.21.15.137:0/1648809171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c0fd460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f16680048c0 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 -- 172.21.15.137:0/1648809171 shutdown_connections 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 --2- 172.21.15.137:0/1648809171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c0fd460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 -- 172.21.15.137:0/1648809171 >> 172.21.15.137:0/1648809171 conn(0x7f166c0f70d0 msgr2=0x7f166c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 -- 172.21.15.137:0/1648809171 shutdown_connections 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.143+0000 7f1673754700 1 -- 172.21.15.137:0/1648809171 wait complete. 2024-03-14T12:52:54.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f1673754700 1 Processor -- start 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f1673754700 1 -- start start 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f1673754700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f1673754700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c10c460 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f16714f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f16714f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:55706/0 (socket says 172.21.15.137:55706) 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f16714f0700 1 -- 172.21.15.137:0/3392856863 learned_addr learned my addr 172.21.15.137:0/3392856863 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f16714f0700 1 -- 172.21.15.137:0/3392856863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668005040 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.144+0000 7f16714f0700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1668000c00 tx=0x7f1668009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.145+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1668009620 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.145+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1668009780 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.145+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166801c5b0 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.145+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c10c660 con 0x7f166c0fd050 2024-03-14T12:52:54.146 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.145+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c10d0f0 con 0x7f166c0fd050 2024-03-14T12:52:54.147 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1668013a60 con 0x7f166c0fd050 2024-03-14T12:52:54.147 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c101d30 con 0x7f166c0fd050 2024-03-14T12:52:54.147 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f165dffb700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 0x7f1658038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:54.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1668049a30 con 0x7f166c0fd050 2024-03-14T12:52:54.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f1670cef700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 0x7f1658038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:54.148 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.146+0000 7f1670cef700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 0x7f1658038d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1660009990 tx=0x7f1660006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:54.150 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.151+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f1668025400 con 0x7f166c0fd050 2024-03-14T12:52:54.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.404+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f166c05f5f0 con 0x7f166c0fd050 2024-03-14T12:52:54.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.405+0000 7f165dffb700 1 -- 172.21.15.137:0/3392856863 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f1668011b60 con 0x7f166c0fd050 2024-03-14T12:52:54.405 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:54.405 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 msgr2=0x7f1658038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 0x7f1658038d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1660009990 tx=0x7f1660006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 msgr2=0x7f166c10bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1668000c00 tx=0x7f1668009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 shutdown_connections 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f16580368d0 0x7f1658038d80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 --2- 172.21.15.137:0/3392856863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f166c0fd050 0x7f166c10bf20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 >> 172.21.15.137:0/3392856863 conn(0x7f166c0f70d0 msgr2=0x7f166c0f7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:54.407 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 shutdown_connections 2024-03-14T12:52:54.408 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:54.407+0000 7f1673754700 1 -- 172.21.15.137:0/3392856863 wait complete. 2024-03-14T12:52:54.408 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:54 smithi111 ceph-mon[39688]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:54 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:52:55.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:55 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3392856863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:56.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:56.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:52:56.305 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:56.370 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:52:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:56 smithi111 ceph-mon[39688]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:58.069 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 -- 172.21.15.137:0/968296671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 msgr2=0x7f4ae80fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/968296671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fb280 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4adc0048c0 tx=0x7f4adc009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 -- 172.21.15.137:0/968296671 shutdown_connections 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/968296671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fb280 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 -- 172.21.15.137:0/968296671 >> 172.21.15.137:0/968296671 conn(0x7f4ae80f6850 msgr2=0x7f4ae80f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 -- 172.21.15.137:0/968296671 shutdown_connections 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.066+0000 7f4aee7c8700 1 -- 172.21.15.137:0/968296671 wait complete. 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4aee7c8700 1 Processor -- start 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4aee7c8700 1 -- start start 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4aee7c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4aee7c8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae80fdde0 con 0x7f4ae80fae70 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4ae7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:58.070 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4ae7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:54604/0 (socket says 172.21.15.137:54604) 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.067+0000 7f4ae7fff700 1 -- 172.21.15.137:0/1058405735 learned_addr learned my addr 172.21.15.137:0/1058405735 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4ae7fff700 1 -- 172.21.15.137:0/1058405735 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4adc005040 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4ae7fff700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009300 tx=0x7f4adc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc009600 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4adc009760 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae80fbeb0 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4adc01c5b0 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.068+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae80fc3a0 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.069+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f4adc013420 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.069+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae805f5f0 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.069+0000 7f4ae4ff9700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 0x7f4ad0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.069+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4adc049b40 con 0x7f4ae80fae70 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.069+0000 7f4ae77fe700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 0x7f4ad0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:52:58.071 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.070+0000 7f4ae77fe700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 0x7f4ad0038d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009990 tx=0x7f4ad8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:52:58.074 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.075+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f4adc04be60 con 0x7f4ae80fae70 2024-03-14T12:52:58.332 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.332+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ae8102970 con 0x7f4ae80fae70 2024-03-14T12:52:58.333 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.333+0000 7f4ae4ff9700 1 -- 172.21.15.137:0/1058405735 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f4adc013960 con 0x7f4ae80fae70 2024-03-14T12:52:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:52:58.333 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:52:58.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 msgr2=0x7f4ad0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:58.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 0x7f4ad0038d30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8009990 tx=0x7f4ad8006e40 comp rx=0 tx=0).stop 2024-03-14T12:52:58.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 msgr2=0x7f4ae80fd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:52:58.335 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4adc009300 tx=0x7f4adc009e30 comp rx=0 tx=0).stop 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 shutdown_connections 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f4ad0036880 0x7f4ad0038d30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 --2- 172.21.15.137:0/1058405735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae80fae70 0x7f4ae80fd8a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.335+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 >> 172.21.15.137:0/1058405735 conn(0x7f4ae80f6850 msgr2=0x7f4ae80f7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.336+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 shutdown_connections 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:52:58.336+0000 7f4aee7c8700 1 -- 172.21.15.137:0/1058405735 wait complete. 2024-03-14T12:52:58.336 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:52:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:58 smithi111 ceph-mon[39688]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:52:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:52:59 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1058405735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:52:59.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:52:59.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:00.260 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:00.329 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:00.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:00 smithi111 ceph-mon[39688]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:02.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 -- 172.21.15.137:0/3759067608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 msgr2=0x7f61940fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:02.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3759067608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f61940fe4c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f617c0048c0 tx=0x7f617c009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:02.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 -- 172.21.15.137:0/3759067608 shutdown_connections 2024-03-14T12:53:02.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3759067608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f61940fe4c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:02.086 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 -- 172.21.15.137:0/3759067608 >> 172.21.15.137:0/3759067608 conn(0x7f61940f6850 msgr2=0x7f61940f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 -- 172.21.15.137:0/3759067608 shutdown_connections 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.084+0000 7f6198f9a700 1 -- 172.21.15.137:0/3759067608 wait complete. 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f6198f9a700 1 Processor -- start 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f6198f9a700 1 -- start start 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f6198f9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f6198f9a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619410d510 con 0x7f61940fe0b0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f619259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f619259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:54620/0 (socket says 172.21.15.137:54620) 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f619259c700 1 -- 172.21.15.137:0/3402778513 learned_addr learned my addr 172.21.15.137:0/3402778513 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.085+0000 7f619259c700 1 -- 172.21.15.137:0/3402778513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c005040 con 0x7f61940fe0b0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f619259c700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f617c009300 tx=0x7f617c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617c009600 con 0x7f61940fe0b0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619410d710 con 0x7f61940fe0b0 2024-03-14T12:53:02.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f617c009760 con 0x7f61940fe0b0 2024-03-14T12:53:02.088 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617c01c5b0 con 0x7f61940fe0b0 2024-03-14T12:53:02.088 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.086+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619410e280 con 0x7f61940fe0b0 2024-03-14T12:53:02.088 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f617c013960 con 0x7f61940fe0b0 2024-03-14T12:53:02.088 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6194102e40 con 0x7f61940fe0b0 2024-03-14T12:53:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f618affd700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 0x7f6180038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f617c049b50 con 0x7f61940fe0b0 2024-03-14T12:53:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f6191d9b700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 0x7f6180038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:02.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.087+0000 7f6191d9b700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 0x7f6180038d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6184009990 tx=0x7f6184006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:02.091 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.092+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f617c0098d0 con 0x7f61940fe0b0 2024-03-14T12:53:02.350 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.350+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f619410ec90 con 0x7f61940fe0b0 2024-03-14T12:53:02.350 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.350+0000 7f618affd700 1 -- 172.21.15.137:0/3402778513 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f617c011370 con 0x7f61940fe0b0 2024-03-14T12:53:02.350 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:02.351 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 msgr2=0x7f6180038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 0x7f6180038d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6184009990 tx=0x7f6184006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 msgr2=0x7f619410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f617c009300 tx=0x7f617c009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 shutdown_connections 2024-03-14T12:53:02.352 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f6180036880 0x7f6180038d30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 --2- 172.21.15.137:0/3402778513 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61940fe0b0 0x7f619410cfd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 >> 172.21.15.137:0/3402778513 conn(0x7f61940f6850 msgr2=0x7f61940f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 shutdown_connections 2024-03-14T12:53:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:02.352+0000 7f6198f9a700 1 -- 172.21.15.137:0/3402778513 wait complete. 2024-03-14T12:53:02.353 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:02 smithi111 ceph-mon[39688]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:03.462 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:03 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3402778513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:03.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:03.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:04.266 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:04.327 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:04.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:04 smithi111 ceph-mon[39688]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:05.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 -- 172.21.15.137:0/935660897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 msgr2=0x7ff99c0fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:05.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 --2- 172.21.15.137:0/935660897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fe4c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff9840048c0 tx=0x7ff984009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:05.882 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 -- 172.21.15.137:0/935660897 shutdown_connections 2024-03-14T12:53:05.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 --2- 172.21.15.137:0/935660897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fe4c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:05.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 -- 172.21.15.137:0/935660897 >> 172.21.15.137:0/935660897 conn(0x7ff99c0f6850 msgr2=0x7ff99c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:05.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.877+0000 7ff9a0948700 1 -- 172.21.15.137:0/935660897 shutdown_connections 2024-03-14T12:53:05.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff9a0948700 1 -- 172.21.15.137:0/935660897 wait complete. 2024-03-14T12:53:05.883 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff9a0948700 1 Processor -- start 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff9a0948700 1 -- start start 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff9a0948700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff9a0948700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c0fdd00 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff99a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:54648/0 (socket says 172.21.15.137:54648) 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.878+0000 7ff99a59c700 1 -- 172.21.15.137:0/3693907274 learned_addr learned my addr 172.21.15.137:0/3693907274 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:05.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff99a59c700 1 -- 172.21.15.137:0/3693907274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984005040 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff99a59c700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff984000c00 tx=0x7ff984009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff984009620 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff984009780 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99c0fbdd0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98401c5b0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.879+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99c0fc2c0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7ff98401c7f0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c05f5f0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.885 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff992ffd700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 0x7ff988038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:05.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff984049cc0 con 0x7ff99c0fe0b0 2024-03-14T12:53:05.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff999d9b700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 0x7ff988038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:05.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.880+0000 7ff999d9b700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 0x7ff988038d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009990 tx=0x7ff98c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:05.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:05.885+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7ff984011dd0 con 0x7ff99c0fe0b0 2024-03-14T12:53:06.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.140+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff99c106b90 con 0x7ff99c0fe0b0 2024-03-14T12:53:06.142 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.140+0000 7ff992ffd700 1 -- 172.21.15.137:0/3693907274 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7ff984011370 con 0x7ff99c0fe0b0 2024-03-14T12:53:06.142 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:06.142 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:06.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 msgr2=0x7ff988038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:06.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 0x7ff988038d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff98c009990 tx=0x7ff98c006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 msgr2=0x7ff99c0fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff984000c00 tx=0x7ff984009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 shutdown_connections 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7ff988036880 0x7ff988038d30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 --2- 172.21.15.137:0/3693907274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff99c0fe0b0 0x7ff99c0fd7c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 >> 172.21.15.137:0/3693907274 conn(0x7ff99c0f6850 msgr2=0x7ff99c0f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 shutdown_connections 2024-03-14T12:53:06.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:06.142+0000 7ff9a0948700 1 -- 172.21.15.137:0/3693907274 wait complete. 2024-03-14T12:53:06.145 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:06.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:53:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:53:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' 2024-03-14T12:53:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-14T12:53:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:06 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3693907274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:07.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:07.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:08.038 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:08.091 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:08.115 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:07 smithi111 ceph-mon[39688]: from='mgr.14160 172.21.15.111:0/80055039' entity='mgr.smithi111.atysez' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-14T12:53:08.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:07 smithi111 ceph-mon[39688]: mgrmap e14: smithi111.atysez(active, since 87s) 2024-03-14T12:53:09.729 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 -- 172.21.15.137:0/176916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 msgr2=0x7f14ec0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 --2- 172.21.15.137:0/176916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0fd460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0048c0 tx=0x7f14dc009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 -- 172.21.15.137:0/176916886 shutdown_connections 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 --2- 172.21.15.137:0/176916886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0fd460 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 -- 172.21.15.137:0/176916886 >> 172.21.15.137:0/176916886 conn(0x7f14ec0f70d0 msgr2=0x7f14ec0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 -- 172.21.15.137:0/176916886 shutdown_connections 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.728+0000 7f14f0952700 1 -- 172.21.15.137:0/176916886 wait complete. 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14f0952700 1 Processor -- start 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14f0952700 1 -- start start 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14f0952700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14f0952700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ec0f95a0 con 0x7f14ec0fd050 2024-03-14T12:53:09.730 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:42400/0 (socket says 172.21.15.137:42400) 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14ea59c700 1 -- 172.21.15.137:0/2022816845 learned_addr learned my addr 172.21.15.137:0/2022816845 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.729+0000 7f14ea59c700 1 -- 172.21.15.137:0/2022816845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc005040 con 0x7f14ec0fd050 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14ea59c700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009300 tx=0x7f14dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14dc009600 con 0x7f14ec0fd050 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14dc009760 con 0x7f14ec0fd050 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14dc01c5b0 con 0x7f14ec0fd050 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec0f7670 con 0x7f14ec0fd050 2024-03-14T12:53:09.731 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.730+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec0f7b60 con 0x7f14ec0fd050 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ec05f5f0 con 0x7f14ec0fd050 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f14dc013960 con 0x7f14ec0fd050 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14e2ffd700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 0x7f14d8038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f14dc049aa0 con 0x7f14ec0fd050 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14e3fff700 1 -- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 msgr2=0x7f14d8038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4041156471 2024-03-14T12:53:09.732 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.731+0000 7f14e3fff700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 0x7f14d8038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:53:09.735 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.736+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f14dc011550 con 0x7f14ec0fd050 2024-03-14T12:53:09.932 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.932+0000 7f14e3fff700 1 -- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 msgr2=0x7f14d8038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/4041156471 2024-03-14T12:53:09.932 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.932+0000 7f14e3fff700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 0x7f14d8038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-14T12:53:09.991 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.991+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14ec0f7e80 con 0x7f14ec0fd050 2024-03-14T12:53:09.992 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.992+0000 7f14e2ffd700 1 -- 172.21.15.137:0/2022816845 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f14dc011370 con 0x7f14ec0fd050 2024-03-14T12:53:09.992 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:09.992 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 msgr2=0x7f14d8038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 0x7f14d8038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 msgr2=0x7f14ec0f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009300 tx=0x7f14dc009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 shutdown_connections 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:6800/4041156471,v1:172.21.15.111:6801/4041156471] conn(0x7f14d8036920 0x7f14d8038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 --2- 172.21.15.137:0/2022816845 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14ec0fd050 0x7f14ec0f9060 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 >> 172.21.15.137:0/2022816845 conn(0x7f14ec0f70d0 msgr2=0x7f14ec065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 shutdown_connections 2024-03-14T12:53:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:09.994+0000 7f14f0952700 1 -- 172.21.15.137:0/2022816845 wait complete. 2024-03-14T12:53:09.995 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:10.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2022816845' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:11.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:11.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:11.822 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:11.883 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: Active manager daemon smithi111.atysez restarted 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: Activating manager daemon smithi111.atysez 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: osdmap e5: 0 total, 0 up, 0 in 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: mgrmap e15: smithi111.atysez(active, starting, since 0.00306982s) 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T12:53:13.326 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: Manager daemon smithi111.atysez is now available 2024-03-14T12:53:13.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:53:13.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:53:13.495 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 -- 172.21.15.137:0/2539512540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 msgr2=0x7f9a540fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:13.495 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 --2- 172.21.15.137:0/2539512540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a540fce50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9a440048c0 tx=0x7f9a44009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 -- 172.21.15.137:0/2539512540 shutdown_connections 2024-03-14T12:53:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 --2- 172.21.15.137:0/2539512540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a540fce50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 -- 172.21.15.137:0/2539512540 >> 172.21.15.137:0/2539512540 conn(0x7f9a540f81c0 msgr2=0x7f9a540fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 -- 172.21.15.137:0/2539512540 shutdown_connections 2024-03-14T12:53:13.496 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.495+0000 7f9a5a05c700 1 -- 172.21.15.137:0/2539512540 wait complete. 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.496+0000 7f9a5a05c700 1 Processor -- start 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.496+0000 7f9a5a05c700 1 -- start start 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.496+0000 7f9a5a05c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.496+0000 7f9a5a05c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5410d4f0 con 0x7f9a540fca40 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:42414/0 (socket says 172.21.15.137:42414) 2024-03-14T12:53:13.497 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a537fe700 1 -- 172.21.15.137:0/1537147229 learned_addr learned my addr 172.21.15.137:0/1537147229 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:13.498 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a537fe700 1 -- 172.21.15.137:0/1537147229 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a44005040 con 0x7f9a540fca40 2024-03-14T12:53:13.498 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a537fe700 1 --2- 172.21.15.137:0/1537147229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009300 tx=0x7f9a44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.497+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a44009600 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.498+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5410d6f0 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.498+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a44009760 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.498+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a4401c5b0 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.498+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5410e260 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.499+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 15) v1 ==== 43162+0+0 (secure 0 0 0) 0x7f9a4401c870 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.499+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a54102e10 con 0x7f9a540fca40 2024-03-14T12:53:13.499 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.499+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9a44049840 con 0x7f9a540fca40 2024-03-14T12:53:13.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.508+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f9a44023300 con 0x7f9a540fca40 2024-03-14T12:53:13.786 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.786+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a5410ec70 con 0x7f9a540fca40 2024-03-14T12:53:13.786 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.786+0000 7f9a4bfff700 1 -- 172.21.15.137:0/1537147229 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9a44011b60 con 0x7f9a540fca40 2024-03-14T12:53:13.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:13.787 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 msgr2=0x7f9a5410cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 --2- 172.21.15.137:0/1537147229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9a44009300 tx=0x7f9a44009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 shutdown_connections 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 --2- 172.21.15.137:0/1537147229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a540fca40 0x7f9a5410cfb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 >> 172.21.15.137:0/1537147229 conn(0x7f9a540f81c0 msgr2=0x7f9a540f8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 shutdown_connections 2024-03-14T12:53:13.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:13.788+0000 7f9a5a05c700 1 -- 172.21.15.137:0/1537147229 wait complete. 2024-03-14T12:53:13.789 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1537147229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:14 smithi111 ceph-mon[39688]: mgrmap e16: smithi111.atysez(active, since 1.00775s) 2024-03-14T12:53:15.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:15.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:15.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: [14/Mar/2024:12:53:14] ENGINE Bus STARTING 2024-03-14T12:53:15.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: [14/Mar/2024:12:53:14] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T12:53:15.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: [14/Mar/2024:12:53:14] ENGINE Bus STARTED 2024-03-14T12:53:15.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:15.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:16.393 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:16.833 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:17.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:17 smithi111 ceph-mon[39688]: mgrmap e17: smithi111.atysez(active, since 3s) 2024-03-14T12:53:19.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 -- 172.21.15.137:0/2139289770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 msgr2=0x7fb8a40fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:19.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/2139289770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fe4c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb8940048c0 tx=0x7fb894009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:19.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 -- 172.21.15.137:0/2139289770 shutdown_connections 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/2139289770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fe4c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 -- 172.21.15.137:0/2139289770 >> 172.21.15.137:0/2139289770 conn(0x7fb8a40f6850 msgr2=0x7fb8a40f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 -- 172.21.15.137:0/2139289770 shutdown_connections 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.085+0000 7fb8aa88a700 1 -- 172.21.15.137:0/2139289770 wait complete. 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8aa88a700 1 Processor -- start 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8aa88a700 1 -- start start 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8aa88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8aa88a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40fdd00 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:47026/0 (socket says 172.21.15.137:47026) 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.086+0000 7fb8a3fff700 1 -- 172.21.15.137:0/1278415690 learned_addr learned my addr 172.21.15.137:0/1278415690 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:19.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8a3fff700 1 -- 172.21.15.137:0/1278415690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894005040 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8a3fff700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb894009300 tx=0x7fb894009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb894009600 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a40fbdd0 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb894009760 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb89401c5b0 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.087+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a40fc2c0 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.088+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a405f5f0 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.088+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fb894013960 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.088+0000 7fb8a0ff9700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 0x7fb88c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.088+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb894049c90 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.088+0000 7fb8a37fe700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 0x7fb88c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.089+0000 7fb8a37fe700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 0x7fb88c038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb898009990 tx=0x7fb898006e40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:19.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.094+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb894011550 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.353+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8a4106b90 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.353+0000 7fb8a0ff9700 1 -- 172.21.15.137:0/1278415690 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb894011370 con 0x7fb8a40fe0b0 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.355+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 msgr2=0x7fb88c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.355+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 0x7fb88c038e20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb898009990 tx=0x7fb898006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.355+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 msgr2=0x7fb8a40fd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.355+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb894009300 tx=0x7fb894009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:19.405 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.355+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 shutdown_connections 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.356+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb88c036970 0x7fb88c038e20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.356+0000 7fb8aa88a700 1 --2- 172.21.15.137:0/1278415690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8a40fe0b0 0x7fb8a40fd7c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.356+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 >> 172.21.15.137:0/1278415690 conn(0x7fb8a40f6850 msgr2=0x7fb8a40f70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.356+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 shutdown_connections 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:19.356+0000 7fb8aa88a700 1 -- 172.21.15.137:0/1278415690 wait complete. 2024-03-14T12:53:19.406 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:53:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:20 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1278415690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:20.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:20.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:21.239 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:21.303 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.857+0000 7f85ae4e5700 1 -- 172.21.15.137:0/1497674638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 msgr2=0x7f85a80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.857+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/1497674638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80fd460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f85980048c0 tx=0x7f8598009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- 172.21.15.137:0/1497674638 shutdown_connections 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/1497674638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80fd460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- 172.21.15.137:0/1497674638 >> 172.21.15.137:0/1497674638 conn(0x7f85a80f70d0 msgr2=0x7f85a80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- 172.21.15.137:0/1497674638 shutdown_connections 2024-03-14T12:53:22.858 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- 172.21.15.137:0/1497674638 wait complete. 2024-03-14T12:53:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 Processor -- start 2024-03-14T12:53:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- start start 2024-03-14T12:53:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.858+0000 7f85ae4e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a80f95a0 con 0x7f85a80fd050 2024-03-14T12:53:22.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:22.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:47036/0 (socket says 172.21.15.137:47036) 2024-03-14T12:53:22.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a7fff700 1 -- 172.21.15.137:0/3936909913 learned_addr learned my addr 172.21.15.137:0/3936909913 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:22.860 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a7fff700 1 -- 172.21.15.137:0/3936909913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598005040 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a7fff700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8598009300 tx=0x7f8598009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8598009600 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a80f7670 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8598009760 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f859801c5b0 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.859+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a80f7b60 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.860+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f859801c870 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.860+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a8101d30 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.860+0000 7f85a4ff9700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 0x7f8590038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.860+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8598049d30 con 0x7f85a80fd050 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.860+0000 7f85a77fe700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 0x7f8590038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:22.861 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.861+0000 7f85a77fe700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 0x7f8590038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f859c009990 tx=0x7f859c006e40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:22.865 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:22.866+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8598023300 con 0x7f85a80fd050 2024-03-14T12:53:23.117 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.117+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85a8101d30 con 0x7f85a80fd050 2024-03-14T12:53:23.117 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.118+0000 7f85a4ff9700 1 -- 172.21.15.137:0/3936909913 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8598011b60 con 0x7f85a80fd050 2024-03-14T12:53:23.118 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:23.118 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:23.119 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 msgr2=0x7f8590038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 0x7f8590038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f859c009990 tx=0x7f859c006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 msgr2=0x7f85a80f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8598009300 tx=0x7f8598009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 shutdown_connections 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8590036970 0x7f8590038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 --2- 172.21.15.137:0/3936909913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a80fd050 0x7f85a80f9060 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 >> 172.21.15.137:0/3936909913 conn(0x7f85a80f70d0 msgr2=0x7f85a8065070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 shutdown_connections 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:23.120+0000 7f85ae4e5700 1 -- 172.21.15.137:0/3936909913 wait complete. 2024-03-14T12:53:23.120 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:23.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:23 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3936909913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:24.740 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:24.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:53:25.041 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T12:53:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T12:53:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-14T12:53:25.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:26 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T12:53:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:26 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T12:53:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:26 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T12:53:26.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:26 smithi111 ceph-mon[39688]: Deploying daemon crash.smithi137 on smithi137 2024-03-14T12:53:27.721 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:27.721 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.402+0000 7fa581619700 1 -- 172.21.15.137:0/3880576882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 msgr2=0x7fa57c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.402+0000 7fa581619700 1 --2- 172.21.15.137:0/3880576882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c101d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5700048c0 tx=0x7fa570009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.402+0000 7fa581619700 1 -- 172.21.15.137:0/3880576882 shutdown_connections 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.402+0000 7fa581619700 1 --2- 172.21.15.137:0/3880576882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c101d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.402+0000 7fa581619700 1 -- 172.21.15.137:0/3880576882 >> 172.21.15.137:0/3880576882 conn(0x7fa57c0f74e0 msgr2=0x7fa57c0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 -- 172.21.15.137:0/3880576882 shutdown_connections 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 -- 172.21.15.137:0/3880576882 wait complete. 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 Processor -- start 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 -- start start 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:27.722 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.403+0000 7fa581619700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c066860 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa57affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa57affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:37232/0 (socket says 172.21.15.137:37232) 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa57affd700 1 -- 172.21.15.137:0/1963978801 learned_addr learned my addr 172.21.15.137:0/1963978801 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa57affd700 1 -- 172.21.15.137:0/1963978801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570005040 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa57affd700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa570007e80 tx=0x7fa570009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5700093c0 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c10e7b0 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa570013420 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa570025560 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.404+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c10ec20 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.405+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57c10eec0 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.405+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fa570009520 con 0x7fa57c1019c0 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.405+0000 7fa5637fe700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 0x7fa564038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:27.723 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.405+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa570049a70 con 0x7fa57c1019c0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.406+0000 7fa57a7fc700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 0x7fa564038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.406+0000 7fa57a7fc700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 0x7fa564038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009990 tx=0x7fa56c006e40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.411+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fa57c10eec0 con 0x7fa57c1019c0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.688+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa57c10eec0 con 0x7fa57c1019c0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.689+0000 7fa5637fe700 1 -- 172.21.15.137:0/1963978801 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa57c10eec0 con 0x7fa57c1019c0 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 msgr2=0x7fa564038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 0x7fa564038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009990 tx=0x7fa56c006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 msgr2=0x7fa57c10e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa570007e80 tx=0x7fa570009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 shutdown_connections 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa564036970 0x7fa564038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 --2- 172.21.15.137:0/1963978801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa57c1019c0 0x7fa57c10e210 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 >> 172.21.15.137:0/1963978801 conn(0x7fa57c0f74e0 msgr2=0x7fa57c0f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 shutdown_connections 2024-03-14T12:53:27.724 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:27.692+0000 7fa581619700 1 -- 172.21.15.137:0/1963978801 wait complete. 2024-03-14T12:53:27.725 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:28.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:27 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1963978801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:29.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:29.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:29.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:29.751 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:30.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:30.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:30.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:30.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:30.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:30 smithi111 ceph-mon[39688]: Deploying daemon node-exporter.smithi137 on smithi137 2024-03-14T12:53:32.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1629662317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 msgr2=0x7fd8dc101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:32.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1629662317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc101da0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd8d80048c0 tx=0x7fd8d8009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1629662317 shutdown_connections 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1629662317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc101da0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1629662317 >> 172.21.15.137:0/1629662317 conn(0x7fd8dc071800 msgr2=0x7fd8dc071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1629662317 shutdown_connections 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.820+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1629662317 wait complete. 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e3b50700 1 Processor -- start 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e3b50700 1 -- start start 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e3b50700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e3b50700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0fdae0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e18ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e18ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:37264/0 (socket says 172.21.15.137:37264) 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e18ec700 1 -- 172.21.15.137:0/1132614809 learned_addr learned my addr 172.21.15.137:0/1132614809 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:32.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.821+0000 7fd8e18ec700 1 -- 172.21.15.137:0/1132614809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d8005040 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8e18ec700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8007e80 tx=0x7fd8d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d80093c0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc0fdce0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8d8013420 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d801c5b0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.822+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc0fe1d0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fd8d8013a00 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc0fd0d0 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8d27fc700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 0x7fd8c8038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd8d8049a90 con 0x7fd8dc1019d0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8e10eb700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 0x7fd8c8038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.823+0000 7fd8e10eb700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 0x7fd8c8038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009990 tx=0x7fd8cc006e40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:32.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:32.828+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd8d8011960 con 0x7fd8dc1019d0 2024-03-14T12:53:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.090+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8dc04bed0 con 0x7fd8dc1019d0 2024-03-14T12:53:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.091+0000 7fd8d27fc700 1 -- 172.21.15.137:0/1132614809 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd8d8011370 con 0x7fd8dc1019d0 2024-03-14T12:53:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:33.091 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 msgr2=0x7fd8c8038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 0x7fd8c8038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009990 tx=0x7fd8cc006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 msgr2=0x7fd8dc0fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8007e80 tx=0x7fd8d8009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 shutdown_connections 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd8c8036970 0x7fd8c8038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 --2- 172.21.15.137:0/1132614809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8dc1019d0 0x7fd8dc0fd5a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 >> 172.21.15.137:0/1132614809 conn(0x7fd8dc071800 msgr2=0x7fd8dc065110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 shutdown_connections 2024-03-14T12:53:33.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:33.093+0000 7fd8e3b50700 1 -- 172.21.15.137:0/1132614809 wait complete. 2024-03-14T12:53:33.094 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:33 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1132614809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:34.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:34.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:34 smithi111 ceph-mon[39688]: Deploying daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T12:53:34.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:34.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:35.184 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:36 smithi111 ceph-mon[39688]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:37.245 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 -- 172.21.15.137:0/917900128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 msgr2=0x7f79a80fbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 --2- 172.21.15.137:0/917900128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a80fbc10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f799c0048c0 tx=0x7f799c009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 -- 172.21.15.137:0/917900128 shutdown_connections 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 --2- 172.21.15.137:0/917900128 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a80fbc10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 -- 172.21.15.137:0/917900128 >> 172.21.15.137:0/917900128 conn(0x7f79a8071800 msgr2=0x7f79a8071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 -- 172.21.15.137:0/917900128 shutdown_connections 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.200+0000 7f79aef32700 1 -- 172.21.15.137:0/917900128 wait complete. 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.201+0000 7f79aef32700 1 Processor -- start 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.201+0000 7f79aef32700 1 -- start start 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.201+0000 7f79aef32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.201+0000 7f79aef32700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8115760 con 0x7f79a80fb840 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.202+0000 7f79accce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:37.246 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.202+0000 7f79accce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.137:37300/0 (socket says 172.21.15.137:37300) 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.202+0000 7f79accce700 1 -- 172.21.15.137:0/2490768244 learned_addr learned my addr 172.21.15.137:0/2490768244 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.202+0000 7f79accce700 1 -- 172.21.15.137:0/2490768244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c005040 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.202+0000 7f79accce700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f799c007e80 tx=0x7f799c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.203+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f799c0093c0 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.203+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a8115960 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.203+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f799c013420 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.203+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f799c01c5b0 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.203+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a8115e50 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f799c01c830 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79a57fa700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 0x7f7998038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f799c049bd0 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a80658f0 con 0x7f79a80fb840 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79a7fff700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 0x7f7998038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.204+0000 7f79a7fff700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 0x7f7998038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7994009990 tx=0x7f7994006e40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:37.247 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.209+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f799c023300 con 0x7f79a80fb840 2024-03-14T12:53:37.527 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.527+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79a804bf60 con 0x7f79a80fb840 2024-03-14T12:53:37.527 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.527+0000 7f79a57fa700 1 -- 172.21.15.137:0/2490768244 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f799c034090 con 0x7f79a80fb840 2024-03-14T12:53:37.527 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:37.527 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":1,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:50:37.255618Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-14T12:53:37.529 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 msgr2=0x7f7998038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:37.529 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 0x7f7998038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7994009990 tx=0x7f7994006e40 comp rx=0 tx=0).stop 2024-03-14T12:53:37.529 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 msgr2=0x7f79a8115220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f799c007e80 tx=0x7f799c009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 shutdown_connections 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7998036970 0x7f7998038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 --2- 172.21.15.137:0/2490768244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79a80fb840 0x7f79a8115220 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.529+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 >> 172.21.15.137:0/2490768244 conn(0x7f79a8071800 msgr2=0x7f79a80f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.530+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 shutdown_connections 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:37.530+0000 7f79aef32700 1 -- 172.21.15.137:0/2490768244 wait complete. 2024-03-14T12:53:37.530 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 1 2024-03-14T12:53:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T12:53:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: Deploying daemon mon.smithi137 on smithi137 2024-03-14T12:53:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:38 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2490768244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:39.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-14T12:53:39.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-14T12:53:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:40.387 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:53:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:40 smithi111 ceph-mon[39688]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:41.375 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:41 smithi137 bash[50276]: dba54c072a9f585b3b40e94276915b2efb3f5a16f5172510b1305188f5b0a4d8 2024-03-14T12:53:41.375 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:41 smithi137 systemd[1]: Started Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 -- 172.21.15.137:0/1554681672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 msgr2=0x7fd924005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/1554681672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd924005a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd9340fba10 tx=0x7fd92800b040 comp rx=0 tx=0).stop 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 -- 172.21.15.137:0/1554681672 shutdown_connections 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/1554681672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd924005a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 -- 172.21.15.137:0/1554681672 >> 172.21.15.137:0/1554681672 conn(0x7fd9340f7620 msgr2=0x7fd9340f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 -- 172.21.15.137:0/1554681672 shutdown_connections 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.271+0000 7fd93ab3b700 1 -- 172.21.15.137:0/1554681672 wait complete. 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.272+0000 7fd93ab3b700 1 Processor -- start 2024-03-14T12:53:46.272 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.272+0000 7fd93ab3b700 1 -- start start 2024-03-14T12:53:46.274 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.272+0000 7fd93ab3b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd93410e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd93ab3b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd93ab3b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93410ecd0 con 0x7fd924003580 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd93ab3b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93410ee40 con 0x7fd93410e860 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd9388d7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd93410e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:53602/0 (socket says 172.21.15.137:53602) 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 -- 172.21.15.137:0/2991521427 learned_addr learned my addr 172.21.15.137:0/2991521427 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 -- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 msgr2=0x7fd9341171d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 -- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 msgr2=0x7fd9341171d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd933fff700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd9388d7700 1 -- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 msgr2=0x7fd9341171d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T12:53:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd9388d7700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd9388d7700 1 -- 172.21.15.137:0/2991521427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd928005040 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.273+0000 7fd9388d7700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd93410e320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd928000ea0 tx=0x7fd92800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd928009360 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd928009970 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd934117710 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd928022440 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd934117b90 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.274+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fd928028020 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.275+0000 7fd9317fa700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 0x7fd918038fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.275+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd92804b7c0 con 0x7fd924003580 2024-03-14T12:53:46.276 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.275+0000 7fd933fff700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 0x7fd918038fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:46.277 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.275+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9340fd1e0 con 0x7fd924003580 2024-03-14T12:53:46.277 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.275+0000 7fd933fff700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 0x7fd918038fa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd920006fd0 tx=0x7fd920008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:46.280 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.280+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd92801b070 con 0x7fd924003580 2024-03-14T12:53:46.325 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.324+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fd928021790 con 0x7fd924003580 2024-03-14T12:53:46.574 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.573+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd93405f5f0 con 0x7fd924003580 2024-03-14T12:53:46.574 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.574+0000 7fd9317fa700 1 -- 172.21.15.137:0/2991521427 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7fd92801b020 con 0x7fd924003580 2024-03-14T12:53:46.574 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-03-14T12:53:46.574 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":2,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","modified":"2024-03-14T12:53:41.212396Z","created":"2024-03-14T12:50:37.255618Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 msgr2=0x7fd918038fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 0x7fd918038fa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd920006fd0 tx=0x7fd920008040 comp rx=0 tx=0).stop 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 msgr2=0x7fd93410e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd93410e320 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd928000ea0 tx=0x7fd92800b040 comp rx=0 tx=0).stop 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 shutdown_connections 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd918036af0 0x7fd918038fa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd924003580 0x7fd93410e320 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 --2- 172.21.15.137:0/2991521427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93410e860 0x7fd9341171d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 >> 172.21.15.137:0/2991521427 conn(0x7fd9340f7620 msgr2=0x7fd934104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 shutdown_connections 2024-03-14T12:53:46.576 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:53:46.576+0000 7fd93ab3b700 1 -- 172.21.15.137:0/2991521427 wait complete. 2024-03-14T12:53:46.577 INFO:teuthology.orchestra.run.smithi137.stderr:dumped monmap epoch 2 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: mon.smithi111 calling monitor election 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: mon.smithi137 calling monitor election 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:46.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: fsmap 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: osdmap e5: 0 total, 0 up, 0 in 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: mgrmap e17: smithi111.atysez(active, since 33s) 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: overall HEALTH_OK 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:46.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T12:53:46.604 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:53:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: mon.smithi111 calling monitor election 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: mon.smithi137 calling monitor election 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T12:53:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: fsmap 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: osdmap e5: 0 total, 0 up, 0 in 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: mgrmap e17: smithi111.atysez(active, since 33s) 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: overall HEALTH_OK 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/535757876' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T12:53:46.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.282 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-14T12:53:47.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-14T12:53:47.560 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: mgrmap e18: smithi111.atysez(active, since 33s), standbys: smithi137.xbyzbp 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:53:47.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2991521427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:47.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:47 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: mgrmap e18: smithi111.atysez(active, since 33s), standbys: smithi137.xbyzbp 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:53:47.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/2991521427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-14T12:53:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:47 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:53:48.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:48 smithi111 ceph-mon[39688]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:48.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:48 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:48.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:48 smithi137 ceph-mon[50336]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:48.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:48 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:49.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.443+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/2857772861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa88102bd0 msgr2=0x7ffa88102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.443+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/2857772861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa88102bd0 0x7ffa88102fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa840048c0 tx=0x7ffa84009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.443+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/2857772861 shutdown_connections 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.443+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/2857772861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa88102bd0 0x7ffa88102fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.443+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/2857772861 >> 172.21.15.111:0/2857772861 conn(0x7ffa880f85e0 msgr2=0x7ffa880fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/2857772861 shutdown_connections 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/2857772861 wait complete. 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 Processor -- start 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 -- start start 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa88102bd0 0x7ffa881132b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa88113e00 con 0x7ffa881137f0 2024-03-14T12:53:49.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8f7c7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8810d750 con 0x7ffa88102bd0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8cd62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8cd62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47428/0 (socket says 172.21.15.111:47428) 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.444+0000 7ffa8cd62700 1 -- 172.21.15.111:0/1804044505 learned_addr learned my addr 172.21.15.111:0/1804044505 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8d563700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa88102bd0 0x7ffa881132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8cd62700 1 -- 172.21.15.111:0/1804044505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa88102bd0 msgr2=0x7ffa881132b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8cd62700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa88102bd0 0x7ffa881132b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8cd62700 1 -- 172.21.15.111:0/1804044505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84005040 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8cd62700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b790 tx=0x7ffa78009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa78010870 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa78010e80 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa7800d3b0 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8810da30 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.445+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8810dff0 con 0x7ffa881137f0 2024-03-14T12:53:49.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.446+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa88101e40 con 0x7ffa881137f0 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.447+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7ffa7800f3e0 con 0x7ffa881137f0 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.447+0000 7ffa7dffb700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 0x7ffa7406b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.447+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ffa780865d0 con 0x7ffa881137f0 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.447+0000 7ffa8d563700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 0x7ffa7406b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.447+0000 7ffa8d563700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 0x7ffa7406b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa840043d0 tx=0x7ffa84009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:53:49.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.452+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7ffa78057470 con 0x7ffa881137f0 2024-03-14T12:53:49.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.663+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ffa8805f5f0 con 0x7ffa881137f0 2024-03-14T12:53:49.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.663+0000 7ffa7dffb700 1 -- 172.21.15.111:0/1804044505 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7ffa78014100 con 0x7ffa881137f0 2024-03-14T12:53:49.664 INFO:teuthology.orchestra.run.smithi111.stdout:# minimal ceph.conf for 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:53:49.664 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-03-14T12:53:49.665 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T12:53:49.665 INFO:teuthology.orchestra.run.smithi111.stdout: mon_host = [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] 2024-03-14T12:53:49.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 msgr2=0x7ffa7406b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 0x7ffa7406b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ffa840043d0 tx=0x7ffa84009e30 comp rx=0 tx=0).stop 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 msgr2=0x7ffa8810d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800b790 tx=0x7ffa78009250 comp rx=0 tx=0).stop 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 shutdown_connections 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa88102bd0 0x7ffa881132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7ffa74069370 0x7ffa7406b820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 --2- 172.21.15.111:0/1804044505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa881137f0 0x7ffa8810d210 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 >> 172.21.15.111:0/1804044505 conn(0x7ffa880f85e0 msgr2=0x7ffa880fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 shutdown_connections 2024-03-14T12:53:49.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:53:49.666+0000 7ffa8f7c7700 1 -- 172.21.15.111:0/1804044505 wait complete. 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T12:53:50.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Reconfiguring mon.smithi111 (unknown last config time)... 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-14T12:53:50.111 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:49 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1804044505' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:50.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Reconfiguring mon.smithi111 (unknown last config time)... 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-14T12:53:50.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:49 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1804044505' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:50.401 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-14T12:53:50.402 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:53:50.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-03-14T12:53:50.433 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:53:50.433 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T12:53:50.500 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:53:50.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-03-14T12:53:50.536 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:53:50.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-14T12:53:50.607 INFO:tasks.cephadm:Deploying OSDs... 2024-03-14T12:53:50.607 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:53:50.607 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-03-14T12:53:50.625 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-14T12:53:50.625 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-03-14T12:53:50.683 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 167182 Links: 1 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-14 12:53:04.929476080 +0000 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-14 12:44:37.069987213 +0000 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-14 12:44:37.069987213 +0000 2024-03-14T12:53:50.684 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-14T12:53:50.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-14T12:53:50.750 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-14T12:53:50.750 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-14T12:53:50.750 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000228891 s, 2.2 MB/s 2024-03-14T12:53:50.751 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-14T12:53:50.810 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-03-14T12:53:50.868 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-14T12:53:50.868 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:50.868 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 166318 Links: 1 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-14 12:53:04.929476080 +0000 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-14 12:44:37.426981252 +0000 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-14 12:44:37.426981252 +0000 2024-03-14T12:53:50.869 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-14T12:53:50.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-14T12:53:50.934 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-14T12:53:50.934 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-14T12:53:50.934 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000214348 s, 2.4 MB/s 2024-03-14T12:53:50.935 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-14T12:53:50.994 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-03-14T12:53:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 163477 Links: 1 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-14 12:53:04.929476080 +0000 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-14 12:44:37.783975292 +0000 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-14 12:44:37.783975292 +0000 2024-03-14T12:53:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-14T12:53:51.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-14T12:53:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-14T12:53:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-14T12:53:51.118 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000181002 s, 2.8 MB/s 2024-03-14T12:53:51.119 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-14T12:53:51.178 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-03-14T12:53:51.236 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-14T12:53:51.236 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:51.236 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 167348 Links: 1 2024-03-14T12:53:51.236 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:51.236 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:51.237 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-03-14 12:53:04.930476062 +0000 2024-03-14T12:53:51.237 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-14 12:44:38.131969481 +0000 2024-03-14T12:53:51.237 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-14 12:44:38.131969481 +0000 2024-03-14T12:53:51.237 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-14T12:53:51.237 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-14T12:53:51.302 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-03-14T12:53:51.302 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-03-14T12:53:51.302 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000235246 s, 2.2 MB/s 2024-03-14T12:53:51.303 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-14T12:53:51.362 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:53:51.362 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-03-14T12:53:51.384 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-14T12:53:51.384 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-03-14T12:53:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-14T12:53:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 165614 Links: 1 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-14 12:53:40.521318122 +0000 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-14 12:44:37.070033897 +0000 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-14 12:44:37.070033897 +0000 2024-03-14T12:53:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-03-14T12:53:51.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-14T12:53:51.515 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-14T12:53:51.516 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-14T12:53:51.516 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000275385 s, 1.9 MB/s 2024-03-14T12:53:51.517 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-14T12:53:51.577 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 165685 Links: 1 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-14 12:53:40.521318122 +0000 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-14 12:44:37.401027776 +0000 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-14 12:44:37.401027776 +0000 2024-03-14T12:53:51.637 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-03-14T12:53:51.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-14T12:53:51.704 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-14T12:53:51.704 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-14T12:53:51.704 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000267876 s, 1.9 MB/s 2024-03-14T12:53:51.705 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-14T12:53:51.765 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-03-14T12:53:51.824 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-14T12:53:51.824 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:51.824 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 168195 Links: 1 2024-03-14T12:53:51.824 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:51.825 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:51.825 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-14 12:53:40.521318122 +0000 2024-03-14T12:53:51.825 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-14 12:44:37.744021435 +0000 2024-03-14T12:53:51.825 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-14 12:44:37.744021435 +0000 2024-03-14T12:53:51.825 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-03-14T12:53:51.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-14T12:53:51.890 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-14T12:53:51.890 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-14T12:53:51.891 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000161794 s, 3.2 MB/s 2024-03-14T12:53:51.892 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-14T12:53:51.950 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-03-14T12:53:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-14T12:53:52.008 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-14T12:53:52.008 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 6h/6d Inode: 170085 Links: 1 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-03-14 12:53:40.521318122 +0000 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-14 12:44:38.105014760 +0000 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-14 12:44:38.105014760 +0000 2024-03-14T12:53:52.009 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-03-14T12:53:52.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-14T12:53:52.075 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-03-14T12:53:52.075 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-03-14T12:53:52.075 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000301331 s, 1.7 MB/s 2024-03-14T12:53:52.076 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-14T12:53:52.135 INFO:tasks.cephadm:Deploying osd.0 on smithi111 with /dev/vg_nvme/lv_4... 2024-03-14T12:53:52.135 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-14T12:53:52.831 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:53:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: Reconfiguring mgr.smithi111.atysez (unknown last config time)... 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:52.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:52 smithi111 ceph-mon[39688]: Reconfiguring daemon mgr.smithi111.atysez on smithi111 2024-03-14T12:53:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: Reconfiguring mgr.smithi111.atysez (unknown last config time)... 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:52 smithi137 ceph-mon[50336]: Reconfiguring daemon mgr.smithi111.atysez on smithi111 2024-03-14T12:53:54.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:54.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:54.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:54.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-14T12:53:54.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:53:54.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:54.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:54 smithi111 ceph-mon[39688]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-14T12:53:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:53:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:54 smithi137 ceph-mon[50336]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-14T12:53:56.235 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:56.235 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:56.235 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:56 smithi111 ceph-mon[39688]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-03-14T12:53:56.235 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:56 smithi111 ceph-mon[39688]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:56.235 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:56 smithi111 ceph-mon[39688]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-03-14T12:53:56.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:56.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:53:56.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:56 smithi137 ceph-mon[50336]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-03-14T12:53:56.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:56 smithi137 ceph-mon[50336]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:56.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:56 smithi137 ceph-mon[50336]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-03-14T12:53:56.953 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-14T12:53:56.954 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-14T12:53:56.954 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-14T12:53:56.954 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-14T12:53:56.954 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122596 s, 855 MB/s 2024-03-14T12:53:56.954 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-14T12:53:58.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:58 smithi137 ceph-mon[50336]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:53:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:58 smithi111 ceph-mon[39688]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:53:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:53:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:53:59.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-03-14T12:54:00.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:00 smithi111 ceph-mon[39688]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:00.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:00 smithi137 ceph-mon[50336]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:00.559 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:54:02.503 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:02 smithi111 ceph-mon[39688]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:02.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:02 smithi137 ceph-mon[50336]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 -- 172.21.15.111:0/2202184020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 msgr2=0x7f1ccc0fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/2202184020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc0fe4d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc004940 tx=0x7f1cbc009e30 comp rx=0 tx=0).stop 2024-03-14T12:54:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 -- 172.21.15.111:0/2202184020 shutdown_connections 2024-03-14T12:54:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/2202184020 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc0fe4d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:02.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/2202184020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 0x7f1ccc0fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 -- 172.21.15.111:0/2202184020 >> 172.21.15.111:0/2202184020 conn(0x7f1ccc0f85b0 msgr2=0x7f1ccc0fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 -- 172.21.15.111:0/2202184020 shutdown_connections 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 -- 172.21.15.111:0/2202184020 wait complete. 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.941+0000 7f1cd37f0700 1 Processor -- start 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 -- start start 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 0x7f1ccc071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc070950 con 0x7f1ccc0fe060 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc070ac0 con 0x7f1ccc0fcef0 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41210/0 (socket says 172.21.15.111:41210) 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 -- 172.21.15.111:0/3550209132 learned_addr learned my addr 172.21.15.111:0/3550209132 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd158c700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 0x7f1ccc071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 -- 172.21.15.111:0/3550209132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 msgr2=0x7f1ccc071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:02.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 0x7f1ccc071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 -- 172.21.15.111:0/3550209132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc005040 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd0d8b700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009fd0 tx=0x7f1cbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cbc00e040 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cbc004290 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc070d40 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.942+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cbc01c3f0 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.943+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc1159a0 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.943+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc05f5f0 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.944+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f1cbc01c660 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.944+0000 7f1cc1ffb700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 0x7f1cb806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.944+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1cbc087500 con 0x7f1ccc0fe060 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.944+0000 7f1cd158c700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 0x7f1cb806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:02.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.945+0000 7f1cd158c700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 0x7f1cb806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8006fd0 tx=0x7f1cc8008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:54:02.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:02.949+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f1cbc0549b0 con 0x7f1ccc0fe060 2024-03-14T12:54:03.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:03.158+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ccc102420 con 0x7f1cb8069210 2024-03-14T12:54:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:54:04.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:54:04.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:04.924 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:04 smithi111 ceph-mon[39688]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:54:04.925 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:04 smithi111 ceph-mon[39688]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:05.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:04 smithi137 ceph-mon[50336]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:54:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:04 smithi137 ceph-mon[50336]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:06.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:06 smithi111 ceph-mon[39688]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:06.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:06 smithi137 ceph-mon[50336]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:08.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:08 smithi111 ceph-mon[39688]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:08.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:08 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:08.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:08 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:08 smithi137 ceph-mon[50336]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:08 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:08 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:09 smithi137 ceph-mon[50336]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T12:54:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:09 smithi137 ceph-mon[50336]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T12:54:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:09 smithi111 ceph-mon[39688]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T12:54:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:09 smithi111 ceph-mon[39688]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1516007031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "726491ca-c4bb-4580-8256-3f3d0890e6d4"}]: dispatch 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1516007031' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "726491ca-c4bb-4580-8256-3f3d0890e6d4"}]': finished 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: osdmap e6: 1 total, 0 up, 1 in 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/304343884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1516007031' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "726491ca-c4bb-4580-8256-3f3d0890e6d4"}]: dispatch 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1516007031' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "726491ca-c4bb-4580-8256-3f3d0890e6d4"}]': finished 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: osdmap e6: 1 total, 0 up, 1 in 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:10 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/304343884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:54:12.602 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:12 smithi111 ceph-mon[39688]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:12 smithi137 ceph-mon[50336]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:13.598 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:13.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:14 smithi111 ceph-mon[39688]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:14 smithi137 ceph-mon[50336]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:16.364 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:16 smithi111 ceph-mon[39688]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:16.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:16 smithi137 ceph-mon[50336]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:18.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:18 smithi111 ceph-mon[39688]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:18 smithi137 ceph-mon[50336]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:20.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:20.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:20.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:20.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: Reconfiguring crash.smithi137 (monmap changed)... 2024-03-14T12:54:20.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:54:20.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:20.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:20 smithi111 ceph-mon[39688]: Reconfiguring daemon crash.smithi137 on smithi137 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: Reconfiguring crash.smithi137 (monmap changed)... 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:21.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:20 smithi137 ceph-mon[50336]: Reconfiguring daemon crash.smithi137 on smithi137 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T12:54:22.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T12:54:22.085 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T12:54:22.086 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:23.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: Deploying daemon osd.0 on smithi111 2024-03-14T12:54:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: Reconfiguring mgr.smithi137.xbyzbp (monmap changed)... 2024-03-14T12:54:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: Reconfiguring daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T12:54:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:23.050 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:23.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: Deploying daemon osd.0 on smithi111 2024-03-14T12:54:23.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: Reconfiguring mgr.smithi137.xbyzbp (monmap changed)... 2024-03-14T12:54:23.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: Reconfiguring daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T12:54:23.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:23.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:24.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:23 smithi111 ceph-mon[39688]: Reconfiguring mon.smithi137 (monmap changed)... 2024-03-14T12:54:24.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:23 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T12:54:24.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:23 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T12:54:24.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:23 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:24.053 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:23 smithi111 ceph-mon[39688]: Reconfiguring daemon mon.smithi137 on smithi137 2024-03-14T12:54:24.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:23 smithi137 ceph-mon[50336]: Reconfiguring mon.smithi137 (monmap changed)... 2024-03-14T12:54:24.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:23 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T12:54:24.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:23 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T12:54:24.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:23 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:24.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:23 smithi137 ceph-mon[50336]: Reconfiguring daemon mon.smithi137 on smithi137 2024-03-14T12:54:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-03-14T12:54:25.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:25.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T12:54:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:25 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-03-14T12:54:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T12:54:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi111.front.sepia.ceph.com:9093"}]: dispatch 2024-03-14T12:54:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T12:54:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-03-14T12:54:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T12:54:26.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:25 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-03-14T12:54:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:26 smithi111 ceph-mon[39688]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:26 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:27.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:26 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:27.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:26 smithi137 ceph-mon[50336]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:27.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:26 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:26 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:28.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:27 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:27 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:27 smithi137 ceph-mon[50336]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:28.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:27 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:28.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:27 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:28.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:27 smithi111 ceph-mon[39688]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:29.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:29.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:30.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:29 smithi111 ceph-mon[39688]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:30.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:29 smithi137 ceph-mon[50336]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:31.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.138+0000 7f1cc1ffb700 1 -- 172.21.15.111:0/3550209132 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1ccc102420 con 0x7f1cb8069210 2024-03-14T12:54:31.139 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 0 on host 'smithi111' 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 msgr2=0x7f1cb806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 0x7f1cb806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8006fd0 tx=0x7f1cc8008040 comp rx=0 tx=0).stop 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 msgr2=0x7f1ccc070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009fd0 tx=0x7f1cbc009e30 comp rx=0 tx=0).stop 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 shutdown_connections 2024-03-14T12:54:31.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ccc0fcef0 0x7f1ccc071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:31.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1cb8069210 0x7f1cb806b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:31.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 --2- 172.21.15.111:0/3550209132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ccc0fe060 0x7f1ccc070310 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:31.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.140+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 >> 172.21.15.111:0/3550209132 conn(0x7f1ccc0f85b0 msgr2=0x7f1ccc0f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:54:31.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.141+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 shutdown_connections 2024-03-14T12:54:31.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:31.141+0000 7f1cd37f0700 1 -- 172.21.15.111:0/3550209132 wait complete. 2024-03-14T12:54:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:31 smithi111 ceph-mon[39688]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T12:54:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:31 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:31 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:31.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:31 smithi137 ceph-mon[50336]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T12:54:31.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:31 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:31.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:31 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: osdmap e7: 1 total, 0 up, 1 in 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:32.489 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:32 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:32.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:32.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T12:54:32.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: osdmap e7: 1 total, 0 up, 1 in 2024-03-14T12:54:32.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:32.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:54:32.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:32.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:32 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:32.510 DEBUG:teuthology.orchestra.run.smithi111:osd.0> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.0.service 2024-03-14T12:54:32.512 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_3... 2024-03-14T12:54:32.513 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-14T12:54:32.542 INFO:journalctl@ceph.osd.0.smithi111.stdout:-- Logs begin at Thu 2024-03-14 12:38:31 UTC. -- 2024-03-14T12:54:32.819 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:54:33.350 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:33 smithi111 ceph-mon[39688]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:54:33.350 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:33 smithi111 ceph-mon[39688]: osdmap e8: 1 total, 0 up, 1 in 2024-03-14T12:54:33.350 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:33 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:33.350 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:33 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:33.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:33 smithi137 ceph-mon[50336]: from='osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:54:33.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:33 smithi137 ceph-mon[50336]: osdmap e8: 1 total, 0 up, 1 in 2024-03-14T12:54:33.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:33 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:33.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:33 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] boot 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: osdmap e9: 1 total, 1 up, 1 in 2024-03-14T12:54:34.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:34.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: osd.0 [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] boot 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: osdmap e9: 1 total, 1 up, 1 in 2024-03-14T12:54:34.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:34 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T12:54:35.501 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:35 smithi111 ceph-mon[39688]: osdmap e10: 1 total, 1 up, 1 in 2024-03-14T12:54:35.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:35 smithi137 ceph-mon[50336]: osdmap e10: 1 total, 1 up, 1 in 2024-03-14T12:54:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-14T12:54:35.640 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-14T12:54:35.641 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-14T12:54:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-14T12:54:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111369 s, 942 MB/s 2024-03-14T12:54:35.641 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 2024-03-14T12:54:35.641 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-14T12:54:36.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:36 smithi137 ceph-mon[50336]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:36.583 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:36 smithi111 ceph-mon[39688]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:36.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-03-14T12:54:36.888 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:54:38.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:54:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:38.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:54:38.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:38.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:38.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:54:38.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 -- 172.21.15.111:0/2730608301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 msgr2=0x7f49880fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2730608301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f49880fd280 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f497c004940 tx=0x7f497c009e30 comp rx=0 tx=0).stop 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 -- 172.21.15.111:0/2730608301 shutdown_connections 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2730608301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 0x7f49880fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2730608301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f49880fd280 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 -- 172.21.15.111:0/2730608301 >> 172.21.15.111:0/2730608301 conn(0x7f49880f8550 msgr2=0x7f49880fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 -- 172.21.15.111:0/2730608301 shutdown_connections 2024-03-14T12:54:39.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 -- 172.21.15.111:0/2730608301 wait complete. 2024-03-14T12:54:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.138+0000 7f498d6a4700 1 Processor -- start 2024-03-14T12:54:39.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 -- start start 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 0x7f498810dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498810e3a0 con 0x7f49880fce70 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498810e510 con 0x7f49880fdfe0 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46522/0 (socket says 172.21.15.111:46522) 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 -- 172.21.15.111:0/2143775818 learned_addr learned my addr 172.21.15.111:0/2143775818 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f49867fc700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 0x7f498810dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 -- 172.21.15.111:0/2143775818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 msgr2=0x7f498810dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 0x7f498810dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 -- 172.21.15.111:0/2143775818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c005040 con 0x7f49880fce70 2024-03-14T12:54:39.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f4986ffd700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f497c00b3b0 tx=0x7f497c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c00e040 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f497c00bc50 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4988112f10 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f497c01c630 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.139+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4988113440 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.140+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498804c710 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.141+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f497c013420 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.141+0000 7f496f7fe700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 0x7f497006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.141+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f497c086fe0 con 0x7f49880fce70 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.141+0000 7f49867fc700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 0x7f497006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:54:39.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.142+0000 7f49867fc700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 0x7f497006b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4978004f60 tx=0x7f497800b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:54:39.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.146+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f497c057c20 con 0x7f49880fce70 2024-03-14T12:54:39.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:39.336+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4988113660 con 0x7f4970069210 2024-03-14T12:54:39.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:39 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi111 2024-03-14T12:54:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:39 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi111 to 12898M 2024-03-14T12:54:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:39 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:39 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi111 2024-03-14T12:54:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:39 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi111 to 12898M 2024-03-14T12:54:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:40.367 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:40.367 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:54:40.367 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:54:40.368 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:54:40.368 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:40.368 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:40.368 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:40 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:40.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:40 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:42.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:42 smithi137 ceph-mon[50336]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:42 smithi111 ceph-mon[39688]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:43.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:43 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2959141847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c5fd1b-e903-43f7-b441-c4427d6ffb78"}]: dispatch 2024-03-14T12:54:43.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:43 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2959141847' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c5fd1b-e903-43f7-b441-c4427d6ffb78"}]': finished 2024-03-14T12:54:43.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:43 smithi137 ceph-mon[50336]: osdmap e11: 2 total, 1 up, 2 in 2024-03-14T12:54:43.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:54:43.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:43 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2959141847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83c5fd1b-e903-43f7-b441-c4427d6ffb78"}]: dispatch 2024-03-14T12:54:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:43 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2959141847' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83c5fd1b-e903-43f7-b441-c4427d6ffb78"}]': finished 2024-03-14T12:54:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:43 smithi111 ceph-mon[39688]: osdmap e11: 2 total, 1 up, 2 in 2024-03-14T12:54:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:54:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:44.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:44 smithi111 ceph-mon[39688]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:44.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:44 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3993156460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:54:44.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:44 smithi137 ceph-mon[50336]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:44.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:44 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3993156460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:54:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:45 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:45.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:45 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:45.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:45 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:45.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:45 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:54:45.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:45 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:45 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:45 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:45 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:45 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:54:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:45 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:46.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:46 smithi111 ceph-mon[39688]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:46 smithi137 ceph-mon[50336]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:48.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:48 smithi111 ceph-mon[39688]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:48 smithi137 ceph-mon[50336]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:50 smithi111 ceph-mon[39688]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:50 smithi137 ceph-mon[50336]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T12:54:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:51 smithi111 ceph-mon[39688]: Deploying daemon osd.1 on smithi111 2024-03-14T12:54:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T12:54:52.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:54:52.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:51 smithi137 ceph-mon[50336]: Deploying daemon osd.1 on smithi111 2024-03-14T12:54:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:52 smithi111 ceph-mon[39688]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:52 smithi137 ceph-mon[50336]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:54 smithi111 ceph-mon[39688]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:54 smithi137 ceph-mon[50336]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:56.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:56 smithi111 ceph-mon[39688]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:56.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:56.834 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:56.834 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:56.901 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:56 smithi137 ceph-mon[50336]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:56.901 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:56.901 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:56.901 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:54:58.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:58 smithi111 ceph-mon[39688]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:58.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:58.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:58.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:54:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:58 smithi137 ceph-mon[50336]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:54:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:54:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:54:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:54:59.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.951+0000 7f496f7fe700 1 -- 172.21.15.111:0/2143775818 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4988113660 con 0x7f4970069210 2024-03-14T12:54:59.951 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-03-14T12:54:59.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 msgr2=0x7f497006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:59.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 0x7f497006b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4978004f60 tx=0x7f497800b040 comp rx=0 tx=0).stop 2024-03-14T12:54:59.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 msgr2=0x7f498810d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:54:59.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f497c00b3b0 tx=0x7f497c009e30 comp rx=0 tx=0).stop 2024-03-14T12:54:59.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 shutdown_connections 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4970069210 0x7f497006b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49880fce70 0x7f498810d860 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 --2- 172.21.15.111:0/2143775818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49880fdfe0 0x7f498810dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 >> 172.21.15.111:0/2143775818 conn(0x7f49880f8550 msgr2=0x7f49880f9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 shutdown_connections 2024-03-14T12:54:59.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:54:59.954+0000 7f498d6a4700 1 -- 172.21.15.111:0/2143775818 wait complete. 2024-03-14T12:55:00.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:00 smithi111 ceph-mon[39688]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:55:00.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:00.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:00.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:00 smithi111 ceph-mon[39688]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T12:55:01.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:00 smithi137 ceph-mon[50336]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:55:01.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:01.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:01.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:00 smithi137 ceph-mon[50336]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T12:55:01.199 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.1.service 2024-03-14T12:55:01.202 INFO:tasks.cephadm:Deploying osd.2 on smithi111 with /dev/vg_nvme/lv_2... 2024-03-14T12:55:01.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-14T12:55:01.232 INFO:journalctl@ceph.osd.1.smithi111.stdout:-- Logs begin at Thu 2024-03-14 12:38:31 UTC. -- 2024-03-14T12:55:01.727 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:55:02.223 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[72179]: 2024-03-14T12:55:01.927+0000 7f67e1daa700 -1 osd.1 0 waiting for initial osdmap 2024-03-14T12:55:02.223 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[72179]: 2024-03-14T12:55:01.932+0000 7f67dcba5700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T12:55:02.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-mon[39688]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T12:55:02.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-mon[39688]: osdmap e12: 2 total, 1 up, 2 in 2024-03-14T12:55:02.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:02.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-mon[39688]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:55:02.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:01 smithi111 ceph-mon[39688]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:55:02.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:01 smithi137 ceph-mon[50336]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T12:55:02.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:01 smithi137 ceph-mon[50336]: osdmap e12: 2 total, 1 up, 2 in 2024-03-14T12:55:02.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:01 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:02.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:01 smithi137 ceph-mon[50336]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:55:02.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:01 smithi137 ceph-mon[50336]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-14T12:55:03.187 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:02 smithi111 ceph-mon[39688]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:55:03.187 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:02 smithi111 ceph-mon[39688]: osdmap e13: 2 total, 1 up, 2 in 2024-03-14T12:55:03.187 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:02 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:03.188 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:02 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:03.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:02 smithi137 ceph-mon[50336]: from='osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:55:03.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:02 smithi137 ceph-mon[50336]: osdmap e13: 2 total, 1 up, 2 in 2024-03-14T12:55:03.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:02 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:03.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:02 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:04.059 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:55:04.060 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:55:04.060 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] boot 2024-03-14T12:55:04.060 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: osdmap e14: 2 total, 2 up, 2 in 2024-03-14T12:55:04.060 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:04.060 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:03 smithi111 ceph-mon[39688]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:04.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:55:04.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:55:04.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: osd.1 [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] boot 2024-03-14T12:55:04.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: osdmap e14: 2 total, 2 up, 2 in 2024-03-14T12:55:04.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T12:55:04.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:03 smithi137 ceph-mon[50336]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126932 s, 826 MB/s 2024-03-14T12:55:04.549 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-03-14T12:55:05.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:04 smithi137 ceph-mon[50336]: osdmap e15: 2 total, 2 up, 2 in 2024-03-14T12:55:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:04 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:04 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:05.262 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:04 smithi111 ceph-mon[39688]: osdmap e15: 2 total, 2 up, 2 in 2024-03-14T12:55:05.262 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:04 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:05.262 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:04 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:05.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-03-14T12:55:06.163 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:55:06.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:06 smithi137 ceph-mon[50336]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:06.539 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:06 smithi111 ceph-mon[39688]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:08.490 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:08 smithi111 ceph-mon[39688]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:08.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:08 smithi137 ceph-mon[50336]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 -- 172.21.15.111:0/2790872418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 msgr2=0x7f72340fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 --2- 172.21.15.111:0/2790872418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f72340fd3e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7228004940 tx=0x7f7228009e30 comp rx=0 tx=0).stop 2024-03-14T12:55:09.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 -- 172.21.15.111:0/2790872418 shutdown_connections 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 --2- 172.21.15.111:0/2790872418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f72340fd3e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 --2- 172.21.15.111:0/2790872418 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 0x7f72340fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 -- 172.21.15.111:0/2790872418 >> 172.21.15.111:0/2790872418 conn(0x7f72340f74e0 msgr2=0x7f72340f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 -- 172.21.15.111:0/2790872418 shutdown_connections 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.010+0000 7f7239d60700 1 -- 172.21.15.111:0/2790872418 wait complete. 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 Processor -- start 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 -- start start 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 0x7f7234110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234066c70 con 0x7f72340fcf70 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7239d60700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234066db0 con 0x7f72340fbe00 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7232ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7232ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53734/0 (socket says 172.21.15.111:53734) 2024-03-14T12:55:09.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.011+0000 7f7232ffd700 1 -- 172.21.15.111:0/111815683 learned_addr learned my addr 172.21.15.111:0/111815683 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f72337fe700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 0x7f7234110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7232ffd700 1 -- 172.21.15.111:0/111815683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 msgr2=0x7f7234110630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7232ffd700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 0x7f7234110630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7232ffd700 1 -- 172.21.15.111:0/111815683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228005040 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7232ffd700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7228009fd0 tx=0x7f7228009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f722800e040 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7228004290 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7234113180 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f722801c3f0 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.012+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7234113710 con 0x7f72340fcf70 2024-03-14T12:55:09.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.013+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723405f5f0 con 0x7f72340fcf70 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.013+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f722800bc50 con 0x7f72340fcf70 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.013+0000 7f721bfff700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 0x7f721c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.014+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f7228087980 con 0x7f72340fcf70 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.014+0000 7f72337fe700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 0x7f721c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.014+0000 7f72337fe700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 0x7f721c06b6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7224006fd0 tx=0x7f7224008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:55:09.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.018+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f7228054980 con 0x7f72340fcf70 2024-03-14T12:55:09.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:09.212+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f72341012a0 con 0x7f721c069210 2024-03-14T12:55:09.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:09 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:55:09.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:09 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:55:09.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:09 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:09.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:09 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:55:09.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:09 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:55:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:09 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:55:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi111 2024-03-14T12:55:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi111 to 6449M 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:10 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:10.761 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:10.761 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:55:10.761 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi111 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi111 to 6449M 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:10.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:10 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:12 smithi137 ceph-mon[50336]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:12 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1517370397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0fe3aa1-1501-4a28-b5ab-2b680418a5ab"}]: dispatch 2024-03-14T12:55:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:12 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1517370397' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0fe3aa1-1501-4a28-b5ab-2b680418a5ab"}]': finished 2024-03-14T12:55:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:12 smithi137 ceph-mon[50336]: osdmap e16: 3 total, 2 up, 3 in 2024-03-14T12:55:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:12 smithi111 ceph-mon[39688]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:12 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1517370397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0fe3aa1-1501-4a28-b5ab-2b680418a5ab"}]: dispatch 2024-03-14T12:55:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:12 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1517370397' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0fe3aa1-1501-4a28-b5ab-2b680418a5ab"}]': finished 2024-03-14T12:55:12.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:12 smithi111 ceph-mon[39688]: osdmap e16: 3 total, 2 up, 3 in 2024-03-14T12:55:12.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:13.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:13 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1076751883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:55:13.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:13 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1076751883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:55:13.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:14 smithi111 ceph-mon[39688]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:14 smithi137 ceph-mon[50336]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:16.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:16.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:16.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:16.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:16.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:16.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:16 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:16 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:18.801 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:18 smithi111 ceph-mon[39688]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:19.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:18 smithi137 ceph-mon[50336]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:20.774 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:20 smithi111 ceph-mon[39688]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:20.774 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T12:55:20.774 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:20 smithi137 ceph-mon[50336]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T12:55:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:21 smithi111 ceph-mon[39688]: Deploying daemon osd.2 on smithi111 2024-03-14T12:55:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:21 smithi137 ceph-mon[50336]: Deploying daemon osd.2 on smithi111 2024-03-14T12:55:22.767 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:22 smithi111 ceph-mon[39688]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:23.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:22 smithi137 ceph-mon[50336]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:24.767 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:24 smithi111 ceph-mon[39688]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:24 smithi137 ceph-mon[50336]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:26 smithi111 ceph-mon[39688]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:26 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:26.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:26 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:26.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:26 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:26 smithi137 ceph-mon[50336]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:26 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:26 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:26 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:29.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:28 smithi137 ceph-mon[50336]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:29.117 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:28 smithi111 ceph-mon[39688]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:29.117 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:29.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.818+0000 7f721bfff700 1 -- 172.21.15.111:0/111815683 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f72341012a0 con 0x7f721c069210 2024-03-14T12:55:29.819 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 2 on host 'smithi111' 2024-03-14T12:55:29.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 msgr2=0x7f721c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:29.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 0x7f721c06b6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7224006fd0 tx=0x7f7224008040 comp rx=0 tx=0).stop 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 msgr2=0x7f7234112b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7228009fd0 tx=0x7f7228009e30 comp rx=0 tx=0).stop 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 shutdown_connections 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72340fbe00 0x7f7234110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f721c069210 0x7f721c06b6c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 --2- 172.21.15.111:0/111815683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72340fcf70 0x7f7234112b80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 >> 172.21.15.111:0/111815683 conn(0x7f72340f74e0 msgr2=0x7f72340f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 shutdown_connections 2024-03-14T12:55:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:55:29.821+0000 7f7239d60700 1 -- 172.21.15.111:0/111815683 wait complete. 2024-03-14T12:55:30.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:29 smithi111 ceph-mon[39688]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T12:55:30.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:29 smithi137 ceph-mon[50336]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T12:55:30.755 DEBUG:teuthology.orchestra.run.smithi111:osd.2> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.2.service 2024-03-14T12:55:30.758 INFO:tasks.cephadm:Deploying osd.3 on smithi137 with /dev/vg_nvme/lv_4... 2024-03-14T12:55:30.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-14T12:55:30.789 INFO:journalctl@ceph.osd.2.smithi111.stdout:-- Logs begin at Thu 2024-03-14 12:38:31 UTC. -- 2024-03-14T12:55:31.027 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: osdmap e17: 3 total, 2 up, 3 in 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:55:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:31.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: osdmap e17: 3 total, 2 up, 3 in 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:31.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:30 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:32.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:31 smithi111 ceph-mon[39688]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:55:32.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:31 smithi111 ceph-mon[39688]: osdmap e18: 3 total, 2 up, 3 in 2024-03-14T12:55:32.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:31 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:32.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:31 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:32.168 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:31 smithi137 ceph-mon[50336]: from='osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-03-14T12:55:32.168 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:31 smithi137 ceph-mon[50336]: osdmap e18: 3 total, 2 up, 3 in 2024-03-14T12:55:32.168 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:31 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:32.168 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:31 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-14T12:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-14T12:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-03-14T12:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-03-14T12:55:33.090 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150559 s, 696 MB/s 2024-03-14T12:55:33.091 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] boot 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: osdmap e19: 3 total, 3 up, 3 in 2024-03-14T12:55:33.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:32 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: osd.2 [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] boot 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: osdmap e19: 3 total, 3 up, 3 in 2024-03-14T12:55:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:32 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T12:55:33.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-03-14T12:55:33.956 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:55:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:33 smithi137 ceph-mon[50336]: osdmap e20: 3 total, 3 up, 3 in 2024-03-14T12:55:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:33 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:33 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:33 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-14T12:55:34.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:33 smithi111 ceph-mon[39688]: osdmap e20: 3 total, 3 up, 3 in 2024-03-14T12:55:34.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:33 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:34.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:33 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:34.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:33 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-14T12:55:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:34 smithi111 ceph-mon[39688]: pgmap v78: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-14T12:55:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:34 smithi111 ceph-mon[39688]: osdmap e21: 3 total, 3 up, 3 in 2024-03-14T12:55:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:34 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T12:55:35.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:34 smithi137 ceph-mon[50336]: pgmap v78: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:35.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:34 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-14T12:55:35.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:34 smithi137 ceph-mon[50336]: osdmap e21: 3 total, 3 up, 3 in 2024-03-14T12:55:35.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:34 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 -- 172.21.15.137:0/3718595807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 msgr2=0x7f6c500fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3718595807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c500fd250 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6c40004a90 tx=0x7f6c40009dd0 comp rx=0 tx=0).stop 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 -- 172.21.15.137:0/3718595807 shutdown_connections 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3718595807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 0x7f6c500fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3718595807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c500fd250 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:35.633 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 -- 172.21.15.137:0/3718595807 >> 172.21.15.137:0/3718595807 conn(0x7f6c500f8570 msgr2=0x7f6c500fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:55:35.634 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 -- 172.21.15.137:0/3718595807 shutdown_connections 2024-03-14T12:55:35.634 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.633+0000 7f6c5844a700 1 -- 172.21.15.137:0/3718595807 wait complete. 2024-03-14T12:55:35.635 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 Processor -- start 2024-03-14T12:55:35.635 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 -- start start 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 0x7f6c5010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5010e480 con 0x7f6c500fe040 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5010ecd0 con 0x7f6c500fce40 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:55496/0 (socket says 172.21.15.137:55496) 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 -- 172.21.15.137:0/3279754632 learned_addr learned my addr 172.21.15.137:0/3279754632 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c559e5700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 0x7f6c5010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 -- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 msgr2=0x7f6c5010de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 0x7f6c5010de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 -- 172.21.15.137:0/3279754632 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c40004650 con 0x7f6c500fce40 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c561e6700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6c400099e0 tx=0x7f6c40009dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c40019070 con 0x7f6c500fce40 2024-03-14T12:55:35.636 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c4000bc20 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5010ef50 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c4000e040 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.634+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5010f430 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.635+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5005f5f0 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.636+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f6c40014430 con 0x7f6c500fce40 2024-03-14T12:55:35.637 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.636+0000 7f6c46ffd700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 0x7f6c3c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:55:35.638 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.636+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f6c40087500 con 0x7f6c500fce40 2024-03-14T12:55:35.638 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.636+0000 7f6c559e5700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 0x7f6c3c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:55:35.638 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.637+0000 7f6c559e5700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 0x7f6c3c06b710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009b80 tx=0x7f6c4c00b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:55:35.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.641+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f6c40057bc0 con 0x7f6c500fce40 2024-03-14T12:55:35.829 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:35.829+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c5010f6c0 con 0x7f6c3c069260 2024-03-14T12:55:36.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: osdmap e22: 3 total, 3 up, 3 in 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:55:36.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:35 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:36.353 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-14T12:55:36.353 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: osdmap e22: 3 total, 3 up, 3 in 2024-03-14T12:55:36.353 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:36.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: from='client.24133 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:55:36.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:55:36.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:55:36.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:35 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[83953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[83953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[83953]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[83953]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[86415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[86415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:36.354 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[86415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:36.622 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[86415]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:36.873 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T12:55:36.873 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:36.873 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87314]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:36.873 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87314]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:37.136 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87323]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T12:55:37.136 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:37.136 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87323]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:37.137 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 12:55:36 smithi111 sudo[87323]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:37.137 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87326]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T12:55:37.137 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:37.137 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87326]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:37.137 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87326]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:37.390 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87399]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T12:55:37.390 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87399]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:37.390 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87399]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:37.390 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87399]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: osdmap e23: 3 total, 3 up, 3 in 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: osdmap e23: 3 total, 3 up, 3 in 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:37 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T12:55:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87470]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:38.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87531]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T12:55:38.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87531]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:38.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87531]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:38.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:37 smithi111 sudo[87531]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54338]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54338]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54338]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54338]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54359]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54359]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54359]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 sudo[54359]: pam_unix(sudo:session): session closed for user root 2024-03-14T12:55:38.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi111 2024-03-14T12:55:38.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:38.400+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6c40054500 con 0x7f6c500fce40 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi111 to 4299M 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:55:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi111 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi111 to 4299M 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:55:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:55:38.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T12:55:38.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: mgrmap e19: smithi111.atysez(active, since 2m), standbys: smithi137.xbyzbp 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/2339413158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]: dispatch 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]: dispatch 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]': finished 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: osdmap e24: 4 total, 3 up, 4 in 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:39 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/3826083645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: mgrmap e19: smithi111.atysez(active, since 2m), standbys: smithi137.xbyzbp 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/2339413158' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]: dispatch 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]: dispatch 2024-03-14T12:55:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6c3f447-d2db-47da-9099-57ca0eff4075"}]': finished 2024-03-14T12:55:39.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: osdmap e24: 4 total, 3 up, 4 in 2024-03-14T12:55:39.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:39.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:39 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3826083645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:55:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:40 smithi111 ceph-mon[39688]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:40 smithi137 ceph-mon[50336]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:43.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:42 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:55:43.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:42 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:44.189 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:44 smithi137 ceph-mon[50336]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.9 KiB/s rd, 170 KiB/s wr, 12 op/s 2024-03-14T12:55:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:44 smithi111 ceph-mon[39688]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 3.9 KiB/s rd, 170 KiB/s wr, 12 op/s 2024-03-14T12:55:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:46 smithi137 ceph-mon[50336]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 119 KiB/s wr, 8 op/s 2024-03-14T12:55:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T12:55:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:46 smithi137 ceph-mon[50336]: Deploying daemon osd.3 on smithi137 2024-03-14T12:55:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:46 smithi111 ceph-mon[39688]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 119 KiB/s wr, 8 op/s 2024-03-14T12:55:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T12:55:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:55:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:46 smithi111 ceph-mon[39688]: Deploying daemon osd.3 on smithi137 2024-03-14T12:55:48.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:48 smithi137 ceph-mon[50336]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2024-03-14T12:55:48.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:48 smithi111 ceph-mon[39688]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 2.1 KiB/s rd, 91 KiB/s wr, 6 op/s 2024-03-14T12:55:50.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:50 smithi137 ceph-mon[50336]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-03-14T12:55:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:50 smithi111 ceph-mon[39688]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-03-14T12:55:52.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:52.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:52.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:52.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:51 smithi137 ceph-mon[50336]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-14T12:55:52.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:52.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:52.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:55:52.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:51 smithi111 ceph-mon[39688]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-14T12:55:54.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.092+0000 7f6c46ffd700 1 -- 172.21.15.137:0/3279754632 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6c5010f6c0 con 0x7f6c3c069260 2024-03-14T12:55:54.093 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 3 on host 'smithi137' 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.094+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 msgr2=0x7f6c3c06b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.094+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 0x7f6c3c06b710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009b80 tx=0x7f6c4c00b040 comp rx=0 tx=0).stop 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.094+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 msgr2=0x7f6c5010d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.094+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6c400099e0 tx=0x7f6c40009dd0 comp rx=0 tx=0).stop 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 shutdown_connections 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c500fce40 0x7f6c5010d940 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6c3c069260 0x7f6c3c06b710 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 --2- 172.21.15.137:0/3279754632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c500fe040 0x7f6c5010de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 >> 172.21.15.137:0/3279754632 conn(0x7f6c500f8570 msgr2=0x7f6c500f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:55:54.095 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 shutdown_connections 2024-03-14T12:55:54.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:55:54.095+0000 7f6c5844a700 1 -- 172.21.15.137:0/3279754632 wait complete. 2024-03-14T12:55:54.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:54 smithi137 ceph-mon[50336]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-14T12:55:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:54 smithi111 ceph-mon[39688]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-14T12:55:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:55.063 DEBUG:teuthology.orchestra.run.smithi137:osd.3> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.3.service 2024-03-14T12:55:55.065 INFO:tasks.cephadm:Deploying osd.4 on smithi137 with /dev/vg_nvme/lv_3... 2024-03-14T12:55:55.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-14T12:55:55.095 INFO:journalctl@ceph.osd.3.smithi137.stdout:-- Logs begin at Thu 2024-03-14 12:38:18 UTC. -- 2024-03-14T12:55:55.469 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:55:55.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:55 smithi137 ceph-mon[50336]: from='osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T12:55:55.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:55 smithi137 ceph-mon[50336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T12:55:55.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:55 smithi111 ceph-mon[39688]: from='osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T12:55:55.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:55 smithi111 ceph-mon[39688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T12:55:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T12:55:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: osdmap e25: 4 total, 3 up, 4 in 2024-03-14T12:55:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:56.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:55:56.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:55:56.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:56.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:56 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:56.644 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:56.644 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T12:55:56.644 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: osdmap e25: 4 total, 3 up, 4 in 2024-03-14T12:55:56.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:56.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:55:56.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:55:56.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:56.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:55:56.645 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[56679]: 2024-03-14T12:55:56.278+0000 7fdd849de700 -1 osd.3 0 waiting for initial osdmap 2024-03-14T12:55:56.645 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 12:55:56 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[56679]: 2024-03-14T12:55:56.284+0000 7fdd7ffda700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T12:55:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:57 smithi111 ceph-mon[39688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:55:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:57 smithi111 ceph-mon[39688]: osdmap e26: 4 total, 3 up, 4 in 2024-03-14T12:55:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:57 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:57.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:57 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:57 smithi137 ceph-mon[50336]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:55:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:57 smithi137 ceph-mon[50336]: osdmap e26: 4 total, 3 up, 4 in 2024-03-14T12:55:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:57 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:57 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:55:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:55:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:58.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] boot 2024-03-14T12:55:58.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: osdmap e27: 4 total, 4 up, 4 in 2024-03-14T12:55:58.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:58.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: osd.3 [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] boot 2024-03-14T12:55:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: osdmap e27: 4 total, 4 up, 4 in 2024-03-14T12:55:58.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T12:55:58.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:55:58.785 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-14T12:55:58.785 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-14T12:55:58.785 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-03-14T12:55:58.785 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-03-14T12:55:58.786 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110367 s, 950 MB/s 2024-03-14T12:55:58.786 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-03-14T12:55:59.494 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:55:59 smithi137 ceph-mon[50336]: osdmap e28: 4 total, 4 up, 4 in 2024-03-14T12:55:59.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-03-14T12:55:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:55:59 smithi111 ceph-mon[39688]: osdmap e28: 4 total, 4 up, 4 in 2024-03-14T12:56:00.205 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: osdmap e29: 4 total, 4 up, 4 in 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:00.446 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:00.446 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.446 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:00 smithi137 ceph-mon[50336]: osdmap e30: 4 total, 4 up, 4 in 2024-03-14T12:56:00.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:00.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: osdmap e29: 4 total, 4 up, 4 in 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:00 smithi111 ceph-mon[39688]: osdmap e30: 4 total, 4 up, 4 in 2024-03-14T12:56:01.320 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi137 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi137 to 15970M 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:01.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:01 smithi137 ceph-mon[50336]: osdmap e31: 4 total, 4 up, 4 in 2024-03-14T12:56:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi137 2024-03-14T12:56:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi137 to 15970M 2024-03-14T12:56:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:01.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:01.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:01 smithi111 ceph-mon[39688]: osdmap e31: 4 total, 4 up, 4 in 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 -- 172.21.15.137:0/377134969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fce40 msgr2=0x7f455c0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 --2- 172.21.15.137:0/377134969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fce40 0x7f455c0fd250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4544004a90 tx=0x7f4544009dd0 comp rx=0 tx=0).stop 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 -- 172.21.15.137:0/377134969 shutdown_connections 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 --2- 172.21.15.137:0/377134969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fe040 0x7f455c0fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 --2- 172.21.15.137:0/377134969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fce40 0x7f455c0fd250 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 -- 172.21.15.137:0/377134969 >> 172.21.15.137:0/377134969 conn(0x7f455c0f8570 msgr2=0x7f455c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:02.033 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 -- 172.21.15.137:0/377134969 shutdown_connections 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.030+0000 7f4561eb7700 1 -- 172.21.15.137:0/377134969 wait complete. 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 Processor -- start 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 -- start start 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fce40 0x7f455c10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c10e530 con 0x7f455c0fce40 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.031+0000 7f4561eb7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455c10ed20 con 0x7f455c0fe040 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:49842/0 (socket says 172.21.15.137:49842) 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 -- 172.21.15.137:0/2542157563 learned_addr learned my addr 172.21.15.137:0/2542157563 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455b7fe700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fce40 0x7f455c10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 -- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fce40 msgr2=0x7f455c10d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:02.034 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fce40 0x7f455c10d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 -- 172.21.15.137:0/2542157563 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544004650 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f455affd700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b790 tx=0x7f454c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454c00f6f0 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f454c00fd00 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f454c0103a0 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455c10f000 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.032+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455c10f540 con 0x7f455c0fe040 2024-03-14T12:56:02.035 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.033+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c05f5f0 con 0x7f455c0fe040 2024-03-14T12:56:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.034+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f454c00f850 con 0x7f455c0fe040 2024-03-14T12:56:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.034+0000 7f4553fff700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 0x7f454806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.034+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f454c086830 con 0x7f455c0fe040 2024-03-14T12:56:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.034+0000 7f455b7fe700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 0x7f454806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:02.036 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.035+0000 7f455b7fe700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 0x7f454806b750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4544000c00 tx=0x7f4544016040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:02.038 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.038+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f454c056d30 con 0x7f455c0fe040 2024-03-14T12:56:02.226 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:02.227+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f455c10f880 con 0x7f45480692a0 2024-03-14T12:56:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:02 smithi111 ceph-mon[39688]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:02.688 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:02 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:56:02.688 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:02 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:56:02.689 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:02 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:02 smithi137 ceph-mon[50336]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:02 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:56:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:02 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:56:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:02 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:03.619 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:03 smithi137 ceph-mon[50336]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:56:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:03 smithi111 ceph-mon[39688]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:56:04.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:04 smithi111 ceph-mon[39688]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-14T12:56:04.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:04 smithi137 ceph-mon[50336]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-14T12:56:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:05 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1442688429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]: dispatch 2024-03-14T12:56:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:05 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]: dispatch 2024-03-14T12:56:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:05 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]': finished 2024-03-14T12:56:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:05 smithi111 ceph-mon[39688]: osdmap e32: 5 total, 4 up, 5 in 2024-03-14T12:56:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:05 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:05 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/1442688429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]: dispatch 2024-03-14T12:56:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:05 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]: dispatch 2024-03-14T12:56:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:05 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1318352-47b7-4cba-8ea1-0b06d8e9843c"}]': finished 2024-03-14T12:56:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:05 smithi137 ceph-mon[50336]: osdmap e32: 5 total, 4 up, 5 in 2024-03-14T12:56:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:05 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:06.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:06 smithi111 ceph-mon[39688]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-14T12:56:06.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:06 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1018221503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:56:06.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:06 smithi137 ceph-mon[50336]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-14T12:56:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:06 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/1018221503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:56:08.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:08 smithi111 ceph-mon[39688]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-14T12:56:08.699 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:08 smithi137 ceph-mon[50336]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-14T12:56:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:10 smithi111 ceph-mon[39688]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-14T12:56:10.711 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:10 smithi137 ceph-mon[50336]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-14T12:56:12.597 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:12 smithi137 ceph-mon[50336]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T12:56:12.597 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T12:56:12.597 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:12 smithi111 ceph-mon[39688]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-14T12:56:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T12:56:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:13.566 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:13 smithi137 ceph-mon[50336]: Deploying daemon osd.4 on smithi137 2024-03-14T12:56:13.566 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:13 smithi111 ceph-mon[39688]: Deploying daemon osd.4 on smithi137 2024-03-14T12:56:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:14.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:14 smithi137 ceph-mon[50336]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:14 smithi111 ceph-mon[39688]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:16.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:16 smithi111 ceph-mon[39688]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:16.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:16 smithi137 ceph-mon[50336]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:18.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:18 smithi137 ceph-mon[50336]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:18.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:18 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:18.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:18 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:18.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:18 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:18.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:18 smithi111 ceph-mon[39688]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:18.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:18 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:18.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:18 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:18.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:18 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:20.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:20 smithi137 ceph-mon[50336]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:20.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.540+0000 7f4553fff700 1 -- 172.21.15.137:0/2542157563 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f455c10f880 con 0x7f45480692a0 2024-03-14T12:56:20.540 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 4 on host 'smithi137' 2024-03-14T12:56:20.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 msgr2=0x7f454806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:20.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 0x7f454806b750 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4544000c00 tx=0x7f4544016040 comp rx=0 tx=0).stop 2024-03-14T12:56:20.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 msgr2=0x7f455c10dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:20.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f454c00b790 tx=0x7f454c009250 comp rx=0 tx=0).stop 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 shutdown_connections 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f45480692a0 0x7f454806b750 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f455c0fce40 0x7f455c10d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 --2- 172.21.15.137:0/2542157563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f455c0fe040 0x7f455c10dea0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 >> 172.21.15.137:0/2542157563 conn(0x7f455c0f8570 msgr2=0x7f455c0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 shutdown_connections 2024-03-14T12:56:20.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:20.542+0000 7f4561eb7700 1 -- 172.21.15.137:0/2542157563 wait complete. 2024-03-14T12:56:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:20 smithi111 ceph-mon[39688]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:21.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:21 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:21.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:21 smithi137 ceph-mon[50336]: from='osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T12:56:21.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:21 smithi137 ceph-mon[50336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T12:56:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:21 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:21 smithi111 ceph-mon[39688]: from='osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T12:56:21.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:21 smithi111 ceph-mon[39688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T12:56:21.963 DEBUG:teuthology.orchestra.run.smithi137:osd.4> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.4.service 2024-03-14T12:56:21.965 INFO:tasks.cephadm:Deploying osd.5 on smithi137 with /dev/vg_nvme/lv_2... 2024-03-14T12:56:21.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-14T12:56:21.996 INFO:journalctl@ceph.osd.4.smithi137.stdout:-- Logs begin at Thu 2024-03-14 12:38:18 UTC. -- 2024-03-14T12:56:22.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: osdmap e33: 5 total, 4 up, 5 in 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:22.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:22.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:23.006 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[64676]: 2024-03-14T12:56:22.540+0000 7f124d3a3700 -1 osd.4 0 waiting for initial osdmap 2024-03-14T12:56:23.006 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[64676]: 2024-03-14T12:56:22.551+0000 7f124819e700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: osdmap e33: 5 total, 4 up, 5 in 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:23.701 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:23 smithi137 ceph-mon[50336]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:56:23.701 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:23 smithi137 ceph-mon[50336]: osdmap e34: 5 total, 4 up, 5 in 2024-03-14T12:56:23.701 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:23 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:23.701 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:23 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:23 smithi111 ceph-mon[39688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:56:23.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:23 smithi111 ceph-mon[39688]: osdmap e34: 5 total, 4 up, 5 in 2024-03-14T12:56:23.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:23 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:23.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:23 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:24.600 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-14T12:56:24.600 INFO:teuthology.orchestra.run.smithi137.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-14T12:56:24.600 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10+0 records in 2024-03-14T12:56:24.601 INFO:teuthology.orchestra.run.smithi137.stderr:10+0 records out 2024-03-14T12:56:24.601 INFO:teuthology.orchestra.run.smithi137.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111224 s, 943 MB/s 2024-03-14T12:56:24.601 INFO:teuthology.orchestra.run.smithi137.stderr:--> Zapping successful for: 2024-03-14T12:56:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:56:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:56:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] boot 2024-03-14T12:56:24.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: osdmap e35: 5 total, 5 up, 5 in 2024-03-14T12:56:24.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: osd.4 [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] boot 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: osdmap e35: 5 total, 5 up, 5 in 2024-03-14T12:56:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T12:56:25.651 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:25 smithi137 ceph-mon[50336]: osdmap e36: 5 total, 5 up, 5 in 2024-03-14T12:56:25.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-03-14T12:56:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:25 smithi111 ceph-mon[39688]: osdmap e36: 5 total, 5 up, 5 in 2024-03-14T12:56:26.208 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:56:26.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:26 smithi137 ceph-mon[50336]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:26 smithi111 ceph-mon[39688]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 -- 172.21.15.137:0/2735599294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 msgr2=0x7fb1580fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 --2- 172.21.15.137:0/2735599294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb1580fd3f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb154004940 tx=0x7fb154009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 -- 172.21.15.137:0/2735599294 shutdown_connections 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 --2- 172.21.15.137:0/2735599294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb1580fd3f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 --2- 172.21.15.137:0/2735599294 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb1580fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 -- 172.21.15.137:0/2735599294 >> 172.21.15.137:0/2735599294 conn(0x7fb1580f7430 msgr2=0x7fb1580f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 -- 172.21.15.137:0/2735599294 shutdown_connections 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.505+0000 7fb15f83e700 1 -- 172.21.15.137:0/2735599294 wait complete. 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 Processor -- start 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 -- start start 2024-03-14T12:56:28.507 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb15810c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15810d400 con 0x7fb1580fbcd0 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15f83e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15810dbf0 con 0x7fb1580fcf80 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:53674/0 (socket says 172.21.15.137:53674) 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 -- 172.21.15.137:0/1444310443 learned_addr learned my addr 172.21.15.137:0/1444310443 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 -- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 msgr2=0x7fb15810c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15d5da700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb15810c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb15810c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15cdd9700 1 -- 172.21.15.137:0/1444310443 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154005040 con 0x7fb1580fcf80 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.506+0000 7fb15d5da700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb15810c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb15cdd9700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb154009fd0 tx=0x7fb154009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb15400e040 con 0x7fb1580fcf80 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb154004290 con 0x7fb1580fcf80 2024-03-14T12:56:28.508 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15810de70 con 0x7fb1580fcf80 2024-03-14T12:56:28.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb15401c3f0 con 0x7fb1580fcf80 2024-03-14T12:56:28.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.507+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15810e430 con 0x7fb1580fcf80 2024-03-14T12:56:28.509 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.508+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15805f5f0 con 0x7fb1580fcf80 2024-03-14T12:56:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.508+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb15400bc50 con 0x7fb1580fcf80 2024-03-14T12:56:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.509+0000 7fb149ffb700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 0x7fb14406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.509+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fb154088190 con 0x7fb1580fcf80 2024-03-14T12:56:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.509+0000 7fb15d5da700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 0x7fb14406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:28.510 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.509+0000 7fb15d5da700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 0x7fb14406b760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb14c006fd0 tx=0x7fb14c008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:28.513 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.513+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb154058440 con 0x7fb1580fcf80 2024-03-14T12:56:28.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:28.702+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb15810e6c0 con 0x7fb1440692b0 2024-03-14T12:56:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi137 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi137 to 7985M 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:29.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi137 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi137 to 7985M 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:29.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:29.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:29 smithi111 ceph-mon[39688]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:56:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:56:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:56:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:29.921 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:29 smithi137 ceph-mon[50336]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:56:29.922 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-14T12:56:29.922 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-14T12:56:29.922 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:30 smithi137 ceph-mon[50336]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:31.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:30 smithi111 ceph-mon[39688]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:31 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/1095989388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]: dispatch 2024-03-14T12:56:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:31 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]: dispatch 2024-03-14T12:56:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:31 smithi137 ceph-mon[50336]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]': finished 2024-03-14T12:56:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:31 smithi137 ceph-mon[50336]: osdmap e37: 6 total, 5 up, 6 in 2024-03-14T12:56:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:31 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:32.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:31 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1095989388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]: dispatch 2024-03-14T12:56:32.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:31 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]: dispatch 2024-03-14T12:56:32.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:31 smithi111 ceph-mon[39688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1323bb35-1d44-4189-8bbf-c192740aaa58"}]': finished 2024-03-14T12:56:32.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:31 smithi111 ceph-mon[39688]: osdmap e37: 6 total, 5 up, 6 in 2024-03-14T12:56:32.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:31 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:33.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:32 smithi137 ceph-mon[50336]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:33.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:32 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/3397419833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:56:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:32 smithi111 ceph-mon[39688]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:33.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:32 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/3397419833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-14T12:56:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:34 smithi137 ceph-mon[50336]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:35.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:34 smithi111 ceph-mon[39688]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:36.941 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:36 smithi137 ceph-mon[50336]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:36 smithi111 ceph-mon[39688]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:39.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:38 smithi137 ceph-mon[50336]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:39.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:38 smithi111 ceph-mon[39688]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:39.846 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:39 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T12:56:39.846 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:39 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:39.846 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:39 smithi137 ceph-mon[50336]: Deploying daemon osd.5 on smithi137 2024-03-14T12:56:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T12:56:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:39 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:40.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:39 smithi111 ceph-mon[39688]: Deploying daemon osd.5 on smithi137 2024-03-14T12:56:40.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:40 smithi137 ceph-mon[50336]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:41.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:40 smithi111 ceph-mon[39688]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:42.952 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:42 smithi137 ceph-mon[50336]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:43.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:42 smithi111 ceph-mon[39688]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:44 smithi111 ceph-mon[39688]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:44 smithi137 ceph-mon[50336]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:46 smithi111 ceph-mon[39688]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:47.176 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:46 smithi137 ceph-mon[50336]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:47.567 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.567+0000 7fb149ffb700 1 -- 172.21.15.137:0/1444310443 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb15810e6c0 con 0x7fb1440692b0 2024-03-14T12:56:47.568 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 5 on host 'smithi137' 2024-03-14T12:56:47.569 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 msgr2=0x7fb14406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:47.569 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 0x7fb14406b760 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb14c006fd0 tx=0x7fb14c008040 comp rx=0 tx=0).stop 2024-03-14T12:56:47.569 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 msgr2=0x7fb15810cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb154009fd0 tx=0x7fb154009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 shutdown_connections 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb1440692b0 0x7fb14406b760 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1580fbcd0 0x7fb15810c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 --2- 172.21.15.137:0/1444310443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1580fcf80 0x7fb15810cd70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 >> 172.21.15.137:0/1444310443 conn(0x7fb1580f7430 msgr2=0x7fb1580f8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 shutdown_connections 2024-03-14T12:56:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:56:47.569+0000 7fb15f83e700 1 -- 172.21.15.137:0/1444310443 wait complete. 2024-03-14T12:56:48.155 DEBUG:teuthology.orchestra.run.smithi137:osd.5> sudo journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.5.service 2024-03-14T12:56:48.157 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-14T12:56:48.157 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-14T12:56:48.190 INFO:journalctl@ceph.osd.5.smithi137.stdout:-- Logs begin at Thu 2024-03-14 12:38:18 UTC. -- 2024-03-14T12:56:48.439 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:56:48.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:48.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:48 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:49.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:49.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T12:56:49.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T12:56:49.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:49.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:48 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:49 smithi111 ceph-mon[39688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T12:56:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:49 smithi111 ceph-mon[39688]: osdmap e38: 6 total, 5 up, 6 in 2024-03-14T12:56:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:49 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:49 smithi111 ceph-mon[39688]: from='osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:49 smithi111 ceph-mon[39688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:49.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-mon[50336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T12:56:49.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-mon[50336]: osdmap e38: 6 total, 5 up, 6 in 2024-03-14T12:56:49.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:49.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-mon[50336]: from='osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:49.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-mon[50336]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T12:56:49.796 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[72682]: 2024-03-14T12:56:49.542+0000 7f4750323700 -1 osd.5 0 waiting for initial osdmap 2024-03-14T12:56:49.796 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 12:56:49 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[72682]: 2024-03-14T12:56:49.550+0000 7f474d122700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T12:56:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 -- 172.21.15.111:0/2229764257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fcf90 msgr2=0x7f390c0fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:50.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 --2- 172.21.15.111:0/2229764257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fcf90 0x7f390c0fd400 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f38fc004940 tx=0x7f38fc009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 -- 172.21.15.111:0/2229764257 shutdown_connections 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 --2- 172.21.15.111:0/2229764257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fcf90 0x7f390c0fd400 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 --2- 172.21.15.111:0/2229764257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fbce0 0x7f390c0fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 -- 172.21.15.111:0/2229764257 >> 172.21.15.111:0/2229764257 conn(0x7f390c0f7440 msgr2=0x7f390c0f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 -- 172.21.15.111:0/2229764257 shutdown_connections 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.251+0000 7f39140e7700 1 -- 172.21.15.111:0/2229764257 wait complete. 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 Processor -- start 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 -- start start 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fcf90 0x7f390c10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c10d390 con 0x7f390c0fbce0 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f39140e7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c10dbe0 con 0x7f390c0fcf90 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46042/0 (socket says 172.21.15.111:46042) 2024-03-14T12:56:50.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 -- 172.21.15.111:0/1111957008 learned_addr learned my addr 172.21.15.111:0/1111957008 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911682700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fcf90 0x7f390c10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 -- 172.21.15.111:0/1111957008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fcf90 msgr2=0x7f390c10cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fcf90 0x7f390c10cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.252+0000 7f3911e83700 1 -- 172.21.15.111:0/1111957008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc005040 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f3911e83700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3908002a50 tx=0x7f39080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3908007cc0 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f390800f040 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c10dec0 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3908009540 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.253+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c10e400 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.254+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c05f5f0 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.254+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f39080096a0 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.254+0000 7f39027fc700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.254+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f3908086a80 con 0x7f390c0fbce0 2024-03-14T12:56:50.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.254+0000 7f3911682700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:50.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.255+0000 7f3911682700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38fc000c00 tx=0x7f38fc009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:50.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.259+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f3908056c90 con 0x7f390c0fbce0 2024-03-14T12:56:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.442+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f390c04bed0 con 0x7f390c0fbce0 2024-03-14T12:56:50.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.443+0000 7f39027fc700 1 -- 172.21.15.111:0/1111957008 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f3908056820 con 0x7f390c0fbce0 2024-03-14T12:56:50.443 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:56:50.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 msgr2=0x7f38f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:50.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38fc000c00 tx=0x7f38fc009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:50.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 msgr2=0x7f390c10c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3908002a50 tx=0x7f39080046b0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 shutdown_connections 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c0fbce0 0x7f390c10c850 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 --2- 172.21.15.111:0/1111957008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c0fcf90 0x7f390c10cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 >> 172.21.15.111:0/1111957008 conn(0x7f390c0f7440 msgr2=0x7f390c0f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 shutdown_connections 2024-03-14T12:56:50.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:50.445+0000 7f39140e7700 1 -- 172.21.15.111:0/1111957008 wait complete. 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: osdmap e39: 6 total, 5 up, 6 in 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:50 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1111957008' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: osdmap e39: 6 total, 5 up, 6 in 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:50.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:50 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1111957008' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T12:56:51.238 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1710420983,"num_in_osds":6,"osd_in_since":1710420991,"num_remapped_pgs":0} 2024-03-14T12:56:51.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:51 smithi137 ceph-mon[50336]: purged_snaps scrub starts 2024-03-14T12:56:51.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:51 smithi137 ceph-mon[50336]: purged_snaps scrub ok 2024-03-14T12:56:51.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:51 smithi137 ceph-mon[50336]: osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] boot 2024-03-14T12:56:51.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:51 smithi137 ceph-mon[50336]: osdmap e40: 6 total, 6 up, 6 in 2024-03-14T12:56:51.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:51 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:51 smithi111 ceph-mon[39688]: purged_snaps scrub starts 2024-03-14T12:56:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:51 smithi111 ceph-mon[39688]: purged_snaps scrub ok 2024-03-14T12:56:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:51 smithi111 ceph-mon[39688]: osd.5 [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] boot 2024-03-14T12:56:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:51 smithi111 ceph-mon[39688]: osdmap e40: 6 total, 6 up, 6 in 2024-03-14T12:56:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:51 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T12:56:52.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-14T12:56:52.506 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: osdmap e41: 6 total, 6 up, 6 in 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:52.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:52.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:52.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:52.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:52.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:52.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: osdmap e41: 6 total, 6 up, 6 in 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:56:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:56:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:53 smithi111 ceph-mon[39688]: Detected new or changed devices on smithi137 2024-03-14T12:56:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:53 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi137 to 5323M 2024-03-14T12:56:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:53 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi137 2024-03-14T12:56:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:53 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi137 to 5323M 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 -- 172.21.15.111:0/3071700546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fbcd0 msgr2=0x7f35c40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 --2- 172.21.15.111:0/3071700546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fbcd0 0x7f35c40fc0e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f35b4004940 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 -- 172.21.15.111:0/3071700546 shutdown_connections 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 --2- 172.21.15.111:0/3071700546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35c40fcf80 0x7f35c40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 --2- 172.21.15.111:0/3071700546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fbcd0 0x7f35c40fc0e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.466+0000 7f35ca725700 1 -- 172.21.15.111:0/3071700546 >> 172.21.15.111:0/3071700546 conn(0x7f35c40f7430 msgr2=0x7f35c40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 -- 172.21.15.111:0/3071700546 shutdown_connections 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 -- 172.21.15.111:0/3071700546 wait complete. 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 Processor -- start 2024-03-14T12:56:54.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 -- start start 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35c40fbcd0 0x7f35c410c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c410d400 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35ca725700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c410dbf0 con 0x7f35c40fbcd0 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57458/0 (socket says 172.21.15.111:57458) 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.467+0000 7f35c37fe700 1 -- 172.21.15.111:0/3130499071 learned_addr learned my addr 172.21.15.111:0/3130499071 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c37fe700 1 -- 172.21.15.111:0/3130499071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35c40fbcd0 msgr2=0x7f35c410c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c37fe700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35c40fbcd0 0x7f35c410c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c37fe700 1 -- 172.21.15.111:0/3130499071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4005040 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c37fe700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b790 tx=0x7f35b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b800f770 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35b800fd80 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c410ded0 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35b800d310 con 0x7f35c40fcf80 2024-03-14T12:56:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.468+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c410e490 con 0x7f35c40fcf80 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.469+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c405f5f0 con 0x7f35c40fcf80 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.469+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f35b8010420 con 0x7f35c40fcf80 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.470+0000 7f35c0ff9700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 0x7f35ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.470+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f35b8086cf0 con 0x7f35c40fcf80 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.470+0000 7f35c3fff700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 0x7f35ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.470+0000 7f35c3fff700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 0x7f35ac06b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35b400b900 tx=0x7f35b4009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:54.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.474+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f35b8056e90 con 0x7f35c40fcf80 2024-03-14T12:56:54.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.659+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f35c410e790 con 0x7f35c40fcf80 2024-03-14T12:56:54.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.659+0000 7f35c0ff9700 1 -- 172.21.15.111:0/3130499071 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f35b8056a20 con 0x7f35c40fcf80 2024-03-14T12:56:54.659 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:56:54.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 msgr2=0x7f35ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:54.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 0x7f35ac06b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35b400b900 tx=0x7f35b4009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 msgr2=0x7f35c410cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f35b800b790 tx=0x7f35b8009250 comp rx=0 tx=0).stop 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 shutdown_connections 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35c40fbcd0 0x7f35c410c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35ac0692a0 0x7f35ac06b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 --2- 172.21.15.111:0/3130499071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35c40fcf80 0x7f35c410cd70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 >> 172.21.15.111:0/3130499071 conn(0x7f35c40f7430 msgr2=0x7f35c40f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 shutdown_connections 2024-03-14T12:56:54.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:54.661+0000 7f35ca725700 1 -- 172.21.15.111:0/3130499071 wait complete. 2024-03-14T12:56:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:54 smithi111 ceph-mon[39688]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:54 smithi137 ceph-mon[50336]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:55.372 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1710421010,"num_in_osds":6,"osd_in_since":1710420991,"num_remapped_pgs":0} 2024-03-14T12:56:55.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T12:56:55.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:55 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3130499071' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T12:56:55.640 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:56:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:55 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3130499071' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-14T12:56:56.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:56 smithi111 ceph-mon[39688]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:57.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:56 smithi137 ceph-mon[50336]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:57.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- 172.21.15.111:0/3529332984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fe850 msgr2=0x7f5184106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:57.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 --2- 172.21.15.111:0/3529332984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fe850 0x7f5184106f20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5174004970 tx=0x7f5174009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:57.422 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- 172.21.15.111:0/3529332984 shutdown_connections 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 --2- 172.21.15.111:0/3529332984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fe850 0x7f5184106f20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 --2- 172.21.15.111:0/3529332984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51840fdeb0 0x7f51840fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- 172.21.15.111:0/3529332984 >> 172.21.15.111:0/3529332984 conn(0x7f5184072bc0 msgr2=0x7f5184072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- 172.21.15.111:0/3529332984 shutdown_connections 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- 172.21.15.111:0/3529332984 wait complete. 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 Processor -- start 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.421+0000 7f518bcb6700 1 -- start start 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51840fe850 0x7f5184113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51841146b0 con 0x7f51840fdeb0 2024-03-14T12:56:57.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518410da10 con 0x7f51840fe850 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57488/0 (socket says 172.21.15.111:57488) 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 -- 172.21.15.111:0/2003361777 learned_addr learned my addr 172.21.15.111:0/2003361777 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 -- 172.21.15.111:0/2003361777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51840fe850 msgr2=0x7f5184113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51840fe850 0x7f5184113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 -- 172.21.15.111:0/2003361777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174005040 con 0x7f51840fdeb0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f5189a52700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f518000b790 tx=0x7f5180009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f518000f770 con 0x7f51840fdeb0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f518000fd80 con 0x7f51840fdeb0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f518000d310 con 0x7f51840fdeb0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518410dcf0 con 0x7f51840fdeb0 2024-03-14T12:56:57.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.422+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518410e1e0 con 0x7f51840fdeb0 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.423+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5184065e10 con 0x7f51840fdeb0 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.424+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5180010420 con 0x7f51840fdeb0 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.424+0000 7f517a7fc700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 0x7f517006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.424+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5180086df0 con 0x7f51840fdeb0 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.424+0000 7f5189251700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 0x7f517006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:56:57.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.424+0000 7f5189251700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 0x7f517006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5174009fd0 tx=0x7f5174009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:56:57.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.428+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f5180056ef0 con 0x7f51840fdeb0 2024-03-14T12:56:57.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.613+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5184109f00 con 0x7f51840fdeb0 2024-03-14T12:56:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.614+0000 7f517a7fc700 1 -- 172.21.15.111:0/2003361777 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11118 (secure 0 0 0) 0x7f5180056a80 con 0x7f51840fdeb0 2024-03-14T12:56:57.614 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:56:57.614 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T12:56:51.546798+0000","last_up_change":"2024-03-14T12:56:50.540579+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6803","nonce":3771249398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6805","nonce":3771249398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6809","nonce":3771249398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6807","nonce":3771249398}]},"public_addr":"172.21.15.111:6803/3771249398","cluster_addr":"172.21.15.111:6805/3771249398","heartbeat_back_addr":"172.21.15.111:6809/3771249398","heartbeat_front_addr":"172.21.15.111:6807/3771249398","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6811","nonce":3746289562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6813","nonce":3746289562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6817","nonce":3746289562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6815","nonce":3746289562}]},"public_addr":"172.21.15.111:6811/3746289562","cluster_addr":"172.21.15.111:6813/3746289562","heartbeat_back_addr":"172.21.15.111:6817/3746289562","heartbeat_front_addr":"172.21.15.111:6815/3746289562","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6819","nonce":337068597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6821","nonce":337068597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6825","nonce":337068597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6823","nonce":337068597}]},"public_addr":"172.21.15.111:6819/337068597","cluster_addr":"172.21.15.111:6821/337068597","heartbeat_back_addr":"172.21.15.111:6825/337068597","heartbeat_front_addr":"172.21.15.111:6823/337068597","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6801","nonce":316104226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6803","nonce":316104226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6807","nonce":316104226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6805","nonce":316104226}]},"public_addr":"172.21.15.137:6801/316104226","cluster_addr":"172.21.15.137:6803/316104226","heartbeat_back_addr":"172.21.15.137:6807/316104226","heartbeat_front_addr":"172.21.15.137:6805/316104226","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6809","nonce":1988641735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6811","nonce":1988641735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6815","nonce":1988641735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6813","nonce":1988641735}]},"public_addr":"172.21.15.137:6809/1988641735","cluster_addr":"172.21.15.137:6811/1988641735","heartbeat_back_addr":"172.21.15.137:6815/1988641735","heartbeat_front_addr":"172.21.15.137:6813/1988641735","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6817","nonce":1979294971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6819","nonce":1979294971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6823","nonce":1979294971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6821","nonce":1979294971}]},"public_addr":"172.21.15.137:6817/1979294971","cluster_addr":"172.21.15.137:6819/1979294971","heartbeat_back_addr":"172.21.15.137:6823/1979294971","heartbeat_front_addr":"172.21.15.137:6821/1979294971","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 msgr2=0x7f517006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 0x7f517006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5174009fd0 tx=0x7f5174009e30 comp rx=0 tx=0).stop 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 msgr2=0x7f5184113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f518000b790 tx=0x7f5180009250 comp rx=0 tx=0).stop 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 shutdown_connections 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f51700692a0 0x7f517006b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51840fdeb0 0x7f5184113a90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 --2- 172.21.15.111:0/2003361777 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51840fe850 0x7f5184113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:56:57.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 >> 172.21.15.111:0/2003361777 conn(0x7f5184072bc0 msgr2=0x7f5184103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:56:57.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 shutdown_connections 2024-03-14T12:56:57.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:56:57.616+0000 7f518bcb6700 1 -- 172.21.15.111:0/2003361777 wait complete. 2024-03-14T12:56:58.391 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T12:55:33.503741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-14T12:56:58.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-14T12:56:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:58 smithi111 ceph-mon[39688]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:58 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2003361777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:56:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:56:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:56:58.658 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:56:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:58 smithi137 ceph-mon[50336]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:56:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:58 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2003361777' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:56:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:56:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 -- 172.21.15.111:0/54235727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c102710 msgr2=0x7f0d9c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 --2- 172.21.15.111:0/54235727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c102710 0x7f0d9c0fbd80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c004940 tx=0x7f0d8c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 -- 172.21.15.111:0/54235727 shutdown_connections 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 --2- 172.21.15.111:0/54235727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c102710 0x7f0d9c0fbd80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 --2- 172.21.15.111:0/54235727 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c101d70 0x7f0d9c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 -- 172.21.15.111:0/54235727 >> 172.21.15.111:0/54235727 conn(0x7f0d9c072bc0 msgr2=0x7f0d9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 -- 172.21.15.111:0/54235727 shutdown_connections 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.478+0000 7f0da0cde700 1 -- 172.21.15.111:0/54235727 wait complete. 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 Processor -- start 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 -- start start 2024-03-14T12:57:00.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c102710 0x7f0d9c113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c114610 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0da0cde700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c10da00 con 0x7f0d9c102710 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52880/0 (socket says 172.21.15.111:52880) 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 -- 172.21.15.111:0/3045697983 learned_addr learned my addr 172.21.15.111:0/3045697983 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d99d9b700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c102710 0x7f0d9c113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 -- 172.21.15.111:0/3045697983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c102710 msgr2=0x7f0d9c113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c102710 0x7f0d9c113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.479+0000 7f0d9a59c700 1 -- 172.21.15.111:0/3045697983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c005040 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0d9a59c700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0d90002a50 tx=0x7f0d900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d90007cc0 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d9000f040 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c10dce0 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d90009540 con 0x7f0d9c101d70 2024-03-14T12:57:00.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.480+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c10e2a0 con 0x7f0d9c101d70 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.481+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c0fd4c0 con 0x7f0d9c101d70 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.481+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0d900096a0 con 0x7f0d9c101d70 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.481+0000 7f0d8affd700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 0x7f0d8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.481+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d90086c60 con 0x7f0d9c101d70 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.482+0000 7f0d99d9b700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 0x7f0d8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.482+0000 7f0d99d9b700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 0x7f0d8406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c10ebc0 tx=0x7f0d8c009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:00.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.486+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f0d90056ce0 con 0x7f0d9c101d70 2024-03-14T12:57:00.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.669+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d9c108d10 con 0x7f0d9c101d70 2024-03-14T12:57:00.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.669+0000 7f0d8affd700 1 -- 172.21.15.111:0/3045697983 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d90056870 con 0x7f0d9c101d70 2024-03-14T12:57:00.669 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 msgr2=0x7f0d8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 0x7f0d8406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c10ebc0 tx=0x7f0d8c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 msgr2=0x7f0d9c113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0d90002a50 tx=0x7f0d900046b0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 shutdown_connections 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0d840692a0 0x7f0d8406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d9c101d70 0x7f0d9c113a80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 --2- 172.21.15.111:0/3045697983 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d9c102710 0x7f0d9c113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 >> 172.21.15.111:0/3045697983 conn(0x7f0d9c072bc0 msgr2=0x7f0d9c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:00.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 shutdown_connections 2024-03-14T12:57:00.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:00.671+0000 7f0da0cde700 1 -- 172.21.15.111:0/3045697983 wait complete. 2024-03-14T12:57:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:00 smithi111 ceph-mon[39688]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:01.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:00 smithi137 ceph-mon[50336]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:01.447 INFO:tasks.cephadm:Setting up client nodes... 2024-03-14T12:57:01.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-14T12:57:01.715 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:01 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3045697983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T12:57:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:01 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3045697983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T12:57:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:02 smithi111 ceph-mon[39688]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:03.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:02 smithi137 ceph-mon[50336]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 -- 172.21.15.111:0/2835659202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 msgr2=0x7f50dc104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2835659202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc104740 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f50cc004940 tx=0x7f50cc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 -- 172.21.15.111:0/2835659202 shutdown_connections 2024-03-14T12:57:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2835659202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc104740 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2835659202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 0x7f50dc101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.604+0000 7f50e28d4700 1 -- 172.21.15.111:0/2835659202 >> 172.21.15.111:0/2835659202 conn(0x7f50dc0f7450 msgr2=0x7f50dc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.605+0000 7f50e28d4700 1 -- 172.21.15.111:0/2835659202 shutdown_connections 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.605+0000 7f50e28d4700 1 -- 172.21.15.111:0/2835659202 wait complete. 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.605+0000 7f50e28d4700 1 Processor -- start 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.605+0000 7f50e28d4700 1 -- start start 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 0x7f50dc10fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc110640 con 0x7f50dc102360 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc1107b0 con 0x7f50dc066f50 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52910/0 (socket says 172.21.15.111:52910) 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 -- 172.21.15.111:0/2633677909 learned_addr learned my addr 172.21.15.111:0/2633677909 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:03.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50dbfff700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 0x7f50dc10fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 -- 172.21.15.111:0/2633677909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 msgr2=0x7f50dc10fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 0x7f50dc10fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 -- 172.21.15.111:0/2633677909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc005040 con 0x7f50dc102360 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50db7fe700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00b3b0 tx=0x7f50cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50cc00e040 con 0x7f50dc102360 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc115110 con 0x7f50dc102360 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50cc00bc50 con 0x7f50dc102360 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50cc01c630 con 0x7f50dc102360 2024-03-14T12:57:03.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.606+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc1156d0 con 0x7f50dc102360 2024-03-14T12:57:03.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.607+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc04c710 con 0x7f50dc102360 2024-03-14T12:57:03.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.608+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f50cc013420 con 0x7f50dc102360 2024-03-14T12:57:03.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.608+0000 7f50d8ff9700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 0x7f50c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:03.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.608+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f50cc087b20 con 0x7f50dc102360 2024-03-14T12:57:03.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.608+0000 7f50dbfff700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 0x7f50c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:03.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.609+0000 7f50dbfff700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 0x7f50c406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50dc066c90 tx=0x7f50d0009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:03.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.613+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f50cc057c20 con 0x7f50dc102360 2024-03-14T12:57:03.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.874+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f50dc115aa0 con 0x7f50dc102360 2024-03-14T12:57:03.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.878+0000 7f50d8ff9700 1 -- 172.21.15.111:0/2633677909 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f50cc0577b0 con 0x7f50dc102360 2024-03-14T12:57:03.878 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-03-14T12:57:03.878 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAf9PJlZeotNBAAijKlQay49plLnZznd5++mw== 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 msgr2=0x7f50c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 0x7f50c406b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f50dc066c90 tx=0x7f50d0009040 comp rx=0 tx=0).stop 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 msgr2=0x7f50dc110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f50cc00b3b0 tx=0x7f50cc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 shutdown_connections 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50dc066f50 0x7f50dc10fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f50c40692a0 0x7f50c406b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 --2- 172.21.15.111:0/2633677909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50dc102360 0x7f50dc110040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 >> 172.21.15.111:0/2633677909 conn(0x7f50dc0f7450 msgr2=0x7f50dc0f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 shutdown_connections 2024-03-14T12:57:03.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:03.880+0000 7f50e28d4700 1 -- 172.21.15.111:0/2633677909 wait complete. 2024-03-14T12:57:04.813 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T12:57:04.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-14T12:57:04.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-14T12:57:04.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:04 smithi111 ceph-mon[39688]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:04.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:04 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2633677909' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:04.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:04 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2633677909' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T12:57:04.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-14T12:57:04.889 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:04 smithi137 ceph-mon[50336]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:04.889 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:04 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2633677909' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:04.890 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:04 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2633677909' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T12:57:05.131 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T12:57:06.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- 172.21.15.137:0/283784988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fcfd0 msgr2=0x7fd9640fd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:06.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 --2- 172.21.15.137:0/283784988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fd440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd958004940 tx=0x7fd958009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:06.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- 172.21.15.137:0/283784988 shutdown_connections 2024-03-14T12:57:06.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 --2- 172.21.15.137:0/283784988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fd440 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:06.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 --2- 172.21.15.137:0/283784988 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fbd20 0x7fd9640fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- 172.21.15.137:0/283784988 >> 172.21.15.137:0/283784988 conn(0x7fd9640f7460 msgr2=0x7fd9640f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- 172.21.15.137:0/283784988 shutdown_connections 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- 172.21.15.137:0/283784988 wait complete. 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 Processor -- start 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.888+0000 7fd968b2e700 1 -- start start 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fe310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd964100210 con 0x7fd9640fcfd0 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640fe850 con 0x7fd9640fbd20 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:48960/0 (socket says 172.21.15.137:48960) 2024-03-14T12:57:06.891 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 -- 172.21.15.137:0/1676418073 learned_addr learned my addr 172.21.15.137:0/1676418073 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd961d9b700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 -- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fcfd0 msgr2=0x7fd9640fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 -- 172.21.15.137:0/1676418073 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958005040 con 0x7fd9640fbd20 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd96259c700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd954002a50 tx=0x7fd9540046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd954007cc0 con 0x7fd9640fbd20 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd95400f040 con 0x7fd9640fbd20 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd95400d9c0 con 0x7fd9640fbd20 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9640feb30 con 0x7fd9640fbd20 2024-03-14T12:57:06.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.889+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9640ff070 con 0x7fd9640fbd20 2024-03-14T12:57:06.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:06 smithi111 ceph-mon[39688]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:06.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.891+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964101b60 con 0x7fd9640fbd20 2024-03-14T12:57:06.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.891+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd95400db20 con 0x7fd9640fbd20 2024-03-14T12:57:06.893 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.891+0000 7fd952ffd700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 0x7fd94c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:06.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.891+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd954086b70 con 0x7fd9640fbd20 2024-03-14T12:57:06.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.891+0000 7fd961d9b700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 0x7fd94c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:06.894 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.892+0000 7fd961d9b700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 0x7fd94c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd95800b900 tx=0x7fd958009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:06.895 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:06.896+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd954056bf0 con 0x7fd9640fbd20 2024-03-14T12:57:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:06 smithi137 ceph-mon[50336]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:07.150 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.150+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd964104780 con 0x7fd9640fbd20 2024-03-14T12:57:07.155 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.155+0000 7fd952ffd700 1 -- 172.21.15.137:0/1676418073 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd954056780 con 0x7fd9640fbd20 2024-03-14T12:57:07.155 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-03-14T12:57:07.155 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQAj9PJlkbEWCRAAWSMbV/IfrXfaBbOrZxLJSQ== 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 msgr2=0x7fd94c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 0x7fd94c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd95800b900 tx=0x7fd958009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 msgr2=0x7fd9640ffc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd954002a50 tx=0x7fd9540046b0 comp rx=0 tx=0).stop 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 shutdown_connections 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9640fbd20 0x7fd9640ffc10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd94c0692a0 0x7fd94c06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:07.157 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 --2- 172.21.15.137:0/1676418073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9640fcfd0 0x7fd9640fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:07.158 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 >> 172.21.15.137:0/1676418073 conn(0x7fd9640f7460 msgr2=0x7fd964065c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:07.158 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 shutdown_connections 2024-03-14T12:57:07.158 INFO:teuthology.orchestra.run.smithi137.stderr:2024-03-14T12:57:07.157+0000 7fd968b2e700 1 -- 172.21.15.137:0/1676418073 wait complete. 2024-03-14T12:57:07.787 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:07 smithi137 ceph-mon[50336]: from='client.? 172.21.15.137:0/1676418073' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:07.787 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:07 smithi137 ceph-mon[50336]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:07.787 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:07 smithi137 ceph-mon[50336]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T12:57:07.807 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T12:57:07.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-14T12:57:07.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-14T12:57:07.889 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-14T12:57:07.890 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-14T12:57:07.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-14T12:57:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:07 smithi111 ceph-mon[39688]: from='client.? 172.21.15.137:0/1676418073' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:07 smithi111 ceph-mon[39688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-14T12:57:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:07 smithi111 ceph-mon[39688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-14T12:57:08.163 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:08.729 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:08 smithi111 ceph-mon[39688]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:08 smithi137 ceph-mon[50336]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 -- 172.21.15.111:0/2469030969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d54102fd0 msgr2=0x7f7d541033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 --2- 172.21.15.111:0/2469030969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d54102fd0 0x7f7d541033a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d44004940 tx=0x7f7d44009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 -- 172.21.15.111:0/2469030969 shutdown_connections 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 --2- 172.21.15.111:0/2469030969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d540fb540 0x7f7d540fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 --2- 172.21.15.111:0/2469030969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d54102fd0 0x7f7d541033a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 -- 172.21.15.111:0/2469030969 >> 172.21.15.111:0/2469030969 conn(0x7f7d54071530 msgr2=0x7f7d54071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 -- 172.21.15.111:0/2469030969 shutdown_connections 2024-03-14T12:57:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.099+0000 7f7d5962f700 1 -- 172.21.15.111:0/2469030969 wait complete. 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 Processor -- start 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 -- start start 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d54102fd0 0x7f7d54113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54114430 con 0x7f7d540fb540 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d5962f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5410da30 con 0x7f7d54102fd0 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:10.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52924/0 (socket says 172.21.15.111:52924) 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 -- 172.21.15.111:0/3380019287 learned_addr learned my addr 172.21.15.111:0/3380019287 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d527fc700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d54102fd0 0x7f7d54113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 -- 172.21.15.111:0/3380019287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d54102fd0 msgr2=0x7f7d54113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d54102fd0 0x7f7d54113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.100+0000 7f7d52ffd700 1 -- 172.21.15.111:0/3380019287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d44005040 con 0x7f7d540fb540 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d52ffd700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400b3b0 tx=0x7f7d44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4400e040 con 0x7f7d540fb540 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d44013420 con 0x7f7d540fb540 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5410dcb0 con 0x7f7d540fb540 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d4401c610 con 0x7f7d540fb540 2024-03-14T12:57:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.101+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5410e270 con 0x7f7d540fb540 2024-03-14T12:57:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.102+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d54100270 con 0x7f7d540fb540 2024-03-14T12:57:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.102+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7d44013a00 con 0x7f7d540fb540 2024-03-14T12:57:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.102+0000 7f7d3b7fe700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 0x7f7d3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.103+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7d44087ff0 con 0x7f7d540fb540 2024-03-14T12:57:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.103+0000 7f7d527fc700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 0x7f7d3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:10.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.103+0000 7f7d527fc700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 0x7f7d3c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d5410ebf0 tx=0x7f7d48009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:10.106 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.107+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f7d440580f0 con 0x7f7d540fb540 2024-03-14T12:57:10.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.345+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7d5410eef0 con 0x7f7d540fb540 2024-03-14T12:57:10.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.348+0000 7f7d3b7fe700 1 -- 172.21.15.111:0/3380019287 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167177 (secure 0 0 0) 0x7f7d44057c80 con 0x7f7d540fb540 2024-03-14T12:57:10.350 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 msgr2=0x7f7d3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 0x7f7d3c06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7d5410ebf0 tx=0x7f7d48009040 comp rx=0 tx=0).stop 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 msgr2=0x7f7d541138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7d4400b3b0 tx=0x7f7d44009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 shutdown_connections 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f7d3c0692a0 0x7f7d3c06b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540fb540 0x7f7d541138a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 --2- 172.21.15.111:0/3380019287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d54102fd0 0x7f7d54113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 >> 172.21.15.111:0/3380019287 conn(0x7f7d54071530 msgr2=0x7f7d540fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:10.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.350+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 shutdown_connections 2024-03-14T12:57:10.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:10.351+0000 7f7d5962f700 1 -- 172.21.15.111:0/3380019287 wait complete. 2024-03-14T12:57:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:10 smithi111 ceph-mon[39688]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:10 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3380019287' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-14T12:57:10.958 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":19,"active_gid":14222,"active_name":"smithi111.atysez","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":1542503054},{"type":"v1","addr":"172.21.15.111:6801","nonce":1542503054}]},"active_addr":"172.21.15.111:6801/1542503054","active_change":"2024-03-14T12:53:13.176619+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14242,"name":"smithi137.xbyzbp","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.111:8443/","prometheus":"http://172.21.15.111:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3904007449}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":4199526358}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3552305380}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":661526538}]}]} 2024-03-14T12:57:10.962 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-14T12:57:10.963 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-14T12:57:10.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T12:57:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:10 smithi137 ceph-mon[50336]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:11.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:10 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3380019287' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-14T12:57:11.229 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:12.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:12 smithi111 ceph-mon[39688]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:13.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:12 smithi137 ceph-mon[50336]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 -- 172.21.15.111:0/2899613867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 msgr2=0x7f6c0c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 --2- 172.21.15.111:0/2899613867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c1020e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6c00004940 tx=0x7f6c00009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 -- 172.21.15.111:0/2899613867 shutdown_connections 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 --2- 172.21.15.111:0/2899613867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 0x7f6c0c0fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 --2- 172.21.15.111:0/2899613867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c1020e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 -- 172.21.15.111:0/2899613867 >> 172.21.15.111:0/2899613867 conn(0x7f6c0c0f7880 msgr2=0x7f6c0c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 -- 172.21.15.111:0/2899613867 shutdown_connections 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.054+0000 7f6c112d7700 1 -- 172.21.15.111:0/2899613867 wait complete. 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 Processor -- start 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 -- start start 2024-03-14T12:57:13.068 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 0x7f6c0c114340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c114a20 con 0x7f6c0c101d10 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c10dd60 con 0x7f6c0c1026b0 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59162/0 (socket says 172.21.15.111:59162) 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 -- 172.21.15.111:0/270660407 learned_addr learned my addr 172.21.15.111:0/270660407 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0a7fc700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 0x7f6c0c114340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 -- 172.21.15.111:0/270660407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 msgr2=0x7f6c0c114340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 0x7f6c0c114340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 -- 172.21.15.111:0/270660407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c00005040 con 0x7f6c0c101d10 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c0affd700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000b3b0 tx=0x7f6c00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0000e040 con 0x7f6c0c101d10 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c10df80 con 0x7f6c0c101d10 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c00013420 con 0x7f6c0c101d10 2024-03-14T12:57:13.069 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c0001c630 con 0x7f6c0c101d10 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.055+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c10e540 con 0x7f6c0c101d10 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.056+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c0fd460 con 0x7f6c0c101d10 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.057+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6c00013970 con 0x7f6c0c101d10 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.057+0000 7f6bf37fe700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 0x7f6bf406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.057+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c000880b0 con 0x7f6c0c101d10 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.057+0000 7f6c0a7fc700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 0x7f6bf406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.057+0000 7f6c0a7fc700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 0x7f6bf406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006fd0 tx=0x7f6bfc008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:13.070 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.061+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f6c000581b0 con 0x7f6c0c101d10 2024-03-14T12:57:13.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.245+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c0c05f5f0 con 0x7f6c0c101d10 2024-03-14T12:57:13.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.246+0000 7f6bf37fe700 1 -- 172.21.15.111:0/270660407 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11118 (secure 0 0 0) 0x7f6c00057d40 con 0x7f6c0c101d10 2024-03-14T12:57:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T12:56:51.546798+0000","last_up_change":"2024-03-14T12:56:50.540579+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6803","nonce":3771249398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6805","nonce":3771249398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6809","nonce":3771249398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6807","nonce":3771249398}]},"public_addr":"172.21.15.111:6803/3771249398","cluster_addr":"172.21.15.111:6805/3771249398","heartbeat_back_addr":"172.21.15.111:6809/3771249398","heartbeat_front_addr":"172.21.15.111:6807/3771249398","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6811","nonce":3746289562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6813","nonce":3746289562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6817","nonce":3746289562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6815","nonce":3746289562}]},"public_addr":"172.21.15.111:6811/3746289562","cluster_addr":"172.21.15.111:6813/3746289562","heartbeat_back_addr":"172.21.15.111:6817/3746289562","heartbeat_front_addr":"172.21.15.111:6815/3746289562","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6819","nonce":337068597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6821","nonce":337068597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6825","nonce":337068597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6823","nonce":337068597}]},"public_addr":"172.21.15.111:6819/337068597","cluster_addr":"172.21.15.111:6821/337068597","heartbeat_back_addr":"172.21.15.111:6825/337068597","heartbeat_front_addr":"172.21.15.111:6823/337068597","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6801","nonce":316104226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6803","nonce":316104226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6807","nonce":316104226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6805","nonce":316104226}]},"public_addr":"172.21.15.137:6801/316104226","cluster_addr":"172.21.15.137:6803/316104226","heartbeat_back_addr":"172.21.15.137:6807/316104226","heartbeat_front_addr":"172.21.15.137:6805/316104226","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6809","nonce":1988641735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6811","nonce":1988641735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6815","nonce":1988641735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6813","nonce":1988641735}]},"public_addr":"172.21.15.137:6809/1988641735","cluster_addr":"172.21.15.137:6811/1988641735","heartbeat_back_addr":"172.21.15.137:6815/1988641735","heartbeat_front_addr":"172.21.15.137:6813/1988641735","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6817","nonce":1979294971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6819","nonce":1979294971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6823","nonce":1979294971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6821","nonce":1979294971}]},"public_addr":"172.21.15.137:6817/1979294971","cluster_addr":"172.21.15.137:6819/1979294971","heartbeat_back_addr":"172.21.15.137:6823/1979294971","heartbeat_front_addr":"172.21.15.137:6821/1979294971","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 msgr2=0x7f6bf406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 0x7f6bf406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc006fd0 tx=0x7f6bfc008040 comp rx=0 tx=0).stop 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 msgr2=0x7f6c0c113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000b3b0 tx=0x7f6c00009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 shutdown_connections 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f6bf40692a0 0x7f6bf406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c0c101d10 0x7f6c0c113e00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 --2- 172.21.15.111:0/270660407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c0c1026b0 0x7f6c0c114340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 >> 172.21.15.111:0/270660407 conn(0x7f6c0c0f7880 msgr2=0x7f6c0c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 shutdown_connections 2024-03-14T12:57:13.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:13.248+0000 7f6c112d7700 1 -- 172.21.15.111:0/270660407 wait complete. 2024-03-14T12:57:13.868 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:13.869 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:13 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/270660407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:57:13.888 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-14T12:57:13.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-14T12:57:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:13 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/270660407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:57:14.152 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:14.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:14 smithi111 ceph-mon[39688]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:15.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:14 smithi137 ceph-mon[50336]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 -- 172.21.15.111:0/3468471827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 msgr2=0x7f8ca80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 --2- 172.21.15.111:0/3468471827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca80fbd80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c98004940 tx=0x7f8c98009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 -- 172.21.15.111:0/3468471827 shutdown_connections 2024-03-14T12:57:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 --2- 172.21.15.111:0/3468471827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca80fbd80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 --2- 172.21.15.111:0/3468471827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 -- 172.21.15.111:0/3468471827 >> 172.21.15.111:0/3468471827 conn(0x7f8ca8072bc0 msgr2=0x7f8ca8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 -- 172.21.15.111:0/3468471827 shutdown_connections 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 -- 172.21.15.111:0/3468471827 wait complete. 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.977+0000 7f8cae526700 1 Processor -- start 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8cae526700 1 -- start start 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8cae526700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca81129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8cae526700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8cae526700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca8113550 con 0x7f8ca8102710 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8cae526700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca810c940 con 0x7f8ca8101d70 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59176/0 (socket says 172.21.15.111:59176) 2024-03-14T12:57:15.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 -- 172.21.15.111:0/2261454918 learned_addr learned my addr 172.21.15.111:0/2261454918 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca7fff700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca81129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 -- 172.21.15.111:0/2261454918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 msgr2=0x7f8ca81129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca81129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca77fe700 1 -- 172.21.15.111:0/2261454918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c98005040 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.978+0000 7f8ca7fff700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8ca77fe700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009180 tx=0x7f8c98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c9800e040 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c98004290 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c9801eea0 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca810cbc0 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.979+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca810d120 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.980+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca80fd4c0 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.980+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8c9801e420 con 0x7f8ca8102710 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.980+0000 7f8ca4ff9700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 0x7f8c9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:15.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.981+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8c9808ac80 con 0x7f8ca8102710 2024-03-14T12:57:15.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.981+0000 7f8ca7fff700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 0x7f8c9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:15.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.981+0000 7f8ca7fff700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 0x7f8c9006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0041d0 tx=0x7f8c9c009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:15.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:15.985+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8c9805ad00 con 0x7f8ca8102710 2024-03-14T12:57:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.166+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ca8108d10 con 0x7f8ca8102710 2024-03-14T12:57:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.166+0000 7f8ca4ff9700 1 -- 172.21.15.111:0/2261454918 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11118 (secure 0 0 0) 0x7f8c9805a890 con 0x7f8ca8102710 2024-03-14T12:57:16.166 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:16.166 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T12:56:51.546798+0000","last_up_change":"2024-03-14T12:56:50.540579+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6803","nonce":3771249398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6805","nonce":3771249398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6809","nonce":3771249398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6807","nonce":3771249398}]},"public_addr":"172.21.15.111:6803/3771249398","cluster_addr":"172.21.15.111:6805/3771249398","heartbeat_back_addr":"172.21.15.111:6809/3771249398","heartbeat_front_addr":"172.21.15.111:6807/3771249398","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6811","nonce":3746289562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6813","nonce":3746289562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6817","nonce":3746289562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6815","nonce":3746289562}]},"public_addr":"172.21.15.111:6811/3746289562","cluster_addr":"172.21.15.111:6813/3746289562","heartbeat_back_addr":"172.21.15.111:6817/3746289562","heartbeat_front_addr":"172.21.15.111:6815/3746289562","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6819","nonce":337068597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6821","nonce":337068597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6825","nonce":337068597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6823","nonce":337068597}]},"public_addr":"172.21.15.111:6819/337068597","cluster_addr":"172.21.15.111:6821/337068597","heartbeat_back_addr":"172.21.15.111:6825/337068597","heartbeat_front_addr":"172.21.15.111:6823/337068597","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6801","nonce":316104226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6803","nonce":316104226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6807","nonce":316104226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6805","nonce":316104226}]},"public_addr":"172.21.15.137:6801/316104226","cluster_addr":"172.21.15.137:6803/316104226","heartbeat_back_addr":"172.21.15.137:6807/316104226","heartbeat_front_addr":"172.21.15.137:6805/316104226","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6809","nonce":1988641735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6811","nonce":1988641735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6815","nonce":1988641735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6813","nonce":1988641735}]},"public_addr":"172.21.15.137:6809/1988641735","cluster_addr":"172.21.15.137:6811/1988641735","heartbeat_back_addr":"172.21.15.137:6815/1988641735","heartbeat_front_addr":"172.21.15.137:6813/1988641735","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6817","nonce":1979294971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6819","nonce":1979294971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6823","nonce":1979294971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6821","nonce":1979294971}]},"public_addr":"172.21.15.137:6817/1979294971","cluster_addr":"172.21.15.137:6819/1979294971","heartbeat_back_addr":"172.21.15.137:6823/1979294971","heartbeat_front_addr":"172.21.15.137:6821/1979294971","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 msgr2=0x7f8c9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 0x7f8c9006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0041d0 tx=0x7f8c9c009040 comp rx=0 tx=0).stop 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 msgr2=0x7f8ca8112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009180 tx=0x7f8c98009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 shutdown_connections 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ca8101d70 0x7f8ca81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c900692a0 0x7f8c9006b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 --2- 172.21.15.111:0/2261454918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ca8102710 0x7f8ca8112f00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.168+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 >> 172.21.15.111:0/2261454918 conn(0x7f8ca8072bc0 msgr2=0x7f8ca80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.169+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 shutdown_connections 2024-03-14T12:57:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:16.169+0000 7f8cae526700 1 -- 172.21.15.111:0/2261454918 wait complete. 2024-03-14T12:57:16.824 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:16 smithi111 ceph-mon[39688]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:16.824 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:16 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2261454918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:57:16.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-14T12:57:16.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-14T12:57:16.845 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-14T12:57:16.845 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-14T12:57:16.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-14T12:57:16.846 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-14T12:57:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:16 smithi137 ceph-mon[50336]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:16 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2261454918' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:57:17.221 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:17.223 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:17.518 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:17.521 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:17.523 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:18.305 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:18 smithi111 ceph-mon[39688]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:19.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:18 smithi137 ceph-mon[50336]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 -- 172.21.15.111:0/1364118103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780fb4a0 msgr2=0x7f46780fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 --2- 172.21.15.111:0/1364118103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780fb4a0 0x7f46780fb910 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4668004940 tx=0x7f4668009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 -- 172.21.15.111:0/1364118103 shutdown_connections 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 --2- 172.21.15.111:0/1364118103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46780fb4a0 0x7f46780fb910 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 --2- 172.21.15.111:0/1364118103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4678102e70 0x7f4678103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 -- 172.21.15.111:0/1364118103 >> 172.21.15.111:0/1364118103 conn(0x7f46780714c0 msgr2=0x7f46780718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 -- 172.21.15.111:0/1364118103 shutdown_connections 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.435+0000 7f468036a700 1 -- 172.21.15.111:0/1364118103 wait complete. 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 Processor -- start 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 -- start start 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46780fb4a0 0x7f4678113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4678114600 con 0x7f4678102e70 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467810d9f0 con 0x7f46780fb4a0 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59670/0 (socket says 172.21.15.111:59670) 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 -- 172.21.15.111:0/372243322 learned_addr learned my addr 172.21.15.111:0/372243322 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467e106700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46780fb4a0 0x7f4678113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 -- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46780fb4a0 msgr2=0x7f4678113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46780fb4a0 0x7f4678113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668005040 con 0x7f4678102e70 2024-03-14T12:57:20.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f467d905700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f4668009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f466800e040 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4668013420 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f466801c630 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467810dc70 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.436+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467810e230 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.437+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f467810e7a0 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4668013590 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 0x7f466406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4668087ec0 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f467e106700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 0x7f466406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 0x7f4664071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4664071a30 con 0x7f466406ef70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.438+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f466808a530 con 0x7f4678102e70 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.439+0000 7f467e106700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 0x7f466406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4674004f40 tx=0x7f467400b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:20.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.439+0000 7f467e907700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 0x7f4664071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.439+0000 7f467e907700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 0x7f4664071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:20.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.440+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== osd.3 v2:172.21.15.137:6800/316104226 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4664071a30 con 0x7f466406ef70 2024-03-14T12:57:20.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.454+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 --> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f467805e6c0 con 0x7f466406ef70 2024-03-14T12:57:20.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f466effd700 1 -- 172.21.15.111:0/372243322 <== osd.3 v2:172.21.15.137:6800/316104226 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f467805e6c0 con 0x7f466406ef70 2024-03-14T12:57:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 msgr2=0x7f4664071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 0x7f4664071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 msgr2=0x7f466406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 0x7f466406b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4674004f40 tx=0x7f467400b040 comp rx=0 tx=0).stop 2024-03-14T12:57:20.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 msgr2=0x7f4678113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4668000c00 tx=0x7f4668009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 shutdown_connections 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46780fb4a0 0x7f4678113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.137:6800/316104226,v1:172.21.15.137:6801/316104226] conn(0x7f466406ef70 0x7f4664071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f46640692a0 0x7f466406b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 --2- 172.21.15.111:0/372243322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4678102e70 0x7f4678113fb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 >> 172.21.15.111:0/372243322 conn(0x7f46780714c0 msgr2=0x7f46780fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 shutdown_connections 2024-03-14T12:57:20.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:20.455+0000 7f468036a700 1 -- 172.21.15.111:0/372243322 wait complete. 2024-03-14T12:57:20.854 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:20 smithi111 ceph-mon[39688]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:21.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:20 smithi137 ceph-mon[50336]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:22 smithi111 ceph-mon[39688]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:23.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:22 smithi137 ceph-mon[50336]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:24.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 -- 172.21.15.111:0/3175317566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 msgr2=0x7fd7f80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3175317566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f80fce10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004940 tx=0x7fd7ec009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:24.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 -- 172.21.15.111:0/3175317566 shutdown_connections 2024-03-14T12:57:24.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3175317566 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f80fce10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3175317566 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 0x7fd7f81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 -- 172.21.15.111:0/3175317566 >> 172.21.15.111:0/3175317566 conn(0x7fd7f80f8970 msgr2=0x7fd7f80fad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 -- 172.21.15.111:0/3175317566 shutdown_connections 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.983+0000 7fd7ff198700 1 -- 172.21.15.111:0/3175317566 wait complete. 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 Processor -- start 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 -- start start 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 0x7fd7f8113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f81146b0 con 0x7fd7f81037a0 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7ff198700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f810da10 con 0x7fd7f8102e00 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59686/0 (socket says 172.21.15.111:59686) 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7f7fff700 1 -- 172.21.15.111:0/3468329450 learned_addr learned my addr 172.21.15.111:0/3468329450 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.984+0000 7fd7fcf34700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 0x7fd7f8113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f7fff700 1 -- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 msgr2=0x7fd7f8113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f7fff700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 0x7fd7f8113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f7fff700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec005040 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f7fff700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0093b0 tx=0x7fd7ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec00e040 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f810dc90 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7ec013420 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec01c630 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f810e250 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.985+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7f810e770 con 0x7fd7f81037a0 2024-03-14T12:57:24.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.986+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd7ec0139e0 con 0x7fd7f81037a0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.986+0000 7fd7f57fa700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 0x7fd7e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd7ec088ad0 con 0x7fd7f81037a0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7fcf34700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 0x7fd7e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7f57fa700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 0x7fd7e0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7e0071a30 con 0x7fd7e006ef70 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7ec088eb0 con 0x7fd7f81037a0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7fcf34700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 0x7fd7e006b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0007580 tx=0x7fd7f0009780 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7fd735700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 0x7fd7e0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.987+0000 7fd7fd735700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 0x7fd7e0071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:23.988+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== osd.5 v2:172.21.15.137:6816/1979294971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd7e0071a30 con 0x7fd7e006ef70 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.003+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 --> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7f80fa360 con 0x7fd7e006ef70 2024-03-14T12:57:24.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.003+0000 7fd7f57fa700 1 -- 172.21.15.111:0/3468329450 <== osd.5 v2:172.21.15.137:6816/1979294971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd7f80fa360 con 0x7fd7e006ef70 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.003+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 msgr2=0x7fd7e0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.003+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 0x7fd7e0071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 msgr2=0x7fd7e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 0x7fd7e006b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0007580 tx=0x7fd7f0009780 comp rx=0 tx=0).stop 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 msgr2=0x7fd7f8113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec0093b0 tx=0x7fd7ec009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:24.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 shutdown_connections 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7f8102e00 0x7fd7f8113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.137:6816/1979294971,v1:172.21.15.137:6817/1979294971] conn(0x7fd7e006ef70 0x7fd7e0071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd7e00692a0 0x7fd7e006b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 --2- 172.21.15.111:0/3468329450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7f81037a0 0x7fd7f8113fd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 >> 172.21.15.111:0/3468329450 conn(0x7fd7f80f8970 msgr2=0x7fd7f80f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 shutdown_connections 2024-03-14T12:57:24.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.004+0000 7fd7ff198700 1 -- 172.21.15.111:0/3468329450 wait complete. 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 -- 172.21.15.111:0/2062279452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 msgr2=0x7f9474106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 --2- 172.21.15.111:0/2062279452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f9474106df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9468004940 tx=0x7f9468009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 -- 172.21.15.111:0/2062279452 shutdown_connections 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 --2- 172.21.15.111:0/2062279452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f9474106df0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 --2- 172.21.15.111:0/2062279452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94740fde10 0x7f94740fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 -- 172.21.15.111:0/2062279452 >> 172.21.15.111:0/2062279452 conn(0x7f94740f7880 msgr2=0x7f94740f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 -- 172.21.15.111:0/2062279452 shutdown_connections 2024-03-14T12:57:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.562+0000 7f947aa17700 1 -- 172.21.15.111:0/2062279452 wait complete. 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 Processor -- start 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 -- start start 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94740fde10 0x7f9474113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9474114920 con 0x7f94740fe7b0 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f947aa17700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947410dcf0 con 0x7f94740fde10 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f94737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f94737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59706/0 (socket says 172.21.15.111:59706) 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.563+0000 7f94737fe700 1 -- 172.21.15.111:0/967606170 learned_addr learned my addr 172.21.15.111:0/967606170 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f94737fe700 1 -- 172.21.15.111:0/967606170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94740fde10 msgr2=0x7f9474113d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f94737fe700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94740fde10 0x7f9474113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f94737fe700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468005040 con 0x7f94740fe7b0 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f94737fe700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9468009380 tx=0x7f9468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946800e040 con 0x7f94740fe7b0 2024-03-14T12:57:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947410df70 con 0x7f94740fe7b0 2024-03-14T12:57:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9468004290 con 0x7f94740fe7b0 2024-03-14T12:57:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f946801c3f0 con 0x7f94740fe7b0 2024-03-14T12:57:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.564+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947410e4b0 con 0x7f94740fe7b0 2024-03-14T12:57:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.565+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9474066770 con 0x7f94740fe7b0 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9468013790 con 0x7f94740fe7b0 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 0x7f945c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9468088bf0 con 0x7f94740fe7b0 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9473fff700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 0x7f945c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 0x7f945c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f945c071a30 con 0x7f945c06ef70 2024-03-14T12:57:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9468088fd0 con 0x7f94740fe7b0 2024-03-14T12:57:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9478fb4700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 0x7f945c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.566+0000 7f9473fff700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 0x7f945c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9464004ad0 tx=0x7f9464007d40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.567+0000 7f9478fb4700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 0x7f945c071380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.567+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== osd.1 v2:172.21.15.111:6810/3746289562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f945c071a30 con 0x7f945c06ef70 2024-03-14T12:57:24.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 --> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f947410ea50 con 0x7f945c06ef70 2024-03-14T12:57:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f9470ff9700 1 -- 172.21.15.111:0/967606170 <== osd.1 v2:172.21.15.111:6810/3746289562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f947410ea50 con 0x7f945c06ef70 2024-03-14T12:57:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 msgr2=0x7f945c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 0x7f945c071380 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 msgr2=0x7f945c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 0x7f945c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9464004ad0 tx=0x7f9464007d40 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 msgr2=0x7f94741142d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.582+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9468009380 tx=0x7f9468009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 shutdown_connections 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6810/3746289562,v1:172.21.15.111:6811/3746289562] conn(0x7f945c06ef70 0x7f945c071380 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94740fde10 0x7f9474113d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f945c0692a0 0x7f945c06b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 --2- 172.21.15.111:0/967606170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94740fe7b0 0x7f94741142d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 >> 172.21.15.111:0/967606170 conn(0x7f94740f7880 msgr2=0x7f94740f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 shutdown_connections 2024-03-14T12:57:24.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:24.583+0000 7f947aa17700 1 -- 172.21.15.111:0/967606170 wait complete. 2024-03-14T12:57:24.871 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:24 smithi111 ceph-mon[39688]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:24.890 INFO:teuthology.orchestra.run.smithi111.stdout:171798691850 2024-03-14T12:57:24.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-14T12:57:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:24 smithi137 ceph-mon[50336]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:25.827 INFO:teuthology.orchestra.run.smithi111.stdout:60129542175 2024-03-14T12:57:25.827 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-14T12:57:26.543 INFO:teuthology.orchestra.run.smithi111.stdout:115964117012 2024-03-14T12:57:26.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-14T12:57:26.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:26 smithi111 ceph-mon[39688]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:27.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:26 smithi137 ceph-mon[50336]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 -- 172.21.15.111:0/3788997640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 msgr2=0x7fc038106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/3788997640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc038106df0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc028004940 tx=0x7fc028009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 -- 172.21.15.111:0/3788997640 shutdown_connections 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/3788997640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc038106df0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/3788997640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc0380fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 -- 172.21.15.111:0/3788997640 >> 172.21.15.111:0/3788997640 conn(0x7fc0380f7880 msgr2=0x7fc0380f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 -- 172.21.15.111:0/3788997640 shutdown_connections 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.438+0000 7fc03cfcb700 1 -- 172.21.15.111:0/3788997640 wait complete. 2024-03-14T12:57:27.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 Processor -- start 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 -- start start 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc038113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc038114930 con 0x7fc0380fe7b0 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03810dd00 con 0x7fc0380fde10 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59732/0 (socket says 172.21.15.111:59732) 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 -- 172.21.15.111:0/503242798 learned_addr learned my addr 172.21.15.111:0/503242798 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:27.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03659c700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc038113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 -- 172.21.15.111:0/503242798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 msgr2=0x7fc038113da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc038113da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc028005040 con 0x7fc0380fe7b0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03659c700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc038113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc035d9b700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc028009380 tx=0x7fc028009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc02800e040 con 0x7fc0380fe7b0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc028004290 con 0x7fc0380fe7b0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03810df80 con 0x7fc0380fe7b0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc02801c3f0 con 0x7fc0380fe7b0 2024-03-14T12:57:27.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.439+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03810e4c0 con 0x7fc0380fe7b0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.440+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc038066770 con 0x7fc0380fe7b0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.441+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc028013790 con 0x7fc0380fe7b0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.441+0000 7fc026ffd700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 0x7fc02006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.441+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc028088bd0 con 0x7fc0380fe7b0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.441+0000 7fc03659c700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 0x7fc02006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.441+0000 7fc026ffd700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 0x7fc020071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.442+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc020071a30 con 0x7fc02006ef70 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.442+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc028088fb0 con 0x7fc0380fe7b0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.442+0000 7fc036d9d700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 0x7fc020071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.442+0000 7fc03659c700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 0x7fc02006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007fd0 tx=0x7fc02c009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.442+0000 7fc036d9d700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 0x7fc020071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.443+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== osd.2 v2:172.21.15.111:6818/337068597 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc020071a30 con 0x7fc02006ef70 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 --> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc03810e9e0 con 0x7fc02006ef70 2024-03-14T12:57:27.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc026ffd700 1 -- 172.21.15.111:0/503242798 <== osd.2 v2:172.21.15.111:6818/337068597 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc03810e9e0 con 0x7fc02006ef70 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 msgr2=0x7fc020071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 0x7fc020071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 msgr2=0x7fc02006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 0x7fc02006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007fd0 tx=0x7fc02c009040 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 msgr2=0x7fc0381142e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc028009380 tx=0x7fc028009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 shutdown_connections 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6818/337068597,v1:172.21.15.111:6819/337068597] conn(0x7fc02006ef70 0x7fc020071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0380fde10 0x7fc038113da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc0200692a0 0x7fc02006b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 --2- 172.21.15.111:0/503242798 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0380fe7b0 0x7fc0381142e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.458+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 >> 172.21.15.111:0/503242798 conn(0x7fc0380f7880 msgr2=0x7fc0380f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:27.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.459+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 shutdown_connections 2024-03-14T12:57:27.683 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:27.459+0000 7fc03cfcb700 1 -- 172.21.15.111:0/503242798 wait complete. 2024-03-14T12:57:28.330 INFO:teuthology.orchestra.run.smithi111.stdout:81604378649 2024-03-14T12:57:28.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-14T12:57:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.545+0000 7fd61500c700 1 -- 172.21.15.111:0/2762835389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 msgr2=0x7fd610106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.545+0000 7fd61500c700 1 --2- 172.21.15.111:0/2762835389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610106f00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd604004940 tx=0x7fd604009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 -- 172.21.15.111:0/2762835389 shutdown_connections 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 --2- 172.21.15.111:0/2762835389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610106f00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 --2- 172.21.15.111:0/2762835389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 0x7fd6100fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 -- 172.21.15.111:0/2762835389 >> 172.21.15.111:0/2762835389 conn(0x7fd610072bc0 msgr2=0x7fd610072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 -- 172.21.15.111:0/2762835389 shutdown_connections 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 -- 172.21.15.111:0/2762835389 wait complete. 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 Processor -- start 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.546+0000 7fd61500c700 1 -- start start 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd61500c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 0x7fd610113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd61500c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd61500c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd610114670 con 0x7fd6100fe830 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd61500c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61010da60 con 0x7fd6100fde90 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:28.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59746/0 (socket says 172.21.15.111:59746) 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 -- 172.21.15.111:0/3696102274 learned_addr learned my addr 172.21.15.111:0/3696102274 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60ed9d700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 0x7fd610113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 -- 172.21.15.111:0/3696102274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 msgr2=0x7fd610113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 0x7fd610113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604005040 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd60e59c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd60400b3b0 tx=0x7fd604009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60400e040 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd604013420 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd60401c5d0 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.547+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61010dce0 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.548+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61010e1d0 con 0x7fd6100fe830 2024-03-14T12:57:28.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.548+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd61010e740 con 0x7fd6100fe830 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd604013590 con 0x7fd6100fe830 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 0x7fd5f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd604088b80 con 0x7fd6100fe830 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 0x7fd5f8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5f8071a30 con 0x7fd5f806ef70 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.549+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd604088f00 con 0x7fd6100fe830 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.550+0000 7fd60ed9d700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 0x7fd5f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.550+0000 7fd60f59e700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 0x7fd5f8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.550+0000 7fd60f59e700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 0x7fd5f8071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.550+0000 7fd60ed9d700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 0x7fd5f806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd600004990 tx=0x7fd600009ee0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.550+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== osd.0 v2:172.21.15.111:6802/3771249398 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd5f8071a30 con 0x7fd5f806ef70 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.565+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 --> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd61010e900 con 0x7fd5f806ef70 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.565+0000 7fd5f77fe700 1 -- 172.21.15.111:0/3696102274 <== osd.0 v2:172.21.15.111:6802/3771249398 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd61010e900 con 0x7fd5f806ef70 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 msgr2=0x7fd5f8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:28.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 0x7fd5f8071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 msgr2=0x7fd5f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 0x7fd5f806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd600004990 tx=0x7fd600009ee0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 msgr2=0x7fd610114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd60400b3b0 tx=0x7fd604009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 shutdown_connections 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6802/3771249398,v1:172.21.15.111:6803/3771249398] conn(0x7fd5f806ef70 0x7fd5f8071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6100fde90 0x7fd610113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd5f80692a0 0x7fd5f806b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 --2- 172.21.15.111:0/3696102274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6100fe830 0x7fd610114020 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 >> 172.21.15.111:0/3696102274 conn(0x7fd610072bc0 msgr2=0x7fd610103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 shutdown_connections 2024-03-14T12:57:28.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:28.566+0000 7fd61500c700 1 -- 172.21.15.111:0/3696102274 wait complete. 2024-03-14T12:57:28.875 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:28.876 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:28 smithi111 ceph-mon[39688]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:29.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:28 smithi137 ceph-mon[50336]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:29.423 INFO:teuthology.orchestra.run.smithi111.stdout:38654705701 2024-03-14T12:57:29.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 -- 172.21.15.111:0/1027441945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc44102e10 msgr2=0x7efc441031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 --2- 172.21.15.111:0/1027441945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc44102e10 0x7efc441031e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc34004940 tx=0x7efc34009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 -- 172.21.15.111:0/1027441945 shutdown_connections 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 --2- 172.21.15.111:0/1027441945 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc441037b0 0x7efc440fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 --2- 172.21.15.111:0/1027441945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc44102e10 0x7efc441031e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 -- 172.21.15.111:0/1027441945 >> 172.21.15.111:0/1027441945 conn(0x7efc440f89a0 msgr2=0x7efc440fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 -- 172.21.15.111:0/1027441945 shutdown_connections 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 -- 172.21.15.111:0/1027441945 wait complete. 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.770+0000 7efc49553700 1 Processor -- start 2024-03-14T12:57:29.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 -- start start 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc44102e10 0x7efc44113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc44114680 con 0x7efc441037b0 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4410da70 con 0x7efc44102e10 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59758/0 (socket says 172.21.15.111:59758) 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 -- 172.21.15.111:0/2094225466 learned_addr learned my addr 172.21.15.111:0/2094225466 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc42ffd700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc44102e10 0x7efc44113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 -- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc44102e10 msgr2=0x7efc44113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc44102e10 0x7efc44113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc34005040 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc427fc700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc3800c800 tx=0x7efc38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc380106f0 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc38010d00 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4410dd50 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.771+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc380113a0 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.772+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4410e310 con 0x7efc441037b0 2024-03-14T12:57:29.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.772+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc4410e850 con 0x7efc441037b0 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efc38012400 con 0x7efc441037b0 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 0x7efc2c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc38087940 con 0x7efc441037b0 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc42ffd700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 0x7efc2c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 0x7efc2c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc2c071a30 con 0x7efc2c06ef70 2024-03-14T12:57:29.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.773+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc38087d20 con 0x7efc441037b0 2024-03-14T12:57:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.774+0000 7efc42ffd700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 0x7efc2c06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc34004f90 tx=0x7efc34009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.774+0000 7efc437fe700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 0x7efc2c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.774+0000 7efc437fe700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 0x7efc2c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:29.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.775+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== osd.4 v2:172.21.15.137:6808/1988641735 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7efc2c071a30 con 0x7efc2c06ef70 2024-03-14T12:57:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:29.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.793+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 --> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efc440fa4b0 con 0x7efc2c06ef70 2024-03-14T12:57:29.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.794+0000 7efc2b7fe700 1 -- 172.21.15.111:0/2094225466 <== osd.4 v2:172.21.15.137:6808/1988641735 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efc440fa4b0 con 0x7efc2c06ef70 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 msgr2=0x7efc2c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 0x7efc2c071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 msgr2=0x7efc2c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 0x7efc2c06b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc34004f90 tx=0x7efc34009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 msgr2=0x7efc44114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc3800c800 tx=0x7efc38004470 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 shutdown_connections 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:6808/1988641735,v1:172.21.15.137:6809/1988641735] conn(0x7efc2c06ef70 0x7efc2c071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc44102e10 0x7efc44113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efc2c0692a0 0x7efc2c06b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 --2- 172.21.15.111:0/2094225466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc441037b0 0x7efc44114030 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 >> 172.21.15.111:0/2094225466 conn(0x7efc440f89a0 msgr2=0x7efc440f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 shutdown_connections 2024-03-14T12:57:29.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:29.795+0000 7efc49553700 1 -- 172.21.15.111:0/2094225466 wait complete. 2024-03-14T12:57:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:30 smithi111 ceph-mon[39688]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:31.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:30 smithi137 ceph-mon[50336]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:31.639 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:31.639 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:32.874 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:32 smithi111 ceph-mon[39688]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:32.894 INFO:teuthology.orchestra.run.smithi111.stdout:150323855377 2024-03-14T12:57:32.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-14T12:57:33.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:32 smithi137 ceph-mon[50336]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:34 smithi111 ceph-mon[39688]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:34 smithi137 ceph-mon[50336]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 -- 172.21.15.111:0/749574303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 msgr2=0x7f0cfc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 --2- 172.21.15.111:0/749574303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0cec004940 tx=0x7f0cec009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 -- 172.21.15.111:0/749574303 shutdown_connections 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 --2- 172.21.15.111:0/749574303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cfc102710 0x7f0cfc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 --2- 172.21.15.111:0/749574303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102140 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 -- 172.21.15.111:0/749574303 >> 172.21.15.111:0/749574303 conn(0x7f0cfc072bc0 msgr2=0x7f0cfc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 -- 172.21.15.111:0/749574303 shutdown_connections 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.799+0000 7f0d0462c700 1 -- 172.21.15.111:0/749574303 wait complete. 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 Processor -- start 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 -- start start 2024-03-14T12:57:36.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cfc102710 0x7f0cfc103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc103750 con 0x7f0cfc101d70 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d0462c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc1038c0 con 0x7f0cfc102710 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d023c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d023c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60678/0 (socket says 172.21.15.111:60678) 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d023c8700 1 -- 172.21.15.111:0/2467368417 learned_addr learned my addr 172.21.15.111:0/2467368417 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d023c8700 1 -- 172.21.15.111:0/2467368417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cfc102710 msgr2=0x7f0cfc103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.800+0000 7f0d023c8700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cfc102710 0x7f0cfc103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.636 INFO:teuthology.orchestra.run.smithi111.stdout:60129542177 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0d023c8700 1 -- 172.21.15.111:0/2467368417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec005040 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0d023c8700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009360 tx=0x7f0cec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cec00e040 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cec013420 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc119e00 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cec01c5f0 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.801+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc11a2b0 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.802+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc0fd4c0 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.802+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0cec013590 con 0x7f0cfc101d70 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.802+0000 7f0cf2ffd700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:36.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.803+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0cec088080 con 0x7f0cfc101d70 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.803+0000 7f0d01bc7700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.803+0000 7f0d01bc7700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 0x7f0ce806b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8006db0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.807+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f0cec0546f0 con 0x7f0cfc101d70 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.002+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0cfc05f5f0 con 0x7f0cfc101d70 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.003+0000 7f0cf2ffd700 1 -- 172.21.15.111:0/2467368417 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0cec057d10 con 0x7f0cfc101d70 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 msgr2=0x7f0ce806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 0x7f0ce806b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8006db0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 msgr2=0x7f0cfc102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009360 tx=0x7f0cec009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 shutdown_connections 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cfc101d70 0x7f0cfc102bd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 --2- 172.21.15.111:0/2467368417 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cfc102710 0x7f0cfc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:36.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.005+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 >> 172.21.15.111:0/2467368417 conn(0x7f0cfc072bc0 msgr2=0x7f0cfc0658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:36.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.006+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 shutdown_connections 2024-03-14T12:57:36.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.006+0000 7f0d0462c700 1 -- 172.21.15.111:0/2467368417 wait complete. 2024-03-14T12:57:36.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:36 smithi111 ceph-mon[39688]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:36.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:36 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2467368417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-14T12:57:36.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:36 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3539647667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-14T12:57:37.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:36 smithi137 ceph-mon[50336]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:37.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:36 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2467368417' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-14T12:57:37.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:36 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3539647667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-14T12:57:37.262 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-03-14T12:57:37.263 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:37.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 -- 172.21.15.111:0/52996759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 msgr2=0x7f3548106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:37.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 --2- 172.21.15.111:0/52996759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548106f00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f353c004940 tx=0x7f353c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:37.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 -- 172.21.15.111:0/52996759 shutdown_connections 2024-03-14T12:57:37.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 --2- 172.21.15.111:0/52996759 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548106f00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 --2- 172.21.15.111:0/52996759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 0x7f35480fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.906+0000 7f354d67e700 1 -- 172.21.15.111:0/52996759 >> 172.21.15.111:0/52996759 conn(0x7f3548072bc0 msgr2=0x7f3548072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.907+0000 7f354d67e700 1 -- 172.21.15.111:0/52996759 shutdown_connections 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.907+0000 7f354d67e700 1 -- 172.21.15.111:0/52996759 wait complete. 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.907+0000 7f354d67e700 1 Processor -- start 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.907+0000 7f354d67e700 1 -- start start 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 0x7f3548113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3548114670 con 0x7f35480fe830 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354810da60 con 0x7f35480fde90 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60706/0 (socket says 172.21.15.111:60706) 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 -- 172.21.15.111:0/3539647667 learned_addr learned my addr 172.21.15.111:0/3539647667 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f3546ffd700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 0x7f3548113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 -- 172.21.15.111:0/3539647667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 msgr2=0x7f3548113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:37.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 0x7f3548113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 -- 172.21.15.111:0/3539647667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c005040 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f35467fc700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f353c00b3b0 tx=0x7f353c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f353c00e040 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f353c013420 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f353c01c5d0 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354810dce0 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.908+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354810e220 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.909+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3548065e30 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.910+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f353c013590 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.911+0000 7f352f7fe700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 0x7f353006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.911+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f353c087fa0 con 0x7f35480fe830 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.911+0000 7f3546ffd700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 0x7f353006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.911+0000 7f3546ffd700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 0x7f353006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3538004990 tx=0x7f3538009ee0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:37.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:35.915+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f353c0580a0 con 0x7f35480fe830 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stdout:115964117015 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.121+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3548109dd0 con 0x7f35480fe830 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.122+0000 7f352f7fe700 1 -- 172.21.15.111:0/3539647667 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f353c057c30 con 0x7f35480fe830 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 msgr2=0x7f353006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 0x7f353006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3538004990 tx=0x7f3538009ee0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 msgr2=0x7f3548114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f353c00b3b0 tx=0x7f353c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 shutdown_connections 2024-03-14T12:57:37.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35480fde90 0x7f3548113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f35300692a0 0x7f353006b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 --2- 172.21.15.111:0/3539647667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35480fe830 0x7f3548114020 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:37.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 >> 172.21.15.111:0/3539647667 conn(0x7f3548072bc0 msgr2=0x7f3548103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:37.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 shutdown_connections 2024-03-14T12:57:37.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.124+0000 7f354d67e700 1 -- 172.21.15.111:0/3539647667 wait complete. 2024-03-14T12:57:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:37 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1708646195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-14T12:57:37.985 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117015 for osd.3 2024-03-14T12:57:37.985 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:37 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1708646195' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-14T12:57:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 -- 172.21.15.111:0/318700257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fe7b0 msgr2=0x7f1140106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 --2- 172.21.15.111:0/318700257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fe7b0 0x7f1140106e80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f112c004940 tx=0x7f112c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 -- 172.21.15.111:0/318700257 shutdown_connections 2024-03-14T12:57:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 --2- 172.21.15.111:0/318700257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fe7b0 0x7f1140106e80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 --2- 172.21.15.111:0/318700257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fde10 0x7f11400fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 -- 172.21.15.111:0/318700257 >> 172.21.15.111:0/318700257 conn(0x7f11400f78d0 msgr2=0x7f11400f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 -- 172.21.15.111:0/318700257 shutdown_connections 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.636+0000 7f1147107700 1 -- 172.21.15.111:0/318700257 wait complete. 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 Processor -- start 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 -- start start 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 0x7f11400fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400fc7a0 con 0x7f11400fde10 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1147107700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11400fc8e0 con 0x7f11400fe7b0 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1144ea3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1144ea3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60718/0 (socket says 172.21.15.111:60718) 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1144ea3700 1 -- 172.21.15.111:0/1708646195 learned_addr learned my addr 172.21.15.111:0/1708646195 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f113ffff700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 0x7f11400fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.637+0000 7f1144ea3700 1 -- 172.21.15.111:0/1708646195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 msgr2=0x7f11400fc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:38.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f1144ea3700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 0x7f11400fc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f1144ea3700 1 -- 172.21.15.111:0/1708646195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c005040 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f113ffff700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 0x7f11400fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f1144ea3700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f113400eb40 tx=0x7f113400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1134007cc0 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f113400f040 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11400fcb30 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1134009570 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.638+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114011a100 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.639+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11400666f0 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.639+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f11340096d0 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.640+0000 7f113d7fa700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 0x7f113006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.640+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1134086ac0 con 0x7f11400fde10 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.640+0000 7f113ffff700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 0x7f113006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:38.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.640+0000 7f113ffff700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 0x7f113006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f112c004560 tx=0x7f112c009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.644+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f113404dd70 con 0x7f11400fde10 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.832+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f114004bf60 con 0x7f11400fde10 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.832+0000 7f113d7fa700 1 -- 172.21.15.111:0/1708646195 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1134053160 con 0x7f11400fde10 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stdout:171798691852 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.834+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 msgr2=0x7f113006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.834+0000 7f1147107700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 0x7f113006b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f112c004560 tx=0x7f112c009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:38.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.834+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 msgr2=0x7f11400fbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.834+0000 7f1147107700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f113400eb40 tx=0x7f113400b040 comp rx=0 tx=0).stop 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 shutdown_connections 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f11300692a0 0x7f113006b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11400fde10 0x7f11400fbcd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 --2- 172.21.15.111:0/1708646195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11400fe7b0 0x7f11400fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 >> 172.21.15.111:0/1708646195 conn(0x7f11400f78d0 msgr2=0x7f11400f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 shutdown_connections 2024-03-14T12:57:38.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:36.835+0000 7f1147107700 1 -- 172.21.15.111:0/1708646195 wait complete. 2024-03-14T12:57:38.735 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:38 smithi111 ceph-mon[39688]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:39.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:38 smithi137 ceph-mon[50336]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:39.168 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:39.191 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691852 for osd.5 2024-03-14T12:57:39.191 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 -- 172.21.15.111:0/2461427191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344102de0 msgr2=0x7f43441031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 --2- 172.21.15.111:0/2461427191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344102de0 0x7f43441031b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4334004940 tx=0x7f4334009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 -- 172.21.15.111:0/2461427191 shutdown_connections 2024-03-14T12:57:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 --2- 172.21.15.111:0/2461427191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344103780 0x7f43440fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 --2- 172.21.15.111:0/2461427191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344102de0 0x7f43441031b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 -- 172.21.15.111:0/2461427191 >> 172.21.15.111:0/2461427191 conn(0x7f43440f8950 msgr2=0x7f43440fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 -- 172.21.15.111:0/2461427191 shutdown_connections 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.763+0000 7f434c19f700 1 -- 172.21.15.111:0/2461427191 wait complete. 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 Processor -- start 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 -- start start 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 0x7f43441138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4344114440 con 0x7f4344103780 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434410da40 con 0x7f4344102de0 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60738/0 (socket says 172.21.15.111:60738) 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 -- 172.21.15.111:0/818386172 learned_addr learned my addr 172.21.15.111:0/818386172 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:39.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f4349f3b700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 0x7f43441138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 -- 172.21.15.111:0/818386172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 msgr2=0x7f43441138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 0x7f43441138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 -- 172.21.15.111:0/818386172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4334005040 con 0x7f4344103780 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f4349f3b700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 0x7f43441138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434973a700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f434000c800 tx=0x7f4340004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f434000a760 con 0x7f4344103780 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434410dd20 con 0x7f4344103780 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f434000ad70 con 0x7f4344103780 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.764+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43400113a0 con 0x7f4344103780 2024-03-14T12:57:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.765+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434410e2e0 con 0x7f4344103780 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.765+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43440fe530 con 0x7f4344103780 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.766+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4340012400 con 0x7f4344103780 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.766+0000 7f433a7fc700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 0x7f433006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.767+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4340087a50 con 0x7f4344103780 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.767+0000 7f4349f3b700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 0x7f433006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.767+0000 7f4349f3b700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 0x7f433006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4334004560 tx=0x7f4334009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:39.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.770+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f4340057bf0 con 0x7f4344103780 2024-03-14T12:57:39.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.959+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f434404c710 con 0x7f4344103780 2024-03-14T12:57:39.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.959+0000 7f433a7fc700 1 -- 172.21.15.111:0/818386172 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4340057780 con 0x7f4344103780 2024-03-14T12:57:39.960 INFO:teuthology.orchestra.run.smithi111.stdout:81604378652 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 msgr2=0x7f433006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 0x7f433006b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4334004560 tx=0x7f4334009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 msgr2=0x7f4344113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f434000c800 tx=0x7f4340004470 comp rx=0 tx=0).stop 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 shutdown_connections 2024-03-14T12:57:39.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4344102de0 0x7f43441138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f43300692a0 0x7f433006b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 --2- 172.21.15.111:0/818386172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4344103780 0x7f4344113df0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.961+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 >> 172.21.15.111:0/818386172 conn(0x7f43440f8950 msgr2=0x7f43440f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.962+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 shutdown_connections 2024-03-14T12:57:39.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:39.962+0000 7f434c19f700 1 -- 172.21.15.111:0/818386172 wait complete. 2024-03-14T12:57:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:40 smithi111 ceph-mon[39688]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:40 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/818386172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-14T12:57:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:40 smithi137 ceph-mon[50336]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:41.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:40 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/818386172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-14T12:57:41.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378652 for osd.2 2024-03-14T12:57:41.011 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 -- 172.21.15.111:0/1239456381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 msgr2=0x7f29a40fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 --2- 172.21.15.111:0/1239456381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a40fbd20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2998004940 tx=0x7f2998009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 -- 172.21.15.111:0/1239456381 shutdown_connections 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 --2- 172.21.15.111:0/1239456381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a40fbd20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 --2- 172.21.15.111:0/1239456381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a41020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 -- 172.21.15.111:0/1239456381 >> 172.21.15.111:0/1239456381 conn(0x7f29a40f7880 msgr2=0x7f29a40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 -- 172.21.15.111:0/1239456381 shutdown_connections 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.508+0000 7f29a895c700 1 -- 172.21.15.111:0/1239456381 wait complete. 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 Processor -- start 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 -- start start 2024-03-14T12:57:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a4102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4103640 con 0x7f29a41026b0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a895c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4103780 con 0x7f29a4101d10 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59988/0 (socket says 172.21.15.111:59988) 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a1d9b700 1 -- 172.21.15.111:0/3389938081 learned_addr learned my addr 172.21.15.111:0/3389938081 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a1d9b700 1 -- 172.21.15.111:0/3389938081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 msgr2=0x7f29a4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.509+0000 7f29a259c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a4102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a1d9b700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a4102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a1d9b700 1 -- 172.21.15.111:0/3389938081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998005040 con 0x7f29a41026b0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a259c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a4102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a1d9b700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2998009380 tx=0x7f2998009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299800e040 con 0x7f29a41026b0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a41039d0 con 0x7f29a41026b0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2998013420 con 0x7f29a41026b0 2024-03-14T12:57:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f299801c630 con 0x7f29a41026b0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.510+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a411a550 con 0x7f29a41026b0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.511+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a40fd460 con 0x7f29a41026b0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.511+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f29980139e0 con 0x7f29a41026b0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.512+0000 7f2992ffd700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 0x7f298c06b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.512+0000 7f29a259c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 0x7f298c06b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.512+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29980880b0 con 0x7f29a41026b0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.512+0000 7f29a259c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 0x7f298c06b7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2994007fd0 tx=0x7f2994009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:41.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.516+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f2998054680 con 0x7f29a41026b0 2024-03-14T12:57:41.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.701+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f29a404bed0 con 0x7f29a41026b0 2024-03-14T12:57:41.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.702+0000 7f2992ffd700 1 -- 172.21.15.111:0/3389938081 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2998057ca0 con 0x7f29a41026b0 2024-03-14T12:57:41.703 INFO:teuthology.orchestra.run.smithi111.stdout:38654705704 2024-03-14T12:57:41.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 msgr2=0x7f298c06b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:41.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 0x7f298c06b7a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2994007fd0 tx=0x7f2994009040 comp rx=0 tx=0).stop 2024-03-14T12:57:41.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 msgr2=0x7f29a41030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:41.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2998009380 tx=0x7f2998009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:41.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 shutdown_connections 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29a4101d10 0x7f29a4102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f298c0692f0 0x7f298c06b7a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 --2- 172.21.15.111:0/3389938081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a41026b0 0x7f29a41030b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 >> 172.21.15.111:0/3389938081 conn(0x7f29a40f7880 msgr2=0x7f29a40f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 shutdown_connections 2024-03-14T12:57:41.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:41.704+0000 7f29a895c700 1 -- 172.21.15.111:0/3389938081 wait complete. 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- 172.21.15.111:0/948886622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 msgr2=0x7f4c6c0fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 --2- 172.21.15.111:0/948886622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c0fbd40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4c68004940 tx=0x7f4c68009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- 172.21.15.111:0/948886622 shutdown_connections 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 --2- 172.21.15.111:0/948886622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c0fbd40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 --2- 172.21.15.111:0/948886622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 0x7f4c6c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- 172.21.15.111:0/948886622 >> 172.21.15.111:0/948886622 conn(0x7f4c6c0f7860 msgr2=0x7f4c6c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:42.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- 172.21.15.111:0/948886622 shutdown_connections 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- 172.21.15.111:0/948886622 wait complete. 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 Processor -- start 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.604+0000 7f4c7410f700 1 -- start start 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c7410f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 0x7f4c6c1138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c7410f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c7410f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c1144f0 con 0x7f4c6c1026d0 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c7410f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6c10da60 con 0x7f4c6c101d30 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60008/0 (socket says 172.21.15.111:60008) 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 -- 172.21.15.111:0/3121645473 learned_addr learned my addr 172.21.15.111:0/3121645473 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c71eab700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 0x7f4c6c1138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 -- 172.21.15.111:0/3121645473 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 msgr2=0x7f4c6c1138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:42.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 0x7f4c6c1138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 -- 172.21.15.111:0/3121645473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68005040 con 0x7f4c6c1026d0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c716aa700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009380 tx=0x7f4c68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c6800e040 con 0x7f4c6c1026d0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6c10dce0 con 0x7f4c6c1026d0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c68013420 con 0x7f4c6c1026d0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.605+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c6801c630 con 0x7f4c6c1026d0 2024-03-14T12:57:42.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.606+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6c10e2a0 con 0x7f4c6c1026d0 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.606+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6c0fd480 con 0x7f4c6c1026d0 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.607+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4c680139e0 con 0x7f4c6c1026d0 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.607+0000 7f4c627fc700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 0x7f4c5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.607+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4c68088010 con 0x7f4c6c1026d0 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.607+0000 7f4c71eab700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 0x7f4c5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:42.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.608+0000 7f4c71eab700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 0x7f4c5806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c006fd0 tx=0x7f4c5c008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:42.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.612+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f4c68058110 con 0x7f4c6c1026d0 2024-03-14T12:57:42.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.797+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4c6c04c710 con 0x7f4c6c1026d0 2024-03-14T12:57:42.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.798+0000 7f4c627fc700 1 -- 172.21.15.111:0/3121645473 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4c68057ca0 con 0x7f4c6c1026d0 2024-03-14T12:57:42.798 INFO:teuthology.orchestra.run.smithi111.stdout:150323855379 2024-03-14T12:57:42.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.799+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 msgr2=0x7f4c5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:42.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.799+0000 7f4c7410f700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 0x7f4c5806b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c006fd0 tx=0x7f4c5c008040 comp rx=0 tx=0).stop 2024-03-14T12:57:42.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 msgr2=0x7f4c6c113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:42.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009380 tx=0x7f4c68009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:42.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 shutdown_connections 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c6c101d30 0x7f4c6c1138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4c580692a0 0x7f4c5806b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 --2- 172.21.15.111:0/3121645473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c6c1026d0 0x7f4c6c113e10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 >> 172.21.15.111:0/3121645473 conn(0x7f4c6c0f7860 msgr2=0x7f4c6c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 shutdown_connections 2024-03-14T12:57:42.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:42.800+0000 7f4c7410f700 1 -- 172.21.15.111:0/3121645473 wait complete. 2024-03-14T12:57:42.919 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:42 smithi111 ceph-mon[39688]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:42.919 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:42 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3389938081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-14T12:57:42.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705704 for osd.0 2024-03-14T12:57:42.939 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:42 smithi137 ceph-mon[50336]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:42 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3389938081' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-14T12:57:43.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855377 got 150323855379 for osd.4 2024-03-14T12:57:43.592 DEBUG:teuthology.parallel:result is None 2024-03-14T12:57:43.592 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-14T12:57:43.592 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-14T12:57:43.861 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:43 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3121645473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-14T12:57:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:43 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3121645473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-14T12:57:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:45.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:44 smithi137 ceph-mon[50336]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:44 smithi111 ceph-mon[39688]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:45.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 -- 172.21.15.111:0/2095305764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734101d70 msgr2=0x7f2734102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:45.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 --2- 172.21.15.111:0/2095305764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734101d70 0x7f2734102140 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2724004940 tx=0x7f2724009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:45.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 -- 172.21.15.111:0/2095305764 shutdown_connections 2024-03-14T12:57:45.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 --2- 172.21.15.111:0/2095305764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734102710 0x7f27340fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 --2- 172.21.15.111:0/2095305764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734101d70 0x7f2734102140 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 -- 172.21.15.111:0/2095305764 >> 172.21.15.111:0/2095305764 conn(0x7f2734072bc0 msgr2=0x7f2734072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 -- 172.21.15.111:0/2095305764 shutdown_connections 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.755+0000 7f2739ed0700 1 -- 172.21.15.111:0/2095305764 wait complete. 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 Processor -- start 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 -- start start 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 0x7f2734113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2734114610 con 0x7f2734102710 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2739ed0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273410da00 con 0x7f2734101d70 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60032/0 (socket says 172.21.15.111:60032) 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 -- 172.21.15.111:0/1272752810 learned_addr learned my addr 172.21.15.111:0/1272752810 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:45.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f27337fe700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 0x7f2734113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 -- 172.21.15.111:0/1272752810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 msgr2=0x7f2734113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 0x7f2734113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f2732ffd700 1 -- 172.21.15.111:0/1272752810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724005040 con 0x7f2734102710 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.756+0000 7f27337fe700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 0x7f2734113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f2732ffd700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f272800c7e0 tx=0x7f2728007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2728012810 con 0x7f2734102710 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273410dce0 con 0x7f2734102710 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2728012e20 con 0x7f2734102710 2024-03-14T12:57:45.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f272800a310 con 0x7f2734102710 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.757+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f273410e2a0 con 0x7f2734102710 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.758+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27340fd4c0 con 0x7f2734102710 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.758+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2728011380 con 0x7f2734102710 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.758+0000 7f271bfff700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 0x7f271c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.759+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2728087260 con 0x7f2734102710 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.759+0000 7f27337fe700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 0x7f271c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:45.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.759+0000 7f27337fe700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 0x7f271c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2724004560 tx=0x7f2724009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:45.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.763+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f27280572e0 con 0x7f2734102710 2024-03-14T12:57:45.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.943+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2734113750 con 0x7f271c0692a0 2024-03-14T12:57:45.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.944+0000 7f271bfff700 1 -- 172.21.15.111:0/1272752810 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28225 (secure 0 0 0) 0x7f2734113750 con 0x7f271c0692a0 2024-03-14T12:57:45.944 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:45.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 msgr2=0x7f271c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:45.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 0x7f271c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2724004560 tx=0x7f2724009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:45.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 msgr2=0x7f2734113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:45.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f272800c7e0 tx=0x7f2728007f90 comp rx=0 tx=0).stop 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 shutdown_connections 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2734101d70 0x7f2734113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f271c0692a0 0x7f271c06b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 --2- 172.21.15.111:0/1272752810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2734102710 0x7f2734113fc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.946+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 >> 172.21.15.111:0/1272752810 conn(0x7f2734072bc0 msgr2=0x7f27340659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.947+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 shutdown_connections 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:45.947+0000 7f2739ed0700 1 -- 172.21.15.111:0/1272752810 wait complete. 2024-03-14T12:57:45.947 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-03-14T12:57:46.546 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-03-14T12:57:45.226227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002193"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-14T12:56:50.547906+0000","last_change":"2024-03-14T12:56:23.548650+0000","last_active":"2024-03-14T12:56:50.547906+0000","last_peered":"2024-03-14T12:56:50.547906+0000","last_clean":"2024-03-14T12:56:50.547906+0000","last_became_active":"2024-03-14T12:56:23.548223+0000","last_became_peered":"2024-03-14T12:56:23.548223+0000","last_unstale":"2024-03-14T12:56:50.547906+0000","last_undegraded":"2024-03-14T12:56:50.547906+0000","last_fullsized":"2024-03-14T12:56:50.547906+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T12:55:33.885196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T12:55:33.885196+0000","last_clean_scrub_stamp":"2024-03-14T12:55:33.885196+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:58:53.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.442}]},{"osd":2,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]}]},{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.436},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Mar 14 12:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35899999999999999,"15min":0.36799999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.437,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.373,"15min":0.38200000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.46300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.315}]},{"osd":1,"last update":"Thu Mar 14 12:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.38500000000000001,"15min":0.39300000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48199999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.44},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 12:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.45300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28599999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.49099999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Mar 14 12:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.442,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.308}]},{"osd":3,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Mar 14 12:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.33500000000000002,"15min":0.32800000000000001},"min":{"1min":0.24099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Mar 14 12:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36399999999999999,"15min":0.36799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.50600000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Mar 14 12:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.375}]},{"osd":4,"last update":"Thu Mar 14 12:57:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T12:57:46.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-14T12:57:46.813 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:46.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:46 smithi111 ceph-mon[39688]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:46 smithi137 ceph-mon[50336]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:47 smithi111 ceph-mon[39688]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T12:57:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:47 smithi137 ceph-mon[50336]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- 172.21.15.111:0/191454248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 msgr2=0x7f5dcc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/191454248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc0fbd80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0004940 tx=0x7f5dc0009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- 172.21.15.111:0/191454248 shutdown_connections 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/191454248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc0fbd80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/191454248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 0x7f5dcc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- 172.21.15.111:0/191454248 >> 172.21.15.111:0/191454248 conn(0x7f5dcc072bc0 msgr2=0x7f5dcc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- 172.21.15.111:0/191454248 shutdown_connections 2024-03-14T12:57:48.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- 172.21.15.111:0/191454248 wait complete. 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 Processor -- start 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.569+0000 7f5dd25c5700 1 -- start start 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 0x7f5dcc10ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc10f6b0 con 0x7f5dcc102710 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc10f7f0 con 0x7f5dcc101d70 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60054/0 (socket says 172.21.15.111:60054) 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 -- 172.21.15.111:0/4129135628 learned_addr learned my addr 172.21.15.111:0/4129135628 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcbfff700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 0x7f5dcc10ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 -- 172.21.15.111:0/4129135628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 msgr2=0x7f5dcc10ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 0x7f5dcc10ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 -- 172.21.15.111:0/4129135628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc0005040 con 0x7f5dcc102710 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dcb7fe700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009380 tx=0x7f5dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:48.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dc000e040 con 0x7f5dcc102710 2024-03-14T12:57:48.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc117b00 con 0x7f5dcc102710 2024-03-14T12:57:48.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5dc0013420 con 0x7f5dcc102710 2024-03-14T12:57:48.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dc001c630 con 0x7f5dcc102710 2024-03-14T12:57:48.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.570+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc118090 con 0x7f5dcc102710 2024-03-14T12:57:48.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.571+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc0ff080 con 0x7f5dcc102710 2024-03-14T12:57:48.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.572+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5dc00139e0 con 0x7f5dcc102710 2024-03-14T12:57:48.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.572+0000 7f5dc8ff9700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 0x7f5db406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:48.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.572+0000 7f5dcbfff700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 0x7f5db406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:48.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.572+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5dc0087fe0 con 0x7f5dcc102710 2024-03-14T12:57:48.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.572+0000 7f5dcbfff700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 0x7f5db406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc006fd0 tx=0x7f5dbc008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:48.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.576+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f5dc0054650 con 0x7f5dcc102710 2024-03-14T12:57:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.757+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5dcc065e50 con 0x7f5db40692a0 2024-03-14T12:57:48.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.758+0000 7f5dc8ff9700 1 -- 172.21.15.111:0/4129135628 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28224 (secure 0 0 0) 0x7f5dcc065e50 con 0x7f5db40692a0 2024-03-14T12:57:48.758 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 msgr2=0x7f5db406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 0x7f5db406b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc006fd0 tx=0x7f5dbc008040 comp rx=0 tx=0).stop 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 msgr2=0x7f5dcc10efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009380 tx=0x7f5dc0009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 shutdown_connections 2024-03-14T12:57:48.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dcc101d70 0x7f5dcc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5db40692a0 0x7f5db406b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 --2- 172.21.15.111:0/4129135628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dcc102710 0x7f5dcc10efd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.760+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 >> 172.21.15.111:0/4129135628 conn(0x7f5dcc072bc0 msgr2=0x7f5dcc0fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.761+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 shutdown_connections 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:48.761+0000 7f5dd25c5700 1 -- 172.21.15.111:0/4129135628 wait complete. 2024-03-14T12:57:48.761 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-03-14T12:57:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:48 smithi137 ceph-mon[50336]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:49.101 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:48 smithi111 ceph-mon[39688]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:49.436 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-14T12:57:47.226499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002118"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-14T12:56:50.547906+0000","last_change":"2024-03-14T12:56:23.548650+0000","last_active":"2024-03-14T12:56:50.547906+0000","last_peered":"2024-03-14T12:56:50.547906+0000","last_clean":"2024-03-14T12:56:50.547906+0000","last_became_active":"2024-03-14T12:56:23.548223+0000","last_became_peered":"2024-03-14T12:56:23.548223+0000","last_unstale":"2024-03-14T12:56:50.547906+0000","last_undegraded":"2024-03-14T12:56:50.547906+0000","last_fullsized":"2024-03-14T12:56:50.547906+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-14T12:55:33.885196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-14T12:55:33.885196+0000","last_clean_scrub_stamp":"2024-03-14T12:55:33.885196+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-15T13:58:53.169720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]}]},{"osd":4,"up_from":35,"seq":150323855380,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.442}]},{"osd":2,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]}]},{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.44600000000000001}]},{"osd":1,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.436},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.47499999999999998}]},{"osd":4,"last update":"Thu Mar 14 12:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.372}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.35899999999999999,"15min":0.36799999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.437,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.373,"15min":0.38200000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.46300000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.315}]},{"osd":1,"last update":"Thu Mar 14 12:57:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.0939999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.38500000000000001,"15min":0.39300000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.48199999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.46700000000000003}]},{"osd":4,"last update":"Thu Mar 14 12:57:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.44},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 14 12:57:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"min":{"1min":0.23000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.45300000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28599999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.49099999999999999,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Thu Mar 14 12:57:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.36699999999999999,"15min":0.372},"min":{"1min":0.27800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.442,"5min":0.47699999999999998,"15min":0.47699999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.308}]},{"osd":3,"last update":"Thu Mar 14 12:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.55800000000000005}]},{"osd":4,"last update":"Thu Mar 14 12:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 14 12:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.33500000000000002,"15min":0.32800000000000001},"min":{"1min":0.24099999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.27800000000000002}]},{"osd":2,"last update":"Thu Mar 14 12:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36399999999999999,"15min":0.36799999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.36599999999999999,"15min":0.372},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.50600000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Mar 14 12:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.375}]},{"osd":4,"last update":"Thu Mar 14 12:57:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-14T12:57:49.437 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-14T12:57:49.437 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-14T12:57:49.437 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-14T12:57:49.437 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-14T12:57:49.709 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:49.930 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:49 smithi111 ceph-mon[39688]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T12:57:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:49 smithi137 ceph-mon[50336]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-14T12:57:50.924 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:50 smithi111 ceph-mon[39688]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:50 smithi137 ceph-mon[50336]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- 172.21.15.111:0/2265670696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c102710 msgr2=0x7f390c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 --2- 172.21.15.111:0/2265670696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c102710 0x7f390c0fbd80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f38fc004940 tx=0x7f38fc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- 172.21.15.111:0/2265670696 shutdown_connections 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 --2- 172.21.15.111:0/2265670696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c102710 0x7f390c0fbd80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 --2- 172.21.15.111:0/2265670696 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c101d70 0x7f390c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- 172.21.15.111:0/2265670696 >> 172.21.15.111:0/2265670696 conn(0x7f390c072bc0 msgr2=0x7f390c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- 172.21.15.111:0/2265670696 shutdown_connections 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- 172.21.15.111:0/2265670696 wait complete. 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 Processor -- start 2024-03-14T12:57:51.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.505+0000 7f3912512700 1 -- start start 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 0x7f390c112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c113550 con 0x7f390c101d70 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c10c940 con 0x7f390c102710 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:51.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33352/0 (socket says 172.21.15.111:33352) 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 -- 172.21.15.111:0/3662089363 learned_addr learned my addr 172.21.15.111:0/3662089363 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390b7fe700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 0x7f390c112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 -- 172.21.15.111:0/3662089363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 msgr2=0x7f390c112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 0x7f390c112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 -- 172.21.15.111:0/3662089363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc005040 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390b7fe700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 0x7f390c112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f390bfff700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f38f4002a50 tx=0x7f38f40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f4007cc0 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38f400f040 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c10cc20 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f4009540 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.506+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c10d1e0 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c0ff080 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f38f40096a0 con 0x7f390c101d70 2024-03-14T12:57:51.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f3908ff9700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f38f4086c60 con 0x7f390c101d70 2024-03-14T12:57:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f390b7fe700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.508+0000 7f390b7fe700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b3b0 tx=0x7f38fc009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.512+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f38f4056ce0 con 0x7f390c101d70 2024-03-14T12:57:51.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.734+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f390c10e010 con 0x7f390c101d70 2024-03-14T12:57:51.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.735+0000 7f3908ff9700 1 -- 172.21.15.111:0/3662089363 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f38f4056870 con 0x7f390c101d70 2024-03-14T12:57:51.735 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:57:51.735 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-14T12:57:51.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 msgr2=0x7f38f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:51.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b3b0 tx=0x7f38fc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:51.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 msgr2=0x7f390c1129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:51.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f38f4002a50 tx=0x7f38f40046b0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 shutdown_connections 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f38f80692a0 0x7f38f806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f390c101d70 0x7f390c1129c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 --2- 172.21.15.111:0/3662089363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f390c102710 0x7f390c112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 >> 172.21.15.111:0/3662089363 conn(0x7f390c072bc0 msgr2=0x7f390c0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 shutdown_connections 2024-03-14T12:57:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:51.737+0000 7f3912512700 1 -- 172.21.15.111:0/3662089363 wait complete. 2024-03-14T12:57:52.392 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-14T12:57:52.392 INFO:tasks.cephadm:Setup complete, yielding 2024-03-14T12:57:52.393 INFO:teuthology.run_tasks:Running task print... 2024-03-14T12:57:52.402 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-14T12:57:52.402 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:57:52.441 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:57:52.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-14T12:57:52.720 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:52.985 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:52 smithi111 ceph-mon[39688]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:52.985 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:52 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3662089363' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-14T12:57:52.985 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:52 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:57:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:52 smithi137 ceph-mon[50336]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:52 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3662089363' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-14T12:57:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:52 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:57:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:53 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:57:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:53 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:57:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:53 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:57:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:53 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:57:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:53 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:57:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:53 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 -- 172.21.15.111:0/2083361260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 msgr2=0x7efe04101e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 --2- 172.21.15.111:0/2083361260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe04101e50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdfc004940 tx=0x7efdfc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 -- 172.21.15.111:0/2083361260 shutdown_connections 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 --2- 172.21.15.111:0/2083361260 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 0x7efe04104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 --2- 172.21.15.111:0/2083361260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe04101e50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 -- 172.21.15.111:0/2083361260 >> 172.21.15.111:0/2083361260 conn(0x7efe040f7480 msgr2=0x7efe040f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 -- 172.21.15.111:0/2083361260 shutdown_connections 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 -- 172.21.15.111:0/2083361260 wait complete. 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.469+0000 7efe0c409700 1 Processor -- start 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 -- start start 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:54.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 0x7efe041100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04110780 con 0x7efe04066af0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04110f70 con 0x7efe04102390 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33364/0 (socket says 172.21.15.111:33364) 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 -- 172.21.15.111:0/909318233 learned_addr learned my addr 172.21.15.111:0/909318233 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe099a4700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 0x7efe041100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 -- 172.21.15.111:0/909318233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 msgr2=0x7efe041100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 0x7efe041100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 -- 172.21.15.111:0/909318233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc005040 con 0x7efe04066af0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0a1a5700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b8a0 tx=0x7efdfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdfc00e040 con 0x7efe04066af0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe041111f0 con 0x7efe04066af0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdfc00bbf0 con 0x7efe04066af0 2024-03-14T12:57:54.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdfc01c3f0 con 0x7efe04066af0 2024-03-14T12:57:54.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.470+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe041117b0 con 0x7efe04066af0 2024-03-14T12:57:54.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.471+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0405f5f0 con 0x7efe04066af0 2024-03-14T12:57:54.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.472+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efdfc00bd60 con 0x7efe04066af0 2024-03-14T12:57:54.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.472+0000 7efe02ffd700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 0x7efdec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:54.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.472+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efdfc0881a0 con 0x7efe04066af0 2024-03-14T12:57:54.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.472+0000 7efe099a4700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 0x7efdec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:54.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.472+0000 7efe099a4700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 0x7efdec06b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006fd0 tx=0x7efdf8008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:54.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.476+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7efdfc0582a0 con 0x7efe04066af0 2024-03-14T12:57:54.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.658+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7efe0404bed0 con 0x7efe04066af0 2024-03-14T12:57:54.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.662+0000 7efe02ffd700 1 -- 172.21.15.111:0/909318233 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7efdfc057e30 con 0x7efe04066af0 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.664+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 msgr2=0x7efdec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.664+0000 7efe0c409700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 0x7efdec06b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efdf8006fd0 tx=0x7efdf8008040 comp rx=0 tx=0).stop 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.664+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 msgr2=0x7efe0410fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.664+0000 7efe0c409700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b8a0 tx=0x7efdfc009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 shutdown_connections 2024-03-14T12:57:54.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efdec0692a0 0x7efdec06b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe04066af0 0x7efe0410fbb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 --2- 172.21.15.111:0/909318233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe04102390 0x7efe041100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 >> 172.21.15.111:0/909318233 conn(0x7efe040f7480 msgr2=0x7efe040f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 shutdown_connections 2024-03-14T12:57:54.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:54.665+0000 7efe0c409700 1 -- 172.21.15.111:0/909318233 wait complete. 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/909318233' entity='client.admin' 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:57:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:54 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/909318233' entity='client.admin' 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:57:55.038 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:54 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:57:55.347 INFO:teuthology.run_tasks:Running task print... 2024-03-14T12:57:55.357 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-14T12:57:55.357 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:57:55.393 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:57:55.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-14T12:57:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:56.851 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:56 smithi111 ceph-mon[39688]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:57.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:56 smithi137 ceph-mon[50336]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 -- 172.21.15.111:0/1597386888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fed90 msgr2=0x7efe340faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 --2- 172.21.15.111:0/1597386888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fed90 0x7efe340faed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efe24004940 tx=0x7efe24009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 -- 172.21.15.111:0/1597386888 shutdown_connections 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 --2- 172.21.15.111:0/1597386888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fed90 0x7efe340faed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 --2- 172.21.15.111:0/1597386888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fe440 0x7efe340fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 -- 172.21.15.111:0/1597386888 >> 172.21.15.111:0/1597386888 conn(0x7efe340f6be0 msgr2=0x7efe340f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 -- 172.21.15.111:0/1597386888 shutdown_connections 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.406+0000 7efe393d0700 1 -- 172.21.15.111:0/1597386888 wait complete. 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 Processor -- start 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 -- start start 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:57.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 0x7efe3410dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3410e3e0 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe393d0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3410e550 con 0x7efe340fed90 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33388/0 (socket says 172.21.15.111:33388) 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 -- 172.21.15.111:0/2908750004 learned_addr learned my addr 172.21.15.111:0/2908750004 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe327fc700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 0x7efe3410dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 -- 172.21.15.111:0/2908750004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 msgr2=0x7efe3410dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 0x7efe3410dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe32ffd700 1 -- 172.21.15.111:0/2908750004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe24005040 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.407+0000 7efe327fc700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 0x7efe3410dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe32ffd700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efe28002a50 tx=0x7efe280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe28007cc0 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe2800f040 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe34112f60 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efe28009540 con 0x7efe340fe440 2024-03-14T12:57:57.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.408+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe34113490 con 0x7efe340fe440 2024-03-14T12:57:57.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.409+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3404c710 con 0x7efe340fe440 2024-03-14T12:57:57.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.409+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efe280096a0 con 0x7efe340fe440 2024-03-14T12:57:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.409+0000 7efe1b7fe700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 0x7efe1c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:57:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.409+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efe28086c80 con 0x7efe340fe440 2024-03-14T12:57:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.409+0000 7efe327fc700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 0x7efe1c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:57:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.410+0000 7efe327fc700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 0x7efe1c06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efe2400b900 tx=0x7efe24009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:57:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.414+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7efe28056d00 con 0x7efe340fe440 2024-03-14T12:57:57.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.611+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7efe341136b0 con 0x7efe1c0692a0 2024-03-14T12:57:57.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.612+0000 7efe1b7fe700 1 -- 172.21.15.111:0/2908750004 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7efe341136b0 con 0x7efe1c0692a0 2024-03-14T12:57:57.612 INFO:teuthology.orchestra.run.smithi111.stdout:Backend: cephadm 2024-03-14T12:57:57.612 INFO:teuthology.orchestra.run.smithi111.stdout:Available: Yes 2024-03-14T12:57:57.612 INFO:teuthology.orchestra.run.smithi111.stdout:Paused: No 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 msgr2=0x7efe1c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 0x7efe1c06b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efe2400b900 tx=0x7efe24009e30 comp rx=0 tx=0).stop 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 msgr2=0x7efe3410d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efe28002a50 tx=0x7efe280046b0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 shutdown_connections 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7efe1c0692a0 0x7efe1c06b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe340fe440 0x7efe3410d8a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 --2- 172.21.15.111:0/2908750004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efe340fed90 0x7efe3410dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:57:57.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 >> 172.21.15.111:0/2908750004 conn(0x7efe340f6be0 msgr2=0x7efe340f8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:57:57.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 shutdown_connections 2024-03-14T12:57:57.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:57:57.614+0000 7efe393d0700 1 -- 172.21.15.111:0/2908750004 wait complete. 2024-03-14T12:57:58.316 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-14T12:57:58.582 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:57:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:58 smithi137 ceph-mon[50336]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:58 smithi137 ceph-mon[50336]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:57:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:57:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:57:59.093 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:58 smithi111 ceph-mon[39688]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:57:59.093 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:58 smithi111 ceph-mon[39688]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:57:59.093 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:57:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 -- 172.21.15.111:0/337520569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 msgr2=0x7fcfe80fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 --2- 172.21.15.111:0/337520569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe80fd3f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8004940 tx=0x7fcfd8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 -- 172.21.15.111:0/337520569 shutdown_connections 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 --2- 172.21.15.111:0/337520569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe80fd3f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 --2- 172.21.15.111:0/337520569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 0x7fcfe80fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 -- 172.21.15.111:0/337520569 >> 172.21.15.111:0/337520569 conn(0x7fcfe80f7430 msgr2=0x7fcfe80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 -- 172.21.15.111:0/337520569 shutdown_connections 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.487+0000 7fcfed55d700 1 -- 172.21.15.111:0/337520569 wait complete. 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 Processor -- start 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 -- start start 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 0x7fcfe810c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:00.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe810d390 con 0x7fcfe80fcf80 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfed55d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe810dbe0 con 0x7fcfe80fbcd0 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40896/0 (socket says 172.21.15.111:40896) 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 -- 172.21.15.111:0/3118312164 learned_addr learned my addr 172.21.15.111:0/3118312164 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe6ffd700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 0x7fcfe810c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 -- 172.21.15.111:0/3118312164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 msgr2=0x7fcfe810c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 0x7fcfe810c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 -- 172.21.15.111:0/3118312164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd8005040 con 0x7fcfe80fcf80 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.488+0000 7fcfe67fc700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b3b0 tx=0x7fcfd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd800e040 con 0x7fcfe80fcf80 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfd800bd20 con 0x7fcfe80fcf80 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe810de60 con 0x7fcfe80fcf80 2024-03-14T12:58:00.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd801c630 con 0x7fcfe80fcf80 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe810e3a0 con 0x7fcfe80fcf80 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.489+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe805f5f0 con 0x7fcfe80fcf80 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.490+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcfd8013420 con 0x7fcfe80fcf80 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.491+0000 7fcfdf7fe700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 0x7fcfd406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.491+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcfd8088300 con 0x7fcfe80fcf80 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.491+0000 7fcfe6ffd700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 0x7fcfd406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.491+0000 7fcfe6ffd700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 0x7fcfd406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004990 tx=0x7fcfd0009ee0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:00.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.495+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fcfd8058400 con 0x7fcfe80fcf80 2024-03-14T12:58:00.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.686+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfe810e740 con 0x7fcfd40692a0 2024-03-14T12:58:00.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.694+0000 7fcfdf7fe700 1 -- 172.21.15.111:0/3118312164 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7fcfe810e740 con 0x7fcfd40692a0 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (3m) 2m ago 5m 26.5M - 0.25.0 c8568f914cd2 a5a721c2d1c2 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (6m) 2m ago 6m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (4m) 72s ago 4m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (3m) 2m ago 5m 80.2M - 9.4.7 954c08fa6188 a0f9b28f5afd 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:9283 running (7m) 2m ago 7m 472M - 17.2.7-646-gb92acd1f a07fb176315d 9d384b708369 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (4m) 72s ago 4m 434M - 17.2.7-646-gb92acd1f a07fb176315d ee0103247ac3 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (7m) 2m ago 7m 42.9M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (4m) 72s ago 4m 41.3M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T12:58:00.694 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (5m) 2m ago 5m 17.6M - 1.5.0 0da6a335fe13 834d8e9211eb 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 72s ago 4m 17.8M - 1.5.0 0da6a335fe13 de8bbb0f9cd8 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (3m) 2m ago 3m 39.1M 4299M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (3m) 2m ago 3m 38.6M 4299M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (2m) 2m ago 2m 39.4M 4299M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (2m) 72s ago 2m 42.0M 5323M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (103s) 72s ago 103s 38.3M 5323M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (77s) 72s ago 76s 10.4M 5323M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T12:58:00.695 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (3m) 2m ago 4m 35.8M - 2.43.0 a07b618ecd1d c5814e8b878b 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 msgr2=0x7fcfd406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 0x7fcfd406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0004990 tx=0x7fcfd0009ee0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 msgr2=0x7fcfe810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b3b0 tx=0x7fcfd8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 shutdown_connections 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe80fbcd0 0x7fcfe810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fcfd40692a0 0x7fcfd406b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 --2- 172.21.15.111:0/3118312164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe80fcf80 0x7fcfe810cd90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:00.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 >> 172.21.15.111:0/3118312164 conn(0x7fcfe80f7430 msgr2=0x7fcfe80f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 shutdown_connections 2024-03-14T12:58:00.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:00.696+0000 7fcfed55d700 1 -- 172.21.15.111:0/3118312164 wait complete. 2024-03-14T12:58:01.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:00 smithi137 ceph-mon[50336]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:01.014 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:00 smithi111 ceph-mon[39688]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:01.341 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-14T12:58:01.608 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:01 smithi137 ceph-mon[50336]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:02.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:01 smithi111 ceph-mon[39688]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:02 smithi111 ceph-mon[39688]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:03.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:02 smithi137 ceph-mon[50336]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 -- 172.21.15.111:0/1251868155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fcfb0 msgr2=0x7fd1380fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/1251868155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fd420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd128004940 tx=0x7fd128009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 -- 172.21.15.111:0/1251868155 shutdown_connections 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/1251868155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fd420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/1251868155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fbd00 0x7fd1380fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 -- 172.21.15.111:0/1251868155 >> 172.21.15.111:0/1251868155 conn(0x7fd1380f7480 msgr2=0x7fd1380f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 -- 172.21.15.111:0/1251868155 shutdown_connections 2024-03-14T12:58:03.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 -- 172.21.15.111:0/1251868155 wait complete. 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.476+0000 7fd13d9a1700 1 Processor -- start 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 -- start start 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd138100230 con 0x7fd1380fbd00 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1380fe830 con 0x7fd1380fcfb0 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40910/0 (socket says 172.21.15.111:40910) 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 -- 172.21.15.111:0/2251472483 learned_addr learned my addr 172.21.15.111:0/2251472483 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd1367fc700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 -- 172.21.15.111:0/2251472483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fcfb0 msgr2=0x7fd1380fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fe2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 -- 172.21.15.111:0/2251472483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd128005040 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd136ffd700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00b790 tx=0x7fd12c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c00f6f0 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1380feb10 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd12c00fd00 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd12c0103a0 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.477+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1380ff0d0 con 0x7fd1380fbd00 2024-03-14T12:58:03.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.478+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13804c710 con 0x7fd1380fbd00 2024-03-14T12:58:03.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.479+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd12c00f850 con 0x7fd1380fbd00 2024-03-14T12:58:03.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.479+0000 7fd11f7fe700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 0x7fd12006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:03.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.479+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd12c086c70 con 0x7fd1380fbd00 2024-03-14T12:58:03.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.479+0000 7fd1367fc700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 0x7fd12006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:03.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.480+0000 7fd1367fc700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 0x7fd12006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd12800b900 tx=0x7fd128009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:03.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.484+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd12c056d70 con 0x7fd1380fbd00 2024-03-14T12:58:03.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.672+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd13802b3b0 con 0x7fd1200692a0 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.676+0000 7fd11f7fe700 1 -- 172.21.15.111:0/2251472483 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7fd13802b3b0 con 0x7fd1200692a0 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stdout:crash 2/2 2m ago 6m * 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-14T12:58:03.676 INFO:teuthology.orchestra.run.smithi111.stdout:mgr 2/2 2m ago 6m count:2 2024-03-14T12:58:03.677 INFO:teuthology.orchestra.run.smithi111.stdout:mon 2/2 2m ago 5m smithi111:172.21.15.111=smithi111;smithi137:172.21.15.137=smithi137;count:2 2024-03-14T12:58:03.677 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-14T12:58:03.677 INFO:teuthology.orchestra.run.smithi111.stdout:osd 6 2m ago - 2024-03-14T12:58:03.677 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-14T12:58:03.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 msgr2=0x7fd12006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 0x7fd12006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd12800b900 tx=0x7fd128009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 msgr2=0x7fd1380ffba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd12c00b790 tx=0x7fd12c009250 comp rx=0 tx=0).stop 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 shutdown_connections 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1200692a0 0x7fd12006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1380fbd00 0x7fd1380ffba0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 --2- 172.21.15.111:0/2251472483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1380fcfb0 0x7fd1380fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 >> 172.21.15.111:0/2251472483 conn(0x7fd1380f7480 msgr2=0x7fd1380668a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 shutdown_connections 2024-03-14T12:58:03.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:03.679+0000 7fd13d9a1700 1 -- 172.21.15.111:0/2251472483 wait complete. 2024-03-14T12:58:04.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-14T12:58:04.675 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:05.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:04 smithi137 ceph-mon[50336]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:04 smithi137 ceph-mon[50336]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:04 smithi111 ceph-mon[39688]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:04 smithi111 ceph-mon[39688]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:06.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 -- 172.21.15.111:0/1672626421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fe440 msgr2=0x7fc41c0fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:06.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 --2- 172.21.15.111:0/1672626421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fe850 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc410004940 tx=0x7fc410009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:06.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 -- 172.21.15.111:0/1672626421 shutdown_connections 2024-03-14T12:58:06.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 --2- 172.21.15.111:0/1672626421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fed90 0x7fc41c0faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 --2- 172.21.15.111:0/1672626421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fe850 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 -- 172.21.15.111:0/1672626421 >> 172.21.15.111:0/1672626421 conn(0x7fc41c0f6be0 msgr2=0x7fc41c0f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 -- 172.21.15.111:0/1672626421 shutdown_connections 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.669+0000 7fc4215d3700 1 -- 172.21.15.111:0/1672626421 wait complete. 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 Processor -- start 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 -- start start 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0fe050 con 0x7fc41c0fed90 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc4215d3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0fc640 con 0x7fc41c0fe440 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40914/0 (socket says 172.21.15.111:40914) 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 -- 172.21.15.111:0/468147914 learned_addr learned my addr 172.21.15.111:0/468147914 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41affd700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 -- 172.21.15.111:0/468147914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fe440 msgr2=0x7fc41c0fda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:06.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 -- 172.21.15.111:0/468147914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc410005040 con 0x7fc41c0fed90 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.670+0000 7fc41a7fc700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c8c0 tx=0x7fc40c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40c00a760 con 0x7fc41c0fed90 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc40c00ad70 con 0x7fc41c0fed90 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c0fc8f0 con 0x7fc41c0fed90 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc40c0113a0 con 0x7fc41c0fed90 2024-03-14T12:58:06.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c0fceb0 con 0x7fc41c0fed90 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.671+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c04c710 con 0x7fc41c0fed90 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.672+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc40c012400 con 0x7fc41c0fed90 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.673+0000 7fc4037fe700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 0x7fc40406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.673+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc40c087a00 con 0x7fc41c0fed90 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.673+0000 7fc41affd700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 0x7fc40406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:06.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.673+0000 7fc41affd700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 0x7fc40406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc410004890 tx=0x7fc410009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:06.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.676+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fc40c057ba0 con 0x7fc41c0fed90 2024-03-14T12:58:06.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.867+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc41c1025b0 con 0x7fc4040692a0 2024-03-14T12:58:06.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.869+0000 7fc4037fe700 1 -- 172.21.15.111:0/468147914 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fc41c1025b0 con 0x7fc4040692a0 2024-03-14T12:58:06.868 INFO:teuthology.orchestra.run.smithi111.stdout:HOST ADDR LABELS STATUS 2024-03-14T12:58:06.869 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 172.21.15.111 2024-03-14T12:58:06.869 INFO:teuthology.orchestra.run.smithi111.stdout:smithi137 172.21.15.137 2024-03-14T12:58:06.869 INFO:teuthology.orchestra.run.smithi111.stdout:2 hosts in cluster 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 msgr2=0x7fc40406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 0x7fc40406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc410004890 tx=0x7fc410009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 msgr2=0x7fc41c0fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00c8c0 tx=0x7fc40c0044c0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 shutdown_connections 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc41c0fe440 0x7fc41c0fda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc4040692a0 0x7fc40406b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 --2- 172.21.15.111:0/468147914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc41c0fed90 0x7fc41c0fc100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:06.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 >> 172.21.15.111:0/468147914 conn(0x7fc41c0f6be0 msgr2=0x7fc41c0f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:06.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 shutdown_connections 2024-03-14T12:58:06.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:06.871+0000 7fc4215d3700 1 -- 172.21.15.111:0/468147914 wait complete. 2024-03-14T12:58:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:06 smithi137 ceph-mon[50336]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:07.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:06 smithi111 ceph-mon[39688]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:07.628 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-14T12:58:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:07 smithi111 ceph-mon[39688]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:07.898 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:08.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:07 smithi137 ceph-mon[50336]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:09.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:08 smithi111 ceph-mon[39688]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:09.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:08 smithi137 ceph-mon[50336]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- 172.21.15.111:0/3099341664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fbcd0 msgr2=0x7f8c900fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 --2- 172.21.15.111:0/3099341664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fbcd0 0x7f8c900fc0e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8c80004940 tx=0x7f8c80009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- 172.21.15.111:0/3099341664 shutdown_connections 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 --2- 172.21.15.111:0/3099341664 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fcf80 0x7f8c900fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 --2- 172.21.15.111:0/3099341664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fbcd0 0x7f8c900fc0e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- 172.21.15.111:0/3099341664 >> 172.21.15.111:0/3099341664 conn(0x7f8c900f7430 msgr2=0x7f8c900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- 172.21.15.111:0/3099341664 shutdown_connections 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- 172.21.15.111:0/3099341664 wait complete. 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 Processor -- start 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- start start 2024-03-14T12:58:09.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fbcd0 0x7f8c9010d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9010e4f0 con 0x7f8c900fcf80 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c94c05700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9010ecf0 con 0x7f8c900fbcd0 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c8dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.766+0000 7f8c8dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40936/0 (socket says 172.21.15.111:40936) 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8dd9b700 1 -- 172.21.15.111:0/183778392 learned_addr learned my addr 172.21.15.111:0/183778392 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8e59c700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fbcd0 0x7f8c9010d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8dd9b700 1 -- 172.21.15.111:0/183778392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fbcd0 msgr2=0x7f8c9010d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8dd9b700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fbcd0 0x7f8c9010d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8dd9b700 1 -- 172.21.15.111:0/183778392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c80005040 con 0x7f8c900fcf80 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c8dd9b700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8c84002a50 tx=0x7f8c840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c84007cc0 con 0x7f8c900fcf80 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c8400f040 con 0x7f8c900fcf80 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9010efd0 con 0x7f8c900fcf80 2024-03-14T12:58:09.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c8400d9c0 con 0x7f8c900fcf80 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.767+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9010f510 con 0x7f8c900fcf80 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.768+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c90101b10 con 0x7f8c900fcf80 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.768+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8c8400db60 con 0x7f8c900fcf80 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.769+0000 7f8c7effd700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 0x7f8c7806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.769+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8c840864d0 con 0x7f8c900fcf80 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.769+0000 7f8c8e59c700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 0x7f8c7806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:09.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.769+0000 7f8c8e59c700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 0x7f8c7806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b900 tx=0x7f8c80009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:09.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.773+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8c8404cd70 con 0x7f8c900fcf80 2024-03-14T12:58:09.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.962+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c9010f730 con 0x7f8c780692a0 2024-03-14T12:58:09.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.963+0000 7f8c7effd700 1 -- 172.21.15.111:0/183778392 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f8c9010f730 con 0x7f8c780692a0 2024-03-14T12:58:09.963 INFO:teuthology.orchestra.run.smithi111.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-14T12:58:09.963 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-14T12:58:09.963 INFO:teuthology.orchestra.run.smithi111.stdout:smithi137 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233004A400BGN 372G No 78s ago Has a FileSystem, LVM detected 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 msgr2=0x7f8c7806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 0x7f8c7806b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000b900 tx=0x7f8c80009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 msgr2=0x7f8c9010def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8c84002a50 tx=0x7f8c840046b0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 shutdown_connections 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c900fbcd0 0x7f8c9010d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8c780692a0 0x7f8c7806b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 --2- 172.21.15.111:0/183778392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c900fcf80 0x7f8c9010def0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 >> 172.21.15.111:0/183778392 conn(0x7f8c900f7430 msgr2=0x7f8c90065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:09.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 shutdown_connections 2024-03-14T12:58:09.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:09.965+0000 7f8c94c05700 1 -- 172.21.15.111:0/183778392 wait complete. 2024-03-14T12:58:10.639 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:58:10.648 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:10.649 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-14T12:58:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:10 smithi111 ceph-mon[39688]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:10.914 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:11.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:10 smithi137 ceph-mon[50336]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:12.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:11 smithi111 ceph-mon[39688]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:12.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:11 smithi137 ceph-mon[50336]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- 172.21.15.111:0/1959969099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fe040 msgr2=0x7f3c340fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/1959969099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fe040 0x7f3c340fe4b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3c28004940 tx=0x7f3c28009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- 172.21.15.111:0/1959969099 shutdown_connections 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/1959969099 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fe040 0x7f3c340fe4b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/1959969099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fce40 0x7f3c340fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- 172.21.15.111:0/1959969099 >> 172.21.15.111:0/1959969099 conn(0x7f3c340f8510 msgr2=0x7f3c340fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- 172.21.15.111:0/1959969099 shutdown_connections 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- 172.21.15.111:0/1959969099 wait complete. 2024-03-14T12:58:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 Processor -- start 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- start start 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fe040 0x7f3c34070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c34070950 con 0x7f3c340fce40 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.783+0000 7f3c3a05e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c34070ac0 con 0x7f3c340fe040 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36440/0 (socket says 172.21.15.111:36440) 2024-03-14T12:58:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 -- 172.21.15.111:0/4109367407 learned_addr learned my addr 172.21.15.111:0/4109367407 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:12.785 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c32ffd700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fe040 0x7f3c34070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 -- 172.21.15.111:0/4109367407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fe040 msgr2=0x7f3c34070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fe040 0x7f3c34070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 -- 172.21.15.111:0/4109367407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c28005040 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c337fe700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b790 tx=0x7f3c24009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c2400f6f0 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c2400fd00 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c34070da0 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c240103a0 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.784+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c34115980 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.785+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3405f5f0 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.786+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3c2400f850 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.786+0000 7f3c1bfff700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 0x7f3c1c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.786+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3c24086be0 con 0x7f3c340fce40 2024-03-14T12:58:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.786+0000 7f3c32ffd700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 0x7f3c1c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:12.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.786+0000 7f3c32ffd700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 0x7f3c1c06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800b3b0 tx=0x7f3c28009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:12.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.790+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f3c24053250 con 0x7f3c340fce40 2024-03-14T12:58:12.998 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:12.998+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c341023a0 con 0x7f3c1c0692a0 2024-03-14T12:58:13.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:12 smithi111 ceph-mon[39688]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:13.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:12 smithi137 ceph-mon[50336]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-14T12:58:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-14T12:58:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:14.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.807+0000 7f3c1bfff700 1 -- 172.21.15.111:0/4109367407 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c341023a0 con 0x7f3c1c0692a0 2024-03-14T12:58:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 msgr2=0x7f3c1c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 0x7f3c1c06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800b3b0 tx=0x7f3c28009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 msgr2=0x7f3c34071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b790 tx=0x7f3c24009220 comp rx=0 tx=0).stop 2024-03-14T12:58:14.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 shutdown_connections 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3c1c0692a0 0x7f3c1c06b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c340fce40 0x7f3c34071e40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 --2- 172.21.15.111:0/4109367407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c340fe040 0x7f3c34070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.809+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 >> 172.21.15.111:0/4109367407 conn(0x7f3c340f8510 msgr2=0x7f3c340f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.810+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 shutdown_connections 2024-03-14T12:58:14.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:14.810+0000 7f3c3a05e700 1 -- 172.21.15.111:0/4109367407 wait complete. 2024-03-14T12:58:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-mon[39688]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-mon[39688]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-14T12:58:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-mon[39688]: osdmap e42: 6 total, 6 up, 6 in 2024-03-14T12:58:15.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-14T12:58:15.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:14 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[39665]: 2024-03-14T12:58:14.784+0000 7f6490817700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T12:58:15.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:14 smithi137 ceph-mon[50336]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:58:15.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:14 smithi137 ceph-mon[50336]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:15.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:14 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-14T12:58:15.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:14 smithi137 ceph-mon[50336]: osdmap e42: 6 total, 6 up, 6 in 2024-03-14T12:58:15.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:14 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-14T12:58:15.535 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: osdmap e43: 6 total, 6 up, 6 in 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T12:58:16.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: osdmap e44: 6 total, 6 up, 6 in 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: fsmap cephfs:0 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: Saving service mds.cephfs spec with placement count:4 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:16.082 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:16.083 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:16.083 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:15 smithi111 ceph-mon[39688]: Deploying daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: osdmap e43: 6 total, 6 up, 6 in 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: osdmap e44: 6 total, 6 up, 6 in 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: fsmap cephfs:0 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: Saving service mds.cephfs spec with placement count:4 2024-03-14T12:58:16.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:16.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:15 smithi137 ceph-mon[50336]: Deploying daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T12:58:16.359 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:16 smithi111 ceph-mon[39688]: pgmap v183: 65 pgs: 6 creating+peering, 1 active+clean, 58 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:16 smithi111 ceph-mon[39688]: osdmap e45: 6 total, 6 up, 6 in 2024-03-14T12:58:17.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:16 smithi137 ceph-mon[50336]: pgmap v183: 65 pgs: 6 creating+peering, 1 active+clean, 58 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:17.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:16 smithi137 ceph-mon[50336]: osdmap e45: 6 total, 6 up, 6 in 2024-03-14T12:58:18.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:17 smithi111 ceph-mon[39688]: osdmap e46: 6 total, 6 up, 6 in 2024-03-14T12:58:18.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:17 smithi137 ceph-mon[50336]: osdmap e46: 6 total, 6 up, 6 in 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 -- 172.21.15.111:0/992944964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fb440 msgr2=0x7fc40c0fb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 --2- 172.21.15.111:0/992944964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fb440 0x7fc40c0fb850 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc408004940 tx=0x7fc408009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 -- 172.21.15.111:0/992944964 shutdown_connections 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 --2- 172.21.15.111:0/992944964 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c0fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 --2- 172.21.15.111:0/992944964 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fb440 0x7fc40c0fb850 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 -- 172.21.15.111:0/992944964 >> 172.21.15.111:0/992944964 conn(0x7fc40c0f6ba0 msgr2=0x7fc40c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 -- 172.21.15.111:0/992944964 shutdown_connections 2024-03-14T12:58:18.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 -- 172.21.15.111:0/992944964 wait complete. 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.512+0000 7fc41338a700 1 Processor -- start 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 -- start start 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fb440 0x7fc40c10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c10e510 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c10ed00 con 0x7fc40c0fb440 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36448/0 (socket says 172.21.15.111:36448) 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 -- 172.21.15.111:0/1480798100 learned_addr learned my addr 172.21.15.111:0/1480798100 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc411126700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fb440 0x7fc40c10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 -- 172.21.15.111:0/1480798100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fb440 msgr2=0x7fc40c10d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fb440 0x7fc40c10d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 -- 172.21.15.111:0/1480798100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc408005040 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc410925700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00b790 tx=0x7fc3fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3fc00f6f0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3fc00fd00 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc3fc0103a0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.513+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c10efe0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.514+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c10f5a0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.514+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c05f5f0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.515+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc3fc00f850 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.515+0000 7fc401ffb700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 0x7fc3f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.515+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc3fc086e80 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.515+0000 7fc411126700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 0x7fc3f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:18.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.516+0000 7fc411126700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 0x7fc3f806b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc40800b900 tx=0x7fc408009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:18.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.520+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fc3fc056ce0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.752+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc40c04bed0 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.753+0000 7fc401ffb700 1 -- 172.21.15.111:0/1480798100 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7fc3fc056870 con 0x7fc40c0fc6f0 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:e2 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 2 2024-03-14T12:58:18.753 INFO:teuthology.orchestra.run.smithi111.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T12:58:14.784583+0000 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 1 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:in 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T12:58:18.754 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 0 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:18.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 msgr2=0x7fc3f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 0x7fc3f806b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc40800b900 tx=0x7fc408009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 msgr2=0x7fc40c10de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00b790 tx=0x7fc3fc009250 comp rx=0 tx=0).stop 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 shutdown_connections 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc40c0fb440 0x7fc40c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fc3f80692a0 0x7fc3f806b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 --2- 172.21.15.111:0/1480798100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc40c0fc6f0 0x7fc40c10de80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 >> 172.21.15.111:0/1480798100 conn(0x7fc40c0f6ba0 msgr2=0x7fc40c0f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 shutdown_connections 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:18.755+0000 7fc41338a700 1 -- 172.21.15.111:0/1480798100 wait complete. 2024-03-14T12:58:18.756 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 2 2024-03-14T12:58:19.091 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:18 smithi111 ceph-mon[39688]: pgmap v186: 65 pgs: 6 creating+peering, 14 active+clean, 45 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:19.091 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:18 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1480798100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T12:58:19.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:18 smithi137 ceph-mon[50336]: pgmap v186: 65 pgs: 6 creating+peering, 14 active+clean, 45 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:19.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:18 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1480798100' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T12:58:19.675 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:58:19.685 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:19.685 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-14T12:58:19.960 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: pgmap v187: 65 pgs: 6 creating+peering, 50 active+clean, 9 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: Deploying daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T12:58:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:boot 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: daemon mds.cephfs.smithi111.zmslha assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: Cluster is now healthy 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: fsmap cephfs:0 1 up:standby 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:creating} 2024-03-14T12:58:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:20 smithi137 ceph-mon[50336]: daemon mds.cephfs.smithi111.zmslha is now active in filesystem cephfs as rank 0 2024-03-14T12:58:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: pgmap v187: 65 pgs: 6 creating+peering, 50 active+clean, 9 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: Deploying daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:boot 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: daemon mds.cephfs.smithi111.zmslha assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: Cluster is now healthy 2024-03-14T12:58:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: fsmap cephfs:0 1 up:standby 2024-03-14T12:58:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T12:58:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:creating} 2024-03-14T12:58:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:20 smithi111 ceph-mon[39688]: daemon mds.cephfs.smithi111.zmslha is now active in filesystem cephfs as rank 0 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 -- 172.21.15.111:0/3187339717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fdc30 msgr2=0x7f73580fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 --2- 172.21.15.111:0/3187339717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fdc30 0x7f73580fc050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f734c004940 tx=0x7f734c009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 -- 172.21.15.111:0/3187339717 shutdown_connections 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 --2- 172.21.15.111:0/3187339717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fdc30 0x7f73580fc050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 --2- 172.21.15.111:0/3187339717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73580fd2e0 0x7f73580fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 -- 172.21.15.111:0/3187339717 >> 172.21.15.111:0/3187339717 conn(0x7f73580f6c10 msgr2=0x7f73580f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 -- 172.21.15.111:0/3187339717 shutdown_connections 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.816+0000 7f735e063700 1 -- 172.21.15.111:0/3187339717 wait complete. 2024-03-14T12:58:21.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 Processor -- start 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 -- start start 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f735810cc20 0x7f7358111c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735810d0e0 con 0x7f73580fd2e0 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f735e063700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735810d220 con 0x7f735810cc20 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37936/0 (socket says 172.21.15.111:37936) 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 -- 172.21.15.111:0/3836242749 learned_addr learned my addr 172.21.15.111:0/3836242749 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f7356ffd700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f735810cc20 0x7f7358111c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 -- 172.21.15.111:0/3836242749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f735810cc20 msgr2=0x7f7358111c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f735810cc20 0x7f7358111c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.817+0000 7f73577fe700 1 -- 172.21.15.111:0/3836242749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c005040 con 0x7f73580fd2e0 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f73577fe700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f734800eb70 tx=0x7f734800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:21.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7348007cc0 con 0x7f73580fd2e0 2024-03-14T12:58:21.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358112190 con 0x7f73580fd2e0 2024-03-14T12:58:21.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f734800f040 con 0x7f73580fd2e0 2024-03-14T12:58:21.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f734800c9d0 con 0x7f73580fd2e0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.818+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7358112700 con 0x7f73580fd2e0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.819+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735804c710 con 0x7f73580fd2e0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.819+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f734800cc40 con 0x7f73580fd2e0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.820+0000 7f733ffff700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 0x7f734006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.820+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7348086c10 con 0x7f73580fd2e0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.820+0000 7f7356ffd700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 0x7f734006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:21.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.820+0000 7f7356ffd700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 0x7f734006b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f734c013680 tx=0x7f734c009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:21.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:21.825+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f734804dd70 con 0x7f73580fd2e0 2024-03-14T12:58:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:21 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:active 2024-03-14T12:58:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:21 smithi111 ceph-mon[39688]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 2024-03-14T12:58:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:21 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:active 2024-03-14T12:58:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:21 smithi137 ceph-mon[50336]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 2024-03-14T12:58:22.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.058+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f73580fddf0 con 0x7f73580fd2e0 2024-03-14T12:58:22.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.893+0000 7f733ffff700 1 -- 172.21.15.111:0/3836242749 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7348056a00 con 0x7f73580fd2e0 2024-03-14T12:58:22.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 msgr2=0x7f734006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 0x7f734006b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f734c013680 tx=0x7f734c009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 msgr2=0x7f735810c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f734800eb70 tx=0x7f734800b040 comp rx=0 tx=0).stop 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 shutdown_connections 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f73400692a0 0x7f734006b750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73580fd2e0 0x7f735810c6e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 --2- 172.21.15.111:0/3836242749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f735810cc20 0x7f7358111c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 >> 172.21.15.111:0/3836242749 conn(0x7f73580f6c10 msgr2=0x7f73580f8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 shutdown_connections 2024-03-14T12:58:22.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:22.896+0000 7f735e063700 1 -- 172.21.15.111:0/3836242749 wait complete. 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: pgmap v188: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:boot 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:23.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:23.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:23.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:22 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3836242749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-14T12:58:23.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: pgmap v188: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:boot 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:23.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:22 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3836242749' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: Deploying daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3836242749' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: daemon mds.cephfs.smithi137.ryucox assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:creating} 2024-03-14T12:58:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:23 smithi137 ceph-mon[50336]: daemon mds.cephfs.smithi137.ryucox is now active in filesystem cephfs as rank 1 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: Deploying daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3836242749' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: daemon mds.cephfs.smithi137.ryucox assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: fsmap cephfs:1 {0=cephfs.smithi111.zmslha=up:active} 1 up:standby 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:creating} 2024-03-14T12:58:24.110 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:23 smithi111 ceph-mon[39688]: daemon mds.cephfs.smithi137.ryucox is now active in filesystem cephfs as rank 1 2024-03-14T12:58:24.131 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:58:24.142 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:24.142 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-14T12:58:24.947 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:24 smithi111 ceph-mon[39688]: pgmap v189: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 3 op/s 2024-03-14T12:58:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:24 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:active 2024-03-14T12:58:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:24 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2024-03-14T12:58:25.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:24 smithi137 ceph-mon[50336]: pgmap v189: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 3 op/s 2024-03-14T12:58:25.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:24 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:active 2024-03-14T12:58:25.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:24 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2024-03-14T12:58:26.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:26.223 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: Deploying daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: Cluster is now healthy 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] up:boot 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby 2024-03-14T12:58:26.224 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:25 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: Deploying daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: Cluster is now healthy 2024-03-14T12:58:26.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] up:boot 2024-03-14T12:58:26.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby 2024-03-14T12:58:26.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:25 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 -- 172.21.15.111:0/4168040357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fbcd0 msgr2=0x7fb3e80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/4168040357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fbcd0 0x7fb3e80fc0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8004940 tx=0x7fb3d8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 -- 172.21.15.111:0/4168040357 shutdown_connections 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/4168040357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3e80fcf80 0x7fb3e80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/4168040357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fbcd0 0x7fb3e80fc0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 -- 172.21.15.111:0/4168040357 >> 172.21.15.111:0/4168040357 conn(0x7fb3e80f7430 msgr2=0x7fb3e80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.750+0000 7fb3edfa7700 1 -- 172.21.15.111:0/4168040357 shutdown_connections 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 -- 172.21.15.111:0/4168040357 wait complete. 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 Processor -- start 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 -- start start 2024-03-14T12:58:26.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3e80fbcd0 0x7fb3e810c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e810d400 con 0x7fb3e80fcf80 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3edfa7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e810dbf0 con 0x7fb3e80fbcd0 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.751+0000 7fb3e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37966/0 (socket says 172.21.15.111:37966) 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 -- 172.21.15.111:0/1681911687 learned_addr learned my addr 172.21.15.111:0/1681911687 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 -- 172.21.15.111:0/1681911687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3e80fbcd0 msgr2=0x7fb3e810c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:26.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3e80fbcd0 0x7fb3e810c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 -- 172.21.15.111:0/1681911687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8005040 con 0x7fb3e80fcf80 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3e6ffd700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00daf0 tx=0x7fb3dc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3dc00ed80 con 0x7fb3e80fcf80 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e810ded0 con 0x7fb3e80fcf80 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3dc017920 con 0x7fb3e80fcf80 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3dc016950 con 0x7fb3e80fcf80 2024-03-14T12:58:26.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.752+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e810e490 con 0x7fb3e80fcf80 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.753+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e805f5f0 con 0x7fb3e80fcf80 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.754+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb3dc014070 con 0x7fb3e80fcf80 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.754+0000 7fb3cffff700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 0x7fb3d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.754+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb3dc0893a0 con 0x7fb3e80fcf80 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.754+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 0x7fb3d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:26.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.754+0000 7fb3e77fe700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 0x7fb3d006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800b900 tx=0x7fb3d8009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:26.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.758+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb3dc059160 con 0x7fb3e80fcf80 2024-03-14T12:58:26.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:26.986+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb3e804bed0 con 0x7fb3e80fcf80 2024-03-14T12:58:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.040+0000 7fb3cffff700 1 -- 172.21.15.111:0/1681911687 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v12) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb3dc058cf0 con 0x7fb3e80fcf80 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 msgr2=0x7fb3d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 0x7fb3d006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb3d800b900 tx=0x7fb3d8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 msgr2=0x7fb3e810cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00daf0 tx=0x7fb3dc00b4f0 comp rx=0 tx=0).stop 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 shutdown_connections 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3e80fbcd0 0x7fb3e810c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb3d00692a0 0x7fb3d006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 --2- 172.21.15.111:0/1681911687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3e80fcf80 0x7fb3e810cd70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 >> 172.21.15.111:0/1681911687 conn(0x7fb3e80f7430 msgr2=0x7fb3e80f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 shutdown_connections 2024-03-14T12:58:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:27.042+0000 7fb3edfa7700 1 -- 172.21.15.111:0/1681911687 wait complete. 2024-03-14T12:58:27.190 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:27 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1681911687' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-14T12:58:27.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:27 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1681911687' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-14T12:58:27.776 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:58:27.792 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:27.792 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data false' 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1681911687' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby-replay 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.317 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.318 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:28.343 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1681911687' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-14T12:58:28.343 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby 2024-03-14T12:58:28.343 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby-replay 2024-03-14T12:58:28.343 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-14T12:58:28.343 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.344 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.344 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.344 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.344 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:28.344 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:28.344 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] up:boot 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:active 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby-replay 1 up:standby 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:29.184 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:29 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] up:boot 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] up:active 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 1 up:standby-replay 1 up:standby 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:29 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:30.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- 172.21.15.111:0/3666021594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 msgr2=0x7facb80fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3666021594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb80fd280 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7facac004940 tx=0x7facac009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- 172.21.15.111:0/3666021594 shutdown_connections 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3666021594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb80fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3666021594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb80fd280 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- 172.21.15.111:0/3666021594 >> 172.21.15.111:0/3666021594 conn(0x7facb80f8550 msgr2=0x7facb80fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- 172.21.15.111:0/3666021594 shutdown_connections 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- 172.21.15.111:0/3666021594 wait complete. 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 Processor -- start 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.433+0000 7facbe3e7700 1 -- start start 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb810dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810e3a0 con 0x7facb80fce70 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810e510 con 0x7facb80fdfe0 2024-03-14T12:58:30.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45818/0 (socket says 172.21.15.111:45818) 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 -- 172.21.15.111:0/3095604830 learned_addr learned my addr 172.21.15.111:0/3095604830 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb77fe700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb810dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 -- 172.21.15.111:0/3095604830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 msgr2=0x7facb810dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb810dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 -- 172.21.15.111:0/3095604830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac005040 con 0x7facb80fce70 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb77fe700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb810dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb7fff700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7facac009380 tx=0x7facac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac00e040 con 0x7facb80fce70 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facac00bd20 con 0x7facb80fce70 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb8112f10 con 0x7facb80fce70 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7facac01c630 con 0x7facb80fce70 2024-03-14T12:58:30.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.434+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb8113440 con 0x7facb80fce70 2024-03-14T12:58:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.435+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb804c710 con 0x7facb80fce70 2024-03-14T12:58:30.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.436+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7facac01c790 con 0x7facb80fce70 2024-03-14T12:58:30.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.437+0000 7facb4ff9700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 0x7faca006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:30.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.437+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7facac088310 con 0x7facb80fce70 2024-03-14T12:58:30.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.437+0000 7facb77fe700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 0x7faca006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:30.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.437+0000 7facb77fe700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 0x7faca006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faca8009fd0 tx=0x7faca8009ee0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:30.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.441+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7facac058170 con 0x7facb80fce70 2024-03-14T12:58:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:30 smithi111 ceph-mon[39688]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-14T12:58:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:30 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] up:standby-replay 2024-03-14T12:58:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:30 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:30 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:30.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:30.674+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7facb8113720 con 0x7facb80fce70 2024-03-14T12:58:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:30 smithi137 ceph-mon[50336]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-14T12:58:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:30 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] up:standby-replay 2024-03-14T12:58:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:30 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:30 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:30.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:30 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:31.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.274+0000 7facb4ff9700 1 -- 172.21.15.111:0/3095604830 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v17) v1 ==== 133+0+0 (secure 0 0 0) 0x7facac057d00 con 0x7facb80fce70 2024-03-14T12:58:31.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 msgr2=0x7faca006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:31.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 0x7faca006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faca8009fd0 tx=0x7faca8009ee0 comp rx=0 tx=0).stop 2024-03-14T12:58:31.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 msgr2=0x7facb810d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:31.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7facac009380 tx=0x7facac009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:31.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 shutdown_connections 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7faca00692a0 0x7faca006b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facb80fce70 0x7facb810d860 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 --2- 172.21.15.111:0/3095604830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb80fdfe0 0x7facb810dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 >> 172.21.15.111:0/3095604830 conn(0x7facb80f8550 msgr2=0x7facb80f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 shutdown_connections 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:31.277+0000 7facbe3e7700 1 -- 172.21.15.111:0/3095604830 wait complete. 2024-03-14T12:58:31.278 INFO:teuthology.orchestra.run.smithi111.stderr:inline data disabled 2024-03-14T12:58:31.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:31 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3095604830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-14T12:58:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:31 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3095604830' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-03-14T12:58:32.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:32 smithi111 ceph-mon[39688]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-14T12:58:32.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:32 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3095604830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-14T12:58:32.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:32 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:active 2024-03-14T12:58:32.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:32 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:32.633 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T12:58:32.644 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:32.645 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-14T12:58:32.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:32 smithi137 ceph-mon[50336]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-14T12:58:32.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:32 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3095604830' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-03-14T12:58:32.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:32 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] up:active 2024-03-14T12:58:32.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:32 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:33.619 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:33 smithi137 ceph-mon[50336]: mds.? [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] up:standby-replay 2024-03-14T12:58:33.619 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:33 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:33 smithi111 ceph-mon[39688]: mds.? [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] up:standby-replay 2024-03-14T12:58:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:33 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T12:58:33.855 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:34 smithi111 ceph-mon[39688]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-14T12:58:34.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:34 smithi137 ceph-mon[50336]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-14T12:58:35.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- 172.21.15.111:0/2371431862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 msgr2=0x7fbb440fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 --2- 172.21.15.111:0/2371431862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fd3e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbb30004940 tx=0x7fbb30009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- 172.21.15.111:0/2371431862 shutdown_connections 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 --2- 172.21.15.111:0/2371431862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fd3e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 --2- 172.21.15.111:0/2371431862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 0x7fbb440fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- 172.21.15.111:0/2371431862 >> 172.21.15.111:0/2371431862 conn(0x7fbb440f74e0 msgr2=0x7fbb440f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- 172.21.15.111:0/2371431862 shutdown_connections 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- 172.21.15.111:0/2371431862 wait complete. 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 Processor -- start 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.733+0000 7fbb4acba700 1 -- start start 2024-03-14T12:58:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 0x7fbb440ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440fe7f0 con 0x7fbb440fcf70 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440fe930 con 0x7fbb440fbe00 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45842/0 (socket says 172.21.15.111:45842) 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 -- 172.21.15.111:0/1227508235 learned_addr learned my addr 172.21.15.111:0/1227508235 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb48a56700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 0x7fbb440ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 -- 172.21.15.111:0/1227508235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 msgr2=0x7fbb440ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 0x7fbb440ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 -- 172.21.15.111:0/1227508235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb440fcf70 2024-03-14T12:58:35.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb43fff700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbb300137b0 tx=0x7fbb30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:35.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb3000e070 con 0x7fbb440fcf70 2024-03-14T12:58:35.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb440febb0 con 0x7fbb440fcf70 2024-03-14T12:58:35.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb3001c400 con 0x7fbb440fcf70 2024-03-14T12:58:35.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb3000b650 con 0x7fbb440fcf70 2024-03-14T12:58:35.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.734+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb440ff110 con 0x7fbb440fcf70 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.735+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb44101b00 con 0x7fbb440fcf70 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.736+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbb3001c9e0 con 0x7fbb440fcf70 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.737+0000 7fbb417fa700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 0x7fbb3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.737+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbb30089220 con 0x7fbb440fcf70 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.737+0000 7fbb48a56700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 0x7fbb3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:35.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.737+0000 7fbb48a56700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 0x7fbb3406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbb440ffad0 tx=0x7fbb3800b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:35.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.741+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fbb300555f0 con 0x7fbb440fcf70 2024-03-14T12:58:36.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:35.999+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbb4405f5f0 con 0x7fbb440fcf70 2024-03-14T12:58:36.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.000+0000 7fbb417fa700 1 -- 172.21.15.111:0/1227508235 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1837 (secure 0 0 0) 0x7fbb30004490 con 0x7fbb440fcf70 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:e18 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 18 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T12:58:32.274858+0000 2024-03-14T12:58:36.001 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T12:58:36.002 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T12:58:36.003 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T12:58:36.004 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T12:58:36.004 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T12:58:36.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:36.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 msgr2=0x7fbb3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 0x7fbb3406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbb440ffad0 tx=0x7fbb3800b040 comp rx=0 tx=0).stop 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 msgr2=0x7fbb440fe2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbb300137b0 tx=0x7fbb30009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 shutdown_connections 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb440fbe00 0x7fbb440ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fbb340692a0 0x7fbb3406b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 --2- 172.21.15.111:0/1227508235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb440fcf70 0x7fbb440fe2b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 >> 172.21.15.111:0/1227508235 conn(0x7fbb440f74e0 msgr2=0x7fbb44066460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 shutdown_connections 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:36.002+0000 7fbb4acba700 1 -- 172.21.15.111:0/1227508235 wait complete. 2024-03-14T12:58:36.005 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 18 2024-03-14T12:58:36.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:36.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:36 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1227508235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T12:58:36.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:58:36.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:36 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1227508235' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T12:58:37.010 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-14T12:58:37.474 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:38 smithi111 ceph-mon[39688]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:58:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:39.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:38 smithi137 ceph-mon[50336]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:58:39.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:39.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 -- 172.21.15.111:0/625148339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fe070 msgr2=0x7f98d40fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 --2- 172.21.15.111:0/625148339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fe070 0x7f98d40fe4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f98bc004940 tx=0x7f98bc009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 -- 172.21.15.111:0/625148339 shutdown_connections 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 --2- 172.21.15.111:0/625148339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fe070 0x7f98d40fe4e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 --2- 172.21.15.111:0/625148339 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fcf00 0x7f98d40fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 -- 172.21.15.111:0/625148339 >> 172.21.15.111:0/625148339 conn(0x7f98d40f85a0 msgr2=0x7f98d40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:39.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 -- 172.21.15.111:0/625148339 shutdown_connections 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.905+0000 7f98d9f76700 1 -- 172.21.15.111:0/625148339 wait complete. 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 Processor -- start 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 -- start start 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 0x7f98d410dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d410e4e0 con 0x7f98d40fcf00 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d9f76700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d410ed30 con 0x7f98d40fe070 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45858/0 (socket says 172.21.15.111:45858) 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98d37fe700 1 -- 172.21.15.111:0/1237278982 learned_addr learned my addr 172.21.15.111:0/1237278982 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.906+0000 7f98caffd700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 0x7f98d410dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d37fe700 1 -- 172.21.15.111:0/1237278982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 msgr2=0x7f98d410dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d37fe700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 0x7f98d410dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d37fe700 1 -- 172.21.15.111:0/1237278982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc005040 con 0x7f98d40fcf00 2024-03-14T12:58:39.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98caffd700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 0x7f98d410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d37fe700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f98c4002a50 tx=0x7f98c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c4007cc0 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98c400f040 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d410f010 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c400da70 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.907+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d410f550 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.908+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d405f5f0 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.908+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f98c400dc10 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.909+0000 7f98d0ff9700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 0x7f98c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.909+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f98c40868c0 con 0x7f98d40fcf00 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.909+0000 7f98caffd700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 0x7f98c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.909+0000 7f98caffd700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 0x7f98c006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b3b0 tx=0x7f98bc009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:39.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:39.913+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f98c4056600 con 0x7f98d40fcf00 2024-03-14T12:58:40.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.143+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d41057d0 con 0x7f98d40fcf00 2024-03-14T12:58:40.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.143+0000 7f98d0ff9700 1 -- 172.21.15.111:0/1237278982 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4729 (secure 0 0 0) 0x7f98c4056190 con 0x7f98d40fcf00 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 msgr2=0x7f98c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 0x7f98c006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f98bc00b3b0 tx=0x7f98bc009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 msgr2=0x7f98d410d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f98c4002a50 tx=0x7f98c40046b0 comp rx=0 tx=0).stop 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 shutdown_connections 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f98c00692a0 0x7f98c006b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:40.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98d40fcf00 0x7f98d410d9a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:40.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 --2- 172.21.15.111:0/1237278982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98d40fe070 0x7f98d410dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:40.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 >> 172.21.15.111:0/1237278982 conn(0x7f98d40f85a0 msgr2=0x7f98d40f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:40.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 shutdown_connections 2024-03-14T12:58:40.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:40.145+0000 7f98d9f76700 1 -- 172.21.15.111:0/1237278982 wait complete. 2024-03-14T12:58:40.147 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 18 2024-03-14T12:58:40.156 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T12:58:40.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:40 smithi111 ceph-mon[39688]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-14T12:58:40.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:40 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1237278982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T12:58:40.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-14T12:58:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:40 smithi137 ceph-mon[50336]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-14T12:58:41.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:40 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1237278982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T12:58:41.313 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:42 smithi111 ceph-mon[39688]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-03-14T12:58:43.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:42 smithi137 ceph-mon[50336]: pgmap v198: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 3 op/s 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 -- 172.21.15.111:0/3524341992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 msgr2=0x7fd96c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 --2- 172.21.15.111:0/3524341992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c0fd3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd960004940 tx=0x7fd960009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 -- 172.21.15.111:0/3524341992 shutdown_connections 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 --2- 172.21.15.111:0/3524341992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c0fd3f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 --2- 172.21.15.111:0/3524341992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 -- 172.21.15.111:0/3524341992 >> 172.21.15.111:0/3524341992 conn(0x7fd96c0f7430 msgr2=0x7fd96c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.741+0000 7fd971b7d700 1 -- 172.21.15.111:0/3524341992 shutdown_connections 2024-03-14T12:58:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 -- 172.21.15.111:0/3524341992 wait complete. 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 Processor -- start 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 -- start start 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c10d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c10e4e0 con 0x7fd96c0fcf80 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd971b7d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96c10ece0 con 0x7fd96c0fbcd0 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd96affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd96affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33450/0 (socket says 172.21.15.111:33450) 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.742+0000 7fd96affd700 1 -- 172.21.15.111:0/2938238512 learned_addr learned my addr 172.21.15.111:0/2938238512 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96b7fe700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c10d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96affd700 1 -- 172.21.15.111:0/2938238512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 msgr2=0x7fd96c10d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96affd700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c10d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96affd700 1 -- 172.21.15.111:0/2938238512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960005040 con 0x7fd96c0fcf80 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96b7fe700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd96affd700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd96000b3b0 tx=0x7fd960009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd96000e040 con 0x7fd96c0fcf80 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd96000bd20 con 0x7fd96c0fcf80 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96c10ef60 con 0x7fd96c0fcf80 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd96001c630 con 0x7fd96c0fcf80 2024-03-14T12:58:43.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.743+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c10f4a0 con 0x7fd96c0fcf80 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.744+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd96c101b10 con 0x7fd96c0fcf80 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.744+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd960013420 con 0x7fd96c0fcf80 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.745+0000 7fd953fff700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 0x7fd95406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.745+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd960087f70 con 0x7fd96c0fcf80 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.745+0000 7fd96b7fe700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 0x7fd95406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:43.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.745+0000 7fd96b7fe700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 0x7fd95406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009fd0 tx=0x7fd95c00b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:43.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:43.749+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd9600b9d70 con 0x7fd96c0fcf80 2024-03-14T12:58:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.013+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fd96c05f5f0 con 0x7fd96c0fcf80 2024-03-14T12:58:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.013+0000 7fd953fff700 1 -- 172.21.15.111:0/2938238512 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+99 (secure 0 0 0) 0x7fd960057cb0 con 0x7fd96c0fcf80 2024-03-14T12:58:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 msgr2=0x7fd95406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 0x7fd95406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009fd0 tx=0x7fd95c00b040 comp rx=0 tx=0).stop 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 msgr2=0x7fd96c10dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd96000b3b0 tx=0x7fd960009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 shutdown_connections 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd96c0fbcd0 0x7fd96c10d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd9540692a0 0x7fd95406b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 --2- 172.21.15.111:0/2938238512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd96c0fcf80 0x7fd96c10dee0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 >> 172.21.15.111:0/2938238512 conn(0x7fd96c0f7430 msgr2=0x7fd96c065c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 shutdown_connections 2024-03-14T12:58:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.015+0000 7fd971b7d700 1 -- 172.21.15.111:0/2938238512 wait complete. 2024-03-14T12:58:44.024 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T12:58:44.798 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:44 smithi111 ceph-mon[39688]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T12:58:44.798 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:44 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2938238512' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-14T12:58:44.820 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-14T12:58:44.833 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T12:58:44.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.972+0000 7f5e6de2f700 1 -- 172.21.15.111:0/3291485763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 msgr2=0x7f5e680fcfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.972+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/3291485763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e680fcfa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5e50004940 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 -- 172.21.15.111:0/3291485763 shutdown_connections 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/3291485763 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e680fcfa0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/3291485763 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 -- 172.21.15.111:0/3291485763 >> 172.21.15.111:0/3291485763 conn(0x7f5e680f6ad0 msgr2=0x7f5e680f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 -- 172.21.15.111:0/3291485763 shutdown_connections 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 -- 172.21.15.111:0/3291485763 wait complete. 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 Processor -- start 2024-03-14T12:58:44.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.973+0000 7f5e6de2f700 1 -- start start 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e68112a50 con 0x7f5e680fcb30 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e68112b90 con 0x7f5e680fc1e0 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33470/0 (socket says 172.21.15.111:33470) 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 -- 172.21.15.111:0/2448821205 learned_addr learned my addr 172.21.15.111:0/2448821205 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e677fe700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 -- 172.21.15.111:0/2448821205 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 msgr2=0x7f5e680fb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 -- 172.21.15.111:0/2448821205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50005040 con 0x7f5e680fcb30 2024-03-14T12:58:44.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e677fe700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e66ffd700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009380 tx=0x7f5e50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e5000e040 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e68112dc0 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e5000bd20 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e5001c630 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.974+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e68113320 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.975+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e68064dc0 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.976+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5e50026440 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.976+0000 7f5e5ffff700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 0x7f5e5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.976+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5e50087fc0 con 0x7f5e680fcb30 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.976+0000 7f5e677fe700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 0x7f5e5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:44.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.977+0000 7f5e677fe700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 0x7f5e5406b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e58007fd0 tx=0x7f5e58009040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:44.981+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f5e50057d80 con 0x7f5e680fcb30 2024-03-14T12:58:45.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:44 smithi137 ceph-mon[50336]: pgmap v199: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T12:58:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:44 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2938238512' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-14T12:58:45.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.193+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e68113770 con 0x7f5e680fcb30 2024-03-14T12:58:45.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.194+0000 7f5e5ffff700 1 -- 172.21.15.111:0/2448821205 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14303 (secure 0 0 0) 0x7f5e50057910 con 0x7f5e680fcb30 2024-03-14T12:58:45.194 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:45.194 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":46,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T12:58:16.792117+0000","last_up_change":"2024-03-14T12:56:50.540579+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-14T12:58:13.229267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-14T12:58:14.001600+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6803","nonce":3771249398}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6805","nonce":3771249398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6809","nonce":3771249398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":3771249398},{"type":"v1","addr":"172.21.15.111:6807","nonce":3771249398}]},"public_addr":"172.21.15.111:6803/3771249398","cluster_addr":"172.21.15.111:6805/3771249398","heartbeat_back_addr":"172.21.15.111:6809/3771249398","heartbeat_front_addr":"172.21.15.111:6807/3771249398","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6811","nonce":3746289562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6813","nonce":3746289562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6817","nonce":3746289562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":3746289562},{"type":"v1","addr":"172.21.15.111:6815","nonce":3746289562}]},"public_addr":"172.21.15.111:6811/3746289562","cluster_addr":"172.21.15.111:6813/3746289562","heartbeat_back_addr":"172.21.15.111:6817/3746289562","heartbeat_front_addr":"172.21.15.111:6815/3746289562","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6819","nonce":337068597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6821","nonce":337068597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6825","nonce":337068597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":337068597},{"type":"v1","addr":"172.21.15.111:6823","nonce":337068597}]},"public_addr":"172.21.15.111:6819/337068597","cluster_addr":"172.21.15.111:6821/337068597","heartbeat_back_addr":"172.21.15.111:6825/337068597","heartbeat_front_addr":"172.21.15.111:6823/337068597","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6801","nonce":316104226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6803","nonce":316104226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6807","nonce":316104226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":316104226},{"type":"v1","addr":"172.21.15.137:6805","nonce":316104226}]},"public_addr":"172.21.15.137:6801/316104226","cluster_addr":"172.21.15.137:6803/316104226","heartbeat_back_addr":"172.21.15.137:6807/316104226","heartbeat_front_addr":"172.21.15.137:6805/316104226","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6809","nonce":1988641735}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6811","nonce":1988641735}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6815","nonce":1988641735}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1988641735},{"type":"v1","addr":"172.21.15.137:6813","nonce":1988641735}]},"public_addr":"172.21.15.137:6809/1988641735","cluster_addr":"172.21.15.137:6811/1988641735","heartbeat_back_addr":"172.21.15.137:6815/1988641735","heartbeat_front_addr":"172.21.15.137:6813/1988641735","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6817","nonce":1979294971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6819","nonce":1979294971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6823","nonce":1979294971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":1979294971},{"type":"v1","addr":"172.21.15.137:6821","nonce":1979294971}]},"public_addr":"172.21.15.137:6817/1979294971","cluster_addr":"172.21.15.137:6819/1979294971","heartbeat_back_addr":"172.21.15.137:6823/1979294971","heartbeat_front_addr":"172.21.15.137:6821/1979294971","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 msgr2=0x7f5e5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 0x7f5e5406b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e58007fd0 tx=0x7f5e58009040 comp rx=0 tx=0).stop 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 msgr2=0x7f5e68112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009380 tx=0x7f5e50009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 shutdown_connections 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e680fc1e0 0x7f5e680fb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5e540692a0 0x7f5e5406b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 --2- 172.21.15.111:0/2448821205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e680fcb30 0x7f5e68112510 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 >> 172.21.15.111:0/2448821205 conn(0x7f5e680f6ad0 msgr2=0x7f5e680f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 shutdown_connections 2024-03-14T12:58:45.196 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.196+0000 7f5e6de2f700 1 -- 172.21.15.111:0/2448821205 wait complete. 2024-03-14T12:58:45.209 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T12:55:33.503741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-14T12:58:13.229267+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-14T12:58:14.001600+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-14T12:58:45.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- 172.21.15.111:0/3218858589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0ffe50 msgr2=0x7f3a6c102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 --2- 172.21.15.111:0/3218858589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0ffe50 0x7f3a6c102230 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a68004940 tx=0x7f3a68009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- 172.21.15.111:0/3218858589 shutdown_connections 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 --2- 172.21.15.111:0/3218858589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0ffe50 0x7f3a6c102230 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 --2- 172.21.15.111:0/3218858589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- 172.21.15.111:0/3218858589 >> 172.21.15.111:0/3218858589 conn(0x7f3a6c0f7300 msgr2=0x7f3a6c0f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- 172.21.15.111:0/3218858589 shutdown_connections 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- 172.21.15.111:0/3218858589 wait complete. 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 Processor -- start 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.332+0000 7f3a73626700 1 -- start start 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0fd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0fd730 con 0x7f3a6c0fb810 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c0fc1c0 con 0x7f3a6c065a20 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33486/0 (socket says 172.21.15.111:33486) 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 -- 172.21.15.111:0/3619673490 learned_addr learned my addr 172.21.15.111:0/3619673490 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:45.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 -- 172.21.15.111:0/3619673490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 msgr2=0x7f3a6c0fd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a713c2700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0fd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 -- 172.21.15.111:0/3619673490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a68005040 con 0x7f3a6c0fb810 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a713c2700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0fd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a70bc1700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800b900 tx=0x7f3a68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6800e040 con 0x7f3a6c0fb810 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c0fc440 con 0x7f3a6c0fb810 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a68004290 con 0x7f3a6c0fb810 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a6801c3f0 con 0x7f3a6c0fb810 2024-03-14T12:58:45.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.333+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c115c20 con 0x7f3a6c0fb810 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.334+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c064970 con 0x7f3a6c0fb810 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.335+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3a68013790 con 0x7f3a6c0fb810 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.335+0000 7f3a61ffb700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 0x7f3a5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.335+0000 7f3a713c2700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 0x7f3a5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.335+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3a680884f0 con 0x7f3a6c0fb810 2024-03-14T12:58:45.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.336+0000 7f3a713c2700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 0x7f3a5806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.340+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f3a68058210 con 0x7f3a6c0fb810 2024-03-14T12:58:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.521+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a6c1056e0 con 0x7f3a6c0fb810 2024-03-14T12:58:45.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.522+0000 7f3a61ffb700 1 -- 172.21.15.111:0/3619673490 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a6c1056e0 con 0x7f3a6c0fb810 2024-03-14T12:58:45.521 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 msgr2=0x7f3a5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 0x7f3a5806b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c006fd0 tx=0x7f3a5c008040 comp rx=0 tx=0).stop 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 msgr2=0x7f3a6c0fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800b900 tx=0x7f3a68009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 shutdown_connections 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a6c065a20 0x7f3a6c0fd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f3a580692a0 0x7f3a5806b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 --2- 172.21.15.111:0/3619673490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a6c0fb810 0x7f3a6c0fbc80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 >> 172.21.15.111:0/3619673490 conn(0x7f3a6c0f7300 msgr2=0x7f3a6c0f7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 shutdown_connections 2024-03-14T12:58:45.524 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.524+0000 7f3a73626700 1 -- 172.21.15.111:0/3619673490 wait complete. 2024-03-14T12:58:45.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- 172.21.15.111:0/344449774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 msgr2=0x7fccec1021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 --2- 172.21.15.111:0/344449774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec1021f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcce8004940 tx=0x7fcce8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- 172.21.15.111:0/344449774 shutdown_connections 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 --2- 172.21.15.111:0/344449774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec1021f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 --2- 172.21.15.111:0/344449774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 0x7fccec0ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- 172.21.15.111:0/344449774 >> 172.21.15.111:0/344449774 conn(0x7fccec0f7300 msgr2=0x7fccec0f9730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- 172.21.15.111:0/344449774 shutdown_connections 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- 172.21.15.111:0/344449774 wait complete. 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 Processor -- start 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.656+0000 7fccf40d5700 1 -- start start 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 0x7fccec10d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec10e320 con 0x7fccec0ffdd0 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec10e490 con 0x7fccec065aa0 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33498/0 (socket says 172.21.15.111:33498) 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 -- 172.21.15.111:0/3522747623 learned_addr learned my addr 172.21.15.111:0/3522747623 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1e71700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 0x7fccec10d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 -- 172.21.15.111:0/3522747623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 msgr2=0x7fccec10d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 0x7fccec10d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 -- 172.21.15.111:0/3522747623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8005040 con 0x7fccec0ffdd0 2024-03-14T12:58:45.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf1670700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b3b0 tx=0x7fcce8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce800e040 con 0x7fccec0ffdd0 2024-03-14T12:58:45.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcce8004290 con 0x7fccec0ffdd0 2024-03-14T12:58:45.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce801c3f0 con 0x7fccec0ffdd0 2024-03-14T12:58:45.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec112e40 con 0x7fccec0ffdd0 2024-03-14T12:58:45.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.657+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec1133d0 con 0x7fccec0ffdd0 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.658+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec064970 con 0x7fccec0ffdd0 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.659+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcce8013420 con 0x7fccec0ffdd0 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.659+0000 7fcce27fc700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 0x7fccd806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.659+0000 7fccf1e71700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 0x7fccd806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.659+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcce8088400 con 0x7fccec0ffdd0 2024-03-14T12:58:45.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.659+0000 7fccf1e71700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 0x7fccd806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fccec0657e0 tx=0x7fccdc009ee0 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.664+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fcce8058260 con 0x7fccec0ffdd0 2024-03-14T12:58:45.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.845+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fccec113640 con 0x7fccec0ffdd0 2024-03-14T12:58:45.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.846+0000 7fcce27fc700 1 -- 172.21.15.111:0/3522747623 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcce8057df0 con 0x7fccec0ffdd0 2024-03-14T12:58:45.845 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T12:58:45.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 msgr2=0x7fccd806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 0x7fccd806b750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fccec0657e0 tx=0x7fccdc009ee0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 msgr2=0x7fccec10dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b3b0 tx=0x7fcce8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 shutdown_connections 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec065aa0 0x7fccec10d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fccd80692a0 0x7fccd806b750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 --2- 172.21.15.111:0/3522747623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccec0ffdd0 0x7fccec10dd20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 >> 172.21.15.111:0/3522747623 conn(0x7fccec0f7300 msgr2=0x7fccec0f7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 shutdown_connections 2024-03-14T12:58:45.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.848+0000 7fccf40d5700 1 -- 172.21.15.111:0/3522747623 wait complete. 2024-03-14T12:58:45.859 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- 172.21.15.111:0/3345390021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b0066850 msgr2=0x7f26b00ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 --2- 172.21.15.111:0/3345390021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b0066850 0x7f26b00ff930 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004940 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- 172.21.15.111:0/3345390021 shutdown_connections 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 --2- 172.21.15.111:0/3345390021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b00ffe70 0x7f26b0102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 --2- 172.21.15.111:0/3345390021 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b0066850 0x7f26b00ff930 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- 172.21.15.111:0/3345390021 >> 172.21.15.111:0/3345390021 conn(0x7f26b00f7360 msgr2=0x7f26b00f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:45.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- 172.21.15.111:0/3345390021 shutdown_connections 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- 172.21.15.111:0/3345390021 wait complete. 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 Processor -- start 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- start start 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0066850 0x7f26b010d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b010e320 con 0x7f26b00ffe70 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.980+0000 7f26b6957700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b010e490 con 0x7f26b0066850 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33512/0 (socket says 172.21.15.111:33512) 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 -- 172.21.15.111:0/803363833 learned_addr learned my addr 172.21.15.111:0/803363833 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26affff700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0066850 0x7f26b010d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 -- 172.21.15.111:0/803363833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0066850 msgr2=0x7f26b010d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:45.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0066850 0x7f26b010d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 -- 172.21.15.111:0/803363833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0005040 con 0x7f26b00ffe70 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26af7fe700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c830 tx=0x7f26a4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a40106f0 con 0x7f26b00ffe70 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26a4010d00 con 0x7f26b00ffe70 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b0112ea0 con 0x7f26b00ffe70 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26a4012e90 con 0x7f26b00ffe70 2024-03-14T12:58:45.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.981+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b0113430 con 0x7f26b00ffe70 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.982+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b00640e0 con 0x7f26b00ffe70 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.982+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f26a4010850 con 0x7f26b00ffe70 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.983+0000 7f26acff9700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 0x7f269806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.983+0000 7f26affff700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 0x7f269806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.983+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f26a4087180 con 0x7f26b00ffe70 2024-03-14T12:58:45.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.983+0000 7f26affff700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 0x7f269806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004890 tx=0x7f26a0009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:45.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:45.987+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f26a4056f40 con 0x7f26b00ffe70 2024-03-14T12:58:46.108 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.108 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.174+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f26b01136b0 con 0x7f26b00ffe70 2024-03-14T12:58:46.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.175+0000 7f26acff9700 1 -- 172.21.15.111:0/803363833 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f26a4056ad0 con 0x7f26b00ffe70 2024-03-14T12:58:46.174 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 msgr2=0x7f269806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 0x7f269806b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f26a0004890 tx=0x7f26a0009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 msgr2=0x7f26b010dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c830 tx=0x7f26a4004470 comp rx=0 tx=0).stop 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 shutdown_connections 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0066850 0x7f26b010d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f26980692a0 0x7f269806b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 --2- 172.21.15.111:0/803363833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26b00ffe70 0x7f26b010dd20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 >> 172.21.15.111:0/803363833 conn(0x7f26b00f7360 msgr2=0x7f26b00f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 shutdown_connections 2024-03-14T12:58:46.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.177+0000 7f26b6957700 1 -- 172.21.15.111:0/803363833 wait complete. 2024-03-14T12:58:46.188 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T12:58:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 -- 172.21.15.111:0/739425776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 msgr2=0x7fcee80fd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 --2- 172.21.15.111:0/739425776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fd6f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fced8004940 tx=0x7fced8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 -- 172.21.15.111:0/739425776 shutdown_connections 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 --2- 172.21.15.111:0/739425776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 0x7fcee80fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 --2- 172.21.15.111:0/739425776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fd6f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 -- 172.21.15.111:0/739425776 >> 172.21.15.111:0/739425776 conn(0x7fcee80f7360 msgr2=0x7fcee80f9790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 -- 172.21.15.111:0/739425776 shutdown_connections 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.306+0000 7fceef788700 1 -- 172.21.15.111:0/739425776 wait complete. 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 Processor -- start 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 -- start start 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 0x7fcee80ffaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee81000f0 con 0x7fcee80fd2e0 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee80fe730 con 0x7fcee80fb8d0 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:46.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33520/0 (socket says 172.21.15.111:33520) 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 -- 172.21.15.111:0/4108876174 learned_addr learned my addr 172.21.15.111:0/4108876174 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceed524700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 0x7fcee80ffaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 -- 172.21.15.111:0/4108876174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 msgr2=0x7fcee80ffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 0x7fcee80ffaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 -- 172.21.15.111:0/4108876174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8005040 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceecd23700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c830 tx=0x7fcee4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee400a760 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee80fea10 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcee400ad70 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.307+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee40113a0 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.308+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee80fef50 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.309+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee8101a40 con 0x7fcee80fd2e0 2024-03-14T12:58:46.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.309+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcee4012400 con 0x7fcee80fd2e0 2024-03-14T12:58:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.309+0000 7fceddffb700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 0x7fced406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.310+0000 7fceed524700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 0x7fced406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.310+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcee4087220 con 0x7fcee80fd2e0 2024-03-14T12:58:46.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.310+0000 7fceed524700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 0x7fced406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fced800b3b0 tx=0x7fced8009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:46.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.314+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fcee4056fe0 con 0x7fcee80fd2e0 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2448821205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:58:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3619673490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T12:58:46.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:46 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3522747623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T12:58:46.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2448821205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: pgmap v200: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3619673490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T12:58:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:46 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3522747623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T12:58:46.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.540+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee80640e0 con 0x7fcee80fd2e0 2024-03-14T12:58:46.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.540+0000 7fceddffb700 1 -- 172.21.15.111:0/4108876174 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4729 (secure 0 0 0) 0x7fcee4056b70 con 0x7fcee80fd2e0 2024-03-14T12:58:46.540 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T12:58:46.540 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T12:58:32.274858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi111.yzwmcx","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6829/1527723924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1527723924},{"type":"v1","addr":"172.21.15.111:6829","nonce":1527723924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/3644044473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3644044473},{"type":"v1","addr":"172.21.15.137:6827","nonce":3644044473}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-14T12:58:46.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 msgr2=0x7fced406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 0x7fced406b750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fced800b3b0 tx=0x7fced8009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 msgr2=0x7fcee80fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcee400c830 tx=0x7fcee4004470 comp rx=0 tx=0).stop 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 shutdown_connections 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee80fb8d0 0x7fcee80ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fced40692a0 0x7fced406b750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 --2- 172.21.15.111:0/4108876174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcee80fd2e0 0x7fcee80fe1f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 >> 172.21.15.111:0/4108876174 conn(0x7fcee80f7360 msgr2=0x7fcee8066260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:46.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 shutdown_connections 2024-03-14T12:58:46.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:46.542+0000 7fceef788700 1 -- 172.21.15.111:0/4108876174 wait complete. 2024-03-14T12:58:46.545 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 18 2024-03-14T12:58:46.554 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-03-14T12:58:46.554 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-14T12:58:46.578 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-14T12:58:46.578 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-03-14T12:58:46.579 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-14T12:58:46.579 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-14T12:58:46.579 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:46.579 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-14T12:58:46.579 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-14T12:58:46.579 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:46.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:46.579 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-03-14T12:58:46.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:46.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-03-14T12:58:46.680 INFO:teuthology.orchestra.run.smithi137.stderr:Cannot find device "ceph-brx" 2024-03-14T12:58:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:58:46.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:46.682 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2024-03-14T12:58:46.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:46.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link delete ceph-brx 2024-03-14T12:58:46.769 INFO:teuthology.orchestra.run.smithi111.stderr:Cannot find device "ceph-brx" 2024-03-14T12:58:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T12:58:46.771 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-14T12:58:46.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-14T12:58:47.074 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:47.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:47 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/803363833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T12:58:47.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:47 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/4108876174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T12:58:47.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:47 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/803363833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T12:58:47.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:47 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/4108876174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T12:58:48.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:48 smithi111 ceph-mon[39688]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:58:48.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:48 smithi137 ceph-mon[50336]: pgmap v201: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:58:48.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/111622803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a180fb4a0 msgr2=0x7f5a180fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:48.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/111622803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a180fb4a0 0x7f5a180fb910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c004940 tx=0x7f5a0c009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:48.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/111622803 shutdown_connections 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/111622803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a180fb4a0 0x7f5a180fb910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/111622803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a18102e70 0x7f5a18103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/111622803 >> 172.21.15.111:0/111622803 conn(0x7f5a180714c0 msgr2=0x7f5a180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/111622803 shutdown_connections 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.992+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/111622803 wait complete. 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.993+0000 7f5a1d0f3700 1 Processor -- start 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.993+0000 7f5a1d0f3700 1 -- start start 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 0x7f5a18113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a181146b0 con 0x7f5a18102e70 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1810da10 con 0x7f5a180fb4a0 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33538/0 (socket says 172.21.15.111:33538) 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 -- 172.21.15.111:0/2678129247 learned_addr learned my addr 172.21.15.111:0/2678129247 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:48.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a16d9d700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 0x7f5a18113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 -- 172.21.15.111:0/2678129247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 msgr2=0x7f5a18113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 0x7f5a18113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 -- 172.21.15.111:0/2678129247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c005040 con 0x7f5a18102e70 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a16d9d700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 0x7f5a18113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1659c700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00b3b0 tx=0x7f5a0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a0c00e040 con 0x7f5a18102e70 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1810dc90 con 0x7f5a18102e70 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a0c013420 con 0x7f5a18102e70 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a0c01c630 con 0x7f5a18102e70 2024-03-14T12:58:48.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.994+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1810e250 con 0x7f5a18102e70 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.995+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18100110 con 0x7f5a18102e70 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.996+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5a0c013970 con 0x7f5a18102e70 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.996+0000 7f59ff7fe700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 0x7f5a0006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.996+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5a0c0883d0 con 0x7f5a18102e70 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.996+0000 7f5a16d9d700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 0x7f5a0006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:48.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:48.997+0000 7f5a16d9d700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 0x7f5a0006b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a08006fd0 tx=0x7f5a08008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:49.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.001+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f5a0c058190 con 0x7f5a18102e70 2024-03-14T12:58:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.225+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5a18065380 con 0x7f5a18102e70 2024-03-14T12:58:49.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.225+0000 7f59ff7fe700 1 -- 172.21.15.111:0/2678129247 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f5a0c057d20 con 0x7f5a18102e70 2024-03-14T12:58:49.225 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 msgr2=0x7f5a0006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 0x7f5a0006b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a08006fd0 tx=0x7f5a08008040 comp rx=0 tx=0).stop 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 msgr2=0x7f5a18113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00b3b0 tx=0x7f5a0c009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 shutdown_connections 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a180fb4a0 0x7f5a18113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f5a000692a0 0x7f5a0006b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 --2- 172.21.15.111:0/2678129247 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a18102e70 0x7f5a18113fd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 >> 172.21.15.111:0/2678129247 conn(0x7f5a180714c0 msgr2=0x7f5a180fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:49.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 shutdown_connections 2024-03-14T12:58:49.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:49.227+0000 7f5a1d0f3700 1 -- 172.21.15.111:0/2678129247 wait complete. 2024-03-14T12:58:49.608 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:49 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2678129247' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T12:58:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:49 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2678129247' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi111.front.sepia.ceph.com 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:50.065 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-14T12:58:50.066 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-14T12:58:50.066 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-14T12:58:50.066 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-14T12:58:50.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.066 DEBUG:teuthology.orchestra.run.smithi111:> ip addr 2024-03-14T12:58:50.085 INFO:teuthology.orchestra.run.smithi111.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-14T12:58:50.085 INFO:teuthology.orchestra.run.smithi111.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-14T12:58:50.085 INFO:teuthology.orchestra.run.smithi111.stdout: inet 127.0.0.1/8 scope host lo 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 ::1/128 scope host 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:24 brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f0 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:d8:a1:25 brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: altname enp7s0f1 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2e brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: link/ether 0c:c4:7a:8f:cc:2f brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:50.086 INFO:teuthology.orchestra.run.smithi111.stdout: inet 172.21.15.111/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-14T12:58:50.087 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft 41995sec preferred_lft 41995sec 2024-03-14T12:58:50.087 INFO:teuthology.orchestra.run.smithi111.stdout: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2024-03-14T12:58:50.087 INFO:teuthology.orchestra.run.smithi111.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:50.087 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-14T12:58:50.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.087 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-14T12:58:50.087 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-14T12:58:50.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add name ceph-brx type bridge 2024-03-14T12:58:50.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr flush dev ceph-brx 2024-03-14T12:58:50.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set ceph-brx up 2024-03-14T12:58:50.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-14T12:58:50.088 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-14T12:58:50.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.257 DEBUG:teuthology.orchestra.run.smithi111:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-14T12:58:50.285 INFO:teuthology.orchestra.run.smithi111.stdout:1 2024-03-14T12:58:50.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.287 DEBUG:teuthology.orchestra.run.smithi111:> route 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run.smithi111.stdout:Kernel IP routing table 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run.smithi111.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run.smithi111.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run.smithi111.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-14T12:58:50.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-14T12:58:50.378 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-14T12:58:50.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:50 smithi111 ceph-mon[39688]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:58:50.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.604 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list 2024-03-14T12:58:50.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.663 DEBUG:teuthology.orchestra.run.smithi111:> ip netns list-id 2024-03-14T12:58:50.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.721 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-14T12:58:50.721 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-14T12:58:50.721 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-14T12:58:50.721 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-14T12:58:50.721 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-14T12:58:50.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:50 smithi137 ceph-mon[50336]: pgmap v202: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:58:50.816 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-14T12:58:50.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-14T12:58:50.816 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-14T12:58:51.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:51.019 DEBUG:teuthology.orchestra.run.smithi111:> (cd / && exec bash -c ' 2024-03-14T12:58:51.019 DEBUG:teuthology.orchestra.run.smithi111:> set -e 2024-03-14T12:58:51.019 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set brx.0 up 2024-03-14T12:58:51.019 DEBUG:teuthology.orchestra.run.smithi111:> sudo ip link set dev brx.0 master ceph-brx 2024-03-14T12:58:51.019 DEBUG:teuthology.orchestra.run.smithi111:> ') 2024-03-14T12:58:51.116 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-14T12:58:51.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T12:58:51.116 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:51.133 INFO:teuthology.orchestra.run.smithi111.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-14T12:58:51.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T12:58:51.133 DEBUG:teuthology.orchestra.run.smithi111:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:51.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2024-03-14T12:58:51.258 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T12:58:51.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T12:58:51.325 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T12:58:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T12:58:51.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:51.327 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2024-03-14T12:58:51.385 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-14T12:58:51.385 DEBUG:teuthology.orchestra.run.smithi111:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-14T12:58:51.427 DEBUG:teuthology.orchestra.run.smithi111:> sudo modprobe fuse 2024-03-14T12:58:51.454 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T12:58:51.454 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T12:58:51.520 INFO:teuthology.orchestra.run.smithi111.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T12:58:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T12:58:51.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:51.521 DEBUG:teuthology.orchestra.run.smithi111:> ls /sys/fs/fuse/connections 2024-03-14T12:58:51.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:2024-03-14T12:58:51.532+0000 7fb0bcb77540 -1 init, newargv = 0x55f868094df0 newargc=15 2024-03-14T12:58:51.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[100002]: starting ceph client 2024-03-14T12:58:51.540 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[100002]: starting fuse 2024-03-14T12:58:51.553 INFO:teuthology.orchestra.run.smithi111.stdout:161 2024-03-14T12:58:51.553 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [161] 2024-03-14T12:58:51.553 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-14T12:58:51.553 DEBUG:teuthology.orchestra.run.smithi111:> import glob 2024-03-14T12:58:51.553 DEBUG:teuthology.orchestra.run.smithi111:> import re 2024-03-14T12:58:51.553 DEBUG:teuthology.orchestra.run.smithi111:> import os 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> import subprocess 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> def _find_admin_socket(client_name): 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> files = glob.glob(asok_path) 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> # Given a non-glob path, it better be there 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> if "*" not in asok_path: 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> assert(len(files) == 1) 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> return files[0] 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> for f in files: 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-14T12:58:51.554 DEBUG:teuthology.orchestra.run.smithi111:> contents = proc_f.read() 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> if mountpoint in contents: 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> return f 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> print(_find_admin_socket("client.0")) 2024-03-14T12:58:51.555 DEBUG:teuthology.orchestra.run.smithi111:> ' 2024-03-14T12:58:51.676 INFO:teuthology.orchestra.run.smithi111.stdout:/var/run/ceph/ceph-client.0.100002.asok 2024-03-14T12:58:51.862 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.100002.asok 2024-03-14T12:58:51.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:51.863 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.100002.asok status 2024-03-14T12:58:52.015 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T12:58:52.015 INFO:teuthology.orchestra.run.smithi111.stdout: "metadata": { 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_sha1": "b92acd1f9b519af8166ff5b5bae92bf72f1fd797", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "entity_id": "0", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "hostname": "smithi111", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "pid": "100002", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "root": "/" 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_count": 0, 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "dentry_pinned_count": 0, 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "id": 14546, 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "inst": { 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "name": { 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "client", 2024-03-14T12:58:52.016 INFO:teuthology.orchestra.run.smithi111.stdout: "num": 14546 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2694268800 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": { 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "192.168.0.1:0", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2694268800 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "inst_str": "client.14546 192.168.0.1:0/2694268800", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "addr_str": "192.168.0.1:0/2694268800", 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "inode_count": 1, 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_epoch": 18, 2024-03-14T12:58:52.017 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch": 46, 2024-03-14T12:58:52.018 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_epoch_barrier": 0, 2024-03-14T12:58:52.018 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklisted": false, 2024-03-14T12:58:52.018 INFO:teuthology.orchestra.run.smithi111.stdout: "fs_name": "cephfs" 2024-03-14T12:58:52.018 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T12:58:52.025 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-14T12:58:52.293 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:58:52.570 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:52 smithi111 ceph-mon[39688]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:58:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:52 smithi137 ceph-mon[50336]: pgmap v203: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:58:54.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 -- 172.21.15.111:0/470767869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 msgr2=0x7fe2141031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/470767869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe2141031e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe204004940 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 -- 172.21.15.111:0/470767869 shutdown_connections 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/470767869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 0x7fe2140fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/470767869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe2141031e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 -- 172.21.15.111:0/470767869 >> 172.21.15.111:0/470767869 conn(0x7fe2140f8980 msgr2=0x7fe2140fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 -- 172.21.15.111:0/470767869 shutdown_connections 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.282+0000 7fe21c7b5700 1 -- 172.21.15.111:0/470767869 wait complete. 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.283+0000 7fe21c7b5700 1 Processor -- start 2024-03-14T12:58:54.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.283+0000 7fe21c7b5700 1 -- start start 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.283+0000 7fe21c7b5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21c7b5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 0x7fe214114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21c7b5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe214114680 con 0x7fe214102e10 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21c7b5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21410da70 con 0x7fe2141037b0 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40338/0 (socket says 172.21.15.111:40338) 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 -- 172.21.15.111:0/3819860116 learned_addr learned my addr 172.21.15.111:0/3819860116 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:58:54.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe219d50700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 0x7fe214114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 -- 172.21.15.111:0/3819860116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 msgr2=0x7fe214114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 0x7fe214114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 -- 172.21.15.111:0/3819860116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204005040 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21a551700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b3b0 tx=0x7fe204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20400e040 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe204013420 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21410dcf0 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe20401c610 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.284+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21410e2b0 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.285+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2140fe560 con 0x7fe214102e10 2024-03-14T12:58:54.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.286+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe204013a00 con 0x7fe214102e10 2024-03-14T12:58:54.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.286+0000 7fe20affd700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 0x7fe20006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:58:54.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.286+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe2040882f0 con 0x7fe214102e10 2024-03-14T12:58:54.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.286+0000 7fe219d50700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 0x7fe20006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:58:54.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.286+0000 7fe219d50700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 0x7fe20006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe21410ec30 tx=0x7fe21000b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:58:54.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.290+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fe2040580b0 con 0x7fe214102e10 2024-03-14T12:58:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.516+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe214109df0 con 0x7fe214102e10 2024-03-14T12:58:54.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.516+0000 7fe20affd700 1 -- 172.21.15.111:0/3819860116 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7fe204057c40 con 0x7fe214102e10 2024-03-14T12:58:54.516 INFO:teuthology.orchestra.run.smithi111.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 msgr2=0x7fe20006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 0x7fe20006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe21410ec30 tx=0x7fe21000b040 comp rx=0 tx=0).stop 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 msgr2=0x7fe214113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe20400b3b0 tx=0x7fe204009e30 comp rx=0 tx=0).stop 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 shutdown_connections 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fe2000692a0 0x7fe20006b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe214102e10 0x7fe214113af0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 --2- 172.21.15.111:0/3819860116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2141037b0 0x7fe214114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 >> 172.21.15.111:0/3819860116 conn(0x7fe2140f8980 msgr2=0x7fe2140f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 shutdown_connections 2024-03-14T12:58:54.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:58:54.518+0000 7fe21c7b5700 1 -- 172.21.15.111:0/3819860116 wait complete. 2024-03-14T12:58:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:54 smithi111 ceph-mon[39688]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-14T12:58:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:54 smithi137 ceph-mon[50336]: pgmap v204: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 170 B/s wr, 4 op/s 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-14T12:58:55.232 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-14T12:58:55.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.232 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-03-14T12:58:55.256 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:55.257 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41977sec preferred_lft 41977sec 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-03-14T12:58:55.258 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-03-14T12:58:55.259 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-14T12:58:55.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add name ceph-brx type bridge 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr flush dev ceph-brx 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set ceph-brx up 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-14T12:58:55.259 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-03-14T12:58:55.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.442 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-14T12:58:55.470 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-03-14T12:58:55.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.471 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-03-14T12:58:55.552 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-03-14T12:58:55.552 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-14T12:58:55.552 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-14T12:58:55.552 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-14T12:58:55.552 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-14T12:58:55.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-14T12:58:55.553 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-03-14T12:58:55.632 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:55 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3819860116' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T12:58:55.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:55 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3819860116' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-14T12:58:55.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.837 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-03-14T12:58:55.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.897 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-03-14T12:58:55.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:55.955 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-03-14T12:58:55.955 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-03-14T12:58:55.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-14T12:58:55.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-14T12:58:55.955 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-03-14T12:58:56.049 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-14T12:58:56.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-14T12:58:56.050 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-03-14T12:58:56.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:56.263 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-03-14T12:58:56.263 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-03-14T12:58:56.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-03-14T12:58:56.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.0 master ceph-brx 2024-03-14T12:58:56.263 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-03-14T12:58:56.362 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-14T12:58:56.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T12:58:56.362 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:56.379 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-14T12:58:56.380 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-14T12:58:56.380 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:56.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-03-14T12:58:56.505 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T12:58:56.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T12:58:56.569 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:56 smithi137 ceph-mon[50336]: pgmap v205: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-14T12:58:56.572 INFO:teuthology.orchestra.run.smithi137.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T12:58:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T12:58:56.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:56.573 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-03-14T12:58:56.631 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-14T12:58:56.632 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-14T12:58:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:56 smithi111 ceph-mon[39688]: pgmap v205: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-14T12:58:56.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-03-14T12:58:56.699 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-14T12:58:56.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-14T12:58:56.765 INFO:teuthology.orchestra.run.smithi137.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-14T12:58:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-14T12:58:56.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:56.767 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-03-14T12:58:56.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-03-14T12:58:56.780+0000 7fbe2fbdb540 -1 init, newargv = 0x559bf9ce7e60 newargc=15 2024-03-14T12:58:56.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[80659]: starting ceph client 2024-03-14T12:58:56.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[80659]: starting fuse 2024-03-14T12:58:56.799 INFO:teuthology.orchestra.run.smithi137.stdout:125 2024-03-14T12:58:56.799 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [125] 2024-03-14T12:58:56.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-14T12:58:56.799 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-03-14T12:58:56.799 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-03-14T12:58:56.799 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-14T12:58:56.800 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-03-14T12:58:56.801 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-03-14T12:58:56.923 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.80659.asok 2024-03-14T12:58:57.105 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.80659.asok 2024-03-14T12:58:57.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:57.106 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.80659.asok status 2024-03-14T12:58:57.258 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-03-14T12:58:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-03-14T12:58:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "b92acd1f9b519af8166ff5b5bae92bf72f1fd797", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "80659", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 24309, 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 24309 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-03-14T12:58:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 584861229 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 584861229 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.24309 192.168.0.1:0/584861229", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/584861229", 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 18, 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 46, 2024-03-14T12:58:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-03-14T12:58:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-03-14T12:58:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-03-14T12:58:57.261 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-03-14T12:58:57.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:57.268 DEBUG:teuthology.orchestra.run.smithi111:> stat --file-system '--printf=%T 2024-03-14T12:58:57.268 DEBUG:teuthology.orchestra.run.smithi111:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:57.292 INFO:teuthology.orchestra.run.smithi111.stdout:fuseblk 2024-03-14T12:58:57.292 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:57.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:57.292 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:57.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:57.372 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-03-14T12:58:57.372 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:57.391 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-03-14T12:58:57.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:57.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T12:58:57.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:58 smithi111 ceph-mon[39688]: pgmap v206: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-14T12:58:58.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:58:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:58.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:58 smithi137 ceph-mon[50336]: pgmap v206: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-14T12:58:58.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:58:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:58:59.101 INFO:teuthology.run_tasks:Running task print... 2024-03-14T12:58:59.110 INFO:teuthology.task.print:**** done client 2024-03-14T12:58:59.110 INFO:teuthology.run_tasks:Running task parallel... 2024-03-14T12:58:59.139 INFO:teuthology.task.parallel:starting parallel... 2024-03-14T12:58:59.139 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-14T12:58:59.140 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-14T12:58:59.140 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T12:58:59.140 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-14T12:58:59.141 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-14T12:58:59.141 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-14T12:58:59.144 INFO:tasks.workunit:Pulling workunits from ref 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:58:59.145 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-14T12:58:59.145 INFO:tasks.workunit:timeout=3h 2024-03-14T12:58:59.145 INFO:tasks.workunit:cleanup=True 2024-03-14T12:58:59.145 DEBUG:teuthology.orchestra.run.smithi111:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:59.168 INFO:teuthology.orchestra.run.smithi111.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Device: a1h/161d Inode: 1 Links: 2 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-03-14 12:58:19.811173889 +0000 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-03-14 12:58:57.369210864 +0000 2024-03-14T12:58:59.169 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2024-03-14T12:58:59.169 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-14T12:58:59.169 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-14T12:58:59.246 DEBUG:teuthology.orchestra.run.smithi137:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:59.270 INFO:teuthology.orchestra.run.smithi137.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:59.270 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-14T12:58:59.270 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 7dh/125d Inode: 1 Links: 3 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-03-14 12:58:59.241107583 +0000 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-03-14 12:58:59.241107583 +0000 2024-03-14T12:58:59.271 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: - 2024-03-14T12:58:59.271 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-14T12:58:59.271 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-14T12:58:59.349 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:58:59.349 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:58:59.368 INFO:tasks.workunit.client.0.smithi111.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-14T12:58:59.411 INFO:tasks.workunit.client.1.smithi137.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-14T12:58:59.432 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:59:00.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:00 smithi111 ceph-mon[39688]: pgmap v207: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-14T12:59:00.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:00 smithi137 ceph-mon[50336]: pgmap v207: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-14T12:59:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 -- 172.21.15.111:0/776481359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 msgr2=0x7f421c0fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 --2- 172.21.15.111:0/776481359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c0fe4c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4204004940 tx=0x7f4204009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 -- 172.21.15.111:0/776481359 shutdown_connections 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 --2- 172.21.15.111:0/776481359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c0fe4c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 --2- 172.21.15.111:0/776481359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 0x7f421c0fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 -- 172.21.15.111:0/776481359 >> 172.21.15.111:0/776481359 conn(0x7f421c0f8580 msgr2=0x7f421c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 -- 172.21.15.111:0/776481359 shutdown_connections 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.257+0000 7f42215f5700 1 -- 172.21.15.111:0/776481359 wait complete. 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 Processor -- start 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 -- start start 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 0x7f421c10d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c10e430 con 0x7f421c0fe050 2024-03-14T12:59:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42215f5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c112dd0 con 0x7f421c0fce50 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47412/0 (socket says 172.21.15.111:47412) 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 -- 172.21.15.111:0/535990297 learned_addr learned my addr 172.21.15.111:0/535990297 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421affd700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 0x7f421c10d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 -- 172.21.15.111:0/535990297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 msgr2=0x7f421c10d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 0x7f421c10d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 -- 172.21.15.111:0/535990297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4204005040 con 0x7f421c0fe050 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f421a7fc700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4204009fd0 tx=0x7f4204009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.258+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f420400e040 con 0x7f421c0fe050 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.259+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4204004290 con 0x7f421c0fe050 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.259+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c112fd0 con 0x7f421c0fe050 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.259+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f420401c3f0 con 0x7f421c0fe050 2024-03-14T12:59:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.259+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c113510 con 0x7f421c0fe050 2024-03-14T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.260+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c04c710 con 0x7f421c0fe050 2024-03-14T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.260+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f420400bc50 con 0x7f421c0fe050 2024-03-14T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.260+0000 7f42137fe700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 0x7f420806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.261+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4204088560 con 0x7f421c0fe050 2024-03-14T12:59:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.261+0000 7f421affd700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 0x7f420806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:01.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.261+0000 7f421affd700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 0x7f420806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f421c0fdbc0 tx=0x7f420c009620 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:01.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.265+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f42040583c0 con 0x7f421c0fe050 2024-03-14T12:59:01.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.450+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f421c105840 con 0x7f421c0fe050 2024-03-14T12:59:01.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.451+0000 7f42137fe700 1 -- 172.21.15.111:0/535990297 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f4204057f50 con 0x7f421c0fe050 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 msgr2=0x7f420806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 0x7f420806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f421c0fdbc0 tx=0x7f420c009620 comp rx=0 tx=0).stop 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 msgr2=0x7f421c10dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4204009fd0 tx=0x7f4204009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 shutdown_connections 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f421c0fce50 0x7f421c10d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f42080692a0 0x7f420806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 --2- 172.21.15.111:0/535990297 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f421c0fe050 0x7f421c10dda0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.452+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 >> 172.21.15.111:0/535990297 conn(0x7f421c0f8580 msgr2=0x7f421c0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.453+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 shutdown_connections 2024-03-14T12:59:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:01.453+0000 7f42215f5700 1 -- 172.21.15.111:0/535990297 wait complete. 2024-03-14T12:59:02.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-14T12:59:02.561 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:59:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:02 smithi111 ceph-mon[39688]: pgmap v208: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:59:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:02 smithi137 ceph-mon[50336]: pgmap v208: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 -- 172.21.15.111:0/3698461106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 msgr2=0x7f8f000fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/3698461106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f000fc0e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0004940 tx=0x7f8ef0009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 -- 172.21.15.111:0/3698461106 shutdown_connections 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/3698461106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f000fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/3698461106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f000fc0e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 -- 172.21.15.111:0/3698461106 >> 172.21.15.111:0/3698461106 conn(0x7f8f000f7430 msgr2=0x7f8f000f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 -- 172.21.15.111:0/3698461106 shutdown_connections 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.541+0000 7f8f07ce3700 1 -- 172.21.15.111:0/3698461106 wait complete. 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 Processor -- start 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 -- start start 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:04.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f0010df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0010e5b0 con 0x7f8f000fbcd0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f07ce3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0010ed50 con 0x7f8f000fcf80 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f05a7f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f05a7f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47418/0 (socket says 172.21.15.111:47418) 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f05a7f700 1 -- 172.21.15.111:0/1613701892 learned_addr learned my addr 172.21.15.111:0/1613701892 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f05a7f700 1 -- 172.21.15.111:0/1613701892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 msgr2=0x7f8f0010df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f0527e700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f0010df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.542+0000 7f8f05a7f700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f0010df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8f05a7f700 1 -- 172.21.15.111:0/1613701892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef0005040 con 0x7f8f000fbcd0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8f0527e700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f0010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8f05a7f700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b3b0 tx=0x7f8ef0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef000e040 con 0x7f8f000fbcd0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0010efd0 con 0x7f8f000fbcd0 2024-03-14T12:59:04.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ef000bd20 con 0x7f8f000fbcd0 2024-03-14T12:59:04.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ef001c630 con 0x7f8f000fbcd0 2024-03-14T12:59:04.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.543+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0010f590 con 0x7f8f000fbcd0 2024-03-14T12:59:04.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.544+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f00101b10 con 0x7f8f000fbcd0 2024-03-14T12:59:04.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.544+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8ef0013420 con 0x7f8f000fbcd0 2024-03-14T12:59:04.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.545+0000 7f8ef67fc700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 0x7f8eec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:04.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.545+0000 7f8f0527e700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 0x7f8eec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:04.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.545+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8ef0087f70 con 0x7f8f000fbcd0 2024-03-14T12:59:04.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.545+0000 7f8f0527e700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 0x7f8eec06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8efc006fd0 tx=0x7f8efc008040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:04.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.549+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8ef00b9d70 con 0x7f8f000fbcd0 2024-03-14T12:59:04.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:04 smithi111 ceph-mon[39688]: pgmap v209: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T12:59:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.732+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f8f00104840 con 0x7f8f000fbcd0 2024-03-14T12:59:04.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.733+0000 7f8ef67fc700 1 -- 172.21.15.111:0/1613701892 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f8ef0057cb0 con 0x7f8f000fbcd0 2024-03-14T12:59:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 msgr2=0x7f8eec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 0x7f8eec06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8efc006fd0 tx=0x7f8efc008040 comp rx=0 tx=0).stop 2024-03-14T12:59:04.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 msgr2=0x7f8f0010d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b3b0 tx=0x7f8ef0009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 shutdown_connections 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f8eec0692a0 0x7f8eec06b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f000fbcd0 0x7f8f0010d9e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 --2- 172.21.15.111:0/1613701892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f000fcf80 0x7f8f0010df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 >> 172.21.15.111:0/1613701892 conn(0x7f8f000f7430 msgr2=0x7f8f00065c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 shutdown_connections 2024-03-14T12:59:04.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:04.735+0000 7f8f07ce3700 1 -- 172.21.15.111:0/1613701892 wait complete. 2024-03-14T12:59:04.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:04 smithi137 ceph-mon[50336]: pgmap v209: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T12:59:05.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-14T12:59:05.840 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:59:06.368 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:06 smithi111 ceph-mon[39688]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-14T12:59:06.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:06 smithi137 ceph-mon[50336]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-14T12:59:08.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 -- 172.21.15.111:0/3476090141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 msgr2=0x7fb0800ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:08.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 --2- 172.21.15.111:0/3476090141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb0800ff9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb074004940 tx=0x7fb074009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 -- 172.21.15.111:0/3476090141 shutdown_connections 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 --2- 172.21.15.111:0/3476090141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 0x7fb080102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 --2- 172.21.15.111:0/3476090141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb0800ff9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 -- 172.21.15.111:0/3476090141 >> 172.21.15.111:0/3476090141 conn(0x7fb0800f7430 msgr2=0x7fb0800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 -- 172.21.15.111:0/3476090141 shutdown_connections 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.997+0000 7fb085e9e700 1 -- 172.21.15.111:0/3476090141 wait complete. 2024-03-14T12:59:08.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 Processor -- start 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 -- start start 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 0x7fb08010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08010e410 con 0x7fb0800662e0 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb085e9e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08010ec60 con 0x7fb0800fff00 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47436/0 (socket says 172.21.15.111:47436) 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 -- 172.21.15.111:0/1412238806 learned_addr learned my addr 172.21.15.111:0/1412238806 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:08.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07effd700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 0x7fb08010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 -- 172.21.15.111:0/1412238806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 msgr2=0x7fb08010de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 0x7fb08010de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.998+0000 7fb07f7fe700 1 -- 172.21.15.111:0/1412238806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074005040 con 0x7fb0800662e0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb07f7fe700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb0740137b0 tx=0x7fb074009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb074021070 con 0x7fb0800662e0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08010eee0 con 0x7fb0800662e0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb07401c400 con 0x7fb0800662e0 2024-03-14T12:59:08.003 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb07400e070 con 0x7fb0800662e0 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:07.999+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb08010f420 con 0x7fb0800662e0 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.000+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb07401c9c0 con 0x7fb0800662e0 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.000+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08005f5f0 con 0x7fb0800662e0 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.001+0000 7fb067fff700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 0x7fb06806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.001+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb0740883e0 con 0x7fb0800662e0 2024-03-14T12:59:08.004 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.001+0000 7fb07effd700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 0x7fb06806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:08.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.001+0000 7fb07effd700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 0x7fb06806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb070009b80 tx=0x7fb07000b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:08.005 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.005+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fb0740581a0 con 0x7fb0800662e0 2024-03-14T12:59:08.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.198+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fb080105780 con 0x7fb0800662e0 2024-03-14T12:59:08.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.198+0000 7fb067fff700 1 -- 172.21.15.111:0/1412238806 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fb074057d30 con 0x7fb0800662e0 2024-03-14T12:59:08.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 msgr2=0x7fb06806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:08.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 0x7fb06806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb070009b80 tx=0x7fb07000b040 comp rx=0 tx=0).stop 2024-03-14T12:59:08.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 msgr2=0x7fb08010d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:08.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb0740137b0 tx=0x7fb074009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:08.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 shutdown_connections 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fb0680692a0 0x7fb06806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0800662e0 0x7fb08010d8d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 --2- 172.21.15.111:0/1412238806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0800fff00 0x7fb08010de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.200+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 >> 172.21.15.111:0/1412238806 conn(0x7fb0800f7430 msgr2=0x7fb0800f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.201+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 shutdown_connections 2024-03-14T12:59:08.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:08.201+0000 7fb085e9e700 1 -- 172.21.15.111:0/1412238806 wait complete. 2024-03-14T12:59:08.629 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:08 smithi111 ceph-mon[39688]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-14T12:59:08.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:08 smithi137 ceph-mon[50336]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-14T12:59:09.034 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-14T12:59:09.317 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:59:10.627 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:10 smithi111 ceph-mon[39688]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-14T12:59:10.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:10 smithi137 ceph-mon[50336]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-14T12:59:11.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- 172.21.15.111:0/1989595204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fb480 msgr2=0x7f48600fb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 --2- 172.21.15.111:0/1989595204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fb480 0x7f48600fb890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4850004940 tx=0x7f4850009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- 172.21.15.111:0/1989595204 shutdown_connections 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 --2- 172.21.15.111:0/1989595204 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fc730 0x7f48600fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 --2- 172.21.15.111:0/1989595204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fb480 0x7f48600fb890 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- 172.21.15.111:0/1989595204 >> 172.21.15.111:0/1989595204 conn(0x7f48600f6be0 msgr2=0x7f48600f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- 172.21.15.111:0/1989595204 shutdown_connections 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- 172.21.15.111:0/1989595204 wait complete. 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 Processor -- start 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.255+0000 7f4866b86700 1 -- start start 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fb480 0x7f48600f8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:11.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860112b60 con 0x7f48600fc730 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860112ca0 con 0x7f48600fb480 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42084/0 (socket says 172.21.15.111:42084) 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 -- 172.21.15.111:0/1288618270 learned_addr learned my addr 172.21.15.111:0/1288618270 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4864922700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fb480 0x7f48600f8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 -- 172.21.15.111:0/1288618270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fb480 msgr2=0x7f48600f8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fb480 0x7f48600f8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 -- 172.21.15.111:0/1288618270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850005040 con 0x7f48600fc730 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485ffff700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f485400eb70 tx=0x7f485400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4854007cc0 con 0x7f48600fc730 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4860112ea0 con 0x7f48600fc730 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f485400f040 con 0x7f48600fc730 2024-03-14T12:59:11.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f485400c9f0 con 0x7f48600fc730 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.256+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4860113430 con 0x7f48600fc730 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.257+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486004c710 con 0x7f48600fc730 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.258+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f48540090d0 con 0x7f48600fc730 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.258+0000 7f485d7fa700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 0x7f484806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.258+0000 7f4864922700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 0x7f484806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.258+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4854086d00 con 0x7f48600fc730 2024-03-14T12:59:11.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.259+0000 7f4864922700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 0x7f484806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4850004890 tx=0x7f4850009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:11.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.263+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f485408a030 con 0x7f48600fc730 2024-03-14T12:59:11.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:11.472+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f4860101130 con 0x7f48480692a0 2024-03-14T12:59:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:12 smithi111 ceph-mon[39688]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-14T12:59:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:12 smithi111 ceph-mon[39688]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:12.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:12 smithi137 ceph-mon[50336]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-14T12:59:12.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:12 smithi137 ceph-mon[50336]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:13.618 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:14 smithi111 ceph-mon[39688]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-14T12:59:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:14 smithi137 ceph-mon[50336]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-14T12:59:16.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:16 smithi111 ceph-mon[39688]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-14T12:59:16.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:16 smithi137 ceph-mon[50336]: pgmap v215: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-14T12:59:18.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:18 smithi111 ceph-mon[39688]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-14T12:59:18.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:18 smithi137 ceph-mon[50336]: pgmap v216: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-14T12:59:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:20 smithi111 ceph-mon[39688]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T12:59:20.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:20 smithi137 ceph-mon[50336]: pgmap v217: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-14T12:59:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:22 smithi111 ceph-mon[39688]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-14T12:59:22.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:22 smithi137 ceph-mon[50336]: pgmap v218: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-14T12:59:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:24 smithi111 ceph-mon[39688]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:59:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:24 smithi137 ceph-mon[50336]: pgmap v219: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-14T12:59:26.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:26 smithi111 ceph-mon[39688]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-14T12:59:26.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:26 smithi137 ceph-mon[50336]: pgmap v220: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 426 B/s wr, 4 op/s 2024-03-14T12:59:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:28 smithi111 ceph-mon[39688]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:28 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:28.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:28 smithi137 ceph-mon[50336]: pgmap v221: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:28.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:28 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:30 smithi111 ceph-mon[39688]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:30 smithi137 ceph-mon[50336]: pgmap v222: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:32.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:32 smithi111 ceph-mon[39688]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:32.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:32 smithi137 ceph-mon[50336]: pgmap v223: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:34.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:34 smithi111 ceph-mon[39688]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:34.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:34 smithi137 ceph-mon[50336]: pgmap v224: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:36.580 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:36 smithi111 ceph-mon[39688]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:36.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:36 smithi137 ceph-mon[50336]: pgmap v225: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:37.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.431+0000 7f485d7fa700 1 -- 172.21.15.111:0/1288618270 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f4860101130 con 0x7f48480692a0 2024-03-14T12:59:37.432 INFO:teuthology.orchestra.run.smithi111.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:59:37.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.433+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 msgr2=0x7f484806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:37.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.433+0000 7f4866b86700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 0x7f484806b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4850004890 tx=0x7f4850009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:37.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.433+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 msgr2=0x7f4860112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.433+0000 7f4866b86700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f485400eb70 tx=0x7f485400b040 comp rx=0 tx=0).stop 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 shutdown_connections 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f48600fb480 0x7f48600f8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f48480692a0 0x7f484806b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 --2- 172.21.15.111:0/1288618270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600fc730 0x7f4860112620 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 >> 172.21.15.111:0/1288618270 conn(0x7f48600f6be0 msgr2=0x7f48600ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 shutdown_connections 2024-03-14T12:59:37.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:37.434+0000 7f4866b86700 1 -- 172.21.15.111:0/1288618270 wait complete. 2024-03-14T12:59:38.211 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-14T12:59:38.506 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:59:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:38 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: pgmap v226: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T12:59:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:38 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:39 smithi137 ceph-mon[50336]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:59:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:39 smithi111 ceph-mon[39688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T12:59:40.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:40 smithi111 ceph-mon[39688]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:40.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:40 smithi137 ceph-mon[50336]: pgmap v227: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:42.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 -- 172.21.15.111:0/3448774880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fce80 msgr2=0x7f4a900fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:42.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 --2- 172.21.15.111:0/3448774880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fce80 0x7f4a900fd290 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c004940 tx=0x7f4a8c009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 -- 172.21.15.111:0/3448774880 shutdown_connections 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 --2- 172.21.15.111:0/3448774880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fdff0 0x7f4a900fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 --2- 172.21.15.111:0/3448774880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fce80 0x7f4a900fd290 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 -- 172.21.15.111:0/3448774880 >> 172.21.15.111:0/3448774880 conn(0x7f4a900f8580 msgr2=0x7f4a900fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 -- 172.21.15.111:0/3448774880 shutdown_connections 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.735+0000 7f4a97427700 1 -- 172.21.15.111:0/3448774880 wait complete. 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 Processor -- start 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 -- start start 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fce80 0x7f4a9010d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9010e460 con 0x7f4a900fdff0 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a97427700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9010ec50 con 0x7f4a900fce80 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:42.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37388/0 (socket says 172.21.15.111:37388) 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 -- 172.21.15.111:0/4179890190 learned_addr learned my addr 172.21.15.111:0/4179890190 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a951c3700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fce80 0x7f4a9010d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 -- 172.21.15.111:0/4179890190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fce80 msgr2=0x7f4a9010d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fce80 0x7f4a9010d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.736+0000 7f4a949c2700 1 -- 172.21.15.111:0/4179890190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c005040 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a949c2700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b790 tx=0x7f4a80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a8000f6f0 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a8000fd00 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9010ef30 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a800103a0 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.737+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9010f4f0 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.738+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9005f5f0 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.738+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4a8000f850 con 0x7f4a900fdff0 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.738+0000 7f4a85ffb700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 0x7f4a7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:42.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.739+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4a80086fb0 con 0x7f4a900fdff0 2024-03-14T12:59:42.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.739+0000 7f4a951c3700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 0x7f4a7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:42.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.739+0000 7f4a951c3700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 0x7f4a7c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00b3b0 tx=0x7f4a8c009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:42.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.743+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f4a80056d70 con 0x7f4a900fdff0 2024-03-14T12:59:42.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:42 smithi137 ceph-mon[50336]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:42 smithi111 ceph-mon[39688]: pgmap v228: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:42.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.951+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a9010f7e0 con 0x7f4a7c0692a0 2024-03-14T12:59:42.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.952+0000 7f4a85ffb700 1 -- 172.21.15.111:0/4179890190 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f4a9010f7e0 con 0x7f4a7c0692a0 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 msgr2=0x7f4a7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 0x7f4a7c06b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00b3b0 tx=0x7f4a8c009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 msgr2=0x7f4a9010ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000b790 tx=0x7f4a80009250 comp rx=0 tx=0).stop 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 shutdown_connections 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a900fce80 0x7f4a9010d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f4a7c0692a0 0x7f4a7c06b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 --2- 172.21.15.111:0/4179890190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a900fdff0 0x7f4a9010ddd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 >> 172.21.15.111:0/4179890190 conn(0x7f4a900f8580 msgr2=0x7f4a900f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 shutdown_connections 2024-03-14T12:59:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:42.955+0000 7f4a97427700 1 -- 172.21.15.111:0/4179890190 wait complete. 2024-03-14T12:59:42.964 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.057+0000 7f97785e2700 1 -- 172.21.15.111:0/1065408774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 msgr2=0x7f97700fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.057+0000 7f97785e2700 1 --2- 172.21.15.111:0/1065408774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f97700fd3f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9760004940 tx=0x7f9760009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- 172.21.15.111:0/1065408774 shutdown_connections 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 --2- 172.21.15.111:0/1065408774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f97700fd3f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 --2- 172.21.15.111:0/1065408774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97700fbcd0 0x7f97700fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- 172.21.15.111:0/1065408774 >> 172.21.15.111:0/1065408774 conn(0x7f97700f7430 msgr2=0x7f97700f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- 172.21.15.111:0/1065408774 shutdown_connections 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- 172.21.15.111:0/1065408774 wait complete. 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 Processor -- start 2024-03-14T12:59:43.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- start start 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97700fbcd0 0x7f9770110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770066350 con 0x7f97700fcf80 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f97785e2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770066490 con 0x7f97700fbcd0 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f9775b7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f9775b7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37420/0 (socket says 172.21.15.111:37420) 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.058+0000 7f9775b7d700 1 -- 172.21.15.111:0/2988507230 learned_addr learned my addr 172.21.15.111:0/2988507230 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9775b7d700 1 -- 172.21.15.111:0/2988507230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97700fbcd0 msgr2=0x7f9770110670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9775b7d700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97700fbcd0 0x7f9770110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9775b7d700 1 -- 172.21.15.111:0/2988507230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760005040 con 0x7f97700fcf80 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9775b7d700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f976000b3b0 tx=0x7f9760009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f976000e040 con 0x7f97700fcf80 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f976000bd20 con 0x7f97700fcf80 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9770113100 con 0x7f97700fcf80 2024-03-14T12:59:43.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f976001c630 con 0x7f97700fcf80 2024-03-14T12:59:43.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.059+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97701136c0 con 0x7f97700fcf80 2024-03-14T12:59:43.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.060+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977005f5f0 con 0x7f97700fcf80 2024-03-14T12:59:43.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.060+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9760013420 con 0x7f97700fcf80 2024-03-14T12:59:43.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.061+0000 7f9766ffd700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 0x7f975c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.061+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9760087fd0 con 0x7f97700fcf80 2024-03-14T12:59:43.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.061+0000 7f977637e700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 0x7f975c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.061+0000 7f977637e700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 0x7f975c06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9770065fd0 tx=0x7f976c007d40 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.064 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.065+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f9760054220 con 0x7f97700fcf80 2024-03-14T12:59:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.274+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9770101380 con 0x7f975c0692a0 2024-03-14T12:59:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.275+0000 7f9766ffd700 1 -- 172.21.15.111:0/2988507230 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f9770101380 con 0x7f975c0692a0 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 msgr2=0x7f975c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 0x7f975c06b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9770065fd0 tx=0x7f976c007d40 comp rx=0 tx=0).stop 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 msgr2=0x7f9770112bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f976000b3b0 tx=0x7f9760009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 shutdown_connections 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97700fbcd0 0x7f9770110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f975c0692a0 0x7f975c06b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 --2- 172.21.15.111:0/2988507230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97700fcf80 0x7f9770112bc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 >> 172.21.15.111:0/2988507230 conn(0x7f97700f7430 msgr2=0x7f97700f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 shutdown_connections 2024-03-14T12:59:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.277+0000 7f97785e2700 1 -- 172.21.15.111:0/2988507230 wait complete. 2024-03-14T12:59:43.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 -- 172.21.15.111:0/93039543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 msgr2=0x7fa2840fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 --2- 172.21.15.111:0/93039543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa2840fb610 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa280004940 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 -- 172.21.15.111:0/93039543 shutdown_connections 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 --2- 172.21.15.111:0/93039543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 0x7fa284102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 --2- 172.21.15.111:0/93039543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa2840fb610 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 -- 172.21.15.111:0/93039543 >> 172.21.15.111:0/93039543 conn(0x7fa2840f6ba0 msgr2=0x7fa2840f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 -- 172.21.15.111:0/93039543 shutdown_connections 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.380+0000 7fa28c31c700 1 -- 172.21.15.111:0/93039543 wait complete. 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 Processor -- start 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 -- start start 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 0x7fa28410deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28410e4b0 con 0x7fa2840fb1e0 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28c31c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28410ed00 con 0x7fa2840fbb50 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37440/0 (socket says 172.21.15.111:37440) 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 -- 172.21.15.111:0/760602211 learned_addr learned my addr 172.21.15.111:0/760602211 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa2898b7700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 0x7fa28410deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 -- 172.21.15.111:0/760602211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 msgr2=0x7fa28410deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 0x7fa28410deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.381+0000 7fa28a0b8700 1 -- 172.21.15.111:0/760602211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280005040 con 0x7fa2840fb1e0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa28a0b8700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa28000b3b0 tx=0x7fa280009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28000e040 con 0x7fa2840fb1e0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa280004290 con 0x7fa2840fb1e0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28410ef80 con 0x7fa2840fb1e0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa28001c3f0 con 0x7fa2840fb1e0 2024-03-14T12:59:43.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.382+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28410f4c0 con 0x7fa2840fb1e0 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.383+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28405f5f0 con 0x7fa2840fb1e0 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.383+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa280013820 con 0x7fa2840fb1e0 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.384+0000 7fa27affd700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 0x7fa27006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.384+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fa280088360 con 0x7fa2840fb1e0 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.384+0000 7fa2898b7700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 0x7fa27006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.384+0000 7fa2898b7700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 0x7fa27006b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa274004f40 tx=0x7fa27400b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.387 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.388+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fa2800581c0 con 0x7fa2840fb1e0 2024-03-14T12:59:43.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.584+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa28410f860 con 0x7fa2700692b0 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.593+0000 7fa27affd700 1 -- 172.21.15.111:0/760602211 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fa28410f860 con 0x7fa2700692b0 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (5m) 58s ago 7m 26.5M - 0.25.0 c8568f914cd2 a5a721c2d1c2 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (7m) 58s ago 7m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (6m) 66s ago 6m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T12:59:43.593 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (5m) 58s ago 6m 87.0M - 9.4.7 954c08fa6188 a0f9b28f5afd 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (79s) 58s ago 78s 12.4M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (85s) 58s ago 84s 16.9M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (76s) 66s ago 76s 15.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (82s) 66s ago 81s 14.5M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:9283 running (8m) 58s ago 8m 481M - 17.2.7-646-gb92acd1f a07fb176315d 9d384b708369 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (6m) 66s ago 6m 434M - 17.2.7-646-gb92acd1f a07fb176315d ee0103247ac3 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (8m) 58s ago 9m 47.4M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (6m) 66s ago 6m 36.1M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 58s ago 7m 17.9M - 1.5.0 0da6a335fe13 834d8e9211eb 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 66s ago 6m 18.2M - 1.5.0 0da6a335fe13 de8bbb0f9cd8 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (5m) 58s ago 5m 46.5M 4299M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (4m) 58s ago 4m 46.1M 4299M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (4m) 58s ago 4m 45.5M 4299M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T12:59:43.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (3m) 66s ago 3m 47.4M 5323M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (3m) 66s ago 3m 43.1M 5323M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (3m) 66s ago 2m 40.7M 5323M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (5m) 58s ago 6m 52.7M - 2.43.0 a07b618ecd1d c5814e8b878b 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.595+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 msgr2=0x7fa27006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.595+0000 7fa28c31c700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 0x7fa27006b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa274004f40 tx=0x7fa27400b040 comp rx=0 tx=0).stop 2024-03-14T12:59:43.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.595+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 msgr2=0x7fa28410d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.595+0000 7fa28c31c700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa28000b3b0 tx=0x7fa280009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 shutdown_connections 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa2700692b0 0x7fa27006b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa2840fb1e0 0x7fa28410d970 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 --2- 172.21.15.111:0/760602211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2840fbb50 0x7fa28410deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 >> 172.21.15.111:0/760602211 conn(0x7fa2840f6ba0 msgr2=0x7fa284100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 shutdown_connections 2024-03-14T12:59:43.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.596+0000 7fa28c31c700 1 -- 172.21.15.111:0/760602211 wait complete. 2024-03-14T12:59:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:43 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 -- 172.21.15.111:0/1340045543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 msgr2=0x7fab640ffa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 --2- 172.21.15.111:0/1340045543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab640ffa90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fab58004940 tx=0x7fab58009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 -- 172.21.15.111:0/1340045543 shutdown_connections 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 --2- 172.21.15.111:0/1340045543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 0x7fab641023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 --2- 172.21.15.111:0/1340045543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab640ffa90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 -- 172.21.15.111:0/1340045543 >> 172.21.15.111:0/1340045543 conn(0x7fab640f74e0 msgr2=0x7fab640f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 -- 172.21.15.111:0/1340045543 shutdown_connections 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.700+0000 7fab6aad7700 1 -- 172.21.15.111:0/1340045543 wait complete. 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 Processor -- start 2024-03-14T12:59:43.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 -- start start 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 0x7fab6410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6410e460 con 0x7fab64066b30 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab6aad7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6410ecb0 con 0x7fab640fffd0 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab68873700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab68873700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37452/0 (socket says 172.21.15.111:37452) 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab68873700 1 -- 172.21.15.111:0/3381639125 learned_addr learned my addr 172.21.15.111:0/3381639125 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.701+0000 7fab63fff700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 0x7fab6410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab68873700 1 -- 172.21.15.111:0/3381639125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 msgr2=0x7fab6410de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab68873700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 0x7fab6410de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab68873700 1 -- 172.21.15.111:0/3381639125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58005040 con 0x7fab64066b30 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab68873700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fab580137b0 tx=0x7fab58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab58021070 con 0x7fab64066b30 2024-03-14T12:59:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6410ef30 con 0x7fab64066b30 2024-03-14T12:59:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab5801c400 con 0x7fab64066b30 2024-03-14T12:59:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab5800e070 con 0x7fab64066b30 2024-03-14T12:59:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.702+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6410f470 con 0x7fab64066b30 2024-03-14T12:59:43.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.703+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6405f5f0 con 0x7fab64066b30 2024-03-14T12:59:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.704+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fab5801c9c0 con 0x7fab64066b30 2024-03-14T12:59:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.704+0000 7fab617fa700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 0x7fab5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.704+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fab58088b00 con 0x7fab64066b30 2024-03-14T12:59:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.704+0000 7fab63fff700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 0x7fab5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:43.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.705+0000 7fab63fff700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 0x7fab5406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fab50009b80 tx=0x7fab5000b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:43.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.708+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fab580588a0 con 0x7fab64066b30 2024-03-14T12:59:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:43 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:43.979 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.980+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fab64105790 con 0x7fab64066b30 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.980+0000 7fab617fa700 1 -- 172.21.15.111:0/3381639125 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fab58058430 con 0x7fab64066b30 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:59:43.980 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 14 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T12:59:43.981 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 msgr2=0x7fab5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 0x7fab5406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fab50009b80 tx=0x7fab5000b040 comp rx=0 tx=0).stop 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 msgr2=0x7fab6410d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fab580137b0 tx=0x7fab58009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 shutdown_connections 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fab540692a0 0x7fab5406b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab64066b30 0x7fab6410d920 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 --2- 172.21.15.111:0/3381639125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab640fffd0 0x7fab6410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:43.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 >> 172.21.15.111:0/3381639125 conn(0x7fab640f74e0 msgr2=0x7fab640f78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:43.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 shutdown_connections 2024-03-14T12:59:43.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:43.982+0000 7fab6aad7700 1 -- 172.21.15.111:0/3381639125 wait complete. 2024-03-14T12:59:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 -- 172.21.15.111:0/4043160456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fe440 msgr2=0x7fa4f80fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/4043160456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fe440 0x7fa4f80fe850 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec004940 tx=0x7fa4ec009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:44.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 -- 172.21.15.111:0/4043160456 shutdown_connections 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/4043160456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fed90 0x7fa4f80faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/4043160456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fe440 0x7fa4f80fe850 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 -- 172.21.15.111:0/4043160456 >> 172.21.15.111:0/4043160456 conn(0x7fa4f80f6be0 msgr2=0x7fa4f80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 -- 172.21.15.111:0/4043160456 shutdown_connections 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 -- 172.21.15.111:0/4043160456 wait complete. 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.094+0000 7fa4fd88d700 1 Processor -- start 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 -- start start 2024-03-14T12:59:44.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fe440 0x7fa4f810d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f810e340 con 0x7fa4f80fed90 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f810e4b0 con 0x7fa4f80fe440 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37478/0 (socket says 172.21.15.111:37478) 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 -- 172.21.15.111:0/3764238639 learned_addr learned my addr 172.21.15.111:0/3764238639 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f6ffd700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fe440 0x7fa4f810d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 -- 172.21.15.111:0/3764238639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fe440 msgr2=0x7fa4f810d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fe440 0x7fa4f810d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 -- 172.21.15.111:0/3764238639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec005040 con 0x7fa4f80fed90 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4f67fc700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800eb40 tx=0x7fa4e800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e8007cc0 con 0x7fa4f80fed90 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4e800f040 con 0x7fa4f80fed90 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f8112e60 con 0x7fa4f80fed90 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4e800c990 con 0x7fa4f80fed90 2024-03-14T12:59:44.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.095+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f81133f0 con 0x7fa4f80fed90 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.096+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f804c710 con 0x7fa4f80fed90 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.097+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa4e800cc00 con 0x7fa4f80fed90 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.097+0000 7fa4df7fe700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 0x7fa4e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.097+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fa4e8086d60 con 0x7fa4f80fed90 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.097+0000 7fa4f6ffd700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 0x7fa4e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T12:59:44.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.098+0000 7fa4f6ffd700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 0x7fa4e006b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec004890 tx=0x7fa4ec009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T12:59:44.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.102+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fa4e804dd70 con 0x7fa4f80fed90 2024-03-14T12:59:44.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.305+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4f8102560 con 0x7fa4e00692a0 2024-03-14T12:59:44.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.306+0000 7fa4df7fe700 1 -- 172.21.15.111:0/3764238639 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa4f8102560 con 0x7fa4e00692a0 2024-03-14T12:59:44.305 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T12:59:44.305 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T12:59:44.305 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T12:59:44.305 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T12:59:44.306 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-03-14T12:59:44.306 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "0/2 daemons upgraded", 2024-03-14T12:59:44.306 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de image on host smithi137", 2024-03-14T12:59:44.306 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T12:59:44.306 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T12:59:44.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 msgr2=0x7fa4e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:44.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 0x7fa4e006b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec004890 tx=0x7fa4ec009e30 comp rx=0 tx=0).stop 2024-03-14T12:59:44.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 msgr2=0x7fa4f810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T12:59:44.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800eb40 tx=0x7fa4e800b040 comp rx=0 tx=0).stop 2024-03-14T12:59:44.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 shutdown_connections 2024-03-14T12:59:44.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa4f80fe440 0x7fa4f810d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fa4e00692a0 0x7fa4e006b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 --2- 172.21.15.111:0/3764238639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa4f80fed90 0x7fa4f810dd40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T12:59:44.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 >> 172.21.15.111:0/3764238639 conn(0x7fa4f80f6be0 msgr2=0x7fa4f80f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T12:59:44.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 shutdown_connections 2024-03-14T12:59:44.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T12:59:44.308+0000 7fa4fd88d700 1 -- 172.21.15.111:0/3764238639 wait complete. 2024-03-14T12:59:45.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: Upgrade: Target is version 18.2.2-825-g2f465319 (unknown) 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de, digests ['quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de'] 2024-03-14T12:59:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:59:45.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:45.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: Upgrade: Need to upgrade myself (mgr.smithi111.atysez) 2024-03-14T12:59:45.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:44 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3381639125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T12:59:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: pgmap v229: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: Upgrade: Target is version 18.2.2-825-g2f465319 (unknown) 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de, digests ['quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de'] 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: Upgrade: Need to upgrade myself (mgr.smithi111.atysez) 2024-03-14T12:59:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:44 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3381639125' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T12:59:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:45 smithi137 ceph-mon[50336]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de on smithi137 2024-03-14T12:59:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:45 smithi137 ceph-mon[50336]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:45 smithi111 ceph-mon[39688]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de on smithi137 2024-03-14T12:59:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:45 smithi111 ceph-mon[39688]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T12:59:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:46 smithi137 ceph-mon[50336]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:46 smithi111 ceph-mon[39688]: pgmap v230: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:48 smithi137 ceph-mon[50336]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:49.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:48 smithi111 ceph-mon[39688]: pgmap v231: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:50 smithi137 ceph-mon[50336]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:51.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:50 smithi111 ceph-mon[39688]: pgmap v232: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:52 smithi137 ceph-mon[50336]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:53.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:52 smithi111 ceph-mon[39688]: pgmap v233: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T12:59:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:54 smithi137 ceph-mon[50336]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:54 smithi111 ceph-mon[39688]: pgmap v234: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:57.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:56 smithi137 ceph-mon[50336]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:57.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:56 smithi111 ceph-mon[39688]: pgmap v235: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T12:59:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:58 smithi137 ceph-mon[50336]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 12:59:58 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T12:59:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:58 smithi111 ceph-mon[39688]: pgmap v236: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T12:59:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 12:59:58 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:01.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:00 smithi137 ceph-mon[50336]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:01.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:00 smithi137 ceph-mon[50336]: overall HEALTH_OK 2024-03-14T13:00:01.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:00 smithi111 ceph-mon[39688]: pgmap v237: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:01.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:00 smithi111 ceph-mon[39688]: overall HEALTH_OK 2024-03-14T13:00:03.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:02 smithi137 ceph-mon[50336]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:03.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:02 smithi111 ceph-mon[39688]: pgmap v238: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:05.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:04 smithi137 ceph-mon[50336]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:04 smithi111 ceph-mon[39688]: pgmap v239: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:06 smithi137 ceph-mon[50336]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:07.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:06 smithi111 ceph-mon[39688]: pgmap v240: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:08.975 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:08 smithi137 ceph-mon[50336]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:00:09.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:08 smithi111 ceph-mon[39688]: pgmap v241: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:00:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:10 smithi137 ceph-mon[50336]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:11.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:10 smithi111 ceph-mon[39688]: pgmap v242: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: Upgrade: Updating mgr.smithi137.xbyzbp 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: Deploying daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:00:12.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:12 smithi137 ceph-mon[50336]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: Upgrade: Updating mgr.smithi137.xbyzbp 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: Deploying daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:00:12.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:12 smithi111 ceph-mon[39688]: pgmap v243: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:13 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:13.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:13 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- 172.21.15.111:0/2828072498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 msgr2=0x7f86340fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 --2- 172.21.15.111:0/2828072498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f86340fd3f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8630004940 tx=0x7f8630009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- 172.21.15.111:0/2828072498 shutdown_connections 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 --2- 172.21.15.111:0/2828072498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f86340fd3f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 --2- 172.21.15.111:0/2828072498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 0x7f86340fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- 172.21.15.111:0/2828072498 >> 172.21.15.111:0/2828072498 conn(0x7f86340f7430 msgr2=0x7f86340f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- 172.21.15.111:0/2828072498 shutdown_connections 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- 172.21.15.111:0/2828072498 wait complete. 2024-03-14T13:00:14.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 Processor -- start 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.422+0000 7f863c180700 1 -- start start 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 0x7f863410faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86341105e0 con 0x7f86340fcf80 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634110e30 con 0x7f86340fbcd0 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48520/0 (socket says 172.21.15.111:48520) 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 -- 172.21.15.111:0/1665129822 learned_addr learned my addr 172.21.15.111:0/1665129822 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f8639f1c700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 0x7f863410faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 -- 172.21.15.111:0/1665129822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 msgr2=0x7f863410faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 0x7f863410faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 -- 172.21.15.111:0/1665129822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8630005040 con 0x7f86340fcf80 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863971b700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8630009380 tx=0x7f8630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863000e040 con 0x7f86340fcf80 2024-03-14T13:00:14.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f863000bd20 con 0x7f86340fcf80 2024-03-14T13:00:14.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86341110b0 con 0x7f86340fcf80 2024-03-14T13:00:14.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f863001c630 con 0x7f86340fcf80 2024-03-14T13:00:14.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.423+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86341115f0 con 0x7f86340fcf80 2024-03-14T13:00:14.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.424+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8634101910 con 0x7f86340fcf80 2024-03-14T13:00:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.425+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8630013420 con 0x7f86340fcf80 2024-03-14T13:00:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.425+0000 7f862a7fc700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 0x7f862006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.425+0000 7f8639f1c700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 0x7f862006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.426+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8630087f00 con 0x7f86340fcf80 2024-03-14T13:00:14.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.426+0000 7f8639f1c700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 0x7f862006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8624009fd0 tx=0x7f862400b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:14.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.430+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f8630057cc0 con 0x7f86340fcf80 2024-03-14T13:00:14.621 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:14 smithi137 ceph-mon[50336]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:14.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.629+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8634111810 con 0x7f86200692a0 2024-03-14T13:00:14.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.631+0000 7f862a7fc700 1 -- 172.21.15.111:0/1665129822 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f8634111810 con 0x7f86200692a0 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.632+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 msgr2=0x7f862006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.632+0000 7f863c180700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 0x7f862006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8624009fd0 tx=0x7f862400b040 comp rx=0 tx=0).stop 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.632+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 msgr2=0x7f863410ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.632+0000 7f863c180700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8630009380 tx=0x7f8630009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 shutdown_connections 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86340fbcd0 0x7f863410faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f86200692a0 0x7f862006b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 --2- 172.21.15.111:0/1665129822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86340fcf80 0x7f863410ffe0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 >> 172.21.15.111:0/1665129822 conn(0x7f86340f7430 msgr2=0x7f8634065c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:14.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 shutdown_connections 2024-03-14T13:00:14.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.633+0000 7f863c180700 1 -- 172.21.15.111:0/1665129822 wait complete. 2024-03-14T13:00:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:14 smithi111 ceph-mon[39688]: pgmap v244: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:14.642 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- 172.21.15.111:0/1246826536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fcf70 msgr2=0x7fd15c0fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 --2- 172.21.15.111:0/1246826536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fcf70 0x7fd15c0fd3e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd150004940 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- 172.21.15.111:0/1246826536 shutdown_connections 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 --2- 172.21.15.111:0/1246826536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fcf70 0x7fd15c0fd3e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 --2- 172.21.15.111:0/1246826536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fbe00 0x7fd15c0fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- 172.21.15.111:0/1246826536 >> 172.21.15.111:0/1246826536 conn(0x7fd15c0f74e0 msgr2=0x7fd15c0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:14.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- 172.21.15.111:0/1246826536 shutdown_connections 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- 172.21.15.111:0/1246826536 wait complete. 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 Processor -- start 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- start start 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fcf70 0x7fd15c10cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c10d320 con 0x7fd15c0fbe00 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.734+0000 7fd160aaa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c10db20 con 0x7fd15c0fcf70 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48532/0 (socket says 172.21.15.111:48532) 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 -- 172.21.15.111:0/2552888254 learned_addr learned my addr 172.21.15.111:0/2552888254 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:14.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd159d9b700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fcf70 0x7fd15c10cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 -- 172.21.15.111:0/2552888254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fcf70 msgr2=0x7fd15c10cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fcf70 0x7fd15c10cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 -- 172.21.15.111:0/2552888254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd150005040 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd15a59c700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00ec90 tx=0x7fd14c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14c007b30 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd14c00f040 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c10de00 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14c00cbd0 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.735+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c10e340 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.736+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c05f5f0 con 0x7fd15c0fbe00 2024-03-14T13:00:14.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.736+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd14c007c90 con 0x7fd15c0fbe00 2024-03-14T13:00:14.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.737+0000 7fd14affd700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 0x7fd14406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:14.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.737+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd14c087590 con 0x7fd15c0fbe00 2024-03-14T13:00:14.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.737+0000 7fd159d9b700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 0x7fd14406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:14.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.737+0000 7fd159d9b700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 0x7fd14406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd150013680 tx=0x7fd150009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:14.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.741+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7fd14c057350 con 0x7fd15c0fbe00 2024-03-14T13:00:14.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.946+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd15c10e710 con 0x7fd1440692b0 2024-03-14T13:00:14.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.947+0000 7fd14affd700 1 -- 172.21.15.111:0/2552888254 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7fd15c10e710 con 0x7fd1440692b0 2024-03-14T13:00:14.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 msgr2=0x7fd14406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 0x7fd14406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd150013680 tx=0x7fd150009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 msgr2=0x7fd15c10c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00ec90 tx=0x7fd14c00b040 comp rx=0 tx=0).stop 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 shutdown_connections 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7fd1440692b0 0x7fd14406b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd15c0fbe00 0x7fd15c10c7e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 --2- 172.21.15.111:0/2552888254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c0fcf70 0x7fd15c10cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 >> 172.21.15.111:0/2552888254 conn(0x7fd15c0f74e0 msgr2=0x7fd15c0f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 shutdown_connections 2024-03-14T13:00:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:14.949+0000 7fd160aaa700 1 -- 172.21.15.111:0/2552888254 wait complete. 2024-03-14T13:00:15.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 -- 172.21.15.111:0/4145136277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 msgr2=0x7f16300fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 --2- 172.21.15.111:0/4145136277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f16300fc0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1620004940 tx=0x7f1620009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 -- 172.21.15.111:0/4145136277 shutdown_connections 2024-03-14T13:00:15.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 --2- 172.21.15.111:0/4145136277 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f16300fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 --2- 172.21.15.111:0/4145136277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f16300fc0e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 -- 172.21.15.111:0/4145136277 >> 172.21.15.111:0/4145136277 conn(0x7f16300f7430 msgr2=0x7f16300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 -- 172.21.15.111:0/4145136277 shutdown_connections 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.050+0000 7f16374d1700 1 -- 172.21.15.111:0/4145136277 wait complete. 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 Processor -- start 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 -- start start 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f1630112a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630066240 con 0x7f16300fbcd0 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f16374d1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1630066380 con 0x7f16300fcf80 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:41186/0 (socket says 172.21.15.111:41186) 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 -- 172.21.15.111:0/75635122 learned_addr learned my addr 172.21.15.111:0/75635122 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:15.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f163526d700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f1630112a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 -- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 msgr2=0x7f1630112a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f1630112a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f1634a6c700 1 -- 172.21.15.111:0/75635122 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1620005040 con 0x7f16300fcf80 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.051+0000 7f163526d700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f1630112a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f1634a6c700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f162c002a50 tx=0x7f162c0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f162c007cc0 con 0x7f16300fcf80 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16301154e0 con 0x7f16300fcf80 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f162c00f040 con 0x7f16300fcf80 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f162c00d9c0 con 0x7f16300fcf80 2024-03-14T13:00:15.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.052+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1630115900 con 0x7f16300fcf80 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.053+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163005f5f0 con 0x7f16300fcf80 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.053+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f162c00db20 con 0x7f16300fcf80 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.054+0000 7f1625ffb700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 0x7f161c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.054+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f162c086f10 con 0x7f16300fcf80 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.054+0000 7f163526d700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 0x7f161c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.054+0000 7f163526d700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 0x7f161c06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1620004890 tx=0x7f1620009e30 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.059+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f162c053160 con 0x7f16300fcf80 2024-03-14T13:00:15.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.255+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f16301013e0 con 0x7f161c0692a0 2024-03-14T13:00:15.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.264+0000 7f1625ffb700 1 -- 172.21.15.111:0/75635122 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f16301013e0 con 0x7f161c0692a0 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 90s ago 8m 26.5M - 0.25.0 c8568f914cd2 a5a721c2d1c2 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (8m) 90s ago 8m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (6m) 97s ago 6m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (6m) 90s ago 7m 87.0M - 9.4.7 954c08fa6188 a0f9b28f5afd 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (110s) 90s ago 110s 12.4M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (116s) 90s ago 116s 16.9M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (108s) 97s ago 107s 15.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (113s) 97s ago 113s 14.5M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:9283 running (9m) 90s ago 9m 481M - 17.2.7-646-gb92acd1f a07fb176315d 9d384b708369 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 starting - - - - 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (9m) 90s ago 9m 47.4M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:00:15.264 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (6m) 97s ago 6m 36.1M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 90s ago 8m 17.9M - 1.5.0 0da6a335fe13 834d8e9211eb 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 97s ago 6m 18.2M - 1.5.0 0da6a335fe13 de8bbb0f9cd8 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (5m) 90s ago 5m 46.5M 4299M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 90s ago 5m 46.1M 4299M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (4m) 90s ago 4m 45.5M 4299M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (4m) 97s ago 4m 47.4M 5323M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (3m) 97s ago 3m 43.1M 5323M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (3m) 97s ago 3m 40.7M 5323M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:00:15.265 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (5m) 90s ago 7m 52.7M - 2.43.0 a07b618ecd1d c5814e8b878b 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 msgr2=0x7f161c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 0x7f161c06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1620004890 tx=0x7f1620009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 msgr2=0x7f1630114fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f162c002a50 tx=0x7f162c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 shutdown_connections 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f161c0692a0 0x7f161c06b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16300fbcd0 0x7f1630112a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 --2- 172.21.15.111:0/75635122 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16300fcf80 0x7f1630114fa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 >> 172.21.15.111:0/75635122 conn(0x7f16300f7430 msgr2=0x7f16300f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.267 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 shutdown_connections 2024-03-14T13:00:15.267 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.266+0000 7f16374d1700 1 -- 172.21.15.111:0/75635122 wait complete. 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/4022919318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 msgr2=0x7f1cb80fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/4022919318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb80fd400 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1cac004940 tx=0x7f1cac009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/4022919318 shutdown_connections 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/4022919318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb80fd400 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/4022919318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 0x7f1cb80fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.365+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/4022919318 >> 172.21.15.111:0/4022919318 conn(0x7f1cb80f7480 msgr2=0x7f1cb80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/4022919318 shutdown_connections 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/4022919318 wait complete. 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 Processor -- start 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 -- start start 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 0x7f1cb81105c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8066bb0 con 0x7f1cb80fcf90 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cbcc2e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8066cf0 con 0x7f1cb80fbce0 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cb5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cb5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48570/0 (socket says 172.21.15.111:48570) 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cb5d9b700 1 -- 172.21.15.111:0/2952252015 learned_addr learned my addr 172.21.15.111:0/2952252015 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.366+0000 7f1cb659c700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 0x7f1cb81105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cb5d9b700 1 -- 172.21.15.111:0/2952252015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 msgr2=0x7f1cb81105c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cb5d9b700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 0x7f1cb81105c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cb5d9b700 1 -- 172.21.15.111:0/2952252015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac005040 con 0x7f1cb80fcf90 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cb5d9b700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0137b0 tx=0x7f1cac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cac00e070 con 0x7f1cb80fcf90 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cac01c400 con 0x7f1cb80fcf90 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cac00b650 con 0x7f1cb80fcf90 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8113050 con 0x7f1cb80fcf90 2024-03-14T13:00:15.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.367+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8113580 con 0x7f1cb80fcf90 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.368+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb805f5f0 con 0x7f1cb80fcf90 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.368+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1cac01c9e0 con 0x7f1cb80fcf90 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.369+0000 7f1ca6ffd700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 0x7f1ca006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.369+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1cac088710 con 0x7f1cb80fcf90 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.369+0000 7f1cb659c700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 0x7f1ca006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.369+0000 7f1cb659c700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 0x7f1ca006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8066830 tx=0x7f1ca800b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.373+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f1cac054ae0 con 0x7f1cb80fcf90 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.637+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1cb804bed0 con 0x7f1cb80fcf90 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.637+0000 7f1ca6ffd700 1 -- 172.21.15.111:0/2952252015 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f1cac004490 con 0x7f1cb80fcf90 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:00:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 14 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:00:15.638 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 msgr2=0x7f1ca006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 0x7f1ca006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8066830 tx=0x7f1ca800b040 comp rx=0 tx=0).stop 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 msgr2=0x7f1cb8112b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0137b0 tx=0x7f1cac009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 shutdown_connections 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fbce0 0x7f1cb81105c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f1ca00692a0 0x7f1ca006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 --2- 172.21.15.111:0/2952252015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fcf90 0x7f1cb8112b10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.639+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 >> 172.21.15.111:0/2952252015 conn(0x7f1cb80f7480 msgr2=0x7f1cb80f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.640+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 shutdown_connections 2024-03-14T13:00:15.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.640+0000 7f1cbcc2e700 1 -- 172.21.15.111:0/2952252015 wait complete. 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- 172.21.15.111:0/4165187221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 msgr2=0x7f20dc102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 --2- 172.21.15.111:0/4165187221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc102340 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f20c4004940 tx=0x7f20c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- 172.21.15.111:0/4165187221 shutdown_connections 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 --2- 172.21.15.111:0/4165187221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc102340 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 --2- 172.21.15.111:0/4165187221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 0x7f20dc0ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- 172.21.15.111:0/4165187221 >> 172.21.15.111:0/4165187221 conn(0x7f20dc0f7430 msgr2=0x7f20dc0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- 172.21.15.111:0/4165187221 shutdown_connections 2024-03-14T13:00:15.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- 172.21.15.111:0/4165187221 wait complete. 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 Processor -- start 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 -- start start 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.743+0000 7f20e2c62700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 0x7f20dc10d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e2c62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e2c62700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc10e420 con 0x7f20dc0fff60 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e2c62700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc10ec70 con 0x7f20dc066280 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48588/0 (socket says 172.21.15.111:48588) 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 -- 172.21.15.111:0/175014533 learned_addr learned my addr 172.21.15.111:0/175014533 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e09fe700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 0x7f20dc10d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 -- 172.21.15.111:0/175014533 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 msgr2=0x7f20dc10d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 0x7f20dc10d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 -- 172.21.15.111:0/175014533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c4005040 con 0x7f20dc0fff60 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20dbfff700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f20c40137b0 tx=0x7f20c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c400e070 con 0x7f20dc0fff60 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20c401c400 con 0x7f20dc0fff60 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc10eef0 con 0x7f20dc0fff60 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20c400b650 con 0x7f20dc0fff60 2024-03-14T13:00:15.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.744+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc10f3d0 con 0x7f20dc0fff60 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.745+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc05f5f0 con 0x7f20dc0fff60 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.746+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f20c400b8c0 con 0x7f20dc0fff60 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.746+0000 7f20d97fa700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 0x7f20c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.746+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f20c40886a0 con 0x7f20dc0fff60 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.746+0000 7f20e09fe700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 0x7f20c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:15.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.746+0000 7f20e09fe700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 0x7f20c806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009b80 tx=0x7f20cc00b040 comp rx=0 tx=0).ready entity=mgr.14222 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:15.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.751+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175445 (secure 0 0 0) 0x7f20c4058570 con 0x7f20dc0fff60 2024-03-14T13:00:15.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.962+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 --> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20dc10f750 con 0x7f20c80692a0 2024-03-14T13:00:15.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.963+0000 7f20d97fa700 1 -- 172.21.15.111:0/175014533 <== mgr.14222 v2:172.21.15.111:6800/1542503054 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+321 (secure 0 0 0) 0x7f20dc10f750 con 0x7f20c80692a0 2024-03-14T13:00:15.963 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:00:15.963 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:00:15.963 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "", 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mgr daemons", 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:00:15.964 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.965+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 msgr2=0x7f20c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.965+0000 7f20e2c62700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 0x7f20c806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20cc009b80 tx=0x7f20cc00b040 comp rx=0 tx=0).stop 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.965+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 msgr2=0x7f20dc10de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.965+0000 7f20e2c62700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f20c40137b0 tx=0x7f20c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 shutdown_connections 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20dc066280 0x7f20dc10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:6800/1542503054,v1:172.21.15.111:6801/1542503054] conn(0x7f20c80692a0 0x7f20c806b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 --2- 172.21.15.111:0/175014533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20dc0fff60 0x7f20dc10de20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:15.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 >> 172.21.15.111:0/175014533 conn(0x7f20dc0f7430 msgr2=0x7f20dc0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:15.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 shutdown_connections 2024-03-14T13:00:15.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:15.966+0000 7f20e2c62700 1 -- 172.21.15.111:0/175014533 wait complete. 2024-03-14T13:00:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:15 smithi137 ceph-mon[50336]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:15 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:15 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2952252015' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:00:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:15 smithi111 ceph-mon[39688]: from='client.14590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:15 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:15 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2952252015' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:00:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:16 smithi137 ceph-mon[50336]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:16 smithi137 ceph-mon[50336]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:16 smithi137 ceph-mon[50336]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:16 smithi111 ceph-mon[39688]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:16 smithi111 ceph-mon[39688]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:16 smithi111 ceph-mon[39688]: pgmap v245: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:17 smithi111 ceph-mon[39688]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:18.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:17 smithi137 ceph-mon[50336]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:19.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:18 smithi111 ceph-mon[39688]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:00:19.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:18 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:19.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:18 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:19.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:18 smithi137 ceph-mon[50336]: pgmap v246: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:00:19.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:18 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:19.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:18 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:20 smithi137 ceph-mon[50336]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:20 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:20 smithi111 ceph-mon[39688]: pgmap v247: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:20 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:22 smithi111 ceph-mon[39688]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:22.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:22 smithi137 ceph-mon[50336]: pgmap v248: 65 pgs: 65 active+clean; 590 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr fail", "who": "smithi111.atysez"}]: dispatch 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: osdmap e47: 6 total, 6 up, 6 in 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "mgr fail", "who": "smithi111.atysez"}]': finished 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: mgrmap e20: smithi137.xbyzbp(active, starting, since 0.00613086s) 2024-03-14T13:00:24.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: Active manager daemon smithi137.xbyzbp restarted 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: Activating manager daemon smithi137.xbyzbp 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: osdmap e48: 6 total, 6 up, 6 in 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: mgrmap e21: smithi137.xbyzbp(active, starting, since 0.00585338s) 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:00:24.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:00:24.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:00:24.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: Manager daemon smithi137.xbyzbp is now available 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:24.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/trash_purge_schedule"}]: dispatch 2024-03-14T13:00:24.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:24 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/trash_purge_schedule"}]: dispatch 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr fail", "who": "smithi111.atysez"}]: dispatch 2024-03-14T13:00:24.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: osdmap e47: 6 total, 6 up, 6 in 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.14222 172.21.15.111:0/1067581686' entity='mgr.smithi111.atysez' cmd='[{"prefix": "mgr fail", "who": "smithi111.atysez"}]': finished 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: mgrmap e20: smithi137.xbyzbp(active, starting, since 0.00613086s) 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: Active manager daemon smithi137.xbyzbp restarted 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: Activating manager daemon smithi137.xbyzbp 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: osdmap e48: 6 total, 6 up, 6 in 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: mgrmap e21: smithi137.xbyzbp(active, starting, since 0.00585338s) 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:00:24.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:00:24.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: Manager daemon smithi137.xbyzbp is now available 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/trash_purge_schedule"}]: dispatch 2024-03-14T13:00:24.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:24 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.xbyzbp/trash_purge_schedule"}]: dispatch 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: Deploying cephadm binary to smithi111 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: Deploying cephadm binary to smithi137 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: mgrmap e22: smithi137.xbyzbp(active, since 1.00844s) 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:25.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:25 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: Deploying cephadm binary to smithi111 2024-03-14T13:00:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: Deploying cephadm binary to smithi137 2024-03-14T13:00:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: mgrmap e22: smithi137.xbyzbp(active, since 1.00844s) 2024-03-14T13:00:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: pgmap v3: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:25 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:26.736 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: [14/Mar/2024:13:00:25] ENGINE Bus STARTING 2024-03-14T13:00:26.737 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: [14/Mar/2024:13:00:25] ENGINE Serving on https://172.21.15.137:7150 2024-03-14T13:00:26.737 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: [14/Mar/2024:13:00:25] ENGINE Serving on http://172.21.15.137:8765 2024-03-14T13:00:26.737 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: [14/Mar/2024:13:00:25] ENGINE Bus STARTED 2024-03-14T13:00:26.737 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:26.737 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:26 smithi111 ceph-mon[39688]: mgrmap e23: smithi137.xbyzbp(active, since 3s) 2024-03-14T13:00:27.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:00:25] ENGINE Bus STARTING 2024-03-14T13:00:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:00:25] ENGINE Serving on https://172.21.15.137:7150 2024-03-14T13:00:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:00:25] ENGINE Serving on http://172.21.15.137:8765 2024-03-14T13:00:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:00:25] ENGINE Bus STARTED 2024-03-14T13:00:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: pgmap v4: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:26 smithi137 ceph-mon[50336]: mgrmap e23: smithi137.xbyzbp(active, since 3s) 2024-03-14T13:00:28.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:28 smithi137 ceph-mon[50336]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:28.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:28 smithi137 ceph-mon[50336]: mgrmap e24: smithi137.xbyzbp(active, since 4s) 2024-03-14T13:00:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:28 smithi111 ceph-mon[39688]: pgmap v5: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:00:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:28 smithi111 ceph-mon[39688]: mgrmap e24: smithi137.xbyzbp(active, since 4s) 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: Standby manager daemon smithi111.atysez started 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/crt"}]: dispatch 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:00:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/key"}]: dispatch 2024-03-14T13:00:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:30 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:00:31.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: pgmap v6: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: Standby manager daemon smithi111.atysez started 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/crt"}]: dispatch 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/key"}]: dispatch 2024-03-14T13:00:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:30 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/1437801540' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:00:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:31 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:31 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:31 smithi137 ceph-mon[50336]: mgrmap e25: smithi137.xbyzbp(active, since 7s), standbys: smithi111.atysez 2024-03-14T13:00:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:31 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:00:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:31 smithi137 ceph-mon[50336]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-14T13:00:32.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:31 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:32.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:31 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:32.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:31 smithi111 ceph-mon[39688]: mgrmap e25: smithi137.xbyzbp(active, since 7s), standbys: smithi111.atysez 2024-03-14T13:00:32.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:31 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:00:32.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:31 smithi111 ceph-mon[39688]: pgmap v7: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:32 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:33.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:32 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:34.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:33 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi111 to 1568M 2024-03-14T13:00:34.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:33 smithi137 ceph-mon[50336]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-14T13:00:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:33 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi111 to 1568M 2024-03-14T13:00:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:33 smithi111 ceph-mon[39688]: pgmap v8: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-14T13:00:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:36 smithi111 ceph-mon[39688]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:00:36.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:36 smithi137 ceph-mon[50336]: pgmap v9: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:00:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:38 smithi111 ceph-mon[39688]: pgmap v10: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-14T13:00:39.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:38 smithi137 ceph-mon[50336]: pgmap v10: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-14T13:00:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:39 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:39 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:40 smithi111 ceph-mon[39688]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-14T13:00:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:40 smithi137 ceph-mon[50336]: pgmap v11: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-14T13:00:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:42 smithi111 ceph-mon[39688]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:00:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:42 smithi137 ceph-mon[50336]: pgmap v12: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Adjusting osd_memory_target on smithi137 to 2592M 2024-03-14T13:00:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:00:44.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:00:44.008 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:00:44.008 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:43 smithi137 ceph-mon[50336]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Adjusting osd_memory_target on smithi137 to 2592M 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:00:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:00:44.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:00:44.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:43 smithi111 ceph-mon[39688]: pgmap v13: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: Reconfiguring node-exporter.smithi111 (dependencies changed)... 2024-03-14T13:00:45.057 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:44 smithi111 ceph-mon[39688]: Reconfiguring daemon node-exporter.smithi111 on smithi111 2024-03-14T13:00:45.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: Reconfiguring node-exporter.smithi111 (dependencies changed)... 2024-03-14T13:00:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:44 smithi137 ceph-mon[50336]: Reconfiguring daemon node-exporter.smithi111 on smithi111 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.125+0000 7f9dd9c83700 1 -- 172.21.15.111:0/3796196498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fbcd0 msgr2=0x7f9dd40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.125+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/3796196498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd40fc0e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4004940 tx=0x7f9dc4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 -- 172.21.15.111:0/3796196498 shutdown_connections 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/3796196498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fcf80 0x7f9dd40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/3796196498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd40fc0e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 -- 172.21.15.111:0/3796196498 >> 172.21.15.111:0/3796196498 conn(0x7f9dd40f7430 msgr2=0x7f9dd40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 -- 172.21.15.111:0/3796196498 shutdown_connections 2024-03-14T13:00:46.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.126+0000 7f9dd9c83700 1 -- 172.21.15.111:0/3796196498 wait complete. 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 Processor -- start 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 -- start start 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fcf80 0x7f9dd410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd410e3f0 con 0x7f9dd40fcf80 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd9c83700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd410ec40 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd37fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd37fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:38712/0 (socket says 172.21.15.111:38712) 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd37fe700 1 -- 172.21.15.111:0/1450170517 learned_addr learned my addr 172.21.15.111:0/1450170517 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.127+0000 7f9dd2ffd700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fcf80 0x7f9dd410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.128+0000 7f9dd37fe700 1 -- 172.21.15.111:0/1450170517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fcf80 msgr2=0x7f9dd410ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.128+0000 7f9dd37fe700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fcf80 0x7f9dd410ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.128+0000 7f9dd37fe700 1 -- 172.21.15.111:0/1450170517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc4005040 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dd37fe700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40137b0 tx=0x7f9dc4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc400e070 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dc401c400 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd410eec0 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dc400b650 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.129+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd410f3a0 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.130+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd405f5f0 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.131+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f9dc401c9e0 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.132+0000 7f9dbbfff700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 0x7f9dbc06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.132+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f9dc408abe0 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.132+0000 7f9dd2ffd700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 0x7f9dbc06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.132+0000 7f9dd2ffd700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 0x7f9dbc06d140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009b80 tx=0x7f9dc800b040 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:46.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.137+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f9dc4058d30 con 0x7f9dd40fbcd0 2024-03-14T13:00:46.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:46 smithi111 ceph-mon[39688]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:00:46.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:46 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:46.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:46 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:46.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.552+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dd410f7e0 con 0x7f9dbc06ac90 2024-03-14T13:00:46.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.554+0000 7f9dbbfff700 1 -- 172.21.15.111:0/1450170517 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f9dd410f7e0 con 0x7f9dbc06ac90 2024-03-14T13:00:46.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.556+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 msgr2=0x7f9dbc06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.556+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 0x7f9dbc06d140 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8009b80 tx=0x7f9dc800b040 comp rx=0 tx=0).stop 2024-03-14T13:00:46.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 msgr2=0x7f9dd410d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9dc40137b0 tx=0x7f9dc4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 shutdown_connections 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f9dbc06ac90 0x7f9dbc06d140 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9dd40fbcd0 0x7f9dd410d8b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 --2- 172.21.15.111:0/1450170517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9dd40fcf80 0x7f9dd410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 >> 172.21.15.111:0/1450170517 conn(0x7f9dd40f7430 msgr2=0x7f9dd40f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 shutdown_connections 2024-03-14T13:00:46.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.557+0000 7f9dd9c83700 1 -- 172.21.15.111:0/1450170517 wait complete. 2024-03-14T13:00:46.569 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.667+0000 7ff795b1f700 1 -- 172.21.15.111:0/1821007698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fcf80 msgr2=0x7ff7900fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.667+0000 7ff795b1f700 1 --2- 172.21.15.111:0/1821007698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fcf80 0x7ff7900fd3f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff784004940 tx=0x7ff784009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 -- 172.21.15.111:0/1821007698 shutdown_connections 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 --2- 172.21.15.111:0/1821007698 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fcf80 0x7ff7900fd3f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 --2- 172.21.15.111:0/1821007698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fbcd0 0x7ff7900fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 -- 172.21.15.111:0/1821007698 >> 172.21.15.111:0/1821007698 conn(0x7ff7900f7430 msgr2=0x7ff7900f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:46.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 -- 172.21.15.111:0/1821007698 shutdown_connections 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 -- 172.21.15.111:0/1821007698 wait complete. 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.668+0000 7ff795b1f700 1 Processor -- start 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff795b1f700 1 -- start start 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff795b1f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fbcd0 0x7ff79010faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff795b1f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff795b1f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff790110630 con 0x7ff7900fcf80 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff795b1f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff790110e80 con 0x7ff7900fbcd0 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53578/0 (socket says 172.21.15.111:53578) 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 -- 172.21.15.111:0/3888819187 learned_addr learned my addr 172.21.15.111:0/3888819187 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78f7fe700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fbcd0 0x7ff79010faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 -- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fbcd0 msgr2=0x7ff79010faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fbcd0 0x7ff79010faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 -- 172.21.15.111:0/3888819187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff780005040 con 0x7ff7900fcf80 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.669+0000 7ff78effd700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff7840137b0 tx=0x7ff784009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.670+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78400e070 con 0x7ff7900fcf80 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.670+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff784005040 con 0x7ff7900fcf80 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.670+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff78401c400 con 0x7ff7900fcf80 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.670+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff78400b650 con 0x7ff7900fcf80 2024-03-14T13:00:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.670+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff790111450 con 0x7ff7900fcf80 2024-03-14T13:00:46.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.671+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7ff78401c9e0 con 0x7ff7900fcf80 2024-03-14T13:00:46.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.671+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79005f5f0 con 0x7ff7900fcf80 2024-03-14T13:00:46.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.672+0000 7ff777fff700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 0x7ff77806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:46.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.672+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7ff78408ac40 con 0x7ff7900fcf80 2024-03-14T13:00:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.672+0000 7ff78f7fe700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 0x7ff77806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.673+0000 7ff78f7fe700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 0x7ff77806d140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff780009fd0 tx=0x7ff78000b040 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:46.679 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.680+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7ff784058d10 con 0x7ff7900fcf80 2024-03-14T13:00:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:46 smithi137 ceph-mon[50336]: pgmap v14: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:00:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:46 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:46 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:46.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.991+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff790111780 con 0x7ff77806ac90 2024-03-14T13:00:46.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.993+0000 7ff777fff700 1 -- 172.21.15.111:0/3888819187 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7ff790111780 con 0x7ff77806ac90 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 msgr2=0x7ff77806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 0x7ff77806d140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff780009fd0 tx=0x7ff78000b040 comp rx=0 tx=0).stop 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 msgr2=0x7ff790110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff7840137b0 tx=0x7ff784009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 shutdown_connections 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7ff77806ac90 0x7ff77806d140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7900fbcd0 0x7ff79010faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 --2- 172.21.15.111:0/3888819187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff7900fcf80 0x7ff790110030 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:46.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.995+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 >> 172.21.15.111:0/3888819187 conn(0x7ff7900f7430 msgr2=0x7ff7900f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:46.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.996+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 shutdown_connections 2024-03-14T13:00:46.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:46.996+0000 7ff795b1f700 1 -- 172.21.15.111:0/3888819187 wait complete. 2024-03-14T13:00:47.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 -- 172.21.15.111:0/3526572107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 msgr2=0x7f6c140fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 --2- 172.21.15.111:0/3526572107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c140fc0e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6c10004940 tx=0x7f6c10009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:47.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 -- 172.21.15.111:0/3526572107 shutdown_connections 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 --2- 172.21.15.111:0/3526572107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c140fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 --2- 172.21.15.111:0/3526572107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c140fc0e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 -- 172.21.15.111:0/3526572107 >> 172.21.15.111:0/3526572107 conn(0x7f6c140f7430 msgr2=0x7f6c140f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 -- 172.21.15.111:0/3526572107 shutdown_connections 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.140+0000 7f6c1b795700 1 -- 172.21.15.111:0/3526572107 wait complete. 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.141+0000 7f6c1b795700 1 Processor -- start 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.142+0000 7f6c1b795700 1 -- start start 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.142+0000 7f6c1b795700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.142+0000 7f6c1b795700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c141022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.142+0000 7f6c1b795700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14102890 con 0x7f6c140fbcd0 2024-03-14T13:00:47.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.142+0000 7f6c1b795700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c14102a00 con 0x7f6c140fcf80 2024-03-14T13:00:47.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53608/0 (socket says 172.21.15.111:53608) 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 -- 172.21.15.111:0/1966382841 learned_addr learned my addr 172.21.15.111:0/1966382841 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c18d30700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c141022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 -- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 msgr2=0x7f6c141022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c141022e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 -- 172.21.15.111:0/1966382841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10005040 con 0x7f6c140fbcd0 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c18d30700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c141022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c19531700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c100137b0 tx=0x7f6c10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.143+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c10021070 con 0x7f6c140fbcd0 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.144+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c14102c80 con 0x7f6c140fbcd0 2024-03-14T13:00:47.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.144+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c14115900 con 0x7f6c140fbcd0 2024-03-14T13:00:47.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.144+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c1001c400 con 0x7f6c140fbcd0 2024-03-14T13:00:47.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.144+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c1000e070 con 0x7f6c140fbcd0 2024-03-14T13:00:47.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.145+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf80052e0 con 0x7f6c140fbcd0 2024-03-14T13:00:47.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.147+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f6c10012740 con 0x7f6c140fbcd0 2024-03-14T13:00:47.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.148+0000 7f6c09ffb700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 0x7f6c0006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.148+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f6c1008b1e0 con 0x7f6c140fbcd0 2024-03-14T13:00:47.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.148+0000 7f6c18d30700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 0x7f6c0006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.149+0000 7f6c18d30700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 0x7f6c0006d140 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c0400b040 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:47.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.155+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f6c10055bd0 con 0x7f6c140fbcd0 2024-03-14T13:00:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.479+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bf8000bf0 con 0x7f6c0006ac90 2024-03-14T13:00:47.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.497+0000 7f6c09ffb700 1 -- 172.21.15.111:0/1966382841 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f6bf8000bf0 con 0x7f6c0006ac90 2024-03-14T13:00:47.496 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:00:47.496 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 15s ago 8m 26.5M - 0.25.0 c8568f914cd2 a5a721c2d1c2 2024-03-14T13:00:47.496 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (8m) 15s ago 8m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:00:47.496 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (7m) 17s ago 7m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (6m) 15s ago 7m 87.0M - 9.4.7 954c08fa6188 a0f9b28f5afd 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (2m) 15s ago 2m 18.0M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (2m) 15s ago 2m 25.6M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (2m) 17s ago 2m 20.7M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (2m) 17s ago 2m 18.9M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:9283 running (10m) 15s ago 10m 288M - 17.2.7-646-gb92acd1f a07fb176315d 9d384b708369 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (33s) 17s ago 7m 479M - 18.2.2-825-g2f465319 a9e65ad305ae 0487350e99e4 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (10m) 15s ago 10m 50.0M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (7m) 17s ago 7m 40.0M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 starting - - - - 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 17s ago 7m 18.6M - 1.5.0 0da6a335fe13 de8bbb0f9cd8 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 15s ago 6m 46.7M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 15s ago 5m 46.8M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 15s ago 5m 47.0M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (4m) 17s ago 4m 48.0M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:00:47.497 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (4m) 17s ago 4m 45.3M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:00:47.498 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (4m) 17s ago 4m 42.5M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:00:47.498 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (6m) 15s ago 7m 59.2M - 2.43.0 a07b618ecd1d c5814e8b878b 2024-03-14T13:00:47.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 msgr2=0x7f6c0006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 0x7f6c0006d140 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6c04009fd0 tx=0x7f6c0400b040 comp rx=0 tx=0).stop 2024-03-14T13:00:47.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 msgr2=0x7f6c14103e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c100137b0 tx=0x7f6c10009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:47.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 shutdown_connections 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f6c0006ac90 0x7f6c0006d140 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c140fbcd0 0x7f6c14103e10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 --2- 172.21.15.111:0/1966382841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6c140fcf80 0x7f6c141022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 >> 172.21.15.111:0/1966382841 conn(0x7f6c140f7430 msgr2=0x7f6c140f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 shutdown_connections 2024-03-14T13:00:47.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.500+0000 7f6c1b795700 1 -- 172.21.15.111:0/1966382841 wait complete. 2024-03-14T13:00:47.523 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:47 smithi111 ceph-mon[39688]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-03-14T13:00:47.523 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:47 smithi111 ceph-mon[39688]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-03-14T13:00:47.523 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:47 smithi111 ceph-mon[39688]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:47.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.607+0000 7f9709995700 1 -- 172.21.15.111:0/1276778823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 msgr2=0x7f97040fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.607+0000 7f9709995700 1 --2- 172.21.15.111:0/1276778823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f97040fc0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f96f4004940 tx=0x7f96f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:47.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 -- 172.21.15.111:0/1276778823 shutdown_connections 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 --2- 172.21.15.111:0/1276778823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 0x7f97040fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 --2- 172.21.15.111:0/1276778823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f97040fc0e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 -- 172.21.15.111:0/1276778823 >> 172.21.15.111:0/1276778823 conn(0x7f97040f7430 msgr2=0x7f97040f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 -- 172.21.15.111:0/1276778823 shutdown_connections 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.612+0000 7f9709995700 1 -- 172.21.15.111:0/1276778823 wait complete. 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 Processor -- start 2024-03-14T13:00:47.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 -- start start 2024-03-14T13:00:47.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 0x7f970410de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970410e490 con 0x7f97040fbcd0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9709995700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970410ece0 con 0x7f97040fcf80 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9702ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9702ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53624/0 (socket says 172.21.15.111:53624) 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f9702ffd700 1 -- 172.21.15.111:0/3415852932 learned_addr learned my addr 172.21.15.111:0/3415852932 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.613+0000 7f97027fc700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 0x7f970410de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9702ffd700 1 -- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 msgr2=0x7f970410de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9702ffd700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 0x7f970410de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9702ffd700 1 -- 172.21.15.111:0/3415852932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4005040 con 0x7f97040fbcd0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9702ffd700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009380 tx=0x7f96f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f400e040 con 0x7f97040fbcd0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970410ef60 con 0x7f97040fbcd0 2024-03-14T13:00:47.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970410f4a0 con 0x7f97040fbcd0 2024-03-14T13:00:47.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.614+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f400bd20 con 0x7f97040fbcd0 2024-03-14T13:00:47.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.615+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96f401c630 con 0x7f97040fbcd0 2024-03-14T13:00:47.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.615+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f00052e0 con 0x7f97040fbcd0 2024-03-14T13:00:47.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.616+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f96f4026440 con 0x7f97040fbcd0 2024-03-14T13:00:47.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.617+0000 7f96eb7fe700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 0x7f96ec06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:47.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.617+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f96f408a070 con 0x7f97040fbcd0 2024-03-14T13:00:47.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.617+0000 7f97027fc700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 0x7f96ec06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:47.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.617+0000 7f97027fc700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 0x7f96ec06d140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f96f80041d0 tx=0x7f96f80043c0 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:47.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:47.625+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f96f40583a0 con 0x7f97040fbcd0 2024-03-14T13:00:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:47 smithi137 ceph-mon[50336]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-03-14T13:00:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:47 smithi137 ceph-mon[50336]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-03-14T13:00:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:47 smithi137 ceph-mon[50336]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:48.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.129+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f96f0005860 con 0x7f97040fbcd0 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.129+0000 7f96eb7fe700 1 -- 172.21.15.111:0/3415852932 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+848 (secure 0 0 0) 0x7f96f4057f30 con 0x7f97040fbcd0 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 1, 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:48.129 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 13, 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:00:48.130 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:00:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 msgr2=0x7f96ec06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 0x7f96ec06d140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f96f80041d0 tx=0x7f96f80043c0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 msgr2=0x7f970410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009380 tx=0x7f96f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 shutdown_connections 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f96ec06ac90 0x7f96ec06d140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97040fbcd0 0x7f970410d950 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 --2- 172.21.15.111:0/3415852932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97040fcf80 0x7f970410de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.132+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 >> 172.21.15.111:0/3415852932 conn(0x7f97040f7430 msgr2=0x7f97040f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.133+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 shutdown_connections 2024-03-14T13:00:48.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.133+0000 7f9709995700 1 -- 172.21.15.111:0/3415852932 wait complete. 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2568298381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 msgr2=0x7fdb4c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2568298381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c102360 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c004880 tx=0x7fdb3c009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2568298381 shutdown_connections 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2568298381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c102360 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2568298381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2568298381 >> 172.21.15.111:0/2568298381 conn(0x7fdb4c0f7430 msgr2=0x7fdb4c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2568298381 shutdown_connections 2024-03-14T13:00:48.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.260+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2568298381 wait complete. 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 Processor -- start 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 -- start start 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c0fb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0fbef0 con 0x7fdb4c0fff80 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb53dc7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0fc060 con 0x7fdb4c066260 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb51b63700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb51b63700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:38794/0 (socket says 172.21.15.111:38794) 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.261+0000 7fdb51b63700 1 -- 172.21.15.111:0/2450150617 learned_addr learned my addr 172.21.15.111:0/2450150617 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:00:48.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51362700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c0fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51b63700 1 -- 172.21.15.111:0/2450150617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 msgr2=0x7fdb4c0fb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51b63700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c0fb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51b63700 1 -- 172.21.15.111:0/2450150617 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48005040 con 0x7fdb4c066260 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51362700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb51b63700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdb480096b0 tx=0x7fdb48009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb4800e840 con 0x7fdb4c066260 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c005040 con 0x7fdb4c066260 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c0fc710 con 0x7fdb4c066260 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb4800f040 con 0x7fdb4c066260 2024-03-14T13:00:48.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.262+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb48013d60 con 0x7fdb4c066260 2024-03-14T13:00:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.263+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb300052e0 con 0x7fdb4c066260 2024-03-14T13:00:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.264+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fdb48013590 con 0x7fdb4c066260 2024-03-14T13:00:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.265+0000 7fdb427fc700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 0x7fdb3806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:00:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.265+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fdb480897d0 con 0x7fdb4c066260 2024-03-14T13:00:48.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.265+0000 7fdb51362700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 0x7fdb3806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:00:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.265+0000 7fdb51362700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 0x7fdb3806d140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c013670 tx=0x7fdb3c009e30 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:00:48.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.272+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fdb48053ed0 con 0x7fdb4c066260 2024-03-14T13:00:48.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.534+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb30000bf0 con 0x7fdb3806ac90 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.535+0000 7fdb427fc700 1 -- 172.21.15.111:0/2450150617 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fdb30000bf0 con 0x7fdb3806ac90 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "1/2 daemons upgraded", 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "", 2024-03-14T13:00:48.535 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:00:48.536 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:00:48.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 msgr2=0x7fdb3806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 0x7fdb3806d140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c013670 tx=0x7fdb3c009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:48.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 msgr2=0x7fdb4c0fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdb480096b0 tx=0x7fdb48009e30 comp rx=0 tx=0).stop 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 shutdown_connections 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fdb3806ac90 0x7fdb3806d140 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb4c066260 0x7fdb4c0fd470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 --2- 172.21.15.111:0/2450150617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb4c0fff80 0x7fdb4c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 >> 172.21.15.111:0/2450150617 conn(0x7fdb4c0f7430 msgr2=0x7fdb4c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 shutdown_connections 2024-03-14T13:00:48.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:00:48.538+0000 7fdb53dc7700 1 -- 172.21.15.111:0/2450150617 wait complete. 2024-03-14T13:00:48.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:48 smithi111 ceph-mon[39688]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:48.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:48 smithi111 ceph-mon[39688]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:48.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:48 smithi111 ceph-mon[39688]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:48.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:48 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3415852932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:00:48.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:48 smithi137 ceph-mon[50336]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:48.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:48 smithi137 ceph-mon[50336]: from='client.14650 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:48.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:48 smithi137 ceph-mon[50336]: pgmap v15: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:48.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:48 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3415852932' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:00:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:49 smithi111 ceph-mon[39688]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:49 smithi137 ceph-mon[50336]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:00:50.325 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:50 smithi111 ceph-mon[39688]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:50.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:50 smithi137 ceph-mon[50336]: pgmap v16: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:Note: switching to '2f465319e3c83d48f999325459bcdfd0346449c0'. 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:state without impacting any branches by switching back to a branch. 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-14T13:00:51.458 INFO:tasks.workunit.client.1.smithi137.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: git switch -c 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr:Or undo this operation with: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: git switch - 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr: 2024-03-14T13:00:51.459 INFO:tasks.workunit.client.1.smithi137.stderr:HEAD is now at 2f465319e3c Merge branch 'wip-64737-reef' of https://github.com/xhernandez/ceph into wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T13:00:51.463 DEBUG:teuthology.orchestra.run.smithi137:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-14T13:00:51.550 INFO:tasks.workunit.client.1.smithi137.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-14T13:00:51.553 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-14T13:00:51.553 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-14T13:00:52.108 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-14T13:00:52.173 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-14T13:00:52.210 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-14T13:00:52.211 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-14T13:00:52.211 INFO:tasks.workunit.client.1.smithi137.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-14T13:00:52.247 INFO:tasks.workunit.client.1.smithi137.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-14T13:00:52.251 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-03-14T13:00:52.251 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-14T13:00:52.312 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-14T13:00:52.314 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-14T13:00:52.314 DEBUG:teuthology.orchestra.run.smithi137:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f465319e3c83d48f999325459bcdfd0346449c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-14T13:00:52.384 INFO:tasks.workunit.client.1.smithi137.stderr:+ mkdir -p fsstress 2024-03-14T13:00:52.386 INFO:tasks.workunit.client.1.smithi137.stderr:+ pushd fsstress 2024-03-14T13:00:52.387 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-14T13:00:52.387 INFO:tasks.workunit.client.1.smithi137.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-14T13:00:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:52 smithi111 ceph-mon[39688]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:00:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:52 smithi137 ceph-mon[50336]: pgmap v17: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:00:53.733 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:53 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:53.734 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:53 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:53.734 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:53 smithi111 ceph-mon[39688]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-03-14T13:00:53.734 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:53 smithi111 ceph-mon[39688]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-03-14T13:00:53.734 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:53 smithi111 ceph-mon[39688]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:53 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:53 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:53 smithi137 ceph-mon[50336]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-03-14T13:00:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:53 smithi137 ceph-mon[50336]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-03-14T13:00:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:53 smithi137 ceph-mon[50336]: pgmap v18: 65 pgs: 65 active+clean; 591 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:00:54.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:54 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:54 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:00:56.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:56 smithi111 ceph-mon[39688]: pgmap v19: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-14T13:00:56.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:56 smithi137 ceph-mon[50336]: pgmap v19: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-14T13:00:58.668 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:58 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:58.669 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:58 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:58.669 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:58 smithi111 ceph-mon[39688]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T13:00:58.669 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:58 smithi111 ceph-mon[39688]: pgmap v20: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-14T13:00:58.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:58 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:58.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:58 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:00:58.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:58 smithi137 ceph-mon[50336]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T13:00:58.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:58 smithi137 ceph-mon[50336]: pgmap v20: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-14T13:00:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:00:59 smithi137 ceph-mon[50336]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T13:00:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:00:59 smithi111 ceph-mon[39688]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T13:01:00.754 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:00 smithi111 ceph-mon[39688]: pgmap v21: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-14T13:01:00.840 INFO:tasks.workunit.client.0.smithi111.stderr:Note: switching to '2f465319e3c83d48f999325459bcdfd0346449c0'. 2024-03-14T13:01:00.840 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:state without impacting any branches by switching back to a branch. 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: git switch -c 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr:Or undo this operation with: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: git switch - 2024-03-14T13:01:00.841 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.842 INFO:tasks.workunit.client.0.smithi111.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-14T13:01:00.842 INFO:tasks.workunit.client.0.smithi111.stderr: 2024-03-14T13:01:00.842 INFO:tasks.workunit.client.0.smithi111.stderr:HEAD is now at 2f465319e3c Merge branch 'wip-64737-reef' of https://github.com/xhernandez/ceph into wip-yuri-testing-2024-03-12-1240-reef 2024-03-14T13:01:00.845 DEBUG:teuthology.orchestra.run.smithi111:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-14T13:01:00.897 INFO:tasks.workunit.client.0.smithi111.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-14T13:01:00.899 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-14T13:01:00.899 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-14T13:01:01.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:00 smithi137 ceph-mon[50336]: pgmap v21: 65 pgs: 65 active+clean; 598 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-14T13:01:02.255 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-14T13:01:02.397 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-14T13:01:02.434 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-14T13:01:02.436 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-14T13:01:02.436 INFO:tasks.workunit.client.0.smithi111.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-14T13:01:02.473 INFO:tasks.workunit.client.0.smithi111.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-14T13:01:02.476 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T13:01:02.476 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-14T13:01:02.495 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-14T13:01:02.496 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-14T13:01:02.496 DEBUG:teuthology.orchestra.run.smithi111:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2f465319e3c83d48f999325459bcdfd0346449c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-14T13:01:02.561 INFO:tasks.workunit.client.0.smithi111.stderr:+ mkdir -p fsstress 2024-03-14T13:01:02.563 INFO:tasks.workunit.client.0.smithi111.stderr:+ pushd fsstress 2024-03-14T13:01:02.565 INFO:tasks.workunit.client.0.smithi111.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-14T13:01:02.565 INFO:tasks.workunit.client.0.smithi111.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-14T13:01:02.834 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:02 smithi111 ceph-mon[39688]: pgmap v22: 65 pgs: 65 active+clean; 602 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-14T13:01:03.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:02 smithi137 ceph-mon[50336]: pgmap v22: 65 pgs: 65 active+clean; 602 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-14T13:01:04.318 INFO:tasks.workunit.client.0.smithi111.stderr:+ tar xzf ltp-full.tgz 2024-03-14T13:01:04.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:04 smithi137 ceph-mon[50336]: pgmap v23: 65 pgs: 65 active+clean; 602 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-14T13:01:04.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:04 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:04.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:04 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:04.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:04 smithi111 ceph-mon[39688]: pgmap v23: 65 pgs: 65 active+clean; 602 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-14T13:01:04.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:04 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:04.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:04 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:05 smithi111 ceph-mon[39688]: Reconfiguring node-exporter.smithi137 (dependencies changed)... 2024-03-14T13:01:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:05 smithi111 ceph-mon[39688]: Reconfiguring daemon node-exporter.smithi137 on smithi137 2024-03-14T13:01:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:05 smithi137 ceph-mon[50336]: Reconfiguring node-exporter.smithi137 (dependencies changed)... 2024-03-14T13:01:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:05 smithi137 ceph-mon[50336]: Reconfiguring daemon node-exporter.smithi137 on smithi137 2024-03-14T13:01:06.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: pgmap v24: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 918 KiB/s wr, 15 op/s 2024-03-14T13:01:06.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:06.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:06.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:01:06.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:01:06.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:01:06.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:06 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: pgmap v24: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.0 KiB/s rd, 918 KiB/s wr, 15 op/s 2024-03-14T13:01:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:01:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:01:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:01:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:06 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:07.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:07 smithi111 ceph-mon[39688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:01:07.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:07 smithi111 ceph-mon[39688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:01:07.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:07 smithi111 ceph-mon[39688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:01:07.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:07 smithi111 ceph-mon[39688]: Upgrade: Need to upgrade myself (mgr.smithi137.xbyzbp) 2024-03-14T13:01:07.831 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:07 smithi111 ceph-mon[39688]: Upgrade: Need to upgrade myself (mgr.smithi137.xbyzbp) 2024-03-14T13:01:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:07 smithi137 ceph-mon[50336]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:01:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:07 smithi137 ceph-mon[50336]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:01:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:07 smithi137 ceph-mon[50336]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:01:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:07 smithi137 ceph-mon[50336]: Upgrade: Need to upgrade myself (mgr.smithi137.xbyzbp) 2024-03-14T13:01:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:07 smithi137 ceph-mon[50336]: Upgrade: Need to upgrade myself (mgr.smithi137.xbyzbp) 2024-03-14T13:01:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: pgmap v25: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 918 KiB/s wr, 13 op/s 2024-03-14T13:01:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:01:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:01:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:01:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:08 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:09.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: pgmap v25: 65 pgs: 65 active+clean; 11 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 918 KiB/s wr, 13 op/s 2024-03-14T13:01:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:01:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:01:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:01:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:08 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:09 smithi111 ceph-mon[39688]: Upgrade: Updating mgr.smithi111.atysez 2024-03-14T13:01:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:09 smithi111 ceph-mon[39688]: Deploying daemon mgr.smithi111.atysez on smithi111 2024-03-14T13:01:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:09 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:09 smithi137 ceph-mon[50336]: Upgrade: Updating mgr.smithi111.atysez 2024-03-14T13:01:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:09 smithi137 ceph-mon[50336]: Deploying daemon mgr.smithi111.atysez on smithi111 2024-03-14T13:01:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:09 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:10.766 INFO:tasks.workunit.client.1.smithi137.stderr:+ tar xzf ltp-full.tgz 2024-03-14T13:01:10.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:10 smithi111 ceph-mon[39688]: pgmap v26: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-03-14T13:01:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:10 smithi137 ceph-mon[50336]: pgmap v26: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 343 KiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-03-14T13:01:12.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:12 smithi111 ceph-mon[39688]: pgmap v27: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-03-14T13:01:12.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:12 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:12.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:12 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:12.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:12 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:12 smithi137 ceph-mon[50336]: pgmap v27: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 342 KiB/s rd, 2.1 MiB/s wr, 160 op/s 2024-03-14T13:01:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:12 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:12 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:12 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:14.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:14 smithi111 ceph-mon[39688]: pgmap v28: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-03-14T13:01:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:14 smithi137 ceph-mon[50336]: pgmap v28: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 341 KiB/s rd, 2.1 MiB/s wr, 159 op/s 2024-03-14T13:01:15.843 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:15 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:01:15.844 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:15 smithi111 ceph-mon[39688]: pgmap v29: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 4.1 MiB/s wr, 336 op/s 2024-03-14T13:01:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:15 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:01:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:15 smithi137 ceph-mon[50336]: pgmap v29: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 4.1 MiB/s wr, 336 op/s 2024-03-14T13:01:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:17 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:17 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:17.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:17 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:17.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:17 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:18.197 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:18 smithi111 ceph-mon[39688]: pgmap v30: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 681 KiB/s rd, 3.2 MiB/s wr, 325 op/s 2024-03-14T13:01:18.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:18 smithi137 ceph-mon[50336]: pgmap v30: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 681 KiB/s rd, 3.2 MiB/s wr, 325 op/s 2024-03-14T13:01:18.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.697+0000 7f53e58c6700 1 -- 172.21.15.111:0/2537259529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 msgr2=0x7f53e006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:18.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.697+0000 7f53e58c6700 1 --2- 172.21.15.111:0/2537259529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e006e2b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f53d0004f20 tx=0x7f53d00049a0 comp rx=0 tx=0).stop 2024-03-14T13:01:18.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 -- 172.21.15.111:0/2537259529 shutdown_connections 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 --2- 172.21.15.111:0/2537259529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 0x7f53e006ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 --2- 172.21.15.111:0/2537259529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e006e2b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 -- 172.21.15.111:0/2537259529 >> 172.21.15.111:0/2537259529 conn(0x7f53e0069de0 msgr2=0x7f53e006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 -- 172.21.15.111:0/2537259529 shutdown_connections 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 -- 172.21.15.111:0/2537259529 wait complete. 2024-03-14T13:01:18.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.698+0000 7f53e58c6700 1 Processor -- start 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.701+0000 7f53e58c6700 1 -- start start 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53e58c6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 0x7f53e00807f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53e58c6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53e58c6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0080e80 con 0x7f53e006e7f0 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53e58c6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0080ff0 con 0x7f53e006fda0 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:49054/0 (socket says 172.21.15.111:49054) 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.702+0000 7f53de7fc700 1 -- 172.21.15.111:0/3512525813 learned_addr learned my addr 172.21.15.111:0/3512525813 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:18.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.703+0000 7f53deffd700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 0x7f53e00807f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:18.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.703+0000 7f53de7fc700 1 -- 172.21.15.111:0/3512525813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 msgr2=0x7f53e00807f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:18.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.703+0000 7f53de7fc700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 0x7f53e00807f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:18.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.703+0000 7f53de7fc700 1 -- 172.21.15.111:0/3512525813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0005040 con 0x7f53e006fda0 2024-03-14T13:01:18.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.703+0000 7f53de7fc700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f53d400b790 tx=0x7f53d4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:18.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.704+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53d400f770 con 0x7f53e006fda0 2024-03-14T13:01:18.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.704+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53d400fd80 con 0x7f53e006fda0 2024-03-14T13:01:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.704+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53d4010e80 con 0x7f53e006fda0 2024-03-14T13:01:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.705+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e007f500 con 0x7f53e006fda0 2024-03-14T13:01:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.705+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e007fa20 con 0x7f53e006fda0 2024-03-14T13:01:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.707+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e0077840 con 0x7f53e006fda0 2024-03-14T13:01:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.709+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f53d40106b0 con 0x7f53e006fda0 2024-03-14T13:01:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.710+0000 7f53c77fe700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 0x7f53c806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.710+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f53d40890e0 con 0x7f53e006fda0 2024-03-14T13:01:18.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.710+0000 7f53deffd700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 0x7f53c806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:18.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.711+0000 7f53deffd700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 0x7f53c806d250 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53d0004c10 tx=0x7f53d000d040 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:18.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:18.714+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f53d4057400 con 0x7f53e006fda0 2024-03-14T13:01:19.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.006+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f53e00733d0 con 0x7f53c806ada0 2024-03-14T13:01:19.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.007+0000 7f53c77fe700 1 -- 172.21.15.111:0/3512525813 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f53e00733d0 con 0x7f53c806ada0 2024-03-14T13:01:19.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 msgr2=0x7f53c806d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 0x7f53c806d250 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53d0004c10 tx=0x7f53d000d040 comp rx=0 tx=0).stop 2024-03-14T13:01:19.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 msgr2=0x7f53e007eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f53d400b790 tx=0x7f53d4009250 comp rx=0 tx=0).stop 2024-03-14T13:01:19.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 shutdown_connections 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f53c806ada0 0x7f53c806d250 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f53d0004c10 tx=0x7f53d000d040 comp rx=0 tx=0).stop 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53e006e7f0 0x7f53e00807f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 --2- 172.21.15.111:0/3512525813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53e006fda0 0x7f53e007eea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.009+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 >> 172.21.15.111:0/3512525813 conn(0x7f53e0069de0 msgr2=0x7f53e006b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.010+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 shutdown_connections 2024-03-14T13:01:19.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.010+0000 7f53e58c6700 1 -- 172.21.15.111:0/3512525813 wait complete. 2024-03-14T13:01:19.024 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 -- 172.21.15.111:0/2445399803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fcf70 msgr2=0x7fb1240fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 --2- 172.21.15.111:0/2445399803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fcf70 0x7fb1240fd3e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb118004880 tx=0x7fb118009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 -- 172.21.15.111:0/2445399803 shutdown_connections 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 --2- 172.21.15.111:0/2445399803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fcf70 0x7fb1240fd3e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 --2- 172.21.15.111:0/2445399803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fbe00 0x7fb1240fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.153+0000 7fb1289a2700 1 -- 172.21.15.111:0/2445399803 >> 172.21.15.111:0/2445399803 conn(0x7fb1240f74e0 msgr2=0x7fb1240f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:19.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.154+0000 7fb1289a2700 1 -- 172.21.15.111:0/2445399803 shutdown_connections 2024-03-14T13:01:19.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.154+0000 7fb1289a2700 1 -- 172.21.15.111:0/2445399803 wait complete. 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.154+0000 7fb1289a2700 1 Processor -- start 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.154+0000 7fb1289a2700 1 -- start start 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.154+0000 7fb1289a2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb1289a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fcf70 0x7fb12410de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb1289a2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12410e400 con 0x7fb1240fcf70 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb1289a2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12410ec50 con 0x7fb1240fbe00 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:49082/0 (socket says 172.21.15.111:49082) 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 -- 172.21.15.111:0/994309134 learned_addr learned my addr 172.21.15.111:0/994309134 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb121d9b700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fcf70 0x7fb12410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 -- 172.21.15.111:0/994309134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fcf70 msgr2=0x7fb12410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fcf70 0x7fb12410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.155+0000 7fb12259c700 1 -- 172.21.15.111:0/994309134 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118005040 con 0x7fb1240fbe00 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb12259c700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1140041d0 tx=0x7fb11400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb114007cc0 con 0x7fb1240fbe00 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb11400f040 con 0x7fb1240fbe00 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb11400c990 con 0x7fb1240fbe00 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12410eed0 con 0x7fb1240fbe00 2024-03-14T13:01:19.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.156+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12410f410 con 0x7fb1240fbe00 2024-03-14T13:01:19.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.157+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12405f5f0 con 0x7fb1240fbe00 2024-03-14T13:01:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.159+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7fb11400caf0 con 0x7fb1240fbe00 2024-03-14T13:01:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.159+0000 7fb112ffd700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 0x7fb10c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.160+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fb114089210 con 0x7fb1240fbe00 2024-03-14T13:01:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.160+0000 7fb121d9b700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 0x7fb10c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.161+0000 7fb121d9b700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 0x7fb10c06d140 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1180137a0 tx=0x7fb118009e30 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:19.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.163+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fb114057360 con 0x7fb1240fbe00 2024-03-14T13:01:19.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.543+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb12410f800 con 0x7fb10c06ac90 2024-03-14T13:01:19.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.545+0000 7fb112ffd700 1 -- 172.21.15.111:0/994309134 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fb12410f800 con 0x7fb10c06ac90 2024-03-14T13:01:19.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 msgr2=0x7fb10c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 0x7fb10c06d140 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb1180137a0 tx=0x7fb118009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 msgr2=0x7fb12410d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1140041d0 tx=0x7fb11400b040 comp rx=0 tx=0).stop 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 shutdown_connections 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7fb10c06ac90 0x7fb10c06d140 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1240fbe00 0x7fb12410d8c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 --2- 172.21.15.111:0/994309134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1240fcf70 0x7fb12410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 >> 172.21.15.111:0/994309134 conn(0x7fb1240f74e0 msgr2=0x7fb1240f8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 shutdown_connections 2024-03-14T13:01:19.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.547+0000 7fb1289a2700 1 -- 172.21.15.111:0/994309134 wait complete. 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 -- 172.21.15.111:0/345605391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fdd20 msgr2=0x7f7604104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 --2- 172.21.15.111:0/345605391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fdd20 0x7f7604104670 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f75f4004940 tx=0x7f75f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 -- 172.21.15.111:0/345605391 shutdown_connections 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 --2- 172.21.15.111:0/345605391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fdd20 0x7f7604104670 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 --2- 172.21.15.111:0/345605391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fd3d0 0x7f76040fd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 -- 172.21.15.111:0/345605391 >> 172.21.15.111:0/345605391 conn(0x7f76040f6ba0 msgr2=0x7f76040f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 -- 172.21.15.111:0/345605391 shutdown_connections 2024-03-14T13:01:19.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.701+0000 7f760c68d700 1 -- 172.21.15.111:0/345605391 wait complete. 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 Processor -- start 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 -- start start 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fd3d0 0x7f760410fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041105d0 con 0x7f76040fd3d0 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760c68d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7604110e20 con 0x7f76040fdd20 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f7609c28700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f7609c28700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:49110/0 (socket says 172.21.15.111:49110) 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f7609c28700 1 -- 172.21.15.111:0/2760082363 learned_addr learned my addr 172.21.15.111:0/2760082363 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:19.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.702+0000 7f760a429700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fd3d0 0x7f760410fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f7609c28700 1 -- 172.21.15.111:0/2760082363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fd3d0 msgr2=0x7f760410fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f7609c28700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fd3d0 0x7f760410fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f7609c28700 1 -- 172.21.15.111:0/2760082363 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4005040 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f7609c28700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75f4007d40 tx=0x7f75f4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75f400e030 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75f401c400 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7604111020 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76041115e0 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.703+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75f400b650 con 0x7f76040fdd20 2024-03-14T13:01:19.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.704+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e80052e0 con 0x7f76040fdd20 2024-03-14T13:01:19.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.706+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f75f4013a50 con 0x7f76040fdd20 2024-03-14T13:01:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.706+0000 7f75faffd700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 0x7f75f006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.706+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f75f408af80 con 0x7f76040fdd20 2024-03-14T13:01:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.706+0000 7f760a429700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 0x7f75f006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:19.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.707+0000 7f760a429700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 0x7f75f006d140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7600009b80 tx=0x7f760000b040 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:19.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:19.710+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f75f40592b0 con 0x7f76040fdd20 2024-03-14T13:01:20.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.001+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f75e8000bf0 con 0x7f75f006ac90 2024-03-14T13:01:20.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.021+0000 7f75faffd700 1 -- 172.21.15.111:0/2760082363 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f75e8000bf0 con 0x7f75f006ac90 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (28s) 1s ago 9m 14.5M - 0.25.0 c8568f914cd2 70fc8e769dee 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (9m) 1s ago 9m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (7m) 3s ago 7m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (23s) 1s ago 8m 80.1M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (2m) 1s ago 2m 37.9M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (3m) 1s ago 3m 67.9M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (2m) 3s ago 2m 23.0M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (2m) 3s ago 2m 20.4M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (8s) 1s ago 10m 48.6M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:01:20.022 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (65s) 3s ago 7m 494M - 18.2.2-825-g2f465319 a9e65ad305ae 0487350e99e4 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (10m) 1s ago 10m 51.7M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (7m) 3s ago 7m 39.0M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (34s) 1s ago 9m 8488k - 1.5.0 0da6a335fe13 c8b729c46aaa 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (14s) 3s ago 7m 8150k - 1.5.0 0da6a335fe13 37967b629a62 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 1s ago 6m 68.3M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (6m) 1s ago 6m 73.5M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 1s ago 5m 62.1M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (5m) 3s ago 5m 68.8M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (5m) 3s ago 5m 65.6M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (4m) 3s ago 4m 64.9M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:01:20.023 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (16s) 1s ago 8m 37.9M - 2.43.0 a07b618ecd1d d3a9067a4910 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 msgr2=0x7f75f006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 0x7f75f006d140 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7600009b80 tx=0x7f760000b040 comp rx=0 tx=0).stop 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 msgr2=0x7f760410ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75f4007d40 tx=0x7f75f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 shutdown_connections 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f75f006ac90 0x7f75f006d140 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76040fd3d0 0x7f760410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 --2- 172.21.15.111:0/2760082363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76040fdd20 0x7f760410ffd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.024+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 >> 172.21.15.111:0/2760082363 conn(0x7f76040f6ba0 msgr2=0x7f76040f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.025+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 shutdown_connections 2024-03-14T13:01:20.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.025+0000 7f760c68d700 1 -- 172.21.15.111:0/2760082363 wait complete. 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: pgmap v31: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 476 op/s 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:19 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.185+0000 7f4e050be700 1 -- 172.21.15.111:0/1120192585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e00104fc0 msgr2=0x7f4e001073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 --2- 172.21.15.111:0/1120192585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e00104fc0 0x7f4e001073a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4df0004940 tx=0x7f4df0009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 -- 172.21.15.111:0/1120192585 shutdown_connections 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 --2- 172.21.15.111:0/1120192585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e001078e0 0x7f4e00109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 --2- 172.21.15.111:0/1120192585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e00104fc0 0x7f4e001073a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 -- 172.21.15.111:0/1120192585 >> 172.21.15.111:0/1120192585 conn(0x7f4e00069de0 msgr2=0x7f4e0006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 -- 172.21.15.111:0/1120192585 shutdown_connections 2024-03-14T13:01:20.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.186+0000 7f4e050be700 1 -- 172.21.15.111:0/1120192585 wait complete. 2024-03-14T13:01:20.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 Processor -- start 2024-03-14T13:01:20.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 -- start start 2024-03-14T13:01:20.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 0x7f4e0011c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 0x7f4e0011cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0011d240 con 0x7f4e001078e0 2024-03-14T13:01:20.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.187+0000 7f4e050be700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00121b80 con 0x7f4e00104fc0 2024-03-14T13:01:20.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfed9d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 0x7f4e0011c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfe59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 0x7f4e0011cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfed9d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 0x7f4e0011c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:49120/0 (socket says 172.21.15.111:49120) 2024-03-14T13:01:20.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfed9d700 1 -- 172.21.15.111:0/2859128673 learned_addr learned my addr 172.21.15.111:0/2859128673 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfe59c700 1 -- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 msgr2=0x7f4e0011c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfe59c700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 0x7f4e0011c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.188+0000 7f4dfe59c700 1 -- 172.21.15.111:0/2859128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0005040 con 0x7f4e001078e0 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4dfe59c700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 0x7f4e0011cbf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e0006e1a0 tx=0x7f4df400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df4004e90 con 0x7f4e001078e0 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4df400f040 con 0x7f4e001078e0 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e00121cc0 con 0x7f4e001078e0 2024-03-14T13:01:20.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df400c920 con 0x7f4e001078e0 2024-03-14T13:01:20.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.189+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e00122280 con 0x7f4e001078e0 2024-03-14T13:01:20.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.191+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0004c710 con 0x7f4e001078e0 2024-03-14T13:01:20.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.192+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f4df4007cc0 con 0x7f4e001078e0 2024-03-14T13:01:20.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.193+0000 7f4de77fe700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 0x7f4de806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.193+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f4df4088f50 con 0x7f4e001078e0 2024-03-14T13:01:20.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.193+0000 7f4dfed9d700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 0x7f4de806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.200 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.194+0000 7f4dfed9d700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 0x7f4de806d150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4df0004560 tx=0x7f4df0009e30 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:20.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.202+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f4df404f4d0 con 0x7f4e001078e0 2024-03-14T13:01:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='client.24399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: pgmap v31: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 476 op/s 2024-03-14T13:01:20.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:19 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:20.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.670+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4e00121e50 con 0x7f4e001078e0 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.670+0000 7f4de77fe700 1 -- 172.21.15.111:0/2859128673 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+848 (secure 0 0 0) 0x7f4e00121e50 con 0x7f4e001078e0 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:20.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 1, 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 13, 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:01:20.672 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 msgr2=0x7f4de806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 0x7f4de806d150 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4df0004560 tx=0x7f4df0009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 msgr2=0x7f4e0011cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 0x7f4e0011cbf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e0006e1a0 tx=0x7f4df400b040 comp rx=0 tx=0).stop 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 shutdown_connections 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f4de806aca0 0x7f4de806d150 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e00104fc0 0x7f4e0011c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 --2- 172.21.15.111:0/2859128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e001078e0 0x7f4e0011cbf0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4e0006e1a0 tx=0x7f4df400b040 comp rx=0 tx=0).stop 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 >> 172.21.15.111:0/2859128673 conn(0x7f4e00069de0 msgr2=0x7f4e0006eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 shutdown_connections 2024-03-14T13:01:20.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.674+0000 7f4e050be700 1 -- 172.21.15.111:0/2859128673 wait complete. 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 -- 172.21.15.111:0/940835353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 msgr2=0x7f3bb806e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 --2- 172.21.15.111:0/940835353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb806e460 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8004940 tx=0x7f3ba8009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 -- 172.21.15.111:0/940835353 shutdown_connections 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 --2- 172.21.15.111:0/940835353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb806ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 --2- 172.21.15.111:0/940835353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb806e460 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 -- 172.21.15.111:0/940835353 >> 172.21.15.111:0/940835353 conn(0x7f3bb8069de0 msgr2=0x7f3bb806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 -- 172.21.15.111:0/940835353 shutdown_connections 2024-03-14T13:01:20.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.825+0000 7f3bbd532700 1 -- 172.21.15.111:0/940835353 wait complete. 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.826+0000 7f3bbd532700 1 Processor -- start 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.826+0000 7f3bbd532700 1 -- start start 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bbd532700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bbd532700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb81050e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bbd532700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb8107030 con 0x7f3bb806fdc0 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bbd532700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb8105620 con 0x7f3bb806e9a0 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb6ffd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb6ffd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:50292/0 (socket says 172.21.15.111:50292) 2024-03-14T13:01:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb6ffd700 1 -- 172.21.15.111:0/2701569246 learned_addr learned my addr 172.21.15.111:0/2701569246 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb67fc700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb81050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb67fc700 1 -- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 msgr2=0x7f3bb8106a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb67fc700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb67fc700 1 -- 172.21.15.111:0/2701569246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8005040 con 0x7f3bb806fdc0 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb6ffd700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.827+0000 7f3bb67fc700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb81050e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009fd0 tx=0x7f3bac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.828+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bac004e90 con 0x7f3bb806fdc0 2024-03-14T13:01:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.828+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb8105870 con 0x7f3bb806fdc0 2024-03-14T13:01:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.828+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb8105e30 con 0x7f3bb806fdc0 2024-03-14T13:01:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.828+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bac00f040 con 0x7f3bb806fdc0 2024-03-14T13:01:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.828+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3bac00c9b0 con 0x7f3bb806fdc0 2024-03-14T13:01:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.829+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba40052e0 con 0x7f3bb806fdc0 2024-03-14T13:01:20.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.830+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 25) v1 ==== 88607+0+0 (secure 0 0 0) 0x7f3bac007cc0 con 0x7f3bb806fdc0 2024-03-14T13:01:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.830+0000 7f3b9f7fe700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 0x7f3ba006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.830+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f3bac088e80 con 0x7f3bb806fdc0 2024-03-14T13:01:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.831+0000 7f3bb6ffd700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 0x7f3ba006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.831+0000 7f3bb6ffd700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 0x7f3ba006d150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ba80135e0 tx=0x7f3ba8009e30 comp rx=0 tx=0).ready entity=mgr.24341 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:20.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:20.837+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f3bac04f4d0 con 0x7f3bb806fdc0 2024-03-14T13:01:21.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.135+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 --> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ba4000bf0 con 0x7f3ba006aca0 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:01:21.140 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:01:21.141 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:01:21.141 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mgr daemons", 2024-03-14T13:01:21.141 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:01:21.141 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.137+0000 7f3b9f7fe700 1 -- 172.21.15.111:0/2701569246 <== mgr.24341 v2:172.21.15.137:6828/1920623081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f3ba4000bf0 con 0x7f3ba006aca0 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 msgr2=0x7f3ba006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 0x7f3ba006d150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3ba80135e0 tx=0x7f3ba8009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 msgr2=0x7f3bb81050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb81050e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009fd0 tx=0x7f3bac00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 shutdown_connections 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:6828/1920623081,v1:172.21.15.137:6829/1920623081] conn(0x7f3ba006aca0 0x7f3ba006d150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bb806e9a0 0x7f3bb8106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:21.142 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 --2- 172.21.15.111:0/2701569246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb806fdc0 0x7f3bb81050e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:21.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 >> 172.21.15.111:0/2701569246 conn(0x7f3bb8069de0 msgr2=0x7f3bb806b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:21.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 shutdown_connections 2024-03-14T13:01:21.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:21.141+0000 7f3bbd532700 1 -- 172.21.15.111:0/2701569246 wait complete. 2024-03-14T13:01:21.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:21 smithi111 ceph-mon[39688]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:21.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:21 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2859128673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:01:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:21 smithi137 ceph-mon[50336]: from='client.24407 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:21.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:21 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2859128673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: pgmap v32: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 328 op/s 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: Standby manager daemon smithi111.atysez restarted 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: Standby manager daemon smithi111.atysez started 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/crt"}]: dispatch 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/key"}]: dispatch 2024-03-14T13:01:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:22 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: pgmap v32: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 328 op/s 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: Standby manager daemon smithi111.atysez restarted 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: Standby manager daemon smithi111.atysez started 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/crt"}]: dispatch 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.atysez/key"}]: dispatch 2024-03-14T13:01:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:22 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:01:23.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:23 smithi111 ceph-mon[39688]: mgrmap e26: smithi137.xbyzbp(active, since 58s), standbys: smithi111.atysez 2024-03-14T13:01:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:23 smithi137 ceph-mon[50336]: mgrmap e26: smithi137.xbyzbp(active, since 58s), standbys: smithi111.atysez 2024-03-14T13:01:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:24 smithi111 ceph-mon[39688]: pgmap v33: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 328 op/s 2024-03-14T13:01:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:24 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:24 smithi137 ceph-mon[50336]: pgmap v33: 65 pgs: 65 active+clean; 69 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.9 MiB/s wr, 328 op/s 2024-03-14T13:01:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:24 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:26.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:26 smithi111 ceph-mon[39688]: pgmap v34: 65 pgs: 65 active+clean; 86 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 533 op/s 2024-03-14T13:01:26.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:26 smithi137 ceph-mon[50336]: pgmap v34: 65 pgs: 65 active+clean; 86 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.4 MiB/s wr, 533 op/s 2024-03-14T13:01:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:28 smithi111 ceph-mon[39688]: pgmap v35: 65 pgs: 65 active+clean; 86 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 356 op/s 2024-03-14T13:01:29.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:28 smithi137 ceph-mon[50336]: pgmap v35: 65 pgs: 65 active+clean; 86 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 3.4 MiB/s wr, 356 op/s 2024-03-14T13:01:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:30 smithi111 ceph-mon[39688]: pgmap v36: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-03-14T13:01:31.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:30 smithi137 ceph-mon[50336]: pgmap v36: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-03-14T13:01:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:32 smithi111 ceph-mon[39688]: pgmap v37: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 394 op/s 2024-03-14T13:01:33.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:32 smithi137 ceph-mon[50336]: pgmap v37: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 394 op/s 2024-03-14T13:01:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:34 smithi111 ceph-mon[39688]: pgmap v38: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 394 op/s 2024-03-14T13:01:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:34 smithi137 ceph-mon[50336]: pgmap v38: 65 pgs: 65 active+clean; 108 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 3.5 MiB/s wr, 394 op/s 2024-03-14T13:01:36.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:36 smithi111 ceph-mon[39688]: pgmap v39: 65 pgs: 65 active+clean; 121 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 564 op/s 2024-03-14T13:01:36.648 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:36 smithi137 ceph-mon[50336]: pgmap v39: 65 pgs: 65 active+clean; 121 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 564 op/s 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: osdmap e49: 6 total, 6 up, 6 in 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd='[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]': finished 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: mgrmap e27: smithi111.atysez(active, starting, since 0.0064487s) 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:01:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: Manager daemon smithi111.atysez is now available 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T13:01:37.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:37 smithi137 ceph-mon[50336]: [14/Mar/2024:13:01:37] ENGINE Bus STARTING 2024-03-14T13:01:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 172.21.15.137:0/2248525542' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: osdmap e49: 6 total, 6 up, 6 in 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.24341 ' entity='mgr.smithi137.xbyzbp' cmd='[{"prefix": "mgr fail", "who": "smithi137.xbyzbp"}]': finished 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: mgrmap e27: smithi111.atysez(active, starting, since 0.0064487s) 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:01:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: Manager daemon smithi111.atysez is now available 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:01:37.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:37.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:01:37.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T13:01:37.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:37 smithi111 ceph-mon[39688]: [14/Mar/2024:13:01:37] ENGINE Bus STARTING 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: [14/Mar/2024:13:01:37] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: [14/Mar/2024:13:01:37] ENGINE Serving on http://172.21.15.111:8765 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: [14/Mar/2024:13:01:37] ENGINE Bus STARTED 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: mgrmap e28: smithi111.atysez(active, since 1.0092s) 2024-03-14T13:01:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:38 smithi137 ceph-mon[50336]: pgmap v3: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: [14/Mar/2024:13:01:37] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T13:01:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: [14/Mar/2024:13:01:37] ENGINE Serving on http://172.21.15.111:8765 2024-03-14T13:01:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: [14/Mar/2024:13:01:37] ENGINE Bus STARTED 2024-03-14T13:01:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:38.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:38.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: mgrmap e28: smithi111.atysez(active, since 1.0092s) 2024-03-14T13:01:38.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:38 smithi111 ceph-mon[39688]: pgmap v3: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:39.986 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:39 smithi111 ceph-mon[39688]: mgrmap e29: smithi111.atysez(active, since 2s) 2024-03-14T13:01:39.987 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:39 smithi111 ceph-mon[39688]: pgmap v4: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:39 smithi137 ceph-mon[50336]: mgrmap e29: smithi111.atysez(active, since 2s) 2024-03-14T13:01:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:39 smithi137 ceph-mon[50336]: pgmap v4: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:41.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:41 smithi137 ceph-mon[50336]: pgmap v5: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:41.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:41 smithi137 ceph-mon[50336]: mgrmap e30: smithi111.atysez(active, since 4s) 2024-03-14T13:01:42.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:41 smithi111 ceph-mon[39688]: pgmap v5: 65 pgs: 65 active+clean; 134 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:01:42.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:41 smithi111 ceph-mon[39688]: mgrmap e30: smithi111.atysez(active, since 4s) 2024-03-14T13:01:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: pgmap v6: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 412 KiB/s wr, 78 op/s 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: pgmap v6: 65 pgs: 65 active+clean; 136 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 412 KiB/s wr, 78 op/s 2024-03-14T13:01:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:44.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:45.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:45.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:45.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:45.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:46.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: pgmap v7: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-14T13:01:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:01:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:01:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:01:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:01:46.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:01:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: pgmap v7: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-14T13:01:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:01:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:01:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:01:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:01:46.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/232229403' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:01:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:47 smithi111 ceph-mon[39688]: mgrmap e31: smithi111.atysez(active, since 9s), standbys: smithi137.xbyzbp 2024-03-14T13:01:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:47 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:47 smithi111 ceph-mon[39688]: pgmap v8: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 305 op/s 2024-03-14T13:01:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:47 smithi137 ceph-mon[50336]: mgrmap e31: smithi111.atysez(active, since 9s), standbys: smithi137.xbyzbp 2024-03-14T13:01:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:47 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:01:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:47 smithi137 ceph-mon[50336]: pgmap v8: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 305 op/s 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:01:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:01:49.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: pgmap v9: 65 pgs: 65 active+clean; 183 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-03-14T13:01:49.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:01:49.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:01:49.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:01:49.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:49 smithi111 ceph-mon[39688]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:01:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: pgmap v9: 65 pgs: 65 active+clean; 183 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:01:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:49 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:01:50.395 INFO:tasks.workunit.client.0.smithi111.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-14T13:01:50.401 INFO:tasks.workunit.client.0.smithi111.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-14T13:01:50.401 INFO:tasks.workunit.client.0.smithi111.stderr:+ make 2024-03-14T13:01:50.744 INFO:tasks.workunit.client.0.smithi111.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T13:01:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:50 smithi111 ceph-mon[39688]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T13:01:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:01:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:01:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:51.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:01:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: Reconfiguring prometheus.smithi111 (dependencies changed)... 2024-03-14T13:01:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:50 smithi137 ceph-mon[50336]: Reconfiguring daemon prometheus.smithi111 on smithi111 2024-03-14T13:01:51.226 INFO:tasks.workunit.client.0.smithi111.stderr:++ readlink -f fsstress 2024-03-14T13:01:51.228 INFO:tasks.workunit.client.0.smithi111.stderr:+ BIN=/home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-14T13:01:51.228 INFO:tasks.workunit.client.0.smithi111.stderr:+ popd 2024-03-14T13:01:51.229 INFO:tasks.workunit.client.0.smithi111.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-14T13:01:51.229 INFO:tasks.workunit.client.0.smithi111.stderr:+ popd 2024-03-14T13:01:51.230 INFO:tasks.workunit.client.0.smithi111.stdout:~/cephtest/mnt.0/client.0/tmp 2024-03-14T13:01:51.230 INFO:tasks.workunit.client.0.smithi111.stderr:++ mktemp -d -p . 2024-03-14T13:01:51.233 INFO:tasks.workunit.client.0.smithi111.stderr:+ T=./tmp.IDB5UJWc6S 2024-03-14T13:01:51.233 INFO:tasks.workunit.client.0.smithi111.stderr:+ /home/ubuntu/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.IDB5UJWc6S -l 1 -n 1000 -p 10 -v 2024-03-14T13:01:51.236 INFO:tasks.workunit.client.0.smithi111.stdout:seed = 1710315505 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/0: dwrite - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/1: dwrite - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/2: link - no file 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/3: fdatasync - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/4: write - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/5: fdatasync - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/6: chown . 78 1 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/7: write - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/8: read - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/9: write - no filename 2024-03-14T13:01:51.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/10: write - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/11: dread - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/12: fdatasync - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/13: fsync - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/14: dwrite - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/15: rename - no filename 2024-03-14T13:01:51.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/16: creat f0 x:0 0 0 2024-03-14T13:01:51.243 INFO:tasks.workunit.client.0.smithi111.stdout:0/17: creat f1 x:0 0 0 2024-03-14T13:01:51.244 INFO:tasks.workunit.client.0.smithi111.stdout:2/0: mknod c0 0 2024-03-14T13:01:51.245 INFO:tasks.workunit.client.0.smithi111.stdout:1/0: dwrite - no filename 2024-03-14T13:01:51.245 INFO:tasks.workunit.client.0.smithi111.stdout:1/1: truncate - no filename 2024-03-14T13:01:51.245 INFO:tasks.workunit.client.0.smithi111.stdout:1/2: truncate - no filename 2024-03-14T13:01:51.245 INFO:tasks.workunit.client.0.smithi111.stdout:1/3: write - no filename 2024-03-14T13:01:51.246 INFO:tasks.workunit.client.0.smithi111.stdout:1/4: stat - no entries 2024-03-14T13:01:51.246 INFO:tasks.workunit.client.0.smithi111.stdout:3/0: mkdir d0 0 2024-03-14T13:01:51.248 INFO:tasks.workunit.client.0.smithi111.stdout:0/18: link f1 f2 0 2024-03-14T13:01:51.249 INFO:tasks.workunit.client.0.smithi111.stdout:0/19: truncate f2 994588 0 2024-03-14T13:01:51.249 INFO:tasks.workunit.client.0.smithi111.stdout:0/20: chown f0 13 1 2024-03-14T13:01:51.250 INFO:tasks.workunit.client.0.smithi111.stdout:0/21: truncate f0 305553 0 2024-03-14T13:01:51.250 INFO:tasks.workunit.client.0.smithi111.stdout:0/22: readlink - no filename 2024-03-14T13:01:51.251 INFO:tasks.workunit.client.0.smithi111.stdout:0/23: write f0 [1239259,80933] 0 2024-03-14T13:01:51.251 INFO:tasks.workunit.client.0.smithi111.stdout:6/0: readlink - no filename 2024-03-14T13:01:51.251 INFO:tasks.workunit.client.0.smithi111.stdout:6/1: write - no filename 2024-03-14T13:01:51.252 INFO:tasks.workunit.client.0.smithi111.stdout:3/1: creat d0/f1 x:0 0 0 2024-03-14T13:01:51.253 INFO:tasks.workunit.client.0.smithi111.stdout:5/0: mknod c0 0 2024-03-14T13:01:51.253 INFO:tasks.workunit.client.0.smithi111.stdout:5/1: dwrite - no filename 2024-03-14T13:01:51.255 INFO:tasks.workunit.client.0.smithi111.stdout:7/0: dread - no filename 2024-03-14T13:01:51.255 INFO:tasks.workunit.client.0.smithi111.stdout:7/1: dwrite - no filename 2024-03-14T13:01:51.255 INFO:tasks.workunit.client.0.smithi111.stdout:7/2: readlink - no filename 2024-03-14T13:01:51.255 INFO:tasks.workunit.client.0.smithi111.stdout:7/3: write - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/4: truncate - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/5: rename - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:0/24: dread f1 [0,4194304] 0 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/6: chown . 397328729 1 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/7: write - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/8: dread - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/9: fsync - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/10: dwrite - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/11: readlink - no filename 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/12: stat - no entries 2024-03-14T13:01:51.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/13: dread - no filename 2024-03-14T13:01:51.257 INFO:tasks.workunit.client.0.smithi111.stdout:3/2: truncate d0/f1 693559 0 2024-03-14T13:01:51.257 INFO:tasks.workunit.client.0.smithi111.stdout:6/2: mkdir d0 0 2024-03-14T13:01:51.257 INFO:tasks.workunit.client.0.smithi111.stdout:0/25: truncate f0 1324654 0 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/2: unlink c0 0 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/3: truncate - no filename 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/4: read - no filename 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/5: rmdir - no directory 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/6: write - no filename 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/7: dwrite - no filename 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/8: stat - no entries 2024-03-14T13:01:51.258 INFO:tasks.workunit.client.0.smithi111.stdout:5/9: fsync - no filename 2024-03-14T13:01:51.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/0: truncate - no filename 2024-03-14T13:01:51.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/1: rmdir - no directory 2024-03-14T13:01:51.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/2: write - no filename 2024-03-14T13:01:51.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/3: dwrite - no filename 2024-03-14T13:01:51.262 INFO:tasks.workunit.client.0.smithi111.stdout:0/26: creat f3 x:0 0 0 2024-03-14T13:01:51.262 INFO:tasks.workunit.client.0.smithi111.stdout:0/27: rmdir - no directory 2024-03-14T13:01:51.262 INFO:tasks.workunit.client.0.smithi111.stdout:5/10: creat f1 x:0 0 0 2024-03-14T13:01:51.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/0: truncate - no filename 2024-03-14T13:01:51.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/1: dread - no filename 2024-03-14T13:01:51.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/2: write - no filename 2024-03-14T13:01:51.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/3: write - no filename 2024-03-14T13:01:51.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/4: chown . 9837109 1 2024-03-14T13:01:51.266 INFO:tasks.workunit.client.0.smithi111.stdout:9/5: chown . 30 1 2024-03-14T13:01:51.266 INFO:tasks.workunit.client.0.smithi111.stdout:9/6: readlink - no filename 2024-03-14T13:01:51.266 INFO:tasks.workunit.client.0.smithi111.stdout:9/7: link - no file 2024-03-14T13:01:51.266 INFO:tasks.workunit.client.0.smithi111.stdout:9/8: link - no file 2024-03-14T13:01:51.266 INFO:tasks.workunit.client.0.smithi111.stdout:7/14: mkdir d0 0 2024-03-14T13:01:51.267 INFO:tasks.workunit.client.0.smithi111.stdout:6/3: creat d0/f1 x:0 0 0 2024-03-14T13:01:51.268 INFO:tasks.workunit.client.0.smithi111.stdout:0/28: creat f4 x:0 0 0 2024-03-14T13:01:51.273 INFO:tasks.workunit.client.0.smithi111.stdout:5/11: dwrite f1 [0,4194304] 0 2024-03-14T13:01:51.274 INFO:tasks.workunit.client.0.smithi111.stdout:8/4: creat f0 x:0 0 0 2024-03-14T13:01:51.274 INFO:tasks.workunit.client.0.smithi111.stdout:8/5: rmdir - no directory 2024-03-14T13:01:51.275 INFO:tasks.workunit.client.0.smithi111.stdout:5/12: read f1 [595345,127150] 0 2024-03-14T13:01:51.276 INFO:tasks.workunit.client.0.smithi111.stdout:6/4: fdatasync d0/f1 0 2024-03-14T13:01:51.277 INFO:tasks.workunit.client.0.smithi111.stdout:9/9: creat f0 x:0 0 0 2024-03-14T13:01:51.277 INFO:tasks.workunit.client.0.smithi111.stdout:7/15: mkdir d0/d1 0 2024-03-14T13:01:51.277 INFO:tasks.workunit.client.0.smithi111.stdout:7/16: fsync - no filename 2024-03-14T13:01:51.278 INFO:tasks.workunit.client.0.smithi111.stdout:4/0: write - no filename 2024-03-14T13:01:51.278 INFO:tasks.workunit.client.0.smithi111.stdout:6/5: chown d0/f1 12631904 1 2024-03-14T13:01:51.280 INFO:tasks.workunit.client.0.smithi111.stdout:6/6: dread - d0/f1 zero size 2024-03-14T13:01:51.281 INFO:tasks.workunit.client.0.smithi111.stdout:8/6: mknod c1 0 2024-03-14T13:01:51.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.281+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/3409604754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 msgr2=0x7fcfe0103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.281+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/3409604754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe0103830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4004940 tx=0x7fcfd4009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:51.282 INFO:tasks.workunit.client.0.smithi111.stdout:0/29: rename f4 to f5 0 2024-03-14T13:01:51.282 INFO:tasks.workunit.client.0.smithi111.stdout:5/13: mkdir d2 0 2024-03-14T13:01:51.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.282+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/3409604754 shutdown_connections 2024-03-14T13:01:51.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.282+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/3409604754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe0104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.282+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/3409604754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe0103830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.282+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/3409604754 >> 172.21.15.111:0/3409604754 conn(0x7fcfe00725a0 msgr2=0x7fcfe00749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:51.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.283+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/3409604754 shutdown_connections 2024-03-14T13:01:51.284 INFO:tasks.workunit.client.0.smithi111.stdout:9/10: mknod c1 0 2024-03-14T13:01:51.284 INFO:tasks.workunit.client.0.smithi111.stdout:9/11: dread - f0 zero size 2024-03-14T13:01:51.284 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.283+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/3409604754 wait complete. 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.283+0000 7fcfe6ff2700 1 Processor -- start 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.283+0000 7fcfe6ff2700 1 -- start start 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfe6ff2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe006e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfe6ff2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfe6ff2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe006fe40 con 0x7fcfe0103420 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfe6ff2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe006ffb0 con 0x7fcfe01046d0 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:52274/0 (socket says 172.21.15.111:52274) 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfdffff700 1 -- 172.21.15.111:0/2561252987 learned_addr learned my addr 172.21.15.111:0/2561252987 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.284+0000 7fcfe4d8e700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe006e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.285+0000 7fcfdffff700 1 -- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 msgr2=0x7fcfe006e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.285+0000 7fcfdffff700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe006e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.285+0000 7fcfdffff700 1 -- 172.21.15.111:0/2561252987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4005040 con 0x7fcfe01046d0 2024-03-14T13:01:51.286 INFO:tasks.workunit.client.0.smithi111.stdout:8/7: dread - f0 zero size 2024-03-14T13:01:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.286+0000 7fcfdffff700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00eb40 tx=0x7fcfcc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.286+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc007cc0 con 0x7fcfe01046d0 2024-03-14T13:01:51.287 INFO:tasks.workunit.client.0.smithi111.stdout:7/17: creat d0/d1/f2 x:0 0 0 2024-03-14T13:01:51.287 INFO:tasks.workunit.client.0.smithi111.stdout:8/8: chown f0 16347367 1 2024-03-14T13:01:51.287 INFO:tasks.workunit.client.0.smithi111.stdout:4/1: symlink l0 0 2024-03-14T13:01:51.287 INFO:tasks.workunit.client.0.smithi111.stdout:4/2: dread - no filename 2024-03-14T13:01:51.288 INFO:tasks.workunit.client.0.smithi111.stdout:4/3: chown l0 2412 1 2024-03-14T13:01:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.286+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfcc00f040 con 0x7fcfe01046d0 2024-03-14T13:01:51.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.286+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe006ef10 con 0x7fcfe01046d0 2024-03-14T13:01:51.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.286+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0076a50 con 0x7fcfe01046d0 2024-03-14T13:01:51.289 INFO:tasks.workunit.client.0.smithi111.stdout:7/18: fdatasync d0/d1/f2 0 2024-03-14T13:01:51.289 INFO:tasks.workunit.client.0.smithi111.stdout:6/7: truncate d0/f1 529399 0 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.287+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfcc009570 con 0x7fcfe01046d0 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.289+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7fcfcc0096d0 con 0x7fcfe01046d0 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.290+0000 7fcfdd7fa700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 0x7fcfd006eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.290+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fcfcc08b320 con 0x7fcfe01046d0 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.290+0000 7fcfe4d8e700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 0x7fcfd006eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.291+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc40052e0 con 0x7fcfe01046d0 2024-03-14T13:01:51.291 INFO:tasks.workunit.client.0.smithi111.stdout:8/9: chown c1 159946 1 2024-03-14T13:01:51.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.291+0000 7fcfe4d8e700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 0x7fcfd006eaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40137b0 tx=0x7fcfd4009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:51.292 INFO:tasks.workunit.client.0.smithi111.stdout:0/30: dread f1 [0,4194304] 0 2024-03-14T13:01:51.292 INFO:tasks.workunit.client.0.smithi111.stdout:6/8: truncate d0/f1 1379545 0 2024-03-14T13:01:51.292 INFO:tasks.workunit.client.0.smithi111.stdout:8/10: fdatasync f0 0 2024-03-14T13:01:51.293 INFO:tasks.workunit.client.0.smithi111.stdout:8/11: dread - f0 zero size 2024-03-14T13:01:51.293 INFO:tasks.workunit.client.0.smithi111.stdout:7/19: truncate d0/d1/f2 38750 0 2024-03-14T13:01:51.293 INFO:tasks.workunit.client.0.smithi111.stdout:5/14: mkdir d2/d3 0 2024-03-14T13:01:51.295 INFO:tasks.workunit.client.0.smithi111.stdout:4/4: rename l0 to l1 0 2024-03-14T13:01:51.296 INFO:tasks.workunit.client.0.smithi111.stdout:7/20: dread d0/d1/f2 [0,4194304] 0 2024-03-14T13:01:51.296 INFO:tasks.workunit.client.0.smithi111.stdout:9/12: link f0 f2 0 2024-03-14T13:01:51.297 INFO:tasks.workunit.client.0.smithi111.stdout:7/21: fdatasync d0/d1/f2 0 2024-03-14T13:01:51.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.299+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fcfcc059340 con 0x7fcfe01046d0 2024-03-14T13:01:51.300 INFO:tasks.workunit.client.0.smithi111.stdout:7/22: write d0/d1/f2 [1050876,119809] 0 2024-03-14T13:01:51.302 INFO:tasks.workunit.client.0.smithi111.stdout:6/9: dwrite d0/f1 [0,4194304] 0 2024-03-14T13:01:51.306 INFO:tasks.workunit.client.0.smithi111.stdout:4/5: creat f2 x:0 0 0 2024-03-14T13:01:51.318 INFO:tasks.workunit.client.0.smithi111.stdout:4/6: mknod c3 0 2024-03-14T13:01:51.322 INFO:tasks.workunit.client.0.smithi111.stdout:4/7: write f2 [594833,105485] 0 2024-03-14T13:01:51.322 INFO:tasks.workunit.client.0.smithi111.stdout:7/23: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:01:51.327 INFO:tasks.workunit.client.0.smithi111.stdout:4/8: write f2 [1350760,79064] 0 2024-03-14T13:01:51.332 INFO:tasks.workunit.client.0.smithi111.stdout:9/13: dwrite f0 [0,4194304] 0 2024-03-14T13:01:51.335 INFO:tasks.workunit.client.0.smithi111.stdout:4/9: rename f2 to f4 0 2024-03-14T13:01:51.338 INFO:tasks.workunit.client.0.smithi111.stdout:7/24: write d0/d1/f2 [3613236,115717] 0 2024-03-14T13:01:51.339 INFO:tasks.workunit.client.0.smithi111.stdout:6/10: dwrite d0/f1 [0,4194304] 0 2024-03-14T13:01:51.339 INFO:tasks.workunit.client.0.smithi111.stdout:7/25: stat d0/d1/f2 0 2024-03-14T13:01:51.339 INFO:tasks.workunit.client.0.smithi111.stdout:7/26: rename d0 to d0/d3 22 2024-03-14T13:01:51.347 INFO:tasks.workunit.client.0.smithi111.stdout:6/11: creat d0/f2 x:0 0 0 2024-03-14T13:01:51.354 INFO:tasks.workunit.client.0.smithi111.stdout:7/27: link d0/d1/f2 d0/d1/f4 0 2024-03-14T13:01:51.355 INFO:tasks.workunit.client.0.smithi111.stdout:6/12: chown d0/f2 99889983 1 2024-03-14T13:01:51.359 INFO:tasks.workunit.client.0.smithi111.stdout:7/28: chown d0/d1/f4 53 1 2024-03-14T13:01:51.361 INFO:tasks.workunit.client.0.smithi111.stdout:4/10: dwrite f4 [0,4194304] 0 2024-03-14T13:01:51.362 INFO:tasks.workunit.client.0.smithi111.stdout:6/13: dread - d0/f2 zero size 2024-03-14T13:01:51.364 INFO:tasks.workunit.client.0.smithi111.stdout:6/14: read d0/f1 [37828,84827] 0 2024-03-14T13:01:51.367 INFO:tasks.workunit.client.0.smithi111.stdout:6/15: write d0/f1 [2586495,26189] 0 2024-03-14T13:01:51.368 INFO:tasks.workunit.client.0.smithi111.stdout:7/29: fdatasync d0/d1/f4 0 2024-03-14T13:01:51.369 INFO:tasks.workunit.client.0.smithi111.stdout:4/11: creat f5 x:0 0 0 2024-03-14T13:01:51.370 INFO:tasks.workunit.client.0.smithi111.stdout:7/30: write d0/d1/f4 [3415162,30997] 0 2024-03-14T13:01:51.375 INFO:tasks.workunit.client.0.smithi111.stdout:7/31: mkdir d0/d1/d5 0 2024-03-14T13:01:51.376 INFO:tasks.workunit.client.0.smithi111.stdout:7/32: chown d0/d1 15163 1 2024-03-14T13:01:51.377 INFO:tasks.workunit.client.0.smithi111.stdout:7/33: truncate d0/d1/f4 5138549 0 2024-03-14T13:01:51.379 INFO:tasks.workunit.client.0.smithi111.stdout:7/34: mkdir d0/d6 0 2024-03-14T13:01:51.381 INFO:tasks.workunit.client.0.smithi111.stdout:7/35: unlink d0/d1/f4 0 2024-03-14T13:01:51.421 INFO:tasks.workunit.client.0.smithi111.stdout:2/1: getdents . 0 2024-03-14T13:01:51.421 INFO:tasks.workunit.client.0.smithi111.stdout:2/2: dread - no filename 2024-03-14T13:01:51.422 INFO:tasks.workunit.client.0.smithi111.stdout:2/3: truncate - no filename 2024-03-14T13:01:51.424 INFO:tasks.workunit.client.0.smithi111.stdout:2/4: creat f1 x:0 0 0 2024-03-14T13:01:51.424 INFO:tasks.workunit.client.0.smithi111.stdout:2/5: truncate f1 411700 0 2024-03-14T13:01:51.425 INFO:tasks.workunit.client.0.smithi111.stdout:2/6: chown c0 0 1 2024-03-14T13:01:51.426 INFO:tasks.workunit.client.0.smithi111.stdout:2/7: symlink l2 0 2024-03-14T13:01:51.432 INFO:tasks.workunit.client.0.smithi111.stdout:5/15: fsync f1 0 2024-03-14T13:01:51.436 INFO:tasks.workunit.client.0.smithi111.stdout:2/8: dwrite f1 [0,4194304] 0 2024-03-14T13:01:51.438 INFO:tasks.workunit.client.0.smithi111.stdout:8/12: getdents . 0 2024-03-14T13:01:51.441 INFO:tasks.workunit.client.0.smithi111.stdout:2/9: creat f3 x:0 0 0 2024-03-14T13:01:51.443 INFO:tasks.workunit.client.0.smithi111.stdout:2/10: write f3 [737799,92718] 0 2024-03-14T13:01:51.443 INFO:tasks.workunit.client.0.smithi111.stdout:8/13: dread - f0 zero size 2024-03-14T13:01:51.447 INFO:tasks.workunit.client.0.smithi111.stdout:9/14: getdents . 0 2024-03-14T13:01:51.447 INFO:tasks.workunit.client.0.smithi111.stdout:8/14: chown f0 1 1 2024-03-14T13:01:51.479 INFO:tasks.workunit.client.0.smithi111.stdout:6/16: write d0/f1 [4108001,116248] 0 2024-03-14T13:01:51.481 INFO:tasks.workunit.client.0.smithi111.stdout:4/12: rename l1 to l6 0 2024-03-14T13:01:51.549 INFO:tasks.workunit.client.0.smithi111.stdout:5/16: dwrite f1 [0,4194304] 0 2024-03-14T13:01:51.551 INFO:tasks.workunit.client.0.smithi111.stdout:5/17: read f1 [3258054,2640] 0 2024-03-14T13:01:51.555 INFO:tasks.workunit.client.0.smithi111.stdout:9/15: dread f2 [0,4194304] 0 2024-03-14T13:01:51.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.587+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfc4000bf0 con 0x7fcfd006c640 2024-03-14T13:01:51.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.588+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2561252987 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fcfc4000bf0 con 0x7fcfd006c640 2024-03-14T13:01:51.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 msgr2=0x7fcfd006eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 0x7fcfd006eaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40137b0 tx=0x7fcfd4009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:51.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 msgr2=0x7fcfe006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00eb40 tx=0x7fcfcc00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 shutdown_connections 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcfd006c640 0x7fcfd006eaf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0103420 0x7fcfe006e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 --2- 172.21.15.111:0/2561252987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01046d0 0x7fcfe006e970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 >> 172.21.15.111:0/2561252987 conn(0x7fcfe00725a0 msgr2=0x7fcfe0073d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 shutdown_connections 2024-03-14T13:01:51.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.591+0000 7fcfe6ff2700 1 -- 172.21.15.111:0/2561252987 wait complete. 2024-03-14T13:01:51.601 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:01:51.638 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:51 smithi111 ceph-mon[39688]: pgmap v10: 65 pgs: 65 active+clean; 183 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 433 op/s 2024-03-14T13:01:51.709 INFO:tasks.workunit.client.0.smithi111.stdout:8/15: rename c1 to c2 0 2024-03-14T13:01:51.712 INFO:tasks.workunit.client.0.smithi111.stdout:7/36: truncate d0/d1/f2 613613 0 2024-03-14T13:01:51.713 INFO:tasks.workunit.client.0.smithi111.stdout:9/16: mknod c3 0 2024-03-14T13:01:51.716 INFO:tasks.workunit.client.0.smithi111.stdout:2/11: getdents . 0 2024-03-14T13:01:51.718 INFO:tasks.workunit.client.0.smithi111.stdout:8/16: mkdir d3 0 2024-03-14T13:01:51.721 INFO:tasks.workunit.client.0.smithi111.stdout:5/18: link f1 d2/d3/f4 0 2024-03-14T13:01:51.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 -- 172.21.15.111:0/764942515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fbce0 msgr2=0x7f6cbc0fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/764942515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc0fc0f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8004940 tx=0x7f6cb8009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 -- 172.21.15.111:0/764942515 shutdown_connections 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/764942515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc0fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/764942515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc0fc0f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.725+0000 7f6cc3d99700 1 -- 172.21.15.111:0/764942515 >> 172.21.15.111:0/764942515 conn(0x7f6cbc0f7480 msgr2=0x7f6cbc0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:51.725 INFO:tasks.workunit.client.0.smithi111.stdout:2/12: rename f3 to f4 0 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.726+0000 7f6cc3d99700 1 -- 172.21.15.111:0/764942515 shutdown_connections 2024-03-14T13:01:51.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.726+0000 7f6cc3d99700 1 -- 172.21.15.111:0/764942515 wait complete. 2024-03-14T13:01:51.726 INFO:tasks.workunit.client.0.smithi111.stdout:4/13: link c3 c7 0 2024-03-14T13:01:51.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.726+0000 7f6cc3d99700 1 Processor -- start 2024-03-14T13:01:51.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.726+0000 7f6cc3d99700 1 -- start start 2024-03-14T13:01:51.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc3d99700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc3d99700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc3d99700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc114a10 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc3d99700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc115260 con 0x7f6cbc0fbce0 2024-03-14T13:01:51.727 INFO:tasks.workunit.client.0.smithi111.stdout:4/14: fdatasync f5 0 2024-03-14T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc1334700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc1334700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43086/0 (socket says 172.21.15.111:43086) 2024-03-14T13:01:51.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.727+0000 7f6cc1334700 1 -- 172.21.15.111:0/420038124 learned_addr learned my addr 172.21.15.111:0/420038124 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc1b35700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc1334700 1 -- 172.21.15.111:0/420038124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fbce0 msgr2=0x7f6cbc113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc1334700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc113ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc1334700 1 -- 172.21.15.111:0/420038124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb8005040 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc1334700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00eb40 tx=0x7f6cac00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac007cc0 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cac00f040 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6cac009570 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.728+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cbc115460 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.729 INFO:tasks.workunit.client.0.smithi111.stdout:4/15: write f4 [3191161,44596] 0 2024-03-14T13:01:51.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.729+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cbc115a20 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.730 INFO:tasks.workunit.client.0.smithi111.stdout:2/13: symlink l5 0 2024-03-14T13:01:51.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.730+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc05f5f0 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.732 INFO:tasks.workunit.client.0.smithi111.stdout:2/14: read f4 [485450,57075] 0 2024-03-14T13:01:51.735 INFO:tasks.workunit.client.0.smithi111.stdout:8/17: getdents d3 0 2024-03-14T13:01:51.736 INFO:tasks.workunit.client.0.smithi111.stdout:8/18: fsync f0 0 2024-03-14T13:01:51.736 INFO:tasks.workunit.client.0.smithi111.stdout:5/19: mknod d2/d3/c5 0 2024-03-14T13:01:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.733+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f6cac0096d0 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.733+0000 7f6cb27fc700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 0x7f6ca806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.733+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f6cac08b640 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.738+0000 7f6cc1b35700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 0x7f6ca806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:51.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.738+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f6cac059630 con 0x7f6cbc0fcf90 2024-03-14T13:01:51.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:51.738+0000 7f6cc1b35700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 0x7f6ca806eae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b0d0 tx=0x7f6cb8009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:51.739 INFO:tasks.workunit.client.0.smithi111.stdout:9/17: dwrite f2 [4194304,4194304] 0 2024-03-14T13:01:51.744 INFO:tasks.workunit.client.0.smithi111.stdout:4/16: rename c3 to c8 0 2024-03-14T13:01:51.746 INFO:tasks.workunit.client.0.smithi111.stdout:8/19: mknod d3/c4 0 2024-03-14T13:01:51.748 INFO:tasks.workunit.client.0.smithi111.stdout:4/17: rename f4 to f9 0 2024-03-14T13:01:51.748 INFO:tasks.workunit.client.0.smithi111.stdout:4/18: fdatasync f5 0 2024-03-14T13:01:51.752 INFO:tasks.workunit.client.0.smithi111.stdout:4/19: creat fa x:0 0 0 2024-03-14T13:01:51.753 INFO:tasks.workunit.client.0.smithi111.stdout:4/20: read - f5 zero size 2024-03-14T13:01:51.756 INFO:tasks.workunit.client.0.smithi111.stdout:4/21: write f5 [4621,18894] 0 2024-03-14T13:01:51.757 INFO:tasks.workunit.client.0.smithi111.stdout:8/20: dwrite f0 [0,4194304] 0 2024-03-14T13:01:51.759 INFO:tasks.workunit.client.0.smithi111.stdout:4/22: creat fb x:0 0 0 2024-03-14T13:01:51.762 INFO:tasks.workunit.client.0.smithi111.stdout:4/23: symlink lc 0 2024-03-14T13:01:51.850 INFO:tasks.workunit.client.0.smithi111.stdout:6/17: truncate d0/f1 3481329 0 2024-03-14T13:01:51.851 INFO:tasks.workunit.client.0.smithi111.stdout:5/20: truncate d2/d3/f4 4177411 0 2024-03-14T13:01:51.860 INFO:tasks.workunit.client.0.smithi111.stdout:4/24: rename c8 to cd 0 2024-03-14T13:01:51.861 INFO:tasks.workunit.client.0.smithi111.stdout:9/18: truncate f0 5216213 0 2024-03-14T13:01:51.865 INFO:tasks.workunit.client.0.smithi111.stdout:8/21: truncate f0 2207890 0 2024-03-14T13:01:51.866 INFO:tasks.workunit.client.0.smithi111.stdout:9/19: getdents . 0 2024-03-14T13:01:51.869 INFO:tasks.workunit.client.0.smithi111.stdout:4/25: dwrite fa [0,4194304] 0 2024-03-14T13:01:51.869 INFO:tasks.workunit.client.0.smithi111.stdout:4/26: dread - fb zero size 2024-03-14T13:01:51.870 INFO:tasks.workunit.client.0.smithi111.stdout:8/22: mkdir d3/d5 0 2024-03-14T13:01:51.872 INFO:tasks.workunit.client.0.smithi111.stdout:4/27: fdatasync f5 0 2024-03-14T13:01:51.875 INFO:tasks.workunit.client.0.smithi111.stdout:9/20: write f2 [1203638,56496] 0 2024-03-14T13:01:51.878 INFO:tasks.workunit.client.0.smithi111.stdout:8/23: write f0 [3109691,44607] 0 2024-03-14T13:01:51.881 INFO:tasks.workunit.client.0.smithi111.stdout:9/21: rename f0 to f4 0 2024-03-14T13:01:51.882 INFO:tasks.workunit.client.0.smithi111.stdout:6/18: link d0/f1 d0/f3 0 2024-03-14T13:01:51.882 INFO:tasks.workunit.client.0.smithi111.stdout:6/19: readlink - no filename 2024-03-14T13:01:51.886 INFO:tasks.workunit.client.0.smithi111.stdout:6/20: creat d0/f4 x:0 0 0 2024-03-14T13:01:51.887 INFO:tasks.workunit.client.0.smithi111.stdout:5/21: write f1 [2057578,19004] 0 2024-03-14T13:01:51.888 INFO:tasks.workunit.client.0.smithi111.stdout:4/28: fsync fa 0 2024-03-14T13:01:51.893 INFO:tasks.workunit.client.0.smithi111.stdout:5/22: dread d2/d3/f4 [0,4194304] 0 2024-03-14T13:01:51.895 INFO:tasks.workunit.client.0.smithi111.stdout:4/29: getdents . 0 2024-03-14T13:01:51.895 INFO:tasks.workunit.client.0.smithi111.stdout:4/30: read f9 [3894594,122497] 0 2024-03-14T13:01:51.896 INFO:tasks.workunit.client.0.smithi111.stdout:5/23: creat d2/f6 x:0 0 0 2024-03-14T13:01:51.897 INFO:tasks.workunit.client.0.smithi111.stdout:5/24: stat d2/d3/f4 0 2024-03-14T13:01:51.898 INFO:tasks.workunit.client.0.smithi111.stdout:4/31: creat fe x:0 0 0 2024-03-14T13:01:51.899 INFO:tasks.workunit.client.0.smithi111.stdout:6/21: rmdir d0 39 2024-03-14T13:01:51.901 INFO:tasks.workunit.client.0.smithi111.stdout:4/32: rename f5 to ff 0 2024-03-14T13:01:51.903 INFO:tasks.workunit.client.0.smithi111.stdout:4/33: truncate ff 526152 0 2024-03-14T13:01:51.904 INFO:tasks.workunit.client.0.smithi111.stdout:6/22: unlink d0/f2 0 2024-03-14T13:01:51.905 INFO:tasks.workunit.client.0.smithi111.stdout:5/25: dwrite f1 [0,4194304] 0 2024-03-14T13:01:51.906 INFO:tasks.workunit.client.0.smithi111.stdout:5/26: stat d2/f6 0 2024-03-14T13:01:51.908 INFO:tasks.workunit.client.0.smithi111.stdout:4/34: rename cd to c10 0 2024-03-14T13:01:51.910 INFO:tasks.workunit.client.0.smithi111.stdout:6/23: mknod d0/c5 0 2024-03-14T13:01:51.913 INFO:tasks.workunit.client.0.smithi111.stdout:5/27: dread d2/d3/f4 [0,4194304] 0 2024-03-14T13:01:51.918 INFO:tasks.workunit.client.0.smithi111.stdout:4/35: dread fa [0,4194304] 0 2024-03-14T13:01:51.919 INFO:tasks.workunit.client.0.smithi111.stdout:4/36: read ff [42930,130673] 0 2024-03-14T13:01:51.919 INFO:tasks.workunit.client.0.smithi111.stdout:5/28: unlink d2/d3/f4 0 2024-03-14T13:01:51.919 INFO:tasks.workunit.client.0.smithi111.stdout:5/29: dread - d2/f6 zero size 2024-03-14T13:01:51.919 INFO:tasks.workunit.client.0.smithi111.stdout:6/24: rename d0/f3 to d0/f6 0 2024-03-14T13:01:51.922 INFO:tasks.workunit.client.0.smithi111.stdout:4/37: write ff [306209,50257] 0 2024-03-14T13:01:51.922 INFO:tasks.workunit.client.0.smithi111.stdout:4/38: chown ff 12690 1 2024-03-14T13:01:51.927 INFO:tasks.workunit.client.0.smithi111.stdout:4/39: rename fa to f11 0 2024-03-14T13:01:51.928 INFO:tasks.workunit.client.0.smithi111.stdout:4/40: write fe [1023182,89200] 0 2024-03-14T13:01:51.931 INFO:tasks.workunit.client.0.smithi111.stdout:4/41: creat f12 x:0 0 0 2024-03-14T13:01:51.934 INFO:tasks.workunit.client.0.smithi111.stdout:6/25: fsync d0/f1 0 2024-03-14T13:01:51.935 INFO:tasks.workunit.client.0.smithi111.stdout:5/30: dwrite f1 [0,4194304] 0 2024-03-14T13:01:51.935 INFO:tasks.workunit.client.0.smithi111.stdout:6/26: rename d0 to d0/d7 22 2024-03-14T13:01:51.940 INFO:tasks.workunit.client.0.smithi111.stdout:6/27: fdatasync d0/f6 0 2024-03-14T13:01:51.940 INFO:tasks.workunit.client.0.smithi111.stdout:6/28: readlink - no filename 2024-03-14T13:01:51.944 INFO:tasks.workunit.client.0.smithi111.stdout:4/42: getdents . 0 2024-03-14T13:01:51.945 INFO:tasks.workunit.client.0.smithi111.stdout:6/29: mknod d0/c8 0 2024-03-14T13:01:51.947 INFO:tasks.workunit.client.0.smithi111.stdout:6/30: symlink d0/l9 0 2024-03-14T13:01:51.947 INFO:tasks.workunit.client.0.smithi111.stdout:4/43: truncate ff 1329122 0 2024-03-14T13:01:51.948 INFO:tasks.workunit.client.0.smithi111.stdout:5/31: dwrite d2/f6 [0,4194304] 0 2024-03-14T13:01:51.949 INFO:tasks.workunit.client.0.smithi111.stdout:6/31: fdatasync d0/f6 0 2024-03-14T13:01:51.951 INFO:tasks.workunit.client.0.smithi111.stdout:6/32: stat d0/f6 0 2024-03-14T13:01:51.951 INFO:tasks.workunit.client.0.smithi111.stdout:4/44: getdents . 0 2024-03-14T13:01:51.952 INFO:tasks.workunit.client.0.smithi111.stdout:6/33: fdatasync d0/f1 0 2024-03-14T13:01:51.958 INFO:tasks.workunit.client.0.smithi111.stdout:4/45: dread fe [0,4194304] 0 2024-03-14T13:01:51.961 INFO:tasks.workunit.client.0.smithi111.stdout:4/46: creat f13 x:0 0 0 2024-03-14T13:01:51.961 INFO:tasks.workunit.client.0.smithi111.stdout:4/47: stat f11 0 2024-03-14T13:01:51.962 INFO:tasks.workunit.client.0.smithi111.stdout:4/48: creat f14 x:0 0 0 2024-03-14T13:01:51.970 INFO:tasks.workunit.client.0.smithi111.stdout:6/34: rmdir d0 39 2024-03-14T13:01:51.972 INFO:tasks.workunit.client.0.smithi111.stdout:5/32: truncate d2/f6 2826839 0 2024-03-14T13:01:51.975 INFO:tasks.workunit.client.0.smithi111.stdout:6/35: rename d0/f1 to d0/fa 0 2024-03-14T13:01:51.976 INFO:tasks.workunit.client.0.smithi111.stdout:5/33: creat d2/f7 x:0 0 0 2024-03-14T13:01:51.977 INFO:tasks.workunit.client.0.smithi111.stdout:5/34: chown d2/d3/c5 786382 1 2024-03-14T13:01:51.977 INFO:tasks.workunit.client.0.smithi111.stdout:5/35: rename d2/d3 to d2/d3/d8 22 2024-03-14T13:01:51.978 INFO:tasks.workunit.client.0.smithi111.stdout:6/36: dread d0/f6 [0,4194304] 0 2024-03-14T13:01:51.979 INFO:tasks.workunit.client.0.smithi111.stdout:6/37: read - d0/f4 zero size 2024-03-14T13:01:51.979 INFO:tasks.workunit.client.0.smithi111.stdout:6/38: chown d0/c8 2106364 1 2024-03-14T13:01:51.985 INFO:tasks.workunit.client.0.smithi111.stdout:5/36: truncate d2/f6 3350334 0 2024-03-14T13:01:51.987 INFO:tasks.workunit.client.0.smithi111.stdout:5/37: mknod d2/c9 0 2024-03-14T13:01:51.987 INFO:tasks.workunit.client.0.smithi111.stdout:6/39: dwrite d0/f6 [0,4194304] 0 2024-03-14T13:01:51.987 INFO:tasks.workunit.client.0.smithi111.stdout:5/38: dread - d2/f7 zero size 2024-03-14T13:01:51.989 INFO:tasks.workunit.client.0.smithi111.stdout:6/40: symlink d0/lb 0 2024-03-14T13:01:51.998 INFO:tasks.workunit.client.0.smithi111.stdout:5/39: dwrite d2/f6 [0,4194304] 0 2024-03-14T13:01:51.998 INFO:tasks.workunit.client.0.smithi111.stdout:5/40: rename d2/d3 to d2/d3/da 22 2024-03-14T13:01:52.001 INFO:tasks.workunit.client.0.smithi111.stdout:5/41: creat d2/d3/fb x:0 0 0 2024-03-14T13:01:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:51 smithi137 ceph-mon[50336]: pgmap v10: 65 pgs: 65 active+clean; 183 MiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.4 MiB/s wr, 433 op/s 2024-03-14T13:01:52.010 INFO:tasks.workunit.client.0.smithi111.stdout:5/42: dwrite d2/d3/fb [0,4194304] 0 2024-03-14T13:01:52.012 INFO:tasks.workunit.client.0.smithi111.stdout:5/43: mknod d2/d3/cc 0 2024-03-14T13:01:52.012 INFO:tasks.workunit.client.0.smithi111.stdout:5/44: rename d2 to d2/d3/dd 22 2024-03-14T13:01:52.013 INFO:tasks.workunit.client.0.smithi111.stdout:5/45: chown d2/d3/cc 67610 1 2024-03-14T13:01:52.017 INFO:tasks.workunit.client.0.smithi111.stdout:5/46: dread d2/d3/fb [0,4194304] 0 2024-03-14T13:01:52.019 INFO:tasks.workunit.client.0.smithi111.stdout:5/47: unlink d2/d3/fb 0 2024-03-14T13:01:52.020 INFO:tasks.workunit.client.0.smithi111.stdout:5/48: write d2/f7 [726044,83559] 0 2024-03-14T13:01:52.028 INFO:tasks.workunit.client.0.smithi111.stdout:5/49: rmdir d2 39 2024-03-14T13:01:52.033 INFO:tasks.workunit.client.0.smithi111.stdout:5/50: getdents d2 0 2024-03-14T13:01:52.037 INFO:tasks.workunit.client.0.smithi111.stdout:5/51: dread f1 [0,4194304] 0 2024-03-14T13:01:52.045 INFO:tasks.workunit.client.0.smithi111.stdout:5/52: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:01:52.046 INFO:tasks.workunit.client.0.smithi111.stdout:5/53: write d2/f6 [1470215,29419] 0 2024-03-14T13:01:52.049 INFO:tasks.workunit.client.0.smithi111.stdout:5/54: mknod d2/d3/ce 0 2024-03-14T13:01:52.050 INFO:tasks.workunit.client.0.smithi111.stdout:5/55: chown f1 9048 1 2024-03-14T13:01:52.061 INFO:tasks.workunit.client.0.smithi111.stdout:5/56: fsync d2/f7 0 2024-03-14T13:01:52.063 INFO:tasks.workunit.client.0.smithi111.stdout:5/57: mknod d2/d3/cf 0 2024-03-14T13:01:52.064 INFO:tasks.workunit.client.0.smithi111.stdout:5/58: symlink d2/l10 0 2024-03-14T13:01:52.065 INFO:tasks.workunit.client.0.smithi111.stdout:5/59: truncate f1 5221586 0 2024-03-14T13:01:52.067 INFO:tasks.workunit.client.0.smithi111.stdout:5/60: creat d2/d3/f11 x:0 0 0 2024-03-14T13:01:52.068 INFO:tasks.workunit.client.0.smithi111.stdout:5/61: fsync d2/f7 0 2024-03-14T13:01:52.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.074+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cbc115e10 con 0x7f6ca806c630 2024-03-14T13:01:52.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.076+0000 7f6cb27fc700 1 -- 172.21.15.111:0/420038124 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f6cbc115e10 con 0x7f6ca806c630 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 msgr2=0x7f6ca806eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 0x7f6ca806eae0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b0d0 tx=0x7f6cb8009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 msgr2=0x7f6cbc114410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6cac00eb40 tx=0x7f6cac00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 shutdown_connections 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cbc0fbce0 0x7f6cbc113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f6ca806c630 0x7f6ca806eae0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 --2- 172.21.15.111:0/420038124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cbc0fcf90 0x7f6cbc114410 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 >> 172.21.15.111:0/420038124 conn(0x7f6cbc0f7480 msgr2=0x7f6cbc0f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 shutdown_connections 2024-03-14T13:01:52.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.079+0000 7f6cc3d99700 1 -- 172.21.15.111:0/420038124 wait complete. 2024-03-14T13:01:52.080 INFO:tasks.workunit.client.0.smithi111.stdout:5/62: dwrite d2/d3/f11 [0,4194304] 0 2024-03-14T13:01:52.086 INFO:tasks.workunit.client.0.smithi111.stdout:5/63: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:01:52.091 INFO:tasks.workunit.client.0.smithi111.stdout:5/64: link d2/d3/f11 d2/d3/f12 0 2024-03-14T13:01:52.093 INFO:tasks.workunit.client.0.smithi111.stdout:5/65: mkdir d2/d13 0 2024-03-14T13:01:52.107 INFO:tasks.workunit.client.0.smithi111.stdout:5/66: fdatasync d2/f7 0 2024-03-14T13:01:52.108 INFO:tasks.workunit.client.0.smithi111.stdout:5/67: truncate d2/f6 5061196 0 2024-03-14T13:01:52.110 INFO:tasks.workunit.client.0.smithi111.stdout:5/68: mkdir d2/d3/d14 0 2024-03-14T13:01:52.112 INFO:tasks.workunit.client.0.smithi111.stdout:5/69: creat d2/d13/f15 x:0 0 0 2024-03-14T13:01:52.113 INFO:tasks.workunit.client.0.smithi111.stdout:5/70: chown d2/c9 45705555 1 2024-03-14T13:01:52.114 INFO:tasks.workunit.client.0.smithi111.stdout:5/71: fsync d2/d3/f11 0 2024-03-14T13:01:52.114 INFO:tasks.workunit.client.0.smithi111.stdout:5/72: fdatasync d2/d3/f11 0 2024-03-14T13:01:52.117 INFO:tasks.workunit.client.0.smithi111.stdout:5/73: rename d2/d3/d14 to d2/d16 0 2024-03-14T13:01:52.124 INFO:tasks.workunit.client.0.smithi111.stdout:5/74: rename d2/d3 to d2/d17 0 2024-03-14T13:01:52.125 INFO:tasks.workunit.client.0.smithi111.stdout:5/75: rmdir d2/d13 39 2024-03-14T13:01:52.126 INFO:tasks.workunit.client.0.smithi111.stdout:5/76: write d2/f7 [1742391,88075] 0 2024-03-14T13:01:52.128 INFO:tasks.workunit.client.0.smithi111.stdout:5/77: mknod d2/d16/c18 0 2024-03-14T13:01:52.135 INFO:tasks.workunit.client.0.smithi111.stdout:5/78: dwrite d2/d17/f12 [4194304,4194304] 0 2024-03-14T13:01:52.136 INFO:tasks.workunit.client.0.smithi111.stdout:5/79: fdatasync d2/f7 0 2024-03-14T13:01:52.142 INFO:tasks.workunit.client.0.smithi111.stdout:5/80: rename d2/d17 to d2/d19 0 2024-03-14T13:01:52.144 INFO:tasks.workunit.client.0.smithi111.stdout:5/81: symlink d2/l1a 0 2024-03-14T13:01:52.146 INFO:tasks.workunit.client.0.smithi111.stdout:5/82: creat d2/d13/f1b x:0 0 0 2024-03-14T13:01:52.148 INFO:tasks.workunit.client.0.smithi111.stdout:5/83: fdatasync f1 0 2024-03-14T13:01:52.149 INFO:tasks.workunit.client.0.smithi111.stdout:5/84: mkdir d2/d1c 0 2024-03-14T13:01:52.151 INFO:tasks.workunit.client.0.smithi111.stdout:5/85: creat d2/d13/f1d x:0 0 0 2024-03-14T13:01:52.152 INFO:tasks.workunit.client.0.smithi111.stdout:5/86: write d2/d13/f1d [555614,5443] 0 2024-03-14T13:01:52.160 INFO:tasks.workunit.client.0.smithi111.stdout:5/87: dwrite d2/d13/f15 [0,4194304] 0 2024-03-14T13:01:52.163 INFO:tasks.workunit.client.0.smithi111.stdout:5/88: creat d2/d19/f1e x:0 0 0 2024-03-14T13:01:52.164 INFO:tasks.workunit.client.0.smithi111.stdout:5/89: write d2/d13/f15 [3826742,20947] 0 2024-03-14T13:01:52.167 INFO:tasks.workunit.client.0.smithi111.stdout:5/90: mkdir d2/d1f 0 2024-03-14T13:01:52.174 INFO:tasks.workunit.client.0.smithi111.stdout:5/91: truncate d2/d19/f12 5071716 0 2024-03-14T13:01:52.175 INFO:tasks.workunit.client.0.smithi111.stdout:5/92: creat d2/d1c/f20 x:0 0 0 2024-03-14T13:01:52.183 INFO:tasks.workunit.client.0.smithi111.stdout:5/93: write d2/d19/f11 [5850833,96036] 0 2024-03-14T13:01:52.184 INFO:tasks.workunit.client.0.smithi111.stdout:5/94: write d2/d13/f15 [1989868,86636] 0 2024-03-14T13:01:52.186 INFO:tasks.workunit.client.0.smithi111.stdout:5/95: link d2/d19/c5 d2/d1c/c21 0 2024-03-14T13:01:52.187 INFO:tasks.workunit.client.0.smithi111.stdout:5/96: write d2/d19/f11 [5204303,62542] 0 2024-03-14T13:01:52.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 -- 172.21.15.111:0/1438459718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 msgr2=0x7f071c101e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 --2- 172.21.15.111:0/1438459718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c101e60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0718004940 tx=0x7f0718009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:52.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 -- 172.21.15.111:0/1438459718 shutdown_connections 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 --2- 172.21.15.111:0/1438459718 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 0x7f071c104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 --2- 172.21.15.111:0/1438459718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c101e60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 -- 172.21.15.111:0/1438459718 >> 172.21.15.111:0/1438459718 conn(0x7f071c0f7450 msgr2=0x7f071c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 -- 172.21.15.111:0/1438459718 shutdown_connections 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.211+0000 7f072450a700 1 -- 172.21.15.111:0/1438459718 wait complete. 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 Processor -- start 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 -- start start 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 0x7f071c110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c110680 con 0x7f071c066af0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c110ed0 con 0x7f071c1023a0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43102/0 (socket says 172.21.15.111:43102) 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 -- 172.21.15.111:0/2194757315 learned_addr learned my addr 172.21.15.111:0/2194757315 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f0721aa5700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 0x7f071c110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 -- 172.21.15.111:0/2194757315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 msgr2=0x7f071c110080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 0x7f071c110080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 -- 172.21.15.111:0/2194757315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718005040 con 0x7f071c066af0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f07222a6700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0718009380 tx=0x7f0718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f071800e040 con 0x7f071c066af0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0718013420 con 0x7f071c066af0 2024-03-14T13:01:52.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c111150 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f071801c630 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.212+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c111690 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.213+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c05f5f0 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.214+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f0718013900 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.214+0000 7f0712ffd700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 0x7f070806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.214+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f071808c9e0 con 0x7f071c066af0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.214+0000 7f0721aa5700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 0x7f070806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.215+0000 7f0721aa5700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 0x7f070806ea10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f070c009fd0 tx=0x7f070c00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:52.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.219+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f071805ab40 con 0x7f071c066af0 2024-03-14T13:01:52.252 INFO:tasks.workunit.client.0.smithi111.stdout:5/97: creat d2/d1c/f22 x:0 0 0 2024-03-14T13:01:52.252 INFO:tasks.workunit.client.0.smithi111.stdout:5/98: chown d2/d1c 8929917 1 2024-03-14T13:01:52.252 INFO:tasks.workunit.client.0.smithi111.stdout:5/99: mknod d2/d1c/c23 0 2024-03-14T13:01:52.252 INFO:tasks.workunit.client.0.smithi111.stdout:5/100: dread - d2/d1c/f22 zero size 2024-03-14T13:01:52.253 INFO:tasks.workunit.client.0.smithi111.stdout:5/101: fsync d2/f7 0 2024-03-14T13:01:52.253 INFO:tasks.workunit.client.0.smithi111.stdout:5/102: chown d2/d19/cf 37 1 2024-03-14T13:01:52.254 INFO:tasks.workunit.client.0.smithi111.stdout:5/103: fsync d2/f6 0 2024-03-14T13:01:52.256 INFO:tasks.workunit.client.0.smithi111.stdout:5/104: creat d2/d1f/f24 x:0 0 0 2024-03-14T13:01:52.257 INFO:tasks.workunit.client.0.smithi111.stdout:5/105: creat d2/d13/f25 x:0 0 0 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (60s) 9s ago 9m 17.0M - 0.25.0 c8568f914cd2 70fc8e769dee 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (9m) 9s ago 9m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (8m) 9s ago 8m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (55s) 9s ago 9m 82.5M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (3m) 9s ago 3m 57.8M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (3m) 9s ago 3m 113M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (3m) 9s ago 3m 53.8M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (3m) 9s ago 3m 81.5M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:01:52.475 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (40s) 9s ago 11m 475M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (98s) 9s ago 8m 61.7M - 18.2.2-825-g2f465319 a9e65ad305ae 0487350e99e4 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (11m) 9s ago 11m 54.1M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (8m) 9s ago 8m 40.2M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (66s) 9s ago 9m 9269k - 1.5.0 0da6a335fe13 c8b729c46aaa 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (46s) 9s ago 8m 8870k - 1.5.0 0da6a335fe13 37967b629a62 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (7m) 9s ago 7m 97.1M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (6m) 9s ago 6m 107M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (6m) 9s ago 6m 89.0M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (6m) 9s ago 6m 111M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (5m) 9s ago 5m 100M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (5m) 9s ago 5m 96.7M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (48s) 9s ago 8m 41.6M - 2.43.0 a07b618ecd1d d3a9067a4910 2024-03-14T13:01:52.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.450+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f071c111a10 con 0x7f070806c560 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.460+0000 7f0712ffd700 1 -- 172.21.15.111:0/2194757315 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f071c111a10 con 0x7f070806c560 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 msgr2=0x7f070806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 0x7f070806ea10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f070c009fd0 tx=0x7f070c00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 msgr2=0x7f071c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0718009380 tx=0x7f0718009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 shutdown_connections 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f070806c560 0x7f070806ea10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f071c066af0 0x7f071c10fb40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 --2- 172.21.15.111:0/2194757315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f071c1023a0 0x7f071c110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 >> 172.21.15.111:0/2194757315 conn(0x7f071c0f7450 msgr2=0x7f071c0f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 shutdown_connections 2024-03-14T13:01:52.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.462+0000 7f072450a700 1 -- 172.21.15.111:0/2194757315 wait complete. 2024-03-14T13:01:52.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- 172.21.15.111:0/3193230680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 msgr2=0x7ff28c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 --2- 172.21.15.111:0/3193230680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c102320 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff280004940 tx=0x7ff280009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- 172.21.15.111:0/3193230680 shutdown_connections 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 --2- 172.21.15.111:0/3193230680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c102320 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 --2- 172.21.15.111:0/3193230680 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- 172.21.15.111:0/3193230680 >> 172.21.15.111:0/3193230680 conn(0x7ff28c0f7430 msgr2=0x7ff28c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- 172.21.15.111:0/3193230680 shutdown_connections 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- 172.21.15.111:0/3193230680 wait complete. 2024-03-14T13:01:52.590 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 Processor -- start 2024-03-14T13:01:52.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.590+0000 7ff292575700 1 -- start start 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff292575700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff292575700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff292575700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c10e460 con 0x7ff28c0fff00 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff292575700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c10ec60 con 0x7ff28c0662e0 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28bfff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28bfff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:52342/0 (socket says 172.21.15.111:52342) 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28bfff700 1 -- 172.21.15.111:0/1162314073 learned_addr learned my addr 172.21.15.111:0/1162314073 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28b7fe700 1 -- 172.21.15.111:0/1162314073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 msgr2=0x7ff28c10d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28b7fe700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28b7fe700 1 -- 172.21.15.111:0/1162314073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff280005040 con 0x7ff28c0fff00 2024-03-14T13:01:52.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28bfff700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.591+0000 7ff28b7fe700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c10de60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff280009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.592+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28000e040 con 0x7ff28c0fff00 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.592+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff28000bc50 con 0x7ff28c0fff00 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.592+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c10eee0 con 0x7ff28c0fff00 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.592+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28001c650 con 0x7ff28c0fff00 2024-03-14T13:01:52.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.592+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c10f420 con 0x7ff28c0fff00 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.593+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2780052e0 con 0x7ff28c0fff00 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.593+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7ff28001c7b0 con 0x7ff28c0fff00 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.594+0000 7ff288ff9700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 0x7ff27406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.594+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff28008c960 con 0x7ff28c0fff00 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.594+0000 7ff28bfff700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 0x7ff27406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:52.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.595+0000 7ff28bfff700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 0x7ff27406ea10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0044e0 tx=0x7ff27c00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:52.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.599+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7ff28005a970 con 0x7ff28c0fff00 2024-03-14T13:01:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:52 smithi111 ceph-mon[39688]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:52 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:52 smithi111 ceph-mon[39688]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:52.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.907+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff278005860 con 0x7ff28c0fff00 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.907+0000 7ff288ff9700 1 -- 172.21.15.111:0/1162314073 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7ff28005a520 con 0x7ff28c0fff00 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:52.907 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:01:52.908 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 msgr2=0x7ff27406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 0x7ff27406ea10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0044e0 tx=0x7ff27c00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 msgr2=0x7ff28c10de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c10de60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff280000c00 tx=0x7ff280009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 shutdown_connections 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff28c0662e0 0x7ff28c10d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff27406c560 0x7ff27406ea10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 --2- 172.21.15.111:0/1162314073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0fff00 0x7ff28c10de60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.909+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 >> 172.21.15.111:0/1162314073 conn(0x7ff28c0f7430 msgr2=0x7ff28c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.910+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 shutdown_connections 2024-03-14T13:01:52.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:52.910+0000 7ff292575700 1 -- 172.21.15.111:0/1162314073 wait complete. 2024-03-14T13:01:53.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:52 smithi137 ceph-mon[50336]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:52 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:01:53.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:52 smithi137 ceph-mon[50336]: from='client.14704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:53.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 -- 172.21.15.111:0/2045518593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 msgr2=0x7f91280fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 --2- 172.21.15.111:0/2045518593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f91280fc0f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9118004940 tx=0x7f9118009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 -- 172.21.15.111:0/2045518593 shutdown_connections 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 --2- 172.21.15.111:0/2045518593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 0x7f91280fd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 --2- 172.21.15.111:0/2045518593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f91280fc0f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 -- 172.21.15.111:0/2045518593 >> 172.21.15.111:0/2045518593 conn(0x7f91280f7440 msgr2=0x7f91280f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 -- 172.21.15.111:0/2045518593 shutdown_connections 2024-03-14T13:01:53.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.023+0000 7f912e6bf700 1 -- 172.21.15.111:0/2045518593 wait complete. 2024-03-14T13:01:53.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 Processor -- start 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 -- start start 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 0x7f91281125a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128112ae0 con 0x7f91280fbce0 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f912e6bf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128112c20 con 0x7f91280fcf90 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f9127fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f9127fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43142/0 (socket says 172.21.15.111:43142) 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f9127fff700 1 -- 172.21.15.111:0/1326216160 learned_addr learned my addr 172.21.15.111:0/1326216160 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.024+0000 7f91277fe700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 0x7f91281125a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9127fff700 1 -- 172.21.15.111:0/1326216160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 msgr2=0x7f91281125a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9127fff700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 0x7f91281125a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9127fff700 1 -- 172.21.15.111:0/1326216160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9118005040 con 0x7f91280fbce0 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9127fff700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9118007d40 tx=0x7f9118009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:53.025 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f911800e030 con 0x7f91280fbce0 2024-03-14T13:01:53.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9128112e50 con 0x7f91280fbce0 2024-03-14T13:01:53.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f911801c400 con 0x7f91280fbce0 2024-03-14T13:01:53.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f911800b650 con 0x7f91280fbce0 2024-03-14T13:01:53.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.025+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9128113330 con 0x7f91280fbce0 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.026+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912804c710 con 0x7f91280fbce0 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.027+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f911801c570 con 0x7f91280fbce0 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.027+0000 7f9124ff9700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 0x7f911006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.027+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f911808cd00 con 0x7f91280fbce0 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.027+0000 7f91277fe700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 0x7f911006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:01:53.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.028+0000 7f91277fe700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 0x7f911006ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f911c009b80 tx=0x7f911c00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:01:53.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.032+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f9118090030 con 0x7f91280fbce0 2024-03-14T13:01:53.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.278+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9128113550 con 0x7f911006c570 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "", 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.279+0000 7f9124ff9700 1 -- 172.21.15.111:0/1326216160 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f9128113550 con 0x7f911006c570 2024-03-14T13:01:53.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 msgr2=0x7f911006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 0x7f911006ea20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f911c009b80 tx=0x7f911c00b040 comp rx=0 tx=0).stop 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 msgr2=0x7f9128066d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9118007d40 tx=0x7f9118009e30 comp rx=0 tx=0).stop 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 shutdown_connections 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f911006c570 0x7f911006ea20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91280fbce0 0x7f9128066d30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 --2- 172.21.15.111:0/1326216160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91280fcf90 0x7f91281125a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.281+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 >> 172.21.15.111:0/1326216160 conn(0x7f91280f7440 msgr2=0x7f91280f8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.282+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 shutdown_connections 2024-03-14T13:01:53.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:01:53.282+0000 7f912e6bf700 1 -- 172.21.15.111:0/1326216160 wait complete. 2024-03-14T13:01:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:53 smithi137 ceph-mon[50336]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:53 smithi137 ceph-mon[50336]: pgmap v11: 65 pgs: 65 active+clean; 188 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 461 op/s 2024-03-14T13:01:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:53 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1162314073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:01:54.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:53 smithi111 ceph-mon[39688]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:54.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:53 smithi111 ceph-mon[39688]: pgmap v11: 65 pgs: 65 active+clean; 188 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 4.7 MiB/s wr, 461 op/s 2024-03-14T13:01:54.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:53 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1162314073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:01:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:54 smithi137 ceph-mon[50336]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:54 smithi111 ceph-mon[39688]: from='client.14716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:01:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:55 smithi137 ceph-mon[50336]: pgmap v12: 65 pgs: 65 active+clean; 262 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 14 MiB/s wr, 595 op/s 2024-03-14T13:01:56.045 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:55 smithi111 ceph-mon[39688]: pgmap v12: 65 pgs: 65 active+clean; 262 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 14 MiB/s wr, 595 op/s 2024-03-14T13:01:56.408 INFO:tasks.workunit.client.1.smithi137.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-14T13:01:56.417 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-14T13:01:56.417 INFO:tasks.workunit.client.1.smithi137.stderr:+ make 2024-03-14T13:01:56.573 INFO:tasks.workunit.client.1.smithi137.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-14T13:01:56.952 INFO:tasks.workunit.client.1.smithi137.stderr:++ readlink -f fsstress 2024-03-14T13:01:56.953 INFO:tasks.workunit.client.1.smithi137.stderr:+ BIN=/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress 2024-03-14T13:01:56.953 INFO:tasks.workunit.client.1.smithi137.stderr:+ popd 2024-03-14T13:01:56.954 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-14T13:01:56.954 INFO:tasks.workunit.client.1.smithi137.stderr:+ popd 2024-03-14T13:01:56.955 INFO:tasks.workunit.client.1.smithi137.stdout:~/cephtest/mnt.1/client.1/tmp 2024-03-14T13:01:56.955 INFO:tasks.workunit.client.1.smithi137.stderr:++ mktemp -d -p . 2024-03-14T13:01:56.958 INFO:tasks.workunit.client.1.smithi137.stderr:+ T=./tmp.CKF3lbLY2Q 2024-03-14T13:01:56.958 INFO:tasks.workunit.client.1.smithi137.stderr:+ /home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress/fsstress -d ./tmp.CKF3lbLY2Q -l 1 -n 1000 -p 10 -v 2024-03-14T13:01:56.960 INFO:tasks.workunit.client.1.smithi137.stdout:seed = 1711036987 2024-03-14T13:01:56.963 INFO:tasks.workunit.client.1.smithi137.stdout:0/0: readlink - no filename 2024-03-14T13:01:56.965 INFO:tasks.workunit.client.1.smithi137.stdout:1/0: fsync - no filename 2024-03-14T13:01:56.965 INFO:tasks.workunit.client.1.smithi137.stdout:1/1: write - no filename 2024-03-14T13:01:56.965 INFO:tasks.workunit.client.1.smithi137.stdout:1/2: dwrite - no filename 2024-03-14T13:01:56.966 INFO:tasks.workunit.client.1.smithi137.stdout:1/3: creat f0 x:0 0 0 2024-03-14T13:01:56.968 INFO:tasks.workunit.client.1.smithi137.stdout:3/0: write - no filename 2024-03-14T13:01:56.968 INFO:tasks.workunit.client.1.smithi137.stdout:1/4: mknod c1 0 2024-03-14T13:01:56.969 INFO:tasks.workunit.client.1.smithi137.stdout:2/0: mknod c0 0 2024-03-14T13:01:56.969 INFO:tasks.workunit.client.1.smithi137.stdout:1/5: read - f0 zero size 2024-03-14T13:01:56.969 INFO:tasks.workunit.client.1.smithi137.stdout:1/6: stat c1 0 2024-03-14T13:01:56.970 INFO:tasks.workunit.client.1.smithi137.stdout:4/0: dread - no filename 2024-03-14T13:01:56.970 INFO:tasks.workunit.client.1.smithi137.stdout:4/1: truncate - no filename 2024-03-14T13:01:56.970 INFO:tasks.workunit.client.1.smithi137.stdout:4/2: rename - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/3: dwrite - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/4: unlink - no file 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/5: rename - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/6: write - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/7: dwrite - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/8: read - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/9: write - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/10: readlink - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/11: dwrite - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/12: read - no filename 2024-03-14T13:01:56.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/13: write - no filename 2024-03-14T13:01:56.972 INFO:tasks.workunit.client.1.smithi137.stdout:3/1: symlink l0 0 2024-03-14T13:01:56.972 INFO:tasks.workunit.client.1.smithi137.stdout:3/2: truncate - no filename 2024-03-14T13:01:56.974 INFO:tasks.workunit.client.1.smithi137.stdout:3/3: creat f1 x:0 0 0 2024-03-14T13:01:56.974 INFO:tasks.workunit.client.1.smithi137.stdout:4/14: creat f0 x:0 0 0 2024-03-14T13:01:56.975 INFO:tasks.workunit.client.1.smithi137.stdout:5/0: chown . 623534 1 2024-03-14T13:01:56.975 INFO:tasks.workunit.client.1.smithi137.stdout:4/15: dread - f0 zero size 2024-03-14T13:01:56.976 INFO:tasks.workunit.client.1.smithi137.stdout:6/0: write - no filename 2024-03-14T13:01:56.977 INFO:tasks.workunit.client.1.smithi137.stdout:4/16: dread - f0 zero size 2024-03-14T13:01:56.979 INFO:tasks.workunit.client.1.smithi137.stdout:5/1: mknod c0 0 2024-03-14T13:01:56.979 INFO:tasks.workunit.client.1.smithi137.stdout:5/2: dread - no filename 2024-03-14T13:01:56.979 INFO:tasks.workunit.client.1.smithi137.stdout:7/0: dwrite - no filename 2024-03-14T13:01:56.979 INFO:tasks.workunit.client.1.smithi137.stdout:7/1: rename - no filename 2024-03-14T13:01:56.980 INFO:tasks.workunit.client.1.smithi137.stdout:7/2: dread - no filename 2024-03-14T13:01:56.980 INFO:tasks.workunit.client.1.smithi137.stdout:3/4: link l0 l2 0 2024-03-14T13:01:56.980 INFO:tasks.workunit.client.1.smithi137.stdout:3/5: rmdir - no directory 2024-03-14T13:01:56.981 INFO:tasks.workunit.client.1.smithi137.stdout:6/1: getdents . 0 2024-03-14T13:01:56.981 INFO:tasks.workunit.client.1.smithi137.stdout:6/2: readlink - no filename 2024-03-14T13:01:56.983 INFO:tasks.workunit.client.1.smithi137.stdout:5/3: rename c0 to c1 0 2024-03-14T13:01:56.985 INFO:tasks.workunit.client.1.smithi137.stdout:8/0: chown . 30255303 1 2024-03-14T13:01:56.985 INFO:tasks.workunit.client.1.smithi137.stdout:8/1: truncate - no filename 2024-03-14T13:01:56.987 INFO:tasks.workunit.client.1.smithi137.stdout:6/3: creat f0 x:0 0 0 2024-03-14T13:01:56.987 INFO:tasks.workunit.client.1.smithi137.stdout:1/7: dwrite f0 [0,4194304] 0 2024-03-14T13:01:56.992 INFO:tasks.workunit.client.1.smithi137.stdout:7/3: creat f0 x:0 0 0 2024-03-14T13:01:56.994 INFO:tasks.workunit.client.1.smithi137.stdout:6/4: symlink l1 0 2024-03-14T13:01:56.999 INFO:tasks.workunit.client.1.smithi137.stdout:6/5: dread - f0 zero size 2024-03-14T13:01:57.002 INFO:tasks.workunit.client.1.smithi137.stdout:4/17: dwrite f0 [0,4194304] 0 2024-03-14T13:01:57.002 INFO:tasks.workunit.client.1.smithi137.stdout:4/18: readlink - no filename 2024-03-14T13:01:57.003 INFO:tasks.workunit.client.1.smithi137.stdout:8/2: creat f0 x:0 0 0 2024-03-14T13:01:57.006 INFO:tasks.workunit.client.1.smithi137.stdout:1/8: dwrite f0 [0,4194304] 0 2024-03-14T13:01:57.006 INFO:tasks.workunit.client.1.smithi137.stdout:1/9: rmdir - no directory 2024-03-14T13:01:57.007 INFO:tasks.workunit.client.1.smithi137.stdout:7/4: creat f1 x:0 0 0 2024-03-14T13:01:57.014 INFO:tasks.workunit.client.1.smithi137.stdout:1/10: dread f0 [0,4194304] 0 2024-03-14T13:01:57.019 INFO:tasks.workunit.client.1.smithi137.stdout:6/6: mkdir d2 0 2024-03-14T13:01:57.020 INFO:tasks.workunit.client.1.smithi137.stdout:4/19: symlink l1 0 2024-03-14T13:01:57.020 INFO:tasks.workunit.client.1.smithi137.stdout:8/3: mkdir d1 0 2024-03-14T13:01:57.020 INFO:tasks.workunit.client.1.smithi137.stdout:9/0: creat f0 x:0 0 0 2024-03-14T13:01:57.021 INFO:tasks.workunit.client.1.smithi137.stdout:3/6: getdents . 0 2024-03-14T13:01:57.021 INFO:tasks.workunit.client.1.smithi137.stdout:6/7: chown l1 26531060 1 2024-03-14T13:01:57.021 INFO:tasks.workunit.client.1.smithi137.stdout:3/7: fdatasync f1 0 2024-03-14T13:01:57.022 INFO:tasks.workunit.client.1.smithi137.stdout:6/8: dread - f0 zero size 2024-03-14T13:01:57.023 INFO:tasks.workunit.client.1.smithi137.stdout:8/4: truncate f0 415709 0 2024-03-14T13:01:57.023 INFO:tasks.workunit.client.1.smithi137.stdout:5/4: chown c1 1 1 2024-03-14T13:01:57.023 INFO:tasks.workunit.client.1.smithi137.stdout:5/5: read - no filename 2024-03-14T13:01:57.024 INFO:tasks.workunit.client.1.smithi137.stdout:5/6: truncate - no filename 2024-03-14T13:01:57.024 INFO:tasks.workunit.client.1.smithi137.stdout:4/20: rename f0 to f2 0 2024-03-14T13:01:57.025 INFO:tasks.workunit.client.1.smithi137.stdout:9/1: creat f1 x:0 0 0 2024-03-14T13:01:57.031 INFO:tasks.workunit.client.1.smithi137.stdout:5/7: symlink l2 0 2024-03-14T13:01:57.031 INFO:tasks.workunit.client.1.smithi137.stdout:5/8: readlink l2 0 2024-03-14T13:01:57.031 INFO:tasks.workunit.client.1.smithi137.stdout:5/9: rmdir - no directory 2024-03-14T13:01:57.031 INFO:tasks.workunit.client.1.smithi137.stdout:4/21: write f2 [2830995,90871] 0 2024-03-14T13:01:57.035 INFO:tasks.workunit.client.1.smithi137.stdout:8/5: creat d1/f2 x:0 0 0 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:8/6: truncate d1/f2 589584 0 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/10: mkdir d3 0 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/11: chown d3 0 1 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/12: dread - no filename 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/13: truncate - no filename 2024-03-14T13:01:57.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/14: dwrite - no filename 2024-03-14T13:01:57.038 INFO:tasks.workunit.client.1.smithi137.stdout:8/7: rename d1 to d1/d3 22 2024-03-14T13:01:57.038 INFO:tasks.workunit.client.1.smithi137.stdout:1/11: truncate f0 3536325 0 2024-03-14T13:01:57.038 INFO:tasks.workunit.client.1.smithi137.stdout:5/15: chown c1 14298 1 2024-03-14T13:01:57.040 INFO:tasks.workunit.client.1.smithi137.stdout:9/2: unlink f0 0 2024-03-14T13:01:57.040 INFO:tasks.workunit.client.1.smithi137.stdout:7/5: getdents . 0 2024-03-14T13:01:57.041 INFO:tasks.workunit.client.1.smithi137.stdout:7/6: truncate f1 37978 0 2024-03-14T13:01:57.042 INFO:tasks.workunit.client.1.smithi137.stdout:8/8: symlink d1/l4 0 2024-03-14T13:01:57.043 INFO:tasks.workunit.client.1.smithi137.stdout:8/9: chown d1/f2 0 1 2024-03-14T13:01:57.045 INFO:tasks.workunit.client.1.smithi137.stdout:9/3: creat f2 x:0 0 0 2024-03-14T13:01:57.048 INFO:tasks.workunit.client.1.smithi137.stdout:1/12: write f0 [1014727,3331] 0 2024-03-14T13:01:57.050 INFO:tasks.workunit.client.1.smithi137.stdout:1/13: read f0 [507369,113306] 0 2024-03-14T13:01:57.051 INFO:tasks.workunit.client.1.smithi137.stdout:7/7: dwrite f1 [0,4194304] 0 2024-03-14T13:01:57.052 INFO:tasks.workunit.client.1.smithi137.stdout:9/4: symlink l3 0 2024-03-14T13:01:57.053 INFO:tasks.workunit.client.1.smithi137.stdout:1/14: chown f0 2 1 2024-03-14T13:01:57.054 INFO:tasks.workunit.client.1.smithi137.stdout:5/16: rmdir d3 0 2024-03-14T13:01:57.055 INFO:tasks.workunit.client.1.smithi137.stdout:8/10: truncate f0 358113 0 2024-03-14T13:01:57.059 INFO:tasks.workunit.client.1.smithi137.stdout:9/5: rename f2 to f4 0 2024-03-14T13:01:57.059 INFO:tasks.workunit.client.1.smithi137.stdout:9/6: stat f4 0 2024-03-14T13:01:57.060 INFO:tasks.workunit.client.1.smithi137.stdout:9/7: chown l3 0 1 2024-03-14T13:01:57.060 INFO:tasks.workunit.client.1.smithi137.stdout:7/8: dread f1 [0,4194304] 0 2024-03-14T13:01:57.062 INFO:tasks.workunit.client.1.smithi137.stdout:9/8: fdatasync f1 0 2024-03-14T13:01:57.062 INFO:tasks.workunit.client.1.smithi137.stdout:1/15: creat f2 x:0 0 0 2024-03-14T13:01:57.063 INFO:tasks.workunit.client.1.smithi137.stdout:7/9: stat f0 0 2024-03-14T13:01:57.064 INFO:tasks.workunit.client.1.smithi137.stdout:5/17: creat f4 x:0 0 0 2024-03-14T13:01:57.064 INFO:tasks.workunit.client.1.smithi137.stdout:7/10: chown f1 61731 1 2024-03-14T13:01:57.064 INFO:tasks.workunit.client.1.smithi137.stdout:1/16: truncate f2 1030296 0 2024-03-14T13:01:57.064 INFO:tasks.workunit.client.1.smithi137.stdout:1/17: rmdir - no directory 2024-03-14T13:01:57.068 INFO:tasks.workunit.client.1.smithi137.stdout:5/18: read - f4 zero size 2024-03-14T13:01:57.070 INFO:tasks.workunit.client.1.smithi137.stdout:7/11: write f1 [4220286,47809] 0 2024-03-14T13:01:57.072 INFO:tasks.workunit.client.1.smithi137.stdout:5/19: creat f5 x:0 0 0 2024-03-14T13:01:57.072 INFO:tasks.workunit.client.1.smithi137.stdout:5/20: stat f4 0 2024-03-14T13:01:57.076 INFO:tasks.workunit.client.1.smithi137.stdout:9/9: rename f4 to f5 0 2024-03-14T13:01:57.080 INFO:tasks.workunit.client.1.smithi137.stdout:9/10: read - f1 zero size 2024-03-14T13:01:57.081 INFO:tasks.workunit.client.1.smithi137.stdout:1/18: dwrite f2 [0,4194304] 0 2024-03-14T13:01:57.082 INFO:tasks.workunit.client.1.smithi137.stdout:8/11: write f0 [959868,116485] 0 2024-03-14T13:01:57.083 INFO:tasks.workunit.client.1.smithi137.stdout:9/11: mkdir d6 0 2024-03-14T13:01:57.090 INFO:tasks.workunit.client.1.smithi137.stdout:7/12: dwrite f1 [0,4194304] 0 2024-03-14T13:01:57.090 INFO:tasks.workunit.client.1.smithi137.stdout:9/12: write f5 [780181,71376] 0 2024-03-14T13:01:57.091 INFO:tasks.workunit.client.1.smithi137.stdout:8/12: dread f0 [0,4194304] 0 2024-03-14T13:01:57.092 INFO:tasks.workunit.client.1.smithi137.stdout:9/13: chown l3 55 1 2024-03-14T13:01:57.094 INFO:tasks.workunit.client.1.smithi137.stdout:7/13: rename f0 to f2 0 2024-03-14T13:01:57.095 INFO:tasks.workunit.client.1.smithi137.stdout:7/14: dread - f2 zero size 2024-03-14T13:01:57.096 INFO:tasks.workunit.client.1.smithi137.stdout:9/14: creat d6/f7 x:0 0 0 2024-03-14T13:01:57.096 INFO:tasks.workunit.client.1.smithi137.stdout:1/19: dwrite f2 [0,4194304] 0 2024-03-14T13:01:57.099 INFO:tasks.workunit.client.1.smithi137.stdout:9/15: mknod d6/c8 0 2024-03-14T13:01:57.099 INFO:tasks.workunit.client.1.smithi137.stdout:1/20: creat f3 x:0 0 0 2024-03-14T13:01:57.100 INFO:tasks.workunit.client.1.smithi137.stdout:9/16: mknod d6/c9 0 2024-03-14T13:01:57.101 INFO:tasks.workunit.client.1.smithi137.stdout:1/21: creat f4 x:0 0 0 2024-03-14T13:01:57.103 INFO:tasks.workunit.client.1.smithi137.stdout:9/17: rename f5 to d6/fa 0 2024-03-14T13:01:57.107 INFO:tasks.workunit.client.1.smithi137.stdout:1/22: rename f2 to f5 0 2024-03-14T13:01:57.107 INFO:tasks.workunit.client.1.smithi137.stdout:1/23: rmdir - no directory 2024-03-14T13:01:57.109 INFO:tasks.workunit.client.1.smithi137.stdout:1/24: write f3 [84671,20162] 0 2024-03-14T13:01:57.111 INFO:tasks.workunit.client.1.smithi137.stdout:1/25: write f5 [4326088,5161] 0 2024-03-14T13:01:57.114 INFO:tasks.workunit.client.1.smithi137.stdout:9/18: dwrite f1 [0,4194304] 0 2024-03-14T13:01:57.114 INFO:tasks.workunit.client.1.smithi137.stdout:1/26: unlink f0 0 2024-03-14T13:01:57.115 INFO:tasks.workunit.client.1.smithi137.stdout:9/19: chown f1 125 1 2024-03-14T13:01:57.116 INFO:tasks.workunit.client.1.smithi137.stdout:9/20: readlink l3 0 2024-03-14T13:01:57.118 INFO:tasks.workunit.client.1.smithi137.stdout:9/21: mknod d6/cb 0 2024-03-14T13:01:57.120 INFO:tasks.workunit.client.1.smithi137.stdout:9/22: unlink l3 0 2024-03-14T13:01:57.122 INFO:tasks.workunit.client.1.smithi137.stdout:1/27: fsync f4 0 2024-03-14T13:01:57.130 INFO:tasks.workunit.client.1.smithi137.stdout:1/28: dwrite f5 [0,4194304] 0 2024-03-14T13:01:57.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/29: dread f3 [0,4194304] 0 2024-03-14T13:01:57.133 INFO:tasks.workunit.client.1.smithi137.stdout:9/23: fdatasync f1 0 2024-03-14T13:01:57.135 INFO:tasks.workunit.client.1.smithi137.stdout:9/24: write f1 [4584376,105860] 0 2024-03-14T13:01:57.135 INFO:tasks.workunit.client.1.smithi137.stdout:1/30: creat f6 x:0 0 0 2024-03-14T13:01:57.135 INFO:tasks.workunit.client.1.smithi137.stdout:9/25: stat d6/c8 0 2024-03-14T13:01:57.140 INFO:tasks.workunit.client.1.smithi137.stdout:9/26: link d6/fa d6/fc 0 2024-03-14T13:01:57.142 INFO:tasks.workunit.client.1.smithi137.stdout:1/31: dwrite f5 [0,4194304] 0 2024-03-14T13:01:57.142 INFO:tasks.workunit.client.1.smithi137.stdout:1/32: dread - f6 zero size 2024-03-14T13:01:57.143 INFO:tasks.workunit.client.1.smithi137.stdout:1/33: stat f3 0 2024-03-14T13:01:57.143 INFO:tasks.workunit.client.1.smithi137.stdout:1/34: rmdir - no directory 2024-03-14T13:01:57.143 INFO:tasks.workunit.client.1.smithi137.stdout:1/35: chown c1 88 1 2024-03-14T13:01:57.147 INFO:tasks.workunit.client.1.smithi137.stdout:9/27: rmdir d6 39 2024-03-14T13:01:57.147 INFO:tasks.workunit.client.1.smithi137.stdout:1/36: dread f5 [0,4194304] 0 2024-03-14T13:01:57.148 INFO:tasks.workunit.client.1.smithi137.stdout:9/28: write f1 [804552,96364] 0 2024-03-14T13:01:57.149 INFO:tasks.workunit.client.1.smithi137.stdout:1/37: write f6 [497291,89155] 0 2024-03-14T13:01:57.149 INFO:tasks.workunit.client.1.smithi137.stdout:9/29: readlink - no filename 2024-03-14T13:01:57.149 INFO:tasks.workunit.client.1.smithi137.stdout:1/38: stat f5 0 2024-03-14T13:01:57.149 INFO:tasks.workunit.client.1.smithi137.stdout:1/39: rmdir - no directory 2024-03-14T13:01:57.151 INFO:tasks.workunit.client.1.smithi137.stdout:1/40: mkdir d7 0 2024-03-14T13:01:57.153 INFO:tasks.workunit.client.1.smithi137.stdout:1/41: symlink d7/l8 0 2024-03-14T13:01:57.155 INFO:tasks.workunit.client.1.smithi137.stdout:1/42: mkdir d7/d9 0 2024-03-14T13:01:57.155 INFO:tasks.workunit.client.1.smithi137.stdout:1/43: fsync f6 0 2024-03-14T13:01:57.159 INFO:tasks.workunit.client.1.smithi137.stdout:1/44: link f3 d7/fa 0 2024-03-14T13:01:57.162 INFO:tasks.workunit.client.1.smithi137.stdout:9/30: dwrite d6/fc [0,4194304] 0 2024-03-14T13:01:57.164 INFO:tasks.workunit.client.1.smithi137.stdout:9/31: rmdir d6 39 2024-03-14T13:01:57.172 INFO:tasks.workunit.client.1.smithi137.stdout:9/32: dwrite d6/f7 [0,4194304] 0 2024-03-14T13:01:57.173 INFO:tasks.workunit.client.1.smithi137.stdout:9/33: rmdir d6 39 2024-03-14T13:01:57.182 INFO:tasks.workunit.client.1.smithi137.stdout:9/34: dwrite d6/fa [4194304,4194304] 0 2024-03-14T13:01:57.189 INFO:tasks.workunit.client.1.smithi137.stdout:9/35: dread d6/f7 [0,4194304] 0 2024-03-14T13:01:57.192 INFO:tasks.workunit.client.1.smithi137.stdout:9/36: getdents d6 0 2024-03-14T13:01:57.202 INFO:tasks.workunit.client.1.smithi137.stdout:9/37: truncate d6/fc 3819683 0 2024-03-14T13:01:57.204 INFO:tasks.workunit.client.1.smithi137.stdout:9/38: read d6/fa [963473,91429] 0 2024-03-14T13:01:57.208 INFO:tasks.workunit.client.1.smithi137.stdout:9/39: dread f1 [0,4194304] 0 2024-03-14T13:01:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:57 smithi137 ceph-mon[50336]: pgmap v13: 65 pgs: 65 active+clean; 262 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 12 MiB/s wr, 399 op/s 2024-03-14T13:01:58.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:57 smithi111 ceph-mon[39688]: pgmap v13: 65 pgs: 65 active+clean; 262 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 12 MiB/s wr, 399 op/s 2024-03-14T13:02:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:01:59 smithi137 ceph-mon[50336]: pgmap v14: 65 pgs: 65 active+clean; 306 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 18 MiB/s wr, 517 op/s 2024-03-14T13:02:00.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:01:59 smithi111 ceph-mon[39688]: pgmap v14: 65 pgs: 65 active+clean; 306 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 18 MiB/s wr, 517 op/s 2024-03-14T13:02:01.583 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:01 smithi111 ceph-mon[39688]: pgmap v15: 65 pgs: 65 active+clean; 306 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 16 MiB/s wr, 312 op/s 2024-03-14T13:02:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:01 smithi137 ceph-mon[50336]: pgmap v15: 65 pgs: 65 active+clean; 306 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 16 MiB/s wr, 312 op/s 2024-03-14T13:02:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:03 smithi137 ceph-mon[50336]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 321 op/s 2024-03-14T13:02:04.039 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:03 smithi111 ceph-mon[39688]: pgmap v16: 65 pgs: 65 active+clean; 310 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 321 op/s 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:8/24: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:2/15: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:3/3: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:9/22: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:7/37: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:1/5: sync 2024-03-14T13:02:05.555 INFO:tasks.workunit.client.0.smithi111.stdout:0/31: sync 2024-03-14T13:02:05.556 INFO:tasks.workunit.client.0.smithi111.stdout:4/49: sync 2024-03-14T13:02:05.556 INFO:tasks.workunit.client.0.smithi111.stdout:5/106: sync 2024-03-14T13:02:05.556 INFO:tasks.workunit.client.0.smithi111.stdout:6/41: sync 2024-03-14T13:02:05.557 INFO:tasks.workunit.client.0.smithi111.stdout:9/23: mkdir d5 0 2024-03-14T13:02:05.565 INFO:tasks.workunit.client.0.smithi111.stdout:3/4: mknod d0/c2 0 2024-03-14T13:02:05.569 INFO:tasks.workunit.client.0.smithi111.stdout:7/38: chown d0/d1/f2 3 1 2024-03-14T13:02:05.572 INFO:tasks.workunit.client.0.smithi111.stdout:8/25: truncate f0 2439411 0 2024-03-14T13:02:05.573 INFO:tasks.workunit.client.0.smithi111.stdout:4/50: rename f14 to f15 0 2024-03-14T13:02:05.574 INFO:tasks.workunit.client.0.smithi111.stdout:2/16: write f1 [3040018,25781] 0 2024-03-14T13:02:05.575 INFO:tasks.workunit.client.0.smithi111.stdout:5/107: fdatasync d2/d19/f12 0 2024-03-14T13:02:05.576 INFO:tasks.workunit.client.0.smithi111.stdout:0/32: write f2 [1944212,122993] 0 2024-03-14T13:02:05.577 INFO:tasks.workunit.client.0.smithi111.stdout:3/5: creat d0/f3 x:0 0 0 2024-03-14T13:02:05.577 INFO:tasks.workunit.client.0.smithi111.stdout:1/6: symlink l0 0 2024-03-14T13:02:05.578 INFO:tasks.workunit.client.0.smithi111.stdout:1/7: read - no filename 2024-03-14T13:02:05.579 INFO:tasks.workunit.client.0.smithi111.stdout:6/42: write d0/f4 [620636,60545] 0 2024-03-14T13:02:05.584 INFO:tasks.workunit.client.0.smithi111.stdout:2/17: symlink l6 0 2024-03-14T13:02:05.586 INFO:tasks.workunit.client.0.smithi111.stdout:9/24: mkdir d5/d6 0 2024-03-14T13:02:05.586 INFO:tasks.workunit.client.0.smithi111.stdout:5/108: dread d2/d19/f12 [0,4194304] 0 2024-03-14T13:02:05.587 INFO:tasks.workunit.client.0.smithi111.stdout:0/33: mkdir d6 0 2024-03-14T13:02:05.588 INFO:tasks.workunit.client.0.smithi111.stdout:3/6: creat d0/f4 x:0 0 0 2024-03-14T13:02:05.591 INFO:tasks.workunit.client.0.smithi111.stdout:8/26: mknod d3/d5/c6 0 2024-03-14T13:02:05.592 INFO:tasks.workunit.client.0.smithi111.stdout:8/27: rename d3 to d3/d5/d7 22 2024-03-14T13:02:05.596 INFO:tasks.workunit.client.0.smithi111.stdout:5/109: symlink d2/d1f/l26 0 2024-03-14T13:02:05.597 INFO:tasks.workunit.client.0.smithi111.stdout:0/34: fsync f0 0 2024-03-14T13:02:05.599 INFO:tasks.workunit.client.0.smithi111.stdout:2/18: dwrite f1 [0,4194304] 0 2024-03-14T13:02:05.599 INFO:tasks.workunit.client.0.smithi111.stdout:7/39: link d0/d1/f2 d0/f7 0 2024-03-14T13:02:05.601 INFO:tasks.workunit.client.0.smithi111.stdout:0/35: read f2 [209526,48631] 0 2024-03-14T13:02:05.601 INFO:tasks.workunit.client.0.smithi111.stdout:9/25: creat d5/d6/f7 x:0 0 0 2024-03-14T13:02:05.604 INFO:tasks.workunit.client.0.smithi111.stdout:9/26: chown c1 46343 1 2024-03-14T13:02:05.604 INFO:tasks.workunit.client.0.smithi111.stdout:0/36: read - f5 zero size 2024-03-14T13:02:05.604 INFO:tasks.workunit.client.0.smithi111.stdout:0/37: readlink - no filename 2024-03-14T13:02:05.607 INFO:tasks.workunit.client.0.smithi111.stdout:9/27: dread - d5/d6/f7 zero size 2024-03-14T13:02:05.607 INFO:tasks.workunit.client.0.smithi111.stdout:6/43: dread d0/fa [0,4194304] 0 2024-03-14T13:02:05.608 INFO:tasks.workunit.client.0.smithi111.stdout:3/7: dwrite d0/f4 [0,4194304] 0 2024-03-14T13:02:05.612 INFO:tasks.workunit.client.0.smithi111.stdout:9/28: truncate d5/d6/f7 790348 0 2024-03-14T13:02:05.613 INFO:tasks.workunit.client.0.smithi111.stdout:2/19: mknod c7 0 2024-03-14T13:02:05.613 INFO:tasks.workunit.client.0.smithi111.stdout:9/29: chown f4 1678485281 1 2024-03-14T13:02:05.613 INFO:tasks.workunit.client.0.smithi111.stdout:9/30: readlink - no filename 2024-03-14T13:02:05.615 INFO:tasks.workunit.client.0.smithi111.stdout:1/8: link l0 l1 0 2024-03-14T13:02:05.615 INFO:tasks.workunit.client.0.smithi111.stdout:1/9: dwrite - no filename 2024-03-14T13:02:05.615 INFO:tasks.workunit.client.0.smithi111.stdout:1/10: dwrite - no filename 2024-03-14T13:02:05.615 INFO:tasks.workunit.client.0.smithi111.stdout:1/11: dwrite - no filename 2024-03-14T13:02:05.617 INFO:tasks.workunit.client.0.smithi111.stdout:7/40: symlink d0/d1/l8 0 2024-03-14T13:02:05.618 INFO:tasks.workunit.client.0.smithi111.stdout:5/110: truncate d2/f7 2069712 0 2024-03-14T13:02:05.619 INFO:tasks.workunit.client.0.smithi111.stdout:9/31: truncate d5/d6/f7 1050248 0 2024-03-14T13:02:05.621 INFO:tasks.workunit.client.0.smithi111.stdout:5/111: read d2/d19/f11 [4351884,55778] 0 2024-03-14T13:02:05.621 INFO:tasks.workunit.client.0.smithi111.stdout:4/51: dwrite f15 [0,4194304] 0 2024-03-14T13:02:05.626 INFO:tasks.workunit.client.0.smithi111.stdout:0/38: unlink f0 0 2024-03-14T13:02:05.628 INFO:tasks.workunit.client.0.smithi111.stdout:3/8: fdatasync d0/f1 0 2024-03-14T13:02:05.629 INFO:tasks.workunit.client.0.smithi111.stdout:2/20: fdatasync f4 0 2024-03-14T13:02:05.630 INFO:tasks.workunit.client.0.smithi111.stdout:2/21: fdatasync f1 0 2024-03-14T13:02:05.630 INFO:tasks.workunit.client.0.smithi111.stdout:2/22: rmdir - no directory 2024-03-14T13:02:05.632 INFO:tasks.workunit.client.0.smithi111.stdout:1/12: chown l0 3703369 1 2024-03-14T13:02:05.632 INFO:tasks.workunit.client.0.smithi111.stdout:4/52: stat f9 0 2024-03-14T13:02:05.633 INFO:tasks.workunit.client.0.smithi111.stdout:4/53: readlink l6 0 2024-03-14T13:02:05.633 INFO:tasks.workunit.client.0.smithi111.stdout:9/32: mkdir d5/d6/d8 0 2024-03-14T13:02:05.634 INFO:tasks.workunit.client.0.smithi111.stdout:9/33: chown c3 2757 1 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/13: stat l1 0 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/14: dwrite - no filename 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/15: readlink l0 0 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/16: write - no filename 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/17: truncate - no filename 2024-03-14T13:02:05.635 INFO:tasks.workunit.client.0.smithi111.stdout:1/18: dwrite - no filename 2024-03-14T13:02:05.636 INFO:tasks.workunit.client.0.smithi111.stdout:1/19: dwrite - no filename 2024-03-14T13:02:05.637 INFO:tasks.workunit.client.0.smithi111.stdout:9/34: write d5/d6/f7 [983455,42537] 0 2024-03-14T13:02:05.637 INFO:tasks.workunit.client.0.smithi111.stdout:7/41: mkdir d0/d6/d9 0 2024-03-14T13:02:05.638 INFO:tasks.workunit.client.0.smithi111.stdout:2/23: mknod c8 0 2024-03-14T13:02:05.639 INFO:tasks.workunit.client.0.smithi111.stdout:2/24: chown l5 256783714 1 2024-03-14T13:02:05.642 INFO:tasks.workunit.client.0.smithi111.stdout:7/42: readlink d0/d1/l8 0 2024-03-14T13:02:05.643 INFO:tasks.workunit.client.0.smithi111.stdout:0/39: symlink d6/l7 0 2024-03-14T13:02:05.643 INFO:tasks.workunit.client.0.smithi111.stdout:7/43: rename d0/d1 to d0/d1/da 22 2024-03-14T13:02:05.643 INFO:tasks.workunit.client.0.smithi111.stdout:1/20: mknod c2 0 2024-03-14T13:02:05.644 INFO:tasks.workunit.client.0.smithi111.stdout:9/35: creat d5/d6/f9 x:0 0 0 2024-03-14T13:02:05.644 INFO:tasks.workunit.client.0.smithi111.stdout:3/9: dread d0/f4 [0,4194304] 0 2024-03-14T13:02:05.644 INFO:tasks.workunit.client.0.smithi111.stdout:0/40: chown f3 8592 1 2024-03-14T13:02:05.645 INFO:tasks.workunit.client.0.smithi111.stdout:2/25: write f4 [1314933,74024] 0 2024-03-14T13:02:05.645 INFO:tasks.workunit.client.0.smithi111.stdout:0/41: stat f1 0 2024-03-14T13:02:05.646 INFO:tasks.workunit.client.0.smithi111.stdout:6/44: truncate d0/f4 197234 0 2024-03-14T13:02:05.648 INFO:tasks.workunit.client.0.smithi111.stdout:1/21: creat f3 x:0 0 0 2024-03-14T13:02:05.651 INFO:tasks.workunit.client.0.smithi111.stdout:7/44: dread d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:05.652 INFO:tasks.workunit.client.0.smithi111.stdout:0/42: rename d6/l7 to d6/l8 0 2024-03-14T13:02:05.654 INFO:tasks.workunit.client.0.smithi111.stdout:5/112: fdatasync d2/f7 0 2024-03-14T13:02:05.656 INFO:tasks.workunit.client.0.smithi111.stdout:1/22: symlink l4 0 2024-03-14T13:02:05.656 INFO:tasks.workunit.client.0.smithi111.stdout:2/26: dwrite f1 [0,4194304] 0 2024-03-14T13:02:05.658 INFO:tasks.workunit.client.0.smithi111.stdout:1/23: truncate f3 428265 0 2024-03-14T13:02:05.659 INFO:tasks.workunit.client.0.smithi111.stdout:6/45: truncate d0/f6 1085864 0 2024-03-14T13:02:05.660 INFO:tasks.workunit.client.0.smithi111.stdout:3/10: truncate d0/f1 702780 0 2024-03-14T13:02:05.662 INFO:tasks.workunit.client.0.smithi111.stdout:7/45: fsync d0/d1/f2 0 2024-03-14T13:02:05.663 INFO:tasks.workunit.client.0.smithi111.stdout:0/43: fdatasync f1 0 2024-03-14T13:02:05.666 INFO:tasks.workunit.client.0.smithi111.stdout:7/46: read d0/d1/f2 [105281,25214] 0 2024-03-14T13:02:05.667 INFO:tasks.workunit.client.0.smithi111.stdout:1/24: rename l0 to l5 0 2024-03-14T13:02:05.668 INFO:tasks.workunit.client.0.smithi111.stdout:1/25: write f3 [351517,69010] 0 2024-03-14T13:02:05.669 INFO:tasks.workunit.client.0.smithi111.stdout:7/47: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:05.669 INFO:tasks.workunit.client.0.smithi111.stdout:1/26: readlink l4 0 2024-03-14T13:02:05.670 INFO:tasks.workunit.client.0.smithi111.stdout:7/48: chown d0/d1 3 1 2024-03-14T13:02:05.675 INFO:tasks.workunit.client.0.smithi111.stdout:0/44: mkdir d6/d9 0 2024-03-14T13:02:05.676 INFO:tasks.workunit.client.0.smithi111.stdout:0/45: chown f2 697 1 2024-03-14T13:02:05.676 INFO:tasks.workunit.client.0.smithi111.stdout:1/27: dread f3 [0,4194304] 0 2024-03-14T13:02:05.677 INFO:tasks.workunit.client.0.smithi111.stdout:3/11: fsync d0/f1 0 2024-03-14T13:02:05.683 INFO:tasks.workunit.client.0.smithi111.stdout:0/46: mknod d6/ca 0 2024-03-14T13:02:05.683 INFO:tasks.workunit.client.0.smithi111.stdout:5/113: dwrite d2/d1c/f22 [0,4194304] 0 2024-03-14T13:02:05.685 INFO:tasks.workunit.client.0.smithi111.stdout:1/28: symlink l6 0 2024-03-14T13:02:05.685 INFO:tasks.workunit.client.0.smithi111.stdout:6/46: chown d0/f6 1 1 2024-03-14T13:02:05.689 INFO:tasks.workunit.client.0.smithi111.stdout:3/12: creat d0/f5 x:0 0 0 2024-03-14T13:02:05.690 INFO:tasks.workunit.client.0.smithi111.stdout:3/13: write d0/f1 [1148435,86801] 0 2024-03-14T13:02:05.692 INFO:tasks.workunit.client.0.smithi111.stdout:0/47: creat d6/fb x:0 0 0 2024-03-14T13:02:05.693 INFO:tasks.workunit.client.0.smithi111.stdout:5/114: rename f1 to d2/d19/f27 0 2024-03-14T13:02:05.694 INFO:tasks.workunit.client.0.smithi111.stdout:1/29: symlink l7 0 2024-03-14T13:02:05.698 INFO:tasks.workunit.client.0.smithi111.stdout:3/14: rename d0/f3 to d0/f6 0 2024-03-14T13:02:05.698 INFO:tasks.workunit.client.0.smithi111.stdout:7/49: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:05.699 INFO:tasks.workunit.client.0.smithi111.stdout:7/50: chown d0/d6 9436326 1 2024-03-14T13:02:05.699 INFO:tasks.workunit.client.0.smithi111.stdout:0/48: mknod d6/cc 0 2024-03-14T13:02:05.703 INFO:tasks.workunit.client.0.smithi111.stdout:5/115: creat d2/d1f/f28 x:0 0 0 2024-03-14T13:02:05.709 INFO:tasks.workunit.client.0.smithi111.stdout:6/47: dread d0/f6 [0,4194304] 0 2024-03-14T13:02:05.710 INFO:tasks.workunit.client.0.smithi111.stdout:5/116: fdatasync d2/d13/f1b 0 2024-03-14T13:02:05.711 INFO:tasks.workunit.client.0.smithi111.stdout:7/51: creat d0/d6/d9/fb x:0 0 0 2024-03-14T13:02:05.712 INFO:tasks.workunit.client.0.smithi111.stdout:6/48: chown d0/f6 5 1 2024-03-14T13:02:05.712 INFO:tasks.workunit.client.0.smithi111.stdout:5/117: chown d2 229694 1 2024-03-14T13:02:05.715 INFO:tasks.workunit.client.0.smithi111.stdout:7/52: write d0/f7 [4822745,93201] 0 2024-03-14T13:02:05.716 INFO:tasks.workunit.client.0.smithi111.stdout:6/49: mknod d0/cc 0 2024-03-14T13:02:05.716 INFO:tasks.workunit.client.0.smithi111.stdout:1/30: dwrite f3 [0,4194304] 0 2024-03-14T13:02:05.719 INFO:tasks.workunit.client.0.smithi111.stdout:7/53: truncate d0/d6/d9/fb 872253 0 2024-03-14T13:02:05.719 INFO:tasks.workunit.client.0.smithi111.stdout:5/118: creat d2/d13/f29 x:0 0 0 2024-03-14T13:02:05.721 INFO:tasks.workunit.client.0.smithi111.stdout:3/15: getdents d0 0 2024-03-14T13:02:05.722 INFO:tasks.workunit.client.0.smithi111.stdout:0/49: getdents d6 0 2024-03-14T13:02:05.724 INFO:tasks.workunit.client.0.smithi111.stdout:3/16: stat d0/c2 0 2024-03-14T13:02:05.724 INFO:tasks.workunit.client.0.smithi111.stdout:6/50: symlink d0/ld 0 2024-03-14T13:02:05.728 INFO:tasks.workunit.client.0.smithi111.stdout:0/50: creat d6/fd x:0 0 0 2024-03-14T13:02:05.728 INFO:tasks.workunit.client.0.smithi111.stdout:7/54: dread d0/d6/d9/fb [0,4194304] 0 2024-03-14T13:02:05.731 INFO:tasks.workunit.client.0.smithi111.stdout:3/17: creat d0/f7 x:0 0 0 2024-03-14T13:02:05.732 INFO:tasks.workunit.client.0.smithi111.stdout:0/51: write d6/fd [477444,91715] 0 2024-03-14T13:02:05.732 INFO:tasks.workunit.client.0.smithi111.stdout:1/31: dwrite f3 [0,4194304] 0 2024-03-14T13:02:05.733 INFO:tasks.workunit.client.0.smithi111.stdout:5/119: write d2/f6 [3546058,20579] 0 2024-03-14T13:02:05.735 INFO:tasks.workunit.client.0.smithi111.stdout:3/18: creat d0/f8 x:0 0 0 2024-03-14T13:02:05.735 INFO:tasks.workunit.client.0.smithi111.stdout:0/52: fdatasync d6/fb 0 2024-03-14T13:02:05.738 INFO:tasks.workunit.client.0.smithi111.stdout:5/120: symlink d2/d1c/l2a 0 2024-03-14T13:02:05.742 INFO:tasks.workunit.client.0.smithi111.stdout:5/121: creat d2/d13/f2b x:0 0 0 2024-03-14T13:02:05.743 INFO:tasks.workunit.client.0.smithi111.stdout:1/32: link l7 l8 0 2024-03-14T13:02:05.743 INFO:tasks.workunit.client.0.smithi111.stdout:6/51: write d0/f6 [23257,12205] 0 2024-03-14T13:02:05.744 INFO:tasks.workunit.client.0.smithi111.stdout:7/55: truncate d0/d6/d9/fb 716598 0 2024-03-14T13:02:05.744 INFO:tasks.workunit.client.0.smithi111.stdout:8/28: sync 2024-03-14T13:02:05.745 INFO:tasks.workunit.client.0.smithi111.stdout:0/53: mkdir d6/d9/de 0 2024-03-14T13:02:05.745 INFO:tasks.workunit.client.0.smithi111.stdout:7/56: fdatasync d0/f7 0 2024-03-14T13:02:05.747 INFO:tasks.workunit.client.0.smithi111.stdout:5/122: chown d2/d19/cc 305687329 1 2024-03-14T13:02:05.747 INFO:tasks.workunit.client.0.smithi111.stdout:1/33: creat f9 x:0 0 0 2024-03-14T13:02:05.747 INFO:tasks.workunit.client.0.smithi111.stdout:6/52: unlink d0/c8 0 2024-03-14T13:02:05.750 INFO:tasks.workunit.client.0.smithi111.stdout:8/29: symlink d3/l8 0 2024-03-14T13:02:05.752 INFO:tasks.workunit.client.0.smithi111.stdout:5/123: dread - d2/d1c/f20 zero size 2024-03-14T13:02:05.755 INFO:tasks.workunit.client.0.smithi111.stdout:0/54: getdents d6/d9/de 0 2024-03-14T13:02:05.757 INFO:tasks.workunit.client.0.smithi111.stdout:3/19: dwrite d0/f8 [0,4194304] 0 2024-03-14T13:02:05.758 INFO:tasks.workunit.client.0.smithi111.stdout:3/20: chown d0/c2 1949 1 2024-03-14T13:02:05.759 INFO:tasks.workunit.client.0.smithi111.stdout:6/53: truncate d0/f6 1141002 0 2024-03-14T13:02:05.760 INFO:tasks.workunit.client.0.smithi111.stdout:5/124: mkdir d2/d16/d2c 0 2024-03-14T13:02:05.762 INFO:tasks.workunit.client.0.smithi111.stdout:3/21: mkdir d0/d9 0 2024-03-14T13:02:05.764 INFO:tasks.workunit.client.0.smithi111.stdout:5/125: chown d2/d19 0 1 2024-03-14T13:02:05.765 INFO:tasks.workunit.client.0.smithi111.stdout:7/57: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:05.767 INFO:tasks.workunit.client.0.smithi111.stdout:6/54: dread d0/f6 [0,4194304] 0 2024-03-14T13:02:05.768 INFO:tasks.workunit.client.0.smithi111.stdout:5/126: fdatasync d2/d1c/f20 0 2024-03-14T13:02:05.770 INFO:tasks.workunit.client.0.smithi111.stdout:3/22: rmdir d0/d9 0 2024-03-14T13:02:05.770 INFO:tasks.workunit.client.0.smithi111.stdout:3/23: dread - d0/f5 zero size 2024-03-14T13:02:05.771 INFO:tasks.workunit.client.0.smithi111.stdout:6/55: mknod d0/ce 0 2024-03-14T13:02:05.771 INFO:tasks.workunit.client.0.smithi111.stdout:3/24: dread - d0/f5 zero size 2024-03-14T13:02:05.771 INFO:tasks.workunit.client.0.smithi111.stdout:5/127: mknod d2/d16/c2d 0 2024-03-14T13:02:05.772 INFO:tasks.workunit.client.0.smithi111.stdout:6/56: dread d0/fa [0,4194304] 0 2024-03-14T13:02:05.774 INFO:tasks.workunit.client.0.smithi111.stdout:3/25: symlink d0/la 0 2024-03-14T13:02:05.780 INFO:tasks.workunit.client.0.smithi111.stdout:6/57: link d0/fa d0/ff 0 2024-03-14T13:02:05.782 INFO:tasks.workunit.client.0.smithi111.stdout:6/58: stat d0/l9 0 2024-03-14T13:02:05.784 INFO:tasks.workunit.client.0.smithi111.stdout:0/55: dwrite f2 [0,4194304] 0 2024-03-14T13:02:05.787 INFO:tasks.workunit.client.0.smithi111.stdout:7/58: truncate d0/d6/d9/fb 735838 0 2024-03-14T13:02:05.790 INFO:tasks.workunit.client.0.smithi111.stdout:5/128: write d2/d19/f27 [2779603,41151] 0 2024-03-14T13:02:05.790 INFO:tasks.workunit.client.0.smithi111.stdout:3/26: getdents d0 0 2024-03-14T13:02:05.790 INFO:tasks.workunit.client.0.smithi111.stdout:6/59: write d0/fa [2177921,47545] 0 2024-03-14T13:02:05.795 INFO:tasks.workunit.client.0.smithi111.stdout:5/129: chown d2/f6 316 1 2024-03-14T13:02:05.797 INFO:tasks.workunit.client.0.smithi111.stdout:3/27: symlink d0/lb 0 2024-03-14T13:02:05.798 INFO:tasks.workunit.client.0.smithi111.stdout:5/130: chown d2/d19/f12 0 1 2024-03-14T13:02:05.799 INFO:tasks.workunit.client.0.smithi111.stdout:0/56: dwrite d6/fb [0,4194304] 0 2024-03-14T13:02:05.801 INFO:tasks.workunit.client.0.smithi111.stdout:5/131: mknod d2/d1c/c2e 0 2024-03-14T13:02:05.801 INFO:tasks.workunit.client.0.smithi111.stdout:0/57: write f1 [130225,13731] 0 2024-03-14T13:02:05.803 INFO:tasks.workunit.client.0.smithi111.stdout:5/132: chown d2/d1c/c23 275069 1 2024-03-14T13:02:05.808 INFO:tasks.workunit.client.0.smithi111.stdout:6/60: dwrite d0/f6 [0,4194304] 0 2024-03-14T13:02:05.809 INFO:tasks.workunit.client.0.smithi111.stdout:3/28: dread d0/f4 [0,4194304] 0 2024-03-14T13:02:05.809 INFO:tasks.workunit.client.0.smithi111.stdout:7/59: truncate d0/f7 3742528 0 2024-03-14T13:02:05.810 INFO:tasks.workunit.client.0.smithi111.stdout:0/58: dread d6/fd [0,4194304] 0 2024-03-14T13:02:05.812 INFO:tasks.workunit.client.0.smithi111.stdout:5/133: rename d2/d13/f1b to d2/d13/f2f 0 2024-03-14T13:02:05.813 INFO:tasks.workunit.client.0.smithi111.stdout:5/134: readlink d2/d1f/l26 0 2024-03-14T13:02:05.813 INFO:tasks.workunit.client.0.smithi111.stdout:7/60: mknod d0/d6/d9/cc 0 2024-03-14T13:02:05.814 INFO:tasks.workunit.client.0.smithi111.stdout:0/59: mknod d6/d9/cf 0 2024-03-14T13:02:05.815 INFO:tasks.workunit.client.0.smithi111.stdout:3/29: rename d0/f5 to d0/fc 0 2024-03-14T13:02:05.816 INFO:tasks.workunit.client.0.smithi111.stdout:0/60: read f2 [1010374,128041] 0 2024-03-14T13:02:05.816 INFO:tasks.workunit.client.0.smithi111.stdout:3/30: dread - d0/fc zero size 2024-03-14T13:02:05.822 INFO:tasks.workunit.client.0.smithi111.stdout:6/61: getdents d0 0 2024-03-14T13:02:05.823 INFO:tasks.workunit.client.0.smithi111.stdout:3/31: link d0/f8 d0/fd 0 2024-03-14T13:02:05.825 INFO:tasks.workunit.client.0.smithi111.stdout:6/62: mknod d0/c10 0 2024-03-14T13:02:05.827 INFO:tasks.workunit.client.0.smithi111.stdout:3/32: mkdir d0/de 0 2024-03-14T13:02:05.828 INFO:tasks.workunit.client.0.smithi111.stdout:3/33: write d0/f7 [854162,80765] 0 2024-03-14T13:02:05.828 INFO:tasks.workunit.client.0.smithi111.stdout:3/34: read - d0/fc zero size 2024-03-14T13:02:05.829 INFO:tasks.workunit.client.0.smithi111.stdout:3/35: readlink d0/lb 0 2024-03-14T13:02:05.830 INFO:tasks.workunit.client.0.smithi111.stdout:5/135: write d2/d13/f25 [754842,73354] 0 2024-03-14T13:02:05.832 INFO:tasks.workunit.client.0.smithi111.stdout:3/36: creat d0/ff x:0 0 0 2024-03-14T13:02:05.833 INFO:tasks.workunit.client.0.smithi111.stdout:3/37: dread - d0/ff zero size 2024-03-14T13:02:05.833 INFO:tasks.workunit.client.0.smithi111.stdout:6/63: link d0/c5 d0/c11 0 2024-03-14T13:02:05.834 INFO:tasks.workunit.client.0.smithi111.stdout:0/61: truncate f2 3309016 0 2024-03-14T13:02:05.838 INFO:tasks.workunit.client.0.smithi111.stdout:7/61: link d0/d1/f2 d0/d1/fd 0 2024-03-14T13:02:05.839 INFO:tasks.workunit.client.0.smithi111.stdout:3/38: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:05.842 INFO:tasks.workunit.client.0.smithi111.stdout:6/64: dwrite d0/fa [0,4194304] 0 2024-03-14T13:02:05.845 INFO:tasks.workunit.client.0.smithi111.stdout:7/62: rename d0/d6 to d0/d1/d5/de 0 2024-03-14T13:02:05.847 INFO:tasks.workunit.client.0.smithi111.stdout:3/39: symlink d0/de/l10 0 2024-03-14T13:02:05.848 INFO:tasks.workunit.client.0.smithi111.stdout:6/65: mknod d0/c12 0 2024-03-14T13:02:05.850 INFO:tasks.workunit.client.0.smithi111.stdout:7/63: symlink d0/d1/lf 0 2024-03-14T13:02:05.851 INFO:tasks.workunit.client.0.smithi111.stdout:6/66: dread d0/f4 [0,4194304] 0 2024-03-14T13:02:05.851 INFO:tasks.workunit.client.0.smithi111.stdout:3/40: creat d0/f11 x:0 0 0 2024-03-14T13:02:05.852 INFO:tasks.workunit.client.0.smithi111.stdout:0/62: write f3 [95647,65826] 0 2024-03-14T13:02:05.854 INFO:tasks.workunit.client.0.smithi111.stdout:0/63: creat d6/f10 x:0 0 0 2024-03-14T13:02:05.855 INFO:tasks.workunit.client.0.smithi111.stdout:6/67: truncate d0/f4 1239520 0 2024-03-14T13:02:05.855 INFO:tasks.workunit.client.0.smithi111.stdout:3/41: write d0/fc [196974,84952] 0 2024-03-14T13:02:05.855 INFO:tasks.workunit.client.0.smithi111.stdout:0/64: chown d6/f10 427206 1 2024-03-14T13:02:05.857 INFO:tasks.workunit.client.0.smithi111.stdout:6/68: symlink d0/l13 0 2024-03-14T13:02:05.858 INFO:tasks.workunit.client.0.smithi111.stdout:6/69: truncate d0/ff 4944439 0 2024-03-14T13:02:05.859 INFO:tasks.workunit.client.0.smithi111.stdout:0/65: symlink d6/d9/l11 0 2024-03-14T13:02:05.860 INFO:tasks.workunit.client.0.smithi111.stdout:3/42: creat d0/de/f12 x:0 0 0 2024-03-14T13:02:05.861 INFO:tasks.workunit.client.0.smithi111.stdout:7/64: fdatasync d0/f7 0 2024-03-14T13:02:05.865 INFO:tasks.workunit.client.0.smithi111.stdout:7/65: dread d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:05.866 INFO:tasks.workunit.client.0.smithi111.stdout:3/43: symlink d0/de/l13 0 2024-03-14T13:02:05.870 INFO:tasks.workunit.client.0.smithi111.stdout:0/66: link f3 d6/d9/f12 0 2024-03-14T13:02:05.871 INFO:tasks.workunit.client.0.smithi111.stdout:7/66: creat d0/f10 x:0 0 0 2024-03-14T13:02:05.871 INFO:tasks.workunit.client.0.smithi111.stdout:7/67: chown d0 14104203 1 2024-03-14T13:02:05.872 INFO:tasks.workunit.client.0.smithi111.stdout:3/44: rmdir d0/de 39 2024-03-14T13:02:05.874 INFO:tasks.workunit.client.0.smithi111.stdout:0/67: creat d6/f13 x:0 0 0 2024-03-14T13:02:05.874 INFO:tasks.workunit.client.0.smithi111.stdout:0/68: chown d6/d9/f12 15 1 2024-03-14T13:02:05.874 INFO:tasks.workunit.client.0.smithi111.stdout:6/70: truncate d0/fa 2865156 0 2024-03-14T13:02:05.875 INFO:tasks.workunit.client.0.smithi111.stdout:7/68: rmdir d0/d1/d5/de/d9 39 2024-03-14T13:02:05.875 INFO:tasks.workunit.client.0.smithi111.stdout:7/69: dread - d0/f10 zero size 2024-03-14T13:02:05.879 INFO:tasks.workunit.client.0.smithi111.stdout:3/45: unlink d0/de/l10 0 2024-03-14T13:02:05.881 INFO:tasks.workunit.client.0.smithi111.stdout:0/69: mknod d6/d9/de/c14 0 2024-03-14T13:02:05.881 INFO:tasks.workunit.client.0.smithi111.stdout:0/70: chown d6/fd 78876 1 2024-03-14T13:02:05.882 INFO:tasks.workunit.client.0.smithi111.stdout:3/46: creat d0/de/f14 x:0 0 0 2024-03-14T13:02:05.882 INFO:tasks.workunit.client.0.smithi111.stdout:7/70: stat d0/d1/d5/de/d9/fb 0 2024-03-14T13:02:05.884 INFO:tasks.workunit.client.0.smithi111.stdout:7/71: mkdir d0/d11 0 2024-03-14T13:02:05.884 INFO:tasks.workunit.client.0.smithi111.stdout:7/72: stat d0/d1/fd 0 2024-03-14T13:02:05.886 INFO:tasks.workunit.client.0.smithi111.stdout:3/47: creat d0/f15 x:0 0 0 2024-03-14T13:02:05.888 INFO:tasks.workunit.client.0.smithi111.stdout:7/73: creat d0/d1/d5/de/d9/f12 x:0 0 0 2024-03-14T13:02:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:05 smithi111 ceph-mon[39688]: pgmap v17: 65 pgs: 65 active+clean; 307 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 297 op/s 2024-03-14T13:02:05.891 INFO:tasks.workunit.client.0.smithi111.stdout:3/48: symlink d0/l16 0 2024-03-14T13:02:05.892 INFO:tasks.workunit.client.0.smithi111.stdout:3/49: chown d0/f8 1 1 2024-03-14T13:02:05.893 INFO:tasks.workunit.client.0.smithi111.stdout:6/71: truncate d0/ff 3029438 0 2024-03-14T13:02:05.894 INFO:tasks.workunit.client.0.smithi111.stdout:6/72: chown d0/l9 796 1 2024-03-14T13:02:05.896 INFO:tasks.workunit.client.0.smithi111.stdout:3/50: rename d0/c2 to d0/de/c17 0 2024-03-14T13:02:05.897 INFO:tasks.workunit.client.0.smithi111.stdout:7/74: dwrite d0/d1/d5/de/d9/f12 [0,4194304] 0 2024-03-14T13:02:05.906 INFO:tasks.workunit.client.0.smithi111.stdout:7/75: link d0/d1/l8 d0/d1/d5/de/d9/l13 0 2024-03-14T13:02:05.909 INFO:tasks.workunit.client.0.smithi111.stdout:3/51: dwrite d0/de/f12 [0,4194304] 0 2024-03-14T13:02:05.912 INFO:tasks.workunit.client.0.smithi111.stdout:6/73: dwrite d0/fa [0,4194304] 0 2024-03-14T13:02:05.916 INFO:tasks.workunit.client.0.smithi111.stdout:6/74: creat d0/f14 x:0 0 0 2024-03-14T13:02:05.924 INFO:tasks.workunit.client.0.smithi111.stdout:7/76: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:05.924 INFO:tasks.workunit.client.0.smithi111.stdout:3/52: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:05.925 INFO:tasks.workunit.client.0.smithi111.stdout:3/53: chown d0/la 110070530 1 2024-03-14T13:02:05.927 INFO:tasks.workunit.client.0.smithi111.stdout:3/54: rmdir d0/de 39 2024-03-14T13:02:05.928 INFO:tasks.workunit.client.0.smithi111.stdout:7/77: mkdir d0/d1/d5/de/d14 0 2024-03-14T13:02:05.930 INFO:tasks.workunit.client.0.smithi111.stdout:7/78: read d0/d1/fd [1982144,797] 0 2024-03-14T13:02:05.932 INFO:tasks.workunit.client.0.smithi111.stdout:6/75: rmdir d0 39 2024-03-14T13:02:05.937 INFO:tasks.workunit.client.0.smithi111.stdout:7/79: link d0/f10 d0/d1/d5/de/d14/f15 0 2024-03-14T13:02:05.938 INFO:tasks.workunit.client.0.smithi111.stdout:3/55: link d0/de/l13 d0/l18 0 2024-03-14T13:02:05.945 INFO:tasks.workunit.client.0.smithi111.stdout:7/80: dread d0/d1/fd [0,4194304] 0 2024-03-14T13:02:05.946 INFO:tasks.workunit.client.0.smithi111.stdout:3/56: dwrite d0/de/f14 [0,4194304] 0 2024-03-14T13:02:05.947 INFO:tasks.workunit.client.0.smithi111.stdout:3/57: dread - d0/f6 zero size 2024-03-14T13:02:05.947 INFO:tasks.workunit.client.0.smithi111.stdout:3/58: fsync d0/f15 0 2024-03-14T13:02:05.948 INFO:tasks.workunit.client.0.smithi111.stdout:7/81: fdatasync d0/d1/d5/de/d14/f15 0 2024-03-14T13:02:05.954 INFO:tasks.workunit.client.0.smithi111.stdout:7/82: creat d0/d1/d5/de/d9/f16 x:0 0 0 2024-03-14T13:02:05.955 INFO:tasks.workunit.client.0.smithi111.stdout:6/76: write d0/f4 [927069,50854] 0 2024-03-14T13:02:05.957 INFO:tasks.workunit.client.0.smithi111.stdout:7/83: mknod d0/d1/d5/de/d14/c17 0 2024-03-14T13:02:05.958 INFO:tasks.workunit.client.0.smithi111.stdout:3/59: write d0/f8 [528605,14075] 0 2024-03-14T13:02:05.959 INFO:tasks.workunit.client.0.smithi111.stdout:6/77: rename d0/lb to d0/l15 0 2024-03-14T13:02:05.960 INFO:tasks.workunit.client.0.smithi111.stdout:3/60: mkdir d0/de/d19 0 2024-03-14T13:02:05.961 INFO:tasks.workunit.client.0.smithi111.stdout:6/78: write d0/f14 [498507,85182] 0 2024-03-14T13:02:05.962 INFO:tasks.workunit.client.0.smithi111.stdout:7/84: dread d0/d1/d5/de/d9/fb [0,4194304] 0 2024-03-14T13:02:05.963 INFO:tasks.workunit.client.0.smithi111.stdout:6/79: write d0/f6 [1978254,120] 0 2024-03-14T13:02:05.966 INFO:tasks.workunit.client.0.smithi111.stdout:3/61: rename d0/f15 to d0/de/d19/f1a 0 2024-03-14T13:02:05.966 INFO:tasks.workunit.client.0.smithi111.stdout:7/85: fdatasync d0/d1/fd 0 2024-03-14T13:02:05.966 INFO:tasks.workunit.client.0.smithi111.stdout:3/62: dread - d0/f11 zero size 2024-03-14T13:02:05.967 INFO:tasks.workunit.client.0.smithi111.stdout:6/80: rename d0/cc to d0/c16 0 2024-03-14T13:02:05.968 INFO:tasks.workunit.client.0.smithi111.stdout:6/81: stat d0/l9 0 2024-03-14T13:02:05.969 INFO:tasks.workunit.client.0.smithi111.stdout:3/63: mknod d0/de/d19/c1b 0 2024-03-14T13:02:05.971 INFO:tasks.workunit.client.0.smithi111.stdout:7/86: rename d0/d1/d5/de/d9/cc to d0/d1/d5/de/d9/c18 0 2024-03-14T13:02:05.973 INFO:tasks.workunit.client.0.smithi111.stdout:4/54: sync 2024-03-14T13:02:05.976 INFO:tasks.workunit.client.0.smithi111.stdout:7/87: unlink d0/f10 0 2024-03-14T13:02:05.982 INFO:tasks.workunit.client.0.smithi111.stdout:4/55: unlink f13 0 2024-03-14T13:02:05.983 INFO:tasks.workunit.client.0.smithi111.stdout:7/88: creat d0/d1/d5/de/d9/f19 x:0 0 0 2024-03-14T13:02:05.983 INFO:tasks.workunit.client.0.smithi111.stdout:6/82: truncate d0/fa 2807976 0 2024-03-14T13:02:05.984 INFO:tasks.workunit.client.0.smithi111.stdout:4/56: mknod c16 0 2024-03-14T13:02:05.985 INFO:tasks.workunit.client.0.smithi111.stdout:3/64: truncate d0/f7 2357217 0 2024-03-14T13:02:05.987 INFO:tasks.workunit.client.0.smithi111.stdout:4/57: write f11 [2161581,77851] 0 2024-03-14T13:02:05.989 INFO:tasks.workunit.client.0.smithi111.stdout:7/89: dwrite d0/d1/fd [0,4194304] 0 2024-03-14T13:02:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:05 smithi137 ceph-mon[50336]: pgmap v17: 65 pgs: 65 active+clean; 307 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 297 op/s 2024-03-14T13:02:06.036 INFO:tasks.workunit.client.0.smithi111.stdout:4/58: mknod c17 0 2024-03-14T13:02:06.036 INFO:tasks.workunit.client.0.smithi111.stdout:7/90: chown d0/d1/fd 0 1 2024-03-14T13:02:06.036 INFO:tasks.workunit.client.0.smithi111.stdout:3/65: rmdir d0 39 2024-03-14T13:02:06.037 INFO:tasks.workunit.client.0.smithi111.stdout:4/59: symlink l18 0 2024-03-14T13:02:06.038 INFO:tasks.workunit.client.0.smithi111.stdout:7/91: read - d0/d1/d5/de/d9/f19 zero size 2024-03-14T13:02:06.039 INFO:tasks.workunit.client.0.smithi111.stdout:3/66: chown d0/l16 462149476 1 2024-03-14T13:02:06.039 INFO:tasks.workunit.client.0.smithi111.stdout:6/83: dread d0/fa [0,4194304] 0 2024-03-14T13:02:06.039 INFO:tasks.workunit.client.0.smithi111.stdout:4/60: symlink l19 0 2024-03-14T13:02:06.040 INFO:tasks.workunit.client.0.smithi111.stdout:6/84: readlink d0/ld 0 2024-03-14T13:02:06.040 INFO:tasks.workunit.client.0.smithi111.stdout:3/67: write d0/f4 [2909100,5242] 0 2024-03-14T13:02:06.041 INFO:tasks.workunit.client.0.smithi111.stdout:7/92: creat d0/d1/d5/f1a x:0 0 0 2024-03-14T13:02:06.042 INFO:tasks.workunit.client.0.smithi111.stdout:6/85: rmdir d0 39 2024-03-14T13:02:06.046 INFO:tasks.workunit.client.0.smithi111.stdout:3/68: rename d0/f11 to d0/f1c 0 2024-03-14T13:02:06.046 INFO:tasks.workunit.client.0.smithi111.stdout:6/86: dread d0/ff [0,4194304] 0 2024-03-14T13:02:06.047 INFO:tasks.workunit.client.0.smithi111.stdout:3/69: stat d0/de/f12 0 2024-03-14T13:02:06.049 INFO:tasks.workunit.client.0.smithi111.stdout:6/87: mkdir d0/d17 0 2024-03-14T13:02:06.050 INFO:tasks.workunit.client.0.smithi111.stdout:4/61: getdents . 0 2024-03-14T13:02:06.051 INFO:tasks.workunit.client.0.smithi111.stdout:3/70: rename d0/lb to d0/de/d19/l1d 0 2024-03-14T13:02:06.054 INFO:tasks.workunit.client.0.smithi111.stdout:3/71: symlink d0/l1e 0 2024-03-14T13:02:06.055 INFO:tasks.workunit.client.0.smithi111.stdout:4/62: link c17 c1a 0 2024-03-14T13:02:06.055 INFO:tasks.workunit.client.0.smithi111.stdout:4/63: chown fe 11720 1 2024-03-14T13:02:06.056 INFO:tasks.workunit.client.0.smithi111.stdout:3/72: chown d0/de/c17 1339406 1 2024-03-14T13:02:06.057 INFO:tasks.workunit.client.0.smithi111.stdout:4/64: creat f1b x:0 0 0 2024-03-14T13:02:06.058 INFO:tasks.workunit.client.0.smithi111.stdout:6/88: dwrite d0/fa [0,4194304] 0 2024-03-14T13:02:06.060 INFO:tasks.workunit.client.0.smithi111.stdout:4/65: mknod c1c 0 2024-03-14T13:02:06.062 INFO:tasks.workunit.client.0.smithi111.stdout:4/66: write f1b [366139,28326] 0 2024-03-14T13:02:06.068 INFO:tasks.workunit.client.0.smithi111.stdout:3/73: getdents d0 0 2024-03-14T13:02:06.069 INFO:tasks.workunit.client.0.smithi111.stdout:6/89: truncate d0/fa 3013757 0 2024-03-14T13:02:06.069 INFO:tasks.workunit.client.0.smithi111.stdout:3/74: dread - d0/ff zero size 2024-03-14T13:02:06.071 INFO:tasks.workunit.client.0.smithi111.stdout:6/90: creat d0/f18 x:0 0 0 2024-03-14T13:02:06.073 INFO:tasks.workunit.client.0.smithi111.stdout:6/91: mknod d0/c19 0 2024-03-14T13:02:06.073 INFO:tasks.workunit.client.0.smithi111.stdout:3/75: dread d0/f1 [0,4194304] 0 2024-03-14T13:02:06.074 INFO:tasks.workunit.client.0.smithi111.stdout:4/67: dread f1b [0,4194304] 0 2024-03-14T13:02:06.074 INFO:tasks.workunit.client.0.smithi111.stdout:6/92: readlink d0/l15 0 2024-03-14T13:02:06.075 INFO:tasks.workunit.client.0.smithi111.stdout:4/68: chown f9 7052 1 2024-03-14T13:02:06.075 INFO:tasks.workunit.client.0.smithi111.stdout:3/76: symlink d0/l1f 0 2024-03-14T13:02:06.077 INFO:tasks.workunit.client.0.smithi111.stdout:3/77: chown d0/ff 14 1 2024-03-14T13:02:06.081 INFO:tasks.workunit.client.0.smithi111.stdout:3/78: unlink d0/f4 0 2024-03-14T13:02:06.081 INFO:tasks.workunit.client.0.smithi111.stdout:4/69: dwrite f15 [0,4194304] 0 2024-03-14T13:02:06.085 INFO:tasks.workunit.client.0.smithi111.stdout:4/70: mkdir d1d 0 2024-03-14T13:02:06.090 INFO:tasks.workunit.client.0.smithi111.stdout:3/79: dwrite d0/de/f14 [0,4194304] 0 2024-03-14T13:02:06.093 INFO:tasks.workunit.client.0.smithi111.stdout:4/71: rename l19 to d1d/l1e 0 2024-03-14T13:02:06.096 INFO:tasks.workunit.client.0.smithi111.stdout:4/72: symlink d1d/l1f 0 2024-03-14T13:02:06.097 INFO:tasks.workunit.client.0.smithi111.stdout:3/80: creat d0/f20 x:0 0 0 2024-03-14T13:02:06.099 INFO:tasks.workunit.client.0.smithi111.stdout:3/81: dread - d0/f1c zero size 2024-03-14T13:02:06.100 INFO:tasks.workunit.client.0.smithi111.stdout:4/73: unlink c17 0 2024-03-14T13:02:06.104 INFO:tasks.workunit.client.0.smithi111.stdout:3/82: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:06.104 INFO:tasks.workunit.client.0.smithi111.stdout:4/74: link f11 d1d/f20 0 2024-03-14T13:02:06.104 INFO:tasks.workunit.client.0.smithi111.stdout:4/75: readlink lc 0 2024-03-14T13:02:06.106 INFO:tasks.workunit.client.0.smithi111.stdout:3/83: unlink d0/de/d19/c1b 0 2024-03-14T13:02:06.107 INFO:tasks.workunit.client.0.smithi111.stdout:3/84: write d0/de/f12 [5129813,15287] 0 2024-03-14T13:02:06.108 INFO:tasks.workunit.client.0.smithi111.stdout:4/76: creat d1d/f21 x:0 0 0 2024-03-14T13:02:06.108 INFO:tasks.workunit.client.0.smithi111.stdout:4/77: chown d1d 109587 1 2024-03-14T13:02:06.110 INFO:tasks.workunit.client.0.smithi111.stdout:4/78: mkdir d1d/d22 0 2024-03-14T13:02:06.111 INFO:tasks.workunit.client.0.smithi111.stdout:3/85: mknod d0/c21 0 2024-03-14T13:02:06.112 INFO:tasks.workunit.client.0.smithi111.stdout:3/86: write d0/de/f12 [4241942,113024] 0 2024-03-14T13:02:06.112 INFO:tasks.workunit.client.0.smithi111.stdout:4/79: mkdir d1d/d22/d23 0 2024-03-14T13:02:06.116 INFO:tasks.workunit.client.0.smithi111.stdout:4/80: mknod d1d/d22/c24 0 2024-03-14T13:02:06.119 INFO:tasks.workunit.client.0.smithi111.stdout:4/81: rename d1d/f21 to d1d/f25 0 2024-03-14T13:02:06.119 INFO:tasks.workunit.client.0.smithi111.stdout:3/87: rmdir d0/de/d19 39 2024-03-14T13:02:06.124 INFO:tasks.workunit.client.0.smithi111.stdout:3/88: dread - d0/f6 zero size 2024-03-14T13:02:06.125 INFO:tasks.workunit.client.0.smithi111.stdout:4/82: truncate fb 825434 0 2024-03-14T13:02:06.130 INFO:tasks.workunit.client.0.smithi111.stdout:4/83: creat d1d/f26 x:0 0 0 2024-03-14T13:02:06.131 INFO:tasks.workunit.client.0.smithi111.stdout:3/89: rename d0/f6 to d0/f22 0 2024-03-14T13:02:06.132 INFO:tasks.workunit.client.0.smithi111.stdout:4/84: write f15 [3975472,101369] 0 2024-03-14T13:02:06.135 INFO:tasks.workunit.client.0.smithi111.stdout:3/90: creat d0/f23 x:0 0 0 2024-03-14T13:02:06.136 INFO:tasks.workunit.client.0.smithi111.stdout:4/85: dread f15 [0,4194304] 0 2024-03-14T13:02:06.137 INFO:tasks.workunit.client.0.smithi111.stdout:3/91: unlink d0/l18 0 2024-03-14T13:02:06.139 INFO:tasks.workunit.client.0.smithi111.stdout:3/92: symlink d0/l24 0 2024-03-14T13:02:06.139 INFO:tasks.workunit.client.0.smithi111.stdout:4/86: dread fe [0,4194304] 0 2024-03-14T13:02:06.140 INFO:tasks.workunit.client.0.smithi111.stdout:4/87: chown f1b 1803 1 2024-03-14T13:02:06.143 INFO:tasks.workunit.client.0.smithi111.stdout:8/30: sync 2024-03-14T13:02:06.143 INFO:tasks.workunit.client.0.smithi111.stdout:2/27: sync 2024-03-14T13:02:06.143 INFO:tasks.workunit.client.0.smithi111.stdout:1/34: sync 2024-03-14T13:02:06.144 INFO:tasks.workunit.client.0.smithi111.stdout:9/36: sync 2024-03-14T13:02:06.145 INFO:tasks.workunit.client.0.smithi111.stdout:4/88: read f9 [610305,127470] 0 2024-03-14T13:02:06.145 INFO:tasks.workunit.client.0.smithi111.stdout:1/35: creat fa x:0 0 0 2024-03-14T13:02:06.146 INFO:tasks.workunit.client.0.smithi111.stdout:2/28: symlink l9 0 2024-03-14T13:02:06.146 INFO:tasks.workunit.client.0.smithi111.stdout:2/29: rmdir - no directory 2024-03-14T13:02:06.146 INFO:tasks.workunit.client.0.smithi111.stdout:9/37: creat d5/fa x:0 0 0 2024-03-14T13:02:06.147 INFO:tasks.workunit.client.0.smithi111.stdout:8/31: symlink d3/d5/l9 0 2024-03-14T13:02:06.148 INFO:tasks.workunit.client.0.smithi111.stdout:9/38: rename d5/d6/d8 to d5/d6/d8/db 22 2024-03-14T13:02:06.148 INFO:tasks.workunit.client.0.smithi111.stdout:4/89: fdatasync f15 0 2024-03-14T13:02:06.149 INFO:tasks.workunit.client.0.smithi111.stdout:1/36: rename c2 to cb 0 2024-03-14T13:02:06.150 INFO:tasks.workunit.client.0.smithi111.stdout:1/37: stat f3 0 2024-03-14T13:02:06.151 INFO:tasks.workunit.client.0.smithi111.stdout:8/32: rename d3/c4 to d3/ca 0 2024-03-14T13:02:06.152 INFO:tasks.workunit.client.0.smithi111.stdout:9/39: creat d5/d6/fc x:0 0 0 2024-03-14T13:02:06.154 INFO:tasks.workunit.client.0.smithi111.stdout:1/38: creat fc x:0 0 0 2024-03-14T13:02:06.154 INFO:tasks.workunit.client.0.smithi111.stdout:3/93: getdents d0/de/d19 0 2024-03-14T13:02:06.155 INFO:tasks.workunit.client.0.smithi111.stdout:8/33: creat d3/d5/fb x:0 0 0 2024-03-14T13:02:06.156 INFO:tasks.workunit.client.0.smithi111.stdout:4/90: creat d1d/d22/d23/f27 x:0 0 0 2024-03-14T13:02:06.157 INFO:tasks.workunit.client.0.smithi111.stdout:9/40: mknod d5/cd 0 2024-03-14T13:02:06.157 INFO:tasks.workunit.client.0.smithi111.stdout:2/30: link l9 la 0 2024-03-14T13:02:06.158 INFO:tasks.workunit.client.0.smithi111.stdout:1/39: readlink l8 0 2024-03-14T13:02:06.159 INFO:tasks.workunit.client.0.smithi111.stdout:2/31: write f4 [153465,17744] 0 2024-03-14T13:02:06.161 INFO:tasks.workunit.client.0.smithi111.stdout:1/40: write f3 [3799688,28008] 0 2024-03-14T13:02:06.163 INFO:tasks.workunit.client.0.smithi111.stdout:1/41: dread - fa zero size 2024-03-14T13:02:06.165 INFO:tasks.workunit.client.0.smithi111.stdout:9/41: link f4 d5/fe 0 2024-03-14T13:02:06.165 INFO:tasks.workunit.client.0.smithi111.stdout:9/42: chown d5 0 1 2024-03-14T13:02:06.166 INFO:tasks.workunit.client.0.smithi111.stdout:3/94: link d0/de/f14 d0/de/f25 0 2024-03-14T13:02:06.167 INFO:tasks.workunit.client.0.smithi111.stdout:4/91: dwrite d1d/f26 [0,4194304] 0 2024-03-14T13:02:06.167 INFO:tasks.workunit.client.0.smithi111.stdout:8/34: truncate f0 1495857 0 2024-03-14T13:02:06.169 INFO:tasks.workunit.client.0.smithi111.stdout:1/42: readlink l1 0 2024-03-14T13:02:06.170 INFO:tasks.workunit.client.0.smithi111.stdout:4/92: chown d1d/f20 13597 1 2024-03-14T13:02:06.170 INFO:tasks.workunit.client.0.smithi111.stdout:2/32: dread f1 [0,4194304] 0 2024-03-14T13:02:06.170 INFO:tasks.workunit.client.0.smithi111.stdout:2/33: rmdir - no directory 2024-03-14T13:02:06.171 INFO:tasks.workunit.client.0.smithi111.stdout:2/34: chown c7 101205 1 2024-03-14T13:02:06.171 INFO:tasks.workunit.client.0.smithi111.stdout:3/95: creat d0/de/d19/f26 x:0 0 0 2024-03-14T13:02:06.173 INFO:tasks.workunit.client.0.smithi111.stdout:9/43: dread d5/d6/f7 [0,4194304] 0 2024-03-14T13:02:06.173 INFO:tasks.workunit.client.0.smithi111.stdout:8/35: mkdir d3/dc 0 2024-03-14T13:02:06.175 INFO:tasks.workunit.client.0.smithi111.stdout:9/44: truncate f4 6060691 0 2024-03-14T13:02:06.179 INFO:tasks.workunit.client.0.smithi111.stdout:2/35: stat l9 0 2024-03-14T13:02:06.179 INFO:tasks.workunit.client.0.smithi111.stdout:2/36: readlink l5 0 2024-03-14T13:02:06.180 INFO:tasks.workunit.client.0.smithi111.stdout:1/43: dwrite fa [0,4194304] 0 2024-03-14T13:02:06.182 INFO:tasks.workunit.client.0.smithi111.stdout:2/37: write f4 [230025,44895] 0 2024-03-14T13:02:06.183 INFO:tasks.workunit.client.0.smithi111.stdout:9/45: creat d5/d6/ff x:0 0 0 2024-03-14T13:02:06.183 INFO:tasks.workunit.client.0.smithi111.stdout:2/38: truncate f1 4746628 0 2024-03-14T13:02:06.187 INFO:tasks.workunit.client.0.smithi111.stdout:1/44: symlink ld 0 2024-03-14T13:02:06.193 INFO:tasks.workunit.client.0.smithi111.stdout:9/46: link d5/fe d5/f10 0 2024-03-14T13:02:06.196 INFO:tasks.workunit.client.0.smithi111.stdout:2/39: dread f4 [0,4194304] 0 2024-03-14T13:02:06.198 INFO:tasks.workunit.client.0.smithi111.stdout:9/47: mknod d5/d6/d8/c11 0 2024-03-14T13:02:06.198 INFO:tasks.workunit.client.0.smithi111.stdout:1/45: dwrite fa [0,4194304] 0 2024-03-14T13:02:06.201 INFO:tasks.workunit.client.0.smithi111.stdout:9/48: write d5/d6/f9 [723277,12717] 0 2024-03-14T13:02:06.202 INFO:tasks.workunit.client.0.smithi111.stdout:2/40: mknod cb 0 2024-03-14T13:02:06.202 INFO:tasks.workunit.client.0.smithi111.stdout:1/46: creat fe x:0 0 0 2024-03-14T13:02:06.206 INFO:tasks.workunit.client.0.smithi111.stdout:2/41: write f4 [2288270,26531] 0 2024-03-14T13:02:06.207 INFO:tasks.workunit.client.0.smithi111.stdout:9/49: creat d5/d6/d8/f12 x:0 0 0 2024-03-14T13:02:06.209 INFO:tasks.workunit.client.0.smithi111.stdout:1/47: creat ff x:0 0 0 2024-03-14T13:02:06.210 INFO:tasks.workunit.client.0.smithi111.stdout:9/50: write d5/d6/d8/f12 [558669,46342] 0 2024-03-14T13:02:06.210 INFO:tasks.workunit.client.0.smithi111.stdout:9/51: readlink - no filename 2024-03-14T13:02:06.212 INFO:tasks.workunit.client.0.smithi111.stdout:1/48: mknod c10 0 2024-03-14T13:02:06.213 INFO:tasks.workunit.client.0.smithi111.stdout:9/52: mknod d5/d6/c13 0 2024-03-14T13:02:06.214 INFO:tasks.workunit.client.0.smithi111.stdout:4/93: dwrite f11 [0,4194304] 0 2024-03-14T13:02:06.216 INFO:tasks.workunit.client.0.smithi111.stdout:1/49: creat f11 x:0 0 0 2024-03-14T13:02:06.217 INFO:tasks.workunit.client.0.smithi111.stdout:8/36: write f0 [1316706,78012] 0 2024-03-14T13:02:06.218 INFO:tasks.workunit.client.0.smithi111.stdout:2/42: dread f1 [0,4194304] 0 2024-03-14T13:02:06.221 INFO:tasks.workunit.client.0.smithi111.stdout:8/37: dread - d3/d5/fb zero size 2024-03-14T13:02:06.221 INFO:tasks.workunit.client.0.smithi111.stdout:2/43: symlink lc 0 2024-03-14T13:02:06.224 INFO:tasks.workunit.client.0.smithi111.stdout:2/44: fdatasync f4 0 2024-03-14T13:02:06.228 INFO:tasks.workunit.client.0.smithi111.stdout:4/94: dread f9 [0,4194304] 0 2024-03-14T13:02:06.229 INFO:tasks.workunit.client.0.smithi111.stdout:9/53: dwrite d5/fa [0,4194304] 0 2024-03-14T13:02:06.230 INFO:tasks.workunit.client.0.smithi111.stdout:1/50: dread f3 [0,4194304] 0 2024-03-14T13:02:06.230 INFO:tasks.workunit.client.0.smithi111.stdout:4/95: read f1b [204944,16405] 0 2024-03-14T13:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout:1/51: write f9 [448239,102307] 0 2024-03-14T13:02:06.233 INFO:tasks.workunit.client.0.smithi111.stdout:9/54: truncate d5/d6/ff 1014447 0 2024-03-14T13:02:06.234 INFO:tasks.workunit.client.0.smithi111.stdout:8/38: dwrite f0 [0,4194304] 0 2024-03-14T13:02:06.237 INFO:tasks.workunit.client.0.smithi111.stdout:1/52: symlink l12 0 2024-03-14T13:02:06.238 INFO:tasks.workunit.client.0.smithi111.stdout:1/53: fdatasync f9 0 2024-03-14T13:02:06.238 INFO:tasks.workunit.client.0.smithi111.stdout:9/55: mkdir d5/d6/d14 0 2024-03-14T13:02:06.239 INFO:tasks.workunit.client.0.smithi111.stdout:4/96: getdents d1d 0 2024-03-14T13:02:06.241 INFO:tasks.workunit.client.0.smithi111.stdout:8/39: unlink d3/d5/c6 0 2024-03-14T13:02:06.242 INFO:tasks.workunit.client.0.smithi111.stdout:1/54: fsync f3 0 2024-03-14T13:02:06.244 INFO:tasks.workunit.client.0.smithi111.stdout:1/55: write ff [421488,24005] 0 2024-03-14T13:02:06.247 INFO:tasks.workunit.client.0.smithi111.stdout:4/97: dread d1d/f20 [0,4194304] 0 2024-03-14T13:02:06.250 INFO:tasks.workunit.client.0.smithi111.stdout:8/40: dwrite f0 [0,4194304] 0 2024-03-14T13:02:06.252 INFO:tasks.workunit.client.0.smithi111.stdout:1/56: dread fa [0,4194304] 0 2024-03-14T13:02:06.252 INFO:tasks.workunit.client.0.smithi111.stdout:8/41: chown d3/ca 3689104 1 2024-03-14T13:02:06.253 INFO:tasks.workunit.client.0.smithi111.stdout:1/57: symlink l13 0 2024-03-14T13:02:06.253 INFO:tasks.workunit.client.0.smithi111.stdout:1/58: rmdir - no directory 2024-03-14T13:02:06.254 INFO:tasks.workunit.client.0.smithi111.stdout:1/59: chown l1 132001318 1 2024-03-14T13:02:06.254 INFO:tasks.workunit.client.0.smithi111.stdout:8/42: creat d3/d5/fd x:0 0 0 2024-03-14T13:02:06.254 INFO:tasks.workunit.client.0.smithi111.stdout:8/43: stat d3/d5/fb 0 2024-03-14T13:02:06.259 INFO:tasks.workunit.client.0.smithi111.stdout:9/56: rmdir d5/d6 39 2024-03-14T13:02:06.262 INFO:tasks.workunit.client.0.smithi111.stdout:1/60: dwrite ff [0,4194304] 0 2024-03-14T13:02:06.262 INFO:tasks.workunit.client.0.smithi111.stdout:8/44: dread f0 [0,4194304] 0 2024-03-14T13:02:06.263 INFO:tasks.workunit.client.0.smithi111.stdout:1/61: fsync f11 0 2024-03-14T13:02:06.264 INFO:tasks.workunit.client.0.smithi111.stdout:9/57: mknod d5/d6/d14/c15 0 2024-03-14T13:02:06.265 INFO:tasks.workunit.client.0.smithi111.stdout:1/62: dread - fe zero size 2024-03-14T13:02:06.269 INFO:tasks.workunit.client.0.smithi111.stdout:1/63: unlink fc 0 2024-03-14T13:02:06.270 INFO:tasks.workunit.client.0.smithi111.stdout:4/98: dwrite f12 [0,4194304] 0 2024-03-14T13:02:06.273 INFO:tasks.workunit.client.0.smithi111.stdout:8/45: dread f0 [0,4194304] 0 2024-03-14T13:02:06.276 INFO:tasks.workunit.client.0.smithi111.stdout:8/46: read - d3/d5/fd zero size 2024-03-14T13:02:06.279 INFO:tasks.workunit.client.0.smithi111.stdout:4/99: dread f12 [0,4194304] 0 2024-03-14T13:02:06.283 INFO:tasks.workunit.client.0.smithi111.stdout:4/100: dread - d1d/d22/d23/f27 zero size 2024-03-14T13:02:06.289 INFO:tasks.workunit.client.0.smithi111.stdout:9/58: dwrite d5/d6/f7 [0,4194304] 0 2024-03-14T13:02:06.290 INFO:tasks.workunit.client.0.smithi111.stdout:8/47: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.297 INFO:tasks.workunit.client.0.smithi111.stdout:1/64: getdents . 0 2024-03-14T13:02:06.303 INFO:tasks.workunit.client.0.smithi111.stdout:1/65: mknod c14 0 2024-03-14T13:02:06.303 INFO:tasks.workunit.client.0.smithi111.stdout:4/101: dwrite f9 [0,4194304] 0 2024-03-14T13:02:06.307 INFO:tasks.workunit.client.0.smithi111.stdout:1/66: rename l4 to l15 0 2024-03-14T13:02:06.307 INFO:tasks.workunit.client.0.smithi111.stdout:8/48: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.307 INFO:tasks.workunit.client.0.smithi111.stdout:4/102: mkdir d1d/d22/d28 0 2024-03-14T13:02:06.309 INFO:tasks.workunit.client.0.smithi111.stdout:1/67: write fa [4315237,1980] 0 2024-03-14T13:02:06.318 INFO:tasks.workunit.client.0.smithi111.stdout:1/68: dread f3 [0,4194304] 0 2024-03-14T13:02:06.320 INFO:tasks.workunit.client.0.smithi111.stdout:4/103: dread fb [0,4194304] 0 2024-03-14T13:02:06.321 INFO:tasks.workunit.client.0.smithi111.stdout:1/69: symlink l16 0 2024-03-14T13:02:06.322 INFO:tasks.workunit.client.0.smithi111.stdout:4/104: readlink d1d/l1f 0 2024-03-14T13:02:06.323 INFO:tasks.workunit.client.0.smithi111.stdout:8/49: dread d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.323 INFO:tasks.workunit.client.0.smithi111.stdout:9/59: dwrite f4 [0,4194304] 0 2024-03-14T13:02:06.324 INFO:tasks.workunit.client.0.smithi111.stdout:8/50: dread - d3/d5/fd zero size 2024-03-14T13:02:06.325 INFO:tasks.workunit.client.0.smithi111.stdout:1/70: creat f17 x:0 0 0 2024-03-14T13:02:06.326 INFO:tasks.workunit.client.0.smithi111.stdout:8/51: fdatasync d3/d5/fd 0 2024-03-14T13:02:06.326 INFO:tasks.workunit.client.0.smithi111.stdout:4/105: rmdir d1d 39 2024-03-14T13:02:06.326 INFO:tasks.workunit.client.0.smithi111.stdout:9/60: creat d5/d6/d8/f16 x:0 0 0 2024-03-14T13:02:06.326 INFO:tasks.workunit.client.0.smithi111.stdout:4/106: readlink lc 0 2024-03-14T13:02:06.327 INFO:tasks.workunit.client.0.smithi111.stdout:1/71: mknod c18 0 2024-03-14T13:02:06.330 INFO:tasks.workunit.client.0.smithi111.stdout:9/61: mknod d5/c17 0 2024-03-14T13:02:06.377 INFO:tasks.workunit.client.0.smithi111.stdout:9/62: write f2 [7009274,25290] 0 2024-03-14T13:02:06.379 INFO:tasks.workunit.client.0.smithi111.stdout:0/71: sync 2024-03-14T13:02:06.379 INFO:tasks.workunit.client.0.smithi111.stdout:5/136: sync 2024-03-14T13:02:06.379 INFO:tasks.workunit.client.0.smithi111.stdout:4/107: creat d1d/d22/d28/f29 x:0 0 0 2024-03-14T13:02:06.383 INFO:tasks.workunit.client.0.smithi111.stdout:8/52: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.383 INFO:tasks.workunit.client.0.smithi111.stdout:0/72: mkdir d6/d9/d15 0 2024-03-14T13:02:06.383 INFO:tasks.workunit.client.0.smithi111.stdout:1/72: dwrite ff [0,4194304] 0 2024-03-14T13:02:06.386 INFO:tasks.workunit.client.0.smithi111.stdout:5/137: unlink d2/d13/f25 0 2024-03-14T13:02:06.390 INFO:tasks.workunit.client.0.smithi111.stdout:4/108: mknod d1d/d22/d23/c2a 0 2024-03-14T13:02:06.391 INFO:tasks.workunit.client.0.smithi111.stdout:8/53: write f0 [5136498,74747] 0 2024-03-14T13:02:06.394 INFO:tasks.workunit.client.0.smithi111.stdout:0/73: creat d6/d9/d15/f16 x:0 0 0 2024-03-14T13:02:06.395 INFO:tasks.workunit.client.0.smithi111.stdout:4/109: rename lc to d1d/d22/l2b 0 2024-03-14T13:02:06.395 INFO:tasks.workunit.client.0.smithi111.stdout:8/54: rename d3 to d3/dc/de 22 2024-03-14T13:02:06.397 INFO:tasks.workunit.client.0.smithi111.stdout:5/138: link d2/l1a d2/d13/l30 0 2024-03-14T13:02:06.398 INFO:tasks.workunit.client.0.smithi111.stdout:4/110: write f9 [2717378,83970] 0 2024-03-14T13:02:06.401 INFO:tasks.workunit.client.0.smithi111.stdout:4/111: creat d1d/f2c x:0 0 0 2024-03-14T13:02:06.402 INFO:tasks.workunit.client.0.smithi111.stdout:8/55: dwrite f0 [0,4194304] 0 2024-03-14T13:02:06.403 INFO:tasks.workunit.client.0.smithi111.stdout:5/139: link d2/d19/f27 d2/d1c/f31 0 2024-03-14T13:02:06.405 INFO:tasks.workunit.client.0.smithi111.stdout:1/73: fsync ff 0 2024-03-14T13:02:06.405 INFO:tasks.workunit.client.0.smithi111.stdout:8/56: dread - d3/d5/fd zero size 2024-03-14T13:02:06.406 INFO:tasks.workunit.client.0.smithi111.stdout:5/140: mknod d2/d1f/c32 0 2024-03-14T13:02:06.407 INFO:tasks.workunit.client.0.smithi111.stdout:8/57: write d3/d5/fd [977086,129600] 0 2024-03-14T13:02:06.409 INFO:tasks.workunit.client.0.smithi111.stdout:5/141: mkdir d2/d19/d33 0 2024-03-14T13:02:06.411 INFO:tasks.workunit.client.0.smithi111.stdout:5/142: chown d2/d19 0 1 2024-03-14T13:02:06.411 INFO:tasks.workunit.client.0.smithi111.stdout:8/58: symlink d3/lf 0 2024-03-14T13:02:06.415 INFO:tasks.workunit.client.0.smithi111.stdout:4/112: dwrite d1d/d22/d28/f29 [0,4194304] 0 2024-03-14T13:02:06.415 INFO:tasks.workunit.client.0.smithi111.stdout:5/143: dread - d2/d19/f1e zero size 2024-03-14T13:02:06.418 INFO:tasks.workunit.client.0.smithi111.stdout:1/74: dwrite f17 [0,4194304] 0 2024-03-14T13:02:06.436 INFO:tasks.workunit.client.0.smithi111.stdout:8/59: truncate d3/d5/fd 777 0 2024-03-14T13:02:06.437 INFO:tasks.workunit.client.0.smithi111.stdout:8/60: stat d3/lf 0 2024-03-14T13:02:06.437 INFO:tasks.workunit.client.0.smithi111.stdout:4/113: truncate f11 3791999 0 2024-03-14T13:02:06.439 INFO:tasks.workunit.client.0.smithi111.stdout:5/144: truncate d2/d1c/f22 4182613 0 2024-03-14T13:02:06.440 INFO:tasks.workunit.client.0.smithi111.stdout:1/75: truncate f17 4131196 0 2024-03-14T13:02:06.443 INFO:tasks.workunit.client.0.smithi111.stdout:1/76: creat f19 x:0 0 0 2024-03-14T13:02:06.444 INFO:tasks.workunit.client.0.smithi111.stdout:8/61: getdents d3/d5 0 2024-03-14T13:02:06.444 INFO:tasks.workunit.client.0.smithi111.stdout:8/62: readlink d3/lf 0 2024-03-14T13:02:06.445 INFO:tasks.workunit.client.0.smithi111.stdout:5/145: dread d2/d13/f15 [0,4194304] 0 2024-03-14T13:02:06.447 INFO:tasks.workunit.client.0.smithi111.stdout:8/63: creat d3/d5/f10 x:0 0 0 2024-03-14T13:02:06.450 INFO:tasks.workunit.client.0.smithi111.stdout:5/146: chown d2/d13/f2f 43107 1 2024-03-14T13:02:06.453 INFO:tasks.workunit.client.0.smithi111.stdout:5/147: fdatasync d2/d13/f15 0 2024-03-14T13:02:06.455 INFO:tasks.workunit.client.0.smithi111.stdout:8/64: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.460 INFO:tasks.workunit.client.0.smithi111.stdout:4/114: write f11 [933418,5625] 0 2024-03-14T13:02:06.470 INFO:tasks.workunit.client.0.smithi111.stdout:5/148: write d2/d1c/f31 [5398194,115039] 0 2024-03-14T13:02:06.471 INFO:tasks.workunit.client.0.smithi111.stdout:4/115: truncate f1b 188978 0 2024-03-14T13:02:06.472 INFO:tasks.workunit.client.0.smithi111.stdout:8/65: truncate d3/d5/fd 1041988 0 2024-03-14T13:02:06.474 INFO:tasks.workunit.client.0.smithi111.stdout:4/116: mknod d1d/d22/d28/c2d 0 2024-03-14T13:02:06.478 INFO:tasks.workunit.client.0.smithi111.stdout:5/149: write d2/d1c/f22 [2520496,69705] 0 2024-03-14T13:02:06.480 INFO:tasks.workunit.client.0.smithi111.stdout:8/66: getdents d3/d5 0 2024-03-14T13:02:06.481 INFO:tasks.workunit.client.0.smithi111.stdout:5/150: creat d2/d1c/f34 x:0 0 0 2024-03-14T13:02:06.483 INFO:tasks.workunit.client.0.smithi111.stdout:8/67: creat d3/dc/f11 x:0 0 0 2024-03-14T13:02:06.484 INFO:tasks.workunit.client.0.smithi111.stdout:4/117: dwrite f9 [4194304,4194304] 0 2024-03-14T13:02:06.485 INFO:tasks.workunit.client.0.smithi111.stdout:8/68: read f0 [4902094,2860] 0 2024-03-14T13:02:06.485 INFO:tasks.workunit.client.0.smithi111.stdout:5/151: creat d2/f35 x:0 0 0 2024-03-14T13:02:06.488 INFO:tasks.workunit.client.0.smithi111.stdout:4/118: creat d1d/f2e x:0 0 0 2024-03-14T13:02:06.490 INFO:tasks.workunit.client.0.smithi111.stdout:5/152: truncate d2/d1c/f20 705855 0 2024-03-14T13:02:06.495 INFO:tasks.workunit.client.0.smithi111.stdout:8/69: getdents d3/dc 0 2024-03-14T13:02:06.497 INFO:tasks.workunit.client.0.smithi111.stdout:8/70: write d3/d5/f10 [624456,49551] 0 2024-03-14T13:02:06.500 INFO:tasks.workunit.client.0.smithi111.stdout:5/153: dwrite d2/d1c/f22 [0,4194304] 0 2024-03-14T13:02:06.500 INFO:tasks.workunit.client.0.smithi111.stdout:4/119: write f1b [1071425,31587] 0 2024-03-14T13:02:06.501 INFO:tasks.workunit.client.0.smithi111.stdout:4/120: read - d1d/d22/d23/f27 zero size 2024-03-14T13:02:06.503 INFO:tasks.workunit.client.0.smithi111.stdout:5/154: write d2/d19/f11 [269248,120063] 0 2024-03-14T13:02:06.506 INFO:tasks.workunit.client.0.smithi111.stdout:4/121: fsync fb 0 2024-03-14T13:02:06.506 INFO:tasks.workunit.client.0.smithi111.stdout:8/71: dread f0 [4194304,4194304] 0 2024-03-14T13:02:06.506 INFO:tasks.workunit.client.0.smithi111.stdout:5/155: truncate d2/d13/f2f 565979 0 2024-03-14T13:02:06.511 INFO:tasks.workunit.client.0.smithi111.stdout:4/122: getdents d1d/d22 0 2024-03-14T13:02:06.516 INFO:tasks.workunit.client.0.smithi111.stdout:4/123: rename f15 to d1d/d22/d23/f2f 0 2024-03-14T13:02:06.517 INFO:tasks.workunit.client.0.smithi111.stdout:8/72: dwrite d3/dc/f11 [0,4194304] 0 2024-03-14T13:02:06.518 INFO:tasks.workunit.client.0.smithi111.stdout:5/156: write d2/d13/f2f [634138,22219] 0 2024-03-14T13:02:06.518 INFO:tasks.workunit.client.0.smithi111.stdout:8/73: chown d3/l8 2030126 1 2024-03-14T13:02:06.520 INFO:tasks.workunit.client.0.smithi111.stdout:4/124: mkdir d1d/d22/d30 0 2024-03-14T13:02:06.530 INFO:tasks.workunit.client.0.smithi111.stdout:5/157: dwrite d2/d19/f11 [0,4194304] 0 2024-03-14T13:02:06.530 INFO:tasks.workunit.client.0.smithi111.stdout:8/74: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.532 INFO:tasks.workunit.client.0.smithi111.stdout:8/75: readlink d3/l8 0 2024-03-14T13:02:06.532 INFO:tasks.workunit.client.0.smithi111.stdout:4/125: dwrite d1d/f26 [0,4194304] 0 2024-03-14T13:02:06.532 INFO:tasks.workunit.client.0.smithi111.stdout:5/158: dread - d2/d13/f29 zero size 2024-03-14T13:02:06.532 INFO:tasks.workunit.client.0.smithi111.stdout:8/76: chown d3/d5 1823388 1 2024-03-14T13:02:06.536 INFO:tasks.workunit.client.0.smithi111.stdout:5/159: mkdir d2/d1f/d36 0 2024-03-14T13:02:06.538 INFO:tasks.workunit.client.0.smithi111.stdout:5/160: read d2/d19/f11 [3301994,4546] 0 2024-03-14T13:02:06.538 INFO:tasks.workunit.client.0.smithi111.stdout:4/126: truncate d1d/f2c 183752 0 2024-03-14T13:02:06.542 INFO:tasks.workunit.client.0.smithi111.stdout:5/161: creat d2/d1f/f37 x:0 0 0 2024-03-14T13:02:06.545 INFO:tasks.workunit.client.0.smithi111.stdout:5/162: creat d2/d16/f38 x:0 0 0 2024-03-14T13:02:06.545 INFO:tasks.workunit.client.0.smithi111.stdout:4/127: getdents d1d/d22/d23 0 2024-03-14T13:02:06.548 INFO:tasks.workunit.client.0.smithi111.stdout:4/128: unlink f9 0 2024-03-14T13:02:06.550 INFO:tasks.workunit.client.0.smithi111.stdout:5/163: symlink d2/d19/d33/l39 0 2024-03-14T13:02:06.553 INFO:tasks.workunit.client.0.smithi111.stdout:5/164: chown d2/d13/f2b 0 1 2024-03-14T13:02:06.556 INFO:tasks.workunit.client.0.smithi111.stdout:5/165: unlink d2/l10 0 2024-03-14T13:02:06.559 INFO:tasks.workunit.client.0.smithi111.stdout:4/129: dwrite f1b [0,4194304] 0 2024-03-14T13:02:06.560 INFO:tasks.workunit.client.0.smithi111.stdout:8/77: dwrite d3/d5/fd [0,4194304] 0 2024-03-14T13:02:06.567 INFO:tasks.workunit.client.0.smithi111.stdout:4/130: write d1d/d22/d23/f2f [4593275,119592] 0 2024-03-14T13:02:06.569 INFO:tasks.workunit.client.0.smithi111.stdout:8/78: creat d3/d5/f12 x:0 0 0 2024-03-14T13:02:06.570 INFO:tasks.workunit.client.0.smithi111.stdout:4/131: stat fe 0 2024-03-14T13:02:06.571 INFO:tasks.workunit.client.0.smithi111.stdout:4/132: stat d1d/d22/d30 0 2024-03-14T13:02:06.574 INFO:tasks.workunit.client.0.smithi111.stdout:8/79: creat d3/f13 x:0 0 0 2024-03-14T13:02:06.580 INFO:tasks.workunit.client.0.smithi111.stdout:4/133: write fb [1616456,85932] 0 2024-03-14T13:02:06.582 INFO:tasks.workunit.client.0.smithi111.stdout:4/134: read d1d/f26 [1620240,80942] 0 2024-03-14T13:02:06.584 INFO:tasks.workunit.client.0.smithi111.stdout:5/166: dwrite d2/d1c/f20 [0,4194304] 0 2024-03-14T13:02:06.585 INFO:tasks.workunit.client.0.smithi111.stdout:4/135: creat d1d/d22/f31 x:0 0 0 2024-03-14T13:02:06.589 INFO:tasks.workunit.client.0.smithi111.stdout:8/80: truncate d3/d5/fd 483272 0 2024-03-14T13:02:06.589 INFO:tasks.workunit.client.0.smithi111.stdout:5/167: truncate d2/d13/f29 686753 0 2024-03-14T13:02:06.591 INFO:tasks.workunit.client.0.smithi111.stdout:8/81: readlink d3/d5/l9 0 2024-03-14T13:02:06.593 INFO:tasks.workunit.client.0.smithi111.stdout:8/82: creat d3/f14 x:0 0 0 2024-03-14T13:02:06.594 INFO:tasks.workunit.client.0.smithi111.stdout:4/136: fsync d1d/d22/f31 0 2024-03-14T13:02:06.598 INFO:tasks.workunit.client.0.smithi111.stdout:5/168: dwrite d2/d1f/f37 [0,4194304] 0 2024-03-14T13:02:06.604 INFO:tasks.workunit.client.0.smithi111.stdout:8/83: dwrite d3/f13 [0,4194304] 0 2024-03-14T13:02:06.605 INFO:tasks.workunit.client.0.smithi111.stdout:4/137: dread f12 [0,4194304] 0 2024-03-14T13:02:06.607 INFO:tasks.workunit.client.0.smithi111.stdout:8/84: symlink d3/d5/l15 0 2024-03-14T13:02:06.610 INFO:tasks.workunit.client.0.smithi111.stdout:5/169: truncate d2/d19/f11 5668179 0 2024-03-14T13:02:06.616 INFO:tasks.workunit.client.0.smithi111.stdout:4/138: dwrite d1d/f2c [0,4194304] 0 2024-03-14T13:02:06.619 INFO:tasks.workunit.client.0.smithi111.stdout:8/85: dwrite d3/d5/f10 [0,4194304] 0 2024-03-14T13:02:06.619 INFO:tasks.workunit.client.0.smithi111.stdout:4/139: rmdir d1d/d22/d23 39 2024-03-14T13:02:06.620 INFO:tasks.workunit.client.0.smithi111.stdout:8/86: stat c2 0 2024-03-14T13:02:06.625 INFO:tasks.workunit.client.0.smithi111.stdout:4/140: unlink d1d/d22/d23/f2f 0 2024-03-14T13:02:06.626 INFO:tasks.workunit.client.0.smithi111.stdout:6/93: sync 2024-03-14T13:02:06.626 INFO:tasks.workunit.client.0.smithi111.stdout:4/141: chown d1d/d22/d28/c2d 27550398 1 2024-03-14T13:02:06.626 INFO:tasks.workunit.client.0.smithi111.stdout:6/94: readlink d0/l9 0 2024-03-14T13:02:06.630 INFO:tasks.workunit.client.0.smithi111.stdout:5/170: truncate d2/d1c/f20 1860927 0 2024-03-14T13:02:06.630 INFO:tasks.workunit.client.0.smithi111.stdout:4/142: chown c10 2 1 2024-03-14T13:02:06.633 INFO:tasks.workunit.client.0.smithi111.stdout:4/143: write d1d/f2e [109140,3653] 0 2024-03-14T13:02:06.644 INFO:tasks.workunit.client.0.smithi111.stdout:8/87: write d3/d5/fd [176065,68119] 0 2024-03-14T13:02:06.649 INFO:tasks.workunit.client.0.smithi111.stdout:6/95: dwrite d0/fa [0,4194304] 0 2024-03-14T13:02:06.649 INFO:tasks.workunit.client.0.smithi111.stdout:5/171: dwrite d2/d1f/f37 [4194304,4194304] 0 2024-03-14T13:02:06.649 INFO:tasks.workunit.client.0.smithi111.stdout:8/88: write d3/f13 [4792427,82409] 0 2024-03-14T13:02:06.651 INFO:tasks.workunit.client.0.smithi111.stdout:5/172: chown d2/d13/f2f 447225 1 2024-03-14T13:02:06.653 INFO:tasks.workunit.client.0.smithi111.stdout:8/89: write d3/d5/f10 [1954782,103661] 0 2024-03-14T13:02:06.656 INFO:tasks.workunit.client.0.smithi111.stdout:6/96: link d0/fa d0/f1a 0 2024-03-14T13:02:06.656 INFO:tasks.workunit.client.0.smithi111.stdout:8/90: read - d3/d5/f12 zero size 2024-03-14T13:02:06.657 INFO:tasks.workunit.client.0.smithi111.stdout:4/144: dwrite f11 [0,4194304] 0 2024-03-14T13:02:06.657 INFO:tasks.workunit.client.0.smithi111.stdout:5/173: chown d2/d13/l30 6994750 1 2024-03-14T13:02:06.661 INFO:tasks.workunit.client.0.smithi111.stdout:8/91: symlink d3/l16 0 2024-03-14T13:02:06.665 INFO:tasks.workunit.client.0.smithi111.stdout:8/92: read d3/d5/fd [319091,108928] 0 2024-03-14T13:02:06.665 INFO:tasks.workunit.client.0.smithi111.stdout:4/145: creat d1d/d22/d30/f32 x:0 0 0 2024-03-14T13:02:06.669 INFO:tasks.workunit.client.0.smithi111.stdout:6/97: dwrite d0/f4 [0,4194304] 0 2024-03-14T13:02:06.670 INFO:tasks.workunit.client.0.smithi111.stdout:5/174: link d2/d19/c5 d2/d16/d2c/c3a 0 2024-03-14T13:02:06.670 INFO:tasks.workunit.client.0.smithi111.stdout:4/146: symlink d1d/d22/d28/l33 0 2024-03-14T13:02:06.671 INFO:tasks.workunit.client.0.smithi111.stdout:6/98: chown d0/l13 260120 1 2024-03-14T13:02:06.674 INFO:tasks.workunit.client.0.smithi111.stdout:4/147: unlink d1d/d22/d28/l33 0 2024-03-14T13:02:06.675 INFO:tasks.workunit.client.0.smithi111.stdout:4/148: readlink d1d/l1f 0 2024-03-14T13:02:06.677 INFO:tasks.workunit.client.0.smithi111.stdout:6/99: rename d0/c12 to d0/c1b 0 2024-03-14T13:02:06.677 INFO:tasks.workunit.client.0.smithi111.stdout:4/149: readlink l6 0 2024-03-14T13:02:06.679 INFO:tasks.workunit.client.0.smithi111.stdout:8/93: dwrite d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.680 INFO:tasks.workunit.client.0.smithi111.stdout:4/150: read d1d/f20 [1510152,30015] 0 2024-03-14T13:02:06.681 INFO:tasks.workunit.client.0.smithi111.stdout:4/151: readlink d1d/l1f 0 2024-03-14T13:02:06.682 INFO:tasks.workunit.client.0.smithi111.stdout:8/94: fsync d3/d5/f10 0 2024-03-14T13:02:06.686 INFO:tasks.workunit.client.0.smithi111.stdout:8/95: unlink d3/d5/l9 0 2024-03-14T13:02:06.687 INFO:tasks.workunit.client.0.smithi111.stdout:4/152: write d1d/f20 [1547068,33750] 0 2024-03-14T13:02:06.689 INFO:tasks.workunit.client.0.smithi111.stdout:8/96: symlink d3/dc/l17 0 2024-03-14T13:02:06.691 INFO:tasks.workunit.client.0.smithi111.stdout:6/100: dwrite d0/f14 [0,4194304] 0 2024-03-14T13:02:06.697 INFO:tasks.workunit.client.0.smithi111.stdout:8/97: creat d3/dc/f18 x:0 0 0 2024-03-14T13:02:06.698 INFO:tasks.workunit.client.0.smithi111.stdout:6/101: creat d0/f1c x:0 0 0 2024-03-14T13:02:06.700 INFO:tasks.workunit.client.0.smithi111.stdout:8/98: truncate d3/d5/f12 346865 0 2024-03-14T13:02:06.701 INFO:tasks.workunit.client.0.smithi111.stdout:6/102: creat d0/f1d x:0 0 0 2024-03-14T13:02:06.702 INFO:tasks.workunit.client.0.smithi111.stdout:4/153: dwrite d1d/f20 [0,4194304] 0 2024-03-14T13:02:06.702 INFO:tasks.workunit.client.0.smithi111.stdout:8/99: creat d3/d5/f19 x:0 0 0 2024-03-14T13:02:06.703 INFO:tasks.workunit.client.0.smithi111.stdout:8/100: fsync d3/dc/f11 0 2024-03-14T13:02:06.804 INFO:tasks.workunit.client.0.smithi111.stdout:8/101: read d3/d5/fd [141010,98817] 0 2024-03-14T13:02:06.804 INFO:tasks.workunit.client.0.smithi111.stdout:6/103: unlink d0/c19 0 2024-03-14T13:02:06.804 INFO:tasks.workunit.client.0.smithi111.stdout:4/154: creat d1d/d22/d23/f34 x:0 0 0 2024-03-14T13:02:06.806 INFO:tasks.workunit.client.0.smithi111.stdout:8/102: mknod d3/c1a 0 2024-03-14T13:02:06.808 INFO:tasks.workunit.client.0.smithi111.stdout:6/104: creat d0/d17/f1e x:0 0 0 2024-03-14T13:02:06.809 INFO:tasks.workunit.client.0.smithi111.stdout:6/105: truncate d0/f18 631319 0 2024-03-14T13:02:06.856 INFO:tasks.workunit.client.0.smithi111.stdout:7/93: sync 2024-03-14T13:02:06.862 INFO:tasks.workunit.client.0.smithi111.stdout:6/106: getdents d0 0 2024-03-14T13:02:06.863 INFO:tasks.workunit.client.0.smithi111.stdout:4/155: dwrite d1d/d22/f31 [0,4194304] 0 2024-03-14T13:02:06.866 INFO:tasks.workunit.client.0.smithi111.stdout:7/94: dread - d0/d1/d5/de/d14/f15 zero size 2024-03-14T13:02:06.867 INFO:tasks.workunit.client.0.smithi111.stdout:6/107: creat d0/d17/f1f x:0 0 0 2024-03-14T13:02:06.868 INFO:tasks.workunit.client.0.smithi111.stdout:8/103: dwrite d3/f14 [0,4194304] 0 2024-03-14T13:02:06.869 INFO:tasks.workunit.client.0.smithi111.stdout:4/156: creat d1d/d22/f35 x:0 0 0 2024-03-14T13:02:06.870 INFO:tasks.workunit.client.0.smithi111.stdout:4/157: chown c10 5881 1 2024-03-14T13:02:06.870 INFO:tasks.workunit.client.0.smithi111.stdout:6/108: mkdir d0/d17/d20 0 2024-03-14T13:02:06.871 INFO:tasks.workunit.client.0.smithi111.stdout:8/104: mkdir d3/d1b 0 2024-03-14T13:02:06.872 INFO:tasks.workunit.client.0.smithi111.stdout:4/158: creat d1d/f36 x:0 0 0 2024-03-14T13:02:06.873 INFO:tasks.workunit.client.0.smithi111.stdout:6/109: mkdir d0/d17/d21 0 2024-03-14T13:02:06.875 INFO:tasks.workunit.client.0.smithi111.stdout:8/105: dread d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.876 INFO:tasks.workunit.client.0.smithi111.stdout:4/159: creat d1d/d22/f37 x:0 0 0 2024-03-14T13:02:06.877 INFO:tasks.workunit.client.0.smithi111.stdout:4/160: dread - d1d/d22/f35 zero size 2024-03-14T13:02:06.879 INFO:tasks.workunit.client.0.smithi111.stdout:8/106: getdents d3/dc 0 2024-03-14T13:02:06.880 INFO:tasks.workunit.client.0.smithi111.stdout:4/161: creat d1d/d22/d30/f38 x:0 0 0 2024-03-14T13:02:06.887 INFO:tasks.workunit.client.0.smithi111.stdout:4/162: dwrite f12 [0,4194304] 0 2024-03-14T13:02:06.888 INFO:tasks.workunit.client.0.smithi111.stdout:4/163: fdatasync d1d/f25 0 2024-03-14T13:02:06.889 INFO:tasks.workunit.client.0.smithi111.stdout:4/164: stat d1d/d22/c24 0 2024-03-14T13:02:06.893 INFO:tasks.workunit.client.0.smithi111.stdout:8/107: rmdir d3 39 2024-03-14T13:02:06.896 INFO:tasks.workunit.client.0.smithi111.stdout:8/108: symlink d3/d5/l1c 0 2024-03-14T13:02:06.897 INFO:tasks.workunit.client.0.smithi111.stdout:8/109: truncate f0 5514375 0 2024-03-14T13:02:06.898 INFO:tasks.workunit.client.0.smithi111.stdout:8/110: chown d3/l8 290023 1 2024-03-14T13:02:06.904 INFO:tasks.workunit.client.0.smithi111.stdout:8/111: dread d3/f14 [0,4194304] 0 2024-03-14T13:02:06.910 INFO:tasks.workunit.client.0.smithi111.stdout:8/112: dwrite d3/f14 [0,4194304] 0 2024-03-14T13:02:06.911 INFO:tasks.workunit.client.0.smithi111.stdout:8/113: chown d3/dc/f11 325 1 2024-03-14T13:02:06.912 INFO:tasks.workunit.client.0.smithi111.stdout:8/114: read d3/f13 [3536575,102755] 0 2024-03-14T13:02:06.914 INFO:tasks.workunit.client.0.smithi111.stdout:8/115: creat d3/d1b/f1d x:0 0 0 2024-03-14T13:02:06.917 INFO:tasks.workunit.client.0.smithi111.stdout:8/116: link d3/d5/f10 d3/d5/f1e 0 2024-03-14T13:02:06.918 INFO:tasks.workunit.client.0.smithi111.stdout:8/117: symlink d3/dc/l1f 0 2024-03-14T13:02:06.920 INFO:tasks.workunit.client.0.smithi111.stdout:8/118: mkdir d3/d1b/d20 0 2024-03-14T13:02:06.927 INFO:tasks.workunit.client.0.smithi111.stdout:8/119: dwrite d3/d5/f1e [0,4194304] 0 2024-03-14T13:02:06.931 INFO:tasks.workunit.client.0.smithi111.stdout:8/120: creat d3/dc/f21 x:0 0 0 2024-03-14T13:02:06.932 INFO:tasks.workunit.client.0.smithi111.stdout:8/121: write d3/dc/f21 [897888,99741] 0 2024-03-14T13:02:06.934 INFO:tasks.workunit.client.0.smithi111.stdout:8/122: mkdir d3/d22 0 2024-03-14T13:02:06.935 INFO:tasks.workunit.client.0.smithi111.stdout:8/123: dread - d3/d1b/f1d zero size 2024-03-14T13:02:06.935 INFO:tasks.workunit.client.0.smithi111.stdout:8/124: chown d3/d5/fb 1924 1 2024-03-14T13:02:06.938 INFO:tasks.workunit.client.0.smithi111.stdout:8/125: rename d3/d5/l1c to d3/d1b/l23 0 2024-03-14T13:02:06.939 INFO:tasks.workunit.client.0.smithi111.stdout:8/126: creat d3/d1b/f24 x:0 0 0 2024-03-14T13:02:06.942 INFO:tasks.workunit.client.0.smithi111.stdout:8/127: dread d3/d5/fb [0,4194304] 0 2024-03-14T13:02:06.943 INFO:tasks.workunit.client.0.smithi111.stdout:8/128: rename d3/d1b to d3/d1b/d20/d25 22 2024-03-14T13:02:06.944 INFO:tasks.workunit.client.0.smithi111.stdout:8/129: read d3/d5/f12 [249445,125474] 0 2024-03-14T13:02:06.945 INFO:tasks.workunit.client.0.smithi111.stdout:8/130: stat d3/d1b 0 2024-03-14T13:02:06.946 INFO:tasks.workunit.client.0.smithi111.stdout:8/131: write d3/d5/fb [1594885,35879] 0 2024-03-14T13:02:06.948 INFO:tasks.workunit.client.0.smithi111.stdout:8/132: creat d3/d5/f26 x:0 0 0 2024-03-14T13:02:06.948 INFO:tasks.workunit.client.0.smithi111.stdout:8/133: truncate d3/dc/f21 1840693 0 2024-03-14T13:02:06.955 INFO:tasks.workunit.client.0.smithi111.stdout:8/134: dwrite d3/d5/fd [0,4194304] 0 2024-03-14T13:02:06.956 INFO:tasks.workunit.client.0.smithi111.stdout:3/96: sync 2024-03-14T13:02:06.956 INFO:tasks.workunit.client.0.smithi111.stdout:2/45: sync 2024-03-14T13:02:06.958 INFO:tasks.workunit.client.0.smithi111.stdout:2/46: chown lc 0 1 2024-03-14T13:02:06.972 INFO:tasks.workunit.client.0.smithi111.stdout:8/135: dwrite d3/d5/f26 [0,4194304] 0 2024-03-14T13:02:06.974 INFO:tasks.workunit.client.0.smithi111.stdout:3/97: dwrite d0/ff [0,4194304] 0 2024-03-14T13:02:06.976 INFO:tasks.workunit.client.0.smithi111.stdout:2/47: dwrite f1 [4194304,4194304] 0 2024-03-14T13:02:06.978 INFO:tasks.workunit.client.0.smithi111.stdout:8/136: mknod d3/d1b/c27 0 2024-03-14T13:02:06.980 INFO:tasks.workunit.client.0.smithi111.stdout:8/137: creat d3/d1b/f28 x:0 0 0 2024-03-14T13:02:06.982 INFO:tasks.workunit.client.0.smithi111.stdout:2/48: dread f1 [0,4194304] 0 2024-03-14T13:02:06.982 INFO:tasks.workunit.client.0.smithi111.stdout:8/138: chown d3/l16 149008339 1 2024-03-14T13:02:06.982 INFO:tasks.workunit.client.0.smithi111.stdout:3/98: rename d0/de/f25 to d0/de/f27 0 2024-03-14T13:02:06.984 INFO:tasks.workunit.client.0.smithi111.stdout:8/139: rmdir d3/dc 39 2024-03-14T13:02:06.988 INFO:tasks.workunit.client.0.smithi111.stdout:8/140: truncate d3/d5/f12 1265768 0 2024-03-14T13:02:06.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/99: dwrite d0/de/d19/f26 [0,4194304] 0 2024-03-14T13:02:06.993 INFO:tasks.workunit.client.0.smithi111.stdout:8/141: rename d3/d1b/c27 to d3/d5/c29 0 2024-03-14T13:02:06.994 INFO:tasks.workunit.client.0.smithi111.stdout:3/100: chown d0/l16 12125 1 2024-03-14T13:02:06.995 INFO:tasks.workunit.client.0.smithi111.stdout:3/101: chown d0/c21 15 1 2024-03-14T13:02:06.997 INFO:tasks.workunit.client.0.smithi111.stdout:3/102: truncate d0/f1c 496871 0 2024-03-14T13:02:06.999 INFO:tasks.workunit.client.0.smithi111.stdout:8/142: dread d3/d5/f1e [0,4194304] 0 2024-03-14T13:02:07.000 INFO:tasks.workunit.client.0.smithi111.stdout:3/103: unlink d0/de/f14 0 2024-03-14T13:02:07.002 INFO:tasks.workunit.client.0.smithi111.stdout:8/143: symlink d3/d22/l2a 0 2024-03-14T13:02:07.002 INFO:tasks.workunit.client.0.smithi111.stdout:8/144: dread - d3/d1b/f28 zero size 2024-03-14T13:02:07.004 INFO:tasks.workunit.client.0.smithi111.stdout:3/104: link d0/f20 d0/de/d19/f28 0 2024-03-14T13:02:07.005 INFO:tasks.workunit.client.0.smithi111.stdout:3/105: truncate d0/f23 550085 0 2024-03-14T13:02:07.005 INFO:tasks.workunit.client.0.smithi111.stdout:8/145: write d3/dc/f18 [245445,28536] 0 2024-03-14T13:02:07.008 INFO:tasks.workunit.client.0.smithi111.stdout:3/106: rename d0/l24 to d0/de/d19/l29 0 2024-03-14T13:02:07.009 INFO:tasks.workunit.client.0.smithi111.stdout:8/146: creat d3/d1b/d20/f2b x:0 0 0 2024-03-14T13:02:07.010 INFO:tasks.workunit.client.0.smithi111.stdout:3/107: mknod d0/de/d19/c2a 0 2024-03-14T13:02:07.015 INFO:tasks.workunit.client.0.smithi111.stdout:8/147: unlink d3/d22/l2a 0 2024-03-14T13:02:07.017 INFO:tasks.workunit.client.0.smithi111.stdout:2/49: fdatasync f1 0 2024-03-14T13:02:07.018 INFO:tasks.workunit.client.0.smithi111.stdout:8/148: creat d3/d1b/f2c x:0 0 0 2024-03-14T13:02:07.018 INFO:tasks.workunit.client.0.smithi111.stdout:2/50: stat l2 0 2024-03-14T13:02:07.018 INFO:tasks.workunit.client.0.smithi111.stdout:3/108: dread d0/de/f12 [4194304,4194304] 0 2024-03-14T13:02:07.018 INFO:tasks.workunit.client.0.smithi111.stdout:8/149: stat d3/ca 0 2024-03-14T13:02:07.019 INFO:tasks.workunit.client.0.smithi111.stdout:8/150: readlink d3/l16 0 2024-03-14T13:02:07.021 INFO:tasks.workunit.client.0.smithi111.stdout:2/51: mkdir dd 0 2024-03-14T13:02:07.024 INFO:tasks.workunit.client.0.smithi111.stdout:3/109: chown d0/fd 861691 1 2024-03-14T13:02:07.026 INFO:tasks.workunit.client.0.smithi111.stdout:8/151: readlink d3/d1b/l23 0 2024-03-14T13:02:07.028 INFO:tasks.workunit.client.0.smithi111.stdout:2/52: dwrite f1 [0,4194304] 0 2024-03-14T13:02:07.035 INFO:tasks.workunit.client.0.smithi111.stdout:8/152: mknod d3/d1b/c2d 0 2024-03-14T13:02:07.037 INFO:tasks.workunit.client.0.smithi111.stdout:8/153: creat d3/d1b/d20/f2e x:0 0 0 2024-03-14T13:02:07.038 INFO:tasks.workunit.client.0.smithi111.stdout:8/154: dread - d3/d1b/d20/f2e zero size 2024-03-14T13:02:07.039 INFO:tasks.workunit.client.0.smithi111.stdout:8/155: fsync d3/d5/fb 0 2024-03-14T13:02:07.040 INFO:tasks.workunit.client.0.smithi111.stdout:8/156: dread - d3/d1b/f28 zero size 2024-03-14T13:02:07.043 INFO:tasks.workunit.client.0.smithi111.stdout:2/53: truncate f1 7355473 0 2024-03-14T13:02:07.044 INFO:tasks.workunit.client.0.smithi111.stdout:2/54: chown f1 198709 1 2024-03-14T13:02:07.047 INFO:tasks.workunit.client.0.smithi111.stdout:3/110: creat d0/de/f2b x:0 0 0 2024-03-14T13:02:07.048 INFO:tasks.workunit.client.0.smithi111.stdout:2/55: creat dd/fe x:0 0 0 2024-03-14T13:02:07.050 INFO:tasks.workunit.client.0.smithi111.stdout:2/56: readlink la 0 2024-03-14T13:02:07.055 INFO:tasks.workunit.client.0.smithi111.stdout:8/157: dwrite d3/f13 [0,4194304] 0 2024-03-14T13:02:07.055 INFO:tasks.workunit.client.0.smithi111.stdout:3/111: read - d0/de/d19/f28 zero size 2024-03-14T13:02:07.055 INFO:tasks.workunit.client.0.smithi111.stdout:2/57: dread f4 [0,4194304] 0 2024-03-14T13:02:07.057 INFO:tasks.workunit.client.0.smithi111.stdout:8/158: mknod d3/d5/c2f 0 2024-03-14T13:02:07.060 INFO:tasks.workunit.client.0.smithi111.stdout:8/159: chown d3/d5/c29 165642943 1 2024-03-14T13:02:07.062 INFO:tasks.workunit.client.0.smithi111.stdout:2/58: link cb dd/cf 0 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:8/160: mkdir d3/d22/d30 0 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:1/77: sync 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:1/78: rmdir - no directory 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:9/63: sync 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:0/74: sync 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:5/175: sync 2024-03-14T13:02:07.064 INFO:tasks.workunit.client.0.smithi111.stdout:3/112: dwrite d0/de/d19/f26 [0,4194304] 0 2024-03-14T13:02:07.065 INFO:tasks.workunit.client.0.smithi111.stdout:2/59: truncate dd/fe 20905 0 2024-03-14T13:02:07.065 INFO:tasks.workunit.client.0.smithi111.stdout:1/79: stat c18 0 2024-03-14T13:02:07.065 INFO:tasks.workunit.client.0.smithi111.stdout:2/60: stat f4 0 2024-03-14T13:02:07.066 INFO:tasks.workunit.client.0.smithi111.stdout:3/113: write d0/de/d19/f26 [3702819,67661] 0 2024-03-14T13:02:07.067 INFO:tasks.workunit.client.0.smithi111.stdout:1/80: creat f1a x:0 0 0 2024-03-14T13:02:07.067 INFO:tasks.workunit.client.0.smithi111.stdout:8/161: write d3/d5/f26 [3612553,49850] 0 2024-03-14T13:02:07.067 INFO:tasks.workunit.client.0.smithi111.stdout:2/61: chown dd/fe 0 1 2024-03-14T13:02:07.068 INFO:tasks.workunit.client.0.smithi111.stdout:9/64: mknod d5/d6/c18 0 2024-03-14T13:02:07.069 INFO:tasks.workunit.client.0.smithi111.stdout:2/62: fsync f4 0 2024-03-14T13:02:07.070 INFO:tasks.workunit.client.0.smithi111.stdout:1/81: write f11 [675076,129363] 0 2024-03-14T13:02:07.070 INFO:tasks.workunit.client.0.smithi111.stdout:8/162: read - d3/d1b/f24 zero size 2024-03-14T13:02:07.072 INFO:tasks.workunit.client.0.smithi111.stdout:5/176: rename d2/d1c/f31 to d2/d16/d2c/f3b 0 2024-03-14T13:02:07.072 INFO:tasks.workunit.client.0.smithi111.stdout:8/163: fsync d3/d1b/f28 0 2024-03-14T13:02:07.075 INFO:tasks.workunit.client.0.smithi111.stdout:1/82: mknod c1b 0 2024-03-14T13:02:07.076 INFO:tasks.workunit.client.0.smithi111.stdout:2/63: creat dd/f10 x:0 0 0 2024-03-14T13:02:07.077 INFO:tasks.workunit.client.0.smithi111.stdout:2/64: chown f1 78 1 2024-03-14T13:02:07.079 INFO:tasks.workunit.client.0.smithi111.stdout:2/65: dread dd/fe [0,4194304] 0 2024-03-14T13:02:07.080 INFO:tasks.workunit.client.0.smithi111.stdout:9/65: symlink d5/l19 0 2024-03-14T13:02:07.081 INFO:tasks.workunit.client.0.smithi111.stdout:8/164: mkdir d3/dc/d31 0 2024-03-14T13:02:07.082 INFO:tasks.workunit.client.0.smithi111.stdout:2/66: truncate dd/f10 953063 0 2024-03-14T13:02:07.083 INFO:tasks.workunit.client.0.smithi111.stdout:9/66: stat d5/d6/d8/f12 0 2024-03-14T13:02:07.083 INFO:tasks.workunit.client.0.smithi111.stdout:2/67: stat c8 0 2024-03-14T13:02:07.083 INFO:tasks.workunit.client.0.smithi111.stdout:9/67: readlink d5/l19 0 2024-03-14T13:02:07.083 INFO:tasks.workunit.client.0.smithi111.stdout:1/83: dwrite ff [0,4194304] 0 2024-03-14T13:02:07.084 INFO:tasks.workunit.client.0.smithi111.stdout:8/165: truncate d3/d1b/f1d 536055 0 2024-03-14T13:02:07.086 INFO:tasks.workunit.client.0.smithi111.stdout:5/177: rmdir d2 39 2024-03-14T13:02:07.088 INFO:tasks.workunit.client.0.smithi111.stdout:1/84: mknod c1c 0 2024-03-14T13:02:07.089 INFO:tasks.workunit.client.0.smithi111.stdout:1/85: readlink l16 0 2024-03-14T13:02:07.091 INFO:tasks.workunit.client.0.smithi111.stdout:9/68: mkdir d5/d6/d8/d1a 0 2024-03-14T13:02:07.095 INFO:tasks.workunit.client.0.smithi111.stdout:3/114: truncate d0/ff 753031 0 2024-03-14T13:02:07.096 INFO:tasks.workunit.client.0.smithi111.stdout:1/86: rename l6 to l1d 0 2024-03-14T13:02:07.096 INFO:tasks.workunit.client.0.smithi111.stdout:2/68: dwrite dd/f10 [0,4194304] 0 2024-03-14T13:02:07.096 INFO:tasks.workunit.client.0.smithi111.stdout:9/69: symlink d5/d6/l1b 0 2024-03-14T13:02:07.096 INFO:tasks.workunit.client.0.smithi111.stdout:9/70: readlink d5/l19 0 2024-03-14T13:02:07.097 INFO:tasks.workunit.client.0.smithi111.stdout:1/87: rename l13 to l1e 0 2024-03-14T13:02:07.099 INFO:tasks.workunit.client.0.smithi111.stdout:2/69: mkdir dd/d11 0 2024-03-14T13:02:07.101 INFO:tasks.workunit.client.0.smithi111.stdout:5/178: creat d2/d1f/d36/f3c x:0 0 0 2024-03-14T13:02:07.103 INFO:tasks.workunit.client.0.smithi111.stdout:8/166: rename d3/d1b/d20/f2e to d3/f32 0 2024-03-14T13:02:07.105 INFO:tasks.workunit.client.0.smithi111.stdout:9/71: creat d5/d6/d14/f1c x:0 0 0 2024-03-14T13:02:07.106 INFO:tasks.workunit.client.0.smithi111.stdout:1/88: mknod c1f 0 2024-03-14T13:02:07.108 INFO:tasks.workunit.client.0.smithi111.stdout:2/70: mknod dd/c12 0 2024-03-14T13:02:07.108 INFO:tasks.workunit.client.0.smithi111.stdout:2/71: chown c0 257750766 1 2024-03-14T13:02:07.110 INFO:tasks.workunit.client.0.smithi111.stdout:2/72: chown lc 20937125 1 2024-03-14T13:02:07.111 INFO:tasks.workunit.client.0.smithi111.stdout:5/179: creat d2/d13/f3d x:0 0 0 2024-03-14T13:02:07.112 INFO:tasks.workunit.client.0.smithi111.stdout:8/167: symlink d3/d1b/d20/l33 0 2024-03-14T13:02:07.114 INFO:tasks.workunit.client.0.smithi111.stdout:1/89: write f17 [4746814,51968] 0 2024-03-14T13:02:07.115 INFO:tasks.workunit.client.0.smithi111.stdout:2/73: mknod dd/d11/c13 0 2024-03-14T13:02:07.115 INFO:tasks.workunit.client.0.smithi111.stdout:9/72: dwrite d5/d6/d8/f12 [0,4194304] 0 2024-03-14T13:02:07.117 INFO:tasks.workunit.client.0.smithi111.stdout:9/73: fdatasync d5/d6/d14/f1c 0 2024-03-14T13:02:07.119 INFO:tasks.workunit.client.0.smithi111.stdout:9/74: dread - d5/d6/fc zero size 2024-03-14T13:02:07.120 INFO:tasks.workunit.client.0.smithi111.stdout:2/74: truncate f4 2232161 0 2024-03-14T13:02:07.122 INFO:tasks.workunit.client.0.smithi111.stdout:8/168: dread d3/f14 [0,4194304] 0 2024-03-14T13:02:07.123 INFO:tasks.workunit.client.0.smithi111.stdout:5/180: dread d2/d13/f15 [0,4194304] 0 2024-03-14T13:02:07.123 INFO:tasks.workunit.client.0.smithi111.stdout:9/75: dread - d5/d6/d8/f16 zero size 2024-03-14T13:02:07.125 INFO:tasks.workunit.client.0.smithi111.stdout:9/76: write f4 [1691781,67348] 0 2024-03-14T13:02:07.126 INFO:tasks.workunit.client.0.smithi111.stdout:8/169: mkdir d3/d1b/d34 0 2024-03-14T13:02:07.126 INFO:tasks.workunit.client.0.smithi111.stdout:1/90: dwrite fe [0,4194304] 0 2024-03-14T13:02:07.126 INFO:tasks.workunit.client.0.smithi111.stdout:9/77: stat d5/c17 0 2024-03-14T13:02:07.128 INFO:tasks.workunit.client.0.smithi111.stdout:9/78: chown d5/d6/d14/c15 659 1 2024-03-14T13:02:07.128 INFO:tasks.workunit.client.0.smithi111.stdout:1/91: mkdir d20 0 2024-03-14T13:02:07.130 INFO:tasks.workunit.client.0.smithi111.stdout:8/170: dread d3/d1b/f1d [0,4194304] 0 2024-03-14T13:02:07.131 INFO:tasks.workunit.client.0.smithi111.stdout:5/181: dread d2/d16/d2c/f3b [0,4194304] 0 2024-03-14T13:02:07.131 INFO:tasks.workunit.client.0.smithi111.stdout:1/92: mkdir d20/d21 0 2024-03-14T13:02:07.132 INFO:tasks.workunit.client.0.smithi111.stdout:9/79: truncate d5/d6/d8/f12 4452869 0 2024-03-14T13:02:07.132 INFO:tasks.workunit.client.0.smithi111.stdout:8/171: chown d3/ca 877926 1 2024-03-14T13:02:07.132 INFO:tasks.workunit.client.0.smithi111.stdout:1/93: chown l16 11915 1 2024-03-14T13:02:07.132 INFO:tasks.workunit.client.0.smithi111.stdout:9/80: fdatasync d5/d6/d14/f1c 0 2024-03-14T13:02:07.133 INFO:tasks.workunit.client.0.smithi111.stdout:1/94: chown d20/d21 33168503 1 2024-03-14T13:02:07.134 INFO:tasks.workunit.client.0.smithi111.stdout:5/182: creat d2/d16/d2c/f3e x:0 0 0 2024-03-14T13:02:07.135 INFO:tasks.workunit.client.0.smithi111.stdout:8/172: mkdir d3/d1b/d20/d35 0 2024-03-14T13:02:07.136 INFO:tasks.workunit.client.0.smithi111.stdout:8/173: dread - d3/d1b/f28 zero size 2024-03-14T13:02:07.138 INFO:tasks.workunit.client.0.smithi111.stdout:1/95: getdents d20/d21 0 2024-03-14T13:02:07.139 INFO:tasks.workunit.client.0.smithi111.stdout:8/174: dread - d3/d1b/d20/f2b zero size 2024-03-14T13:02:07.139 INFO:tasks.workunit.client.0.smithi111.stdout:9/81: dread d5/fe [0,4194304] 0 2024-03-14T13:02:07.141 INFO:tasks.workunit.client.0.smithi111.stdout:1/96: symlink d20/l22 0 2024-03-14T13:02:07.144 INFO:tasks.workunit.client.0.smithi111.stdout:1/97: creat d20/f23 x:0 0 0 2024-03-14T13:02:07.146 INFO:tasks.workunit.client.0.smithi111.stdout:5/183: dwrite d2/d16/d2c/f3e [0,4194304] 0 2024-03-14T13:02:07.150 INFO:tasks.workunit.client.0.smithi111.stdout:1/98: dread f9 [0,4194304] 0 2024-03-14T13:02:07.151 INFO:tasks.workunit.client.0.smithi111.stdout:9/82: dwrite f2 [0,4194304] 0 2024-03-14T13:02:07.152 INFO:tasks.workunit.client.0.smithi111.stdout:9/83: chown d5/d6/fc 3551668 1 2024-03-14T13:02:07.154 INFO:tasks.workunit.client.0.smithi111.stdout:1/99: link f17 d20/f24 0 2024-03-14T13:02:07.155 INFO:tasks.workunit.client.0.smithi111.stdout:1/100: write fa [670375,98047] 0 2024-03-14T13:02:07.163 INFO:tasks.workunit.client.0.smithi111.stdout:9/84: dread d5/d6/f7 [0,4194304] 0 2024-03-14T13:02:07.163 INFO:tasks.workunit.client.0.smithi111.stdout:9/85: readlink d5/d6/l1b 0 2024-03-14T13:02:07.165 INFO:tasks.workunit.client.0.smithi111.stdout:9/86: mknod d5/d6/d8/c1d 0 2024-03-14T13:02:07.171 INFO:tasks.workunit.client.0.smithi111.stdout:8/175: getdents d3 0 2024-03-14T13:02:07.173 INFO:tasks.workunit.client.0.smithi111.stdout:8/176: symlink d3/d5/l36 0 2024-03-14T13:02:07.179 INFO:tasks.workunit.client.0.smithi111.stdout:9/87: fdatasync d5/d6/d8/f12 0 2024-03-14T13:02:07.186 INFO:tasks.workunit.client.0.smithi111.stdout:8/177: dwrite d3/f13 [0,4194304] 0 2024-03-14T13:02:07.186 INFO:tasks.workunit.client.0.smithi111.stdout:5/184: write d2/d19/f12 [1481229,31252] 0 2024-03-14T13:02:07.187 INFO:tasks.workunit.client.0.smithi111.stdout:1/101: getdents d20 0 2024-03-14T13:02:07.187 INFO:tasks.workunit.client.0.smithi111.stdout:2/75: dwrite dd/fe [0,4194304] 0 2024-03-14T13:02:07.190 INFO:tasks.workunit.client.0.smithi111.stdout:5/185: fsync d2/d1f/f24 0 2024-03-14T13:02:07.191 INFO:tasks.workunit.client.0.smithi111.stdout:8/178: write d3/d1b/f24 [555075,67660] 0 2024-03-14T13:02:07.191 INFO:tasks.workunit.client.0.smithi111.stdout:1/102: mkdir d20/d25 0 2024-03-14T13:02:07.192 INFO:tasks.workunit.client.0.smithi111.stdout:1/103: truncate f11 1450197 0 2024-03-14T13:02:07.193 INFO:tasks.workunit.client.0.smithi111.stdout:2/76: symlink dd/l14 0 2024-03-14T13:02:07.195 INFO:tasks.workunit.client.0.smithi111.stdout:5/186: fsync d2/d1f/f24 0 2024-03-14T13:02:07.196 INFO:tasks.workunit.client.0.smithi111.stdout:8/179: unlink d3/d1b/l23 0 2024-03-14T13:02:07.198 INFO:tasks.workunit.client.0.smithi111.stdout:1/104: symlink d20/l26 0 2024-03-14T13:02:07.200 INFO:tasks.workunit.client.0.smithi111.stdout:5/187: dread d2/d16/d2c/f3b [0,4194304] 0 2024-03-14T13:02:07.201 INFO:tasks.workunit.client.0.smithi111.stdout:8/180: creat d3/d1b/d20/f37 x:0 0 0 2024-03-14T13:02:07.203 INFO:tasks.workunit.client.0.smithi111.stdout:2/77: truncate f4 864757 0 2024-03-14T13:02:07.203 INFO:tasks.workunit.client.0.smithi111.stdout:5/188: mknod d2/d13/c3f 0 2024-03-14T13:02:07.204 INFO:tasks.workunit.client.0.smithi111.stdout:1/105: dread d20/f24 [4194304,4194304] 0 2024-03-14T13:02:07.205 INFO:tasks.workunit.client.0.smithi111.stdout:2/78: mknod dd/c15 0 2024-03-14T13:02:07.206 INFO:tasks.workunit.client.0.smithi111.stdout:8/181: creat d3/dc/d31/f38 x:0 0 0 2024-03-14T13:02:07.207 INFO:tasks.workunit.client.0.smithi111.stdout:8/182: readlink d3/d5/l15 0 2024-03-14T13:02:07.210 INFO:tasks.workunit.client.0.smithi111.stdout:8/183: write d3/f32 [812125,23237] 0 2024-03-14T13:02:07.211 INFO:tasks.workunit.client.0.smithi111.stdout:8/184: fsync d3/d5/fb 0 2024-03-14T13:02:07.212 INFO:tasks.workunit.client.0.smithi111.stdout:5/189: dwrite d2/d13/f3d [0,4194304] 0 2024-03-14T13:02:07.212 INFO:tasks.workunit.client.0.smithi111.stdout:8/185: stat d3 0 2024-03-14T13:02:07.214 INFO:tasks.workunit.client.0.smithi111.stdout:5/190: truncate d2/d1f/d36/f3c 135531 0 2024-03-14T13:02:07.217 INFO:tasks.workunit.client.0.smithi111.stdout:8/186: rename d3/f13 to d3/d1b/d20/d35/f39 0 2024-03-14T13:02:07.223 INFO:tasks.workunit.client.0.smithi111.stdout:2/79: fsync f4 0 2024-03-14T13:02:07.224 INFO:tasks.workunit.client.0.smithi111.stdout:2/80: fsync dd/f10 0 2024-03-14T13:02:07.224 INFO:tasks.workunit.client.0.smithi111.stdout:6/110: sync 2024-03-14T13:02:07.225 INFO:tasks.workunit.client.0.smithi111.stdout:7/95: sync 2024-03-14T13:02:07.227 INFO:tasks.workunit.client.0.smithi111.stdout:1/106: truncate f17 5774232 0 2024-03-14T13:02:07.227 INFO:tasks.workunit.client.0.smithi111.stdout:7/96: readlink d0/d1/l8 0 2024-03-14T13:02:07.229 INFO:tasks.workunit.client.0.smithi111.stdout:6/111: rename d0/fa to d0/d17/d21/f22 0 2024-03-14T13:02:07.231 INFO:tasks.workunit.client.0.smithi111.stdout:7/97: write d0/d1/fd [732270,117389] 0 2024-03-14T13:02:07.233 INFO:tasks.workunit.client.0.smithi111.stdout:2/81: dwrite dd/f10 [0,4194304] 0 2024-03-14T13:02:07.235 INFO:tasks.workunit.client.0.smithi111.stdout:6/112: symlink d0/l23 0 2024-03-14T13:02:07.236 INFO:tasks.workunit.client.0.smithi111.stdout:1/107: dread fa [0,4194304] 0 2024-03-14T13:02:07.236 INFO:tasks.workunit.client.0.smithi111.stdout:6/113: dread - d0/f1d zero size 2024-03-14T13:02:07.237 INFO:tasks.workunit.client.0.smithi111.stdout:7/98: mkdir d0/d1/d5/de/d1b 0 2024-03-14T13:02:07.238 INFO:tasks.workunit.client.0.smithi111.stdout:1/108: fdatasync f1a 0 2024-03-14T13:02:07.239 INFO:tasks.workunit.client.0.smithi111.stdout:6/114: fdatasync d0/d17/f1f 0 2024-03-14T13:02:07.240 INFO:tasks.workunit.client.0.smithi111.stdout:1/109: readlink l1d 0 2024-03-14T13:02:07.241 INFO:tasks.workunit.client.0.smithi111.stdout:7/99: mknod d0/c1c 0 2024-03-14T13:02:07.242 INFO:tasks.workunit.client.0.smithi111.stdout:6/115: rmdir d0 39 2024-03-14T13:02:07.243 INFO:tasks.workunit.client.0.smithi111.stdout:8/187: dwrite f0 [0,4194304] 0 2024-03-14T13:02:07.244 INFO:tasks.workunit.client.0.smithi111.stdout:1/110: truncate d20/f23 789044 0 2024-03-14T13:02:07.247 INFO:tasks.workunit.client.0.smithi111.stdout:2/82: dwrite dd/f10 [0,4194304] 0 2024-03-14T13:02:07.247 INFO:tasks.workunit.client.0.smithi111.stdout:6/116: mkdir d0/d17/d24 0 2024-03-14T13:02:07.248 INFO:tasks.workunit.client.0.smithi111.stdout:6/117: chown d0/f1d 542736 1 2024-03-14T13:02:07.250 INFO:tasks.workunit.client.0.smithi111.stdout:7/100: rename d0/d1/l8 to d0/d1/d5/de/d9/l1d 0 2024-03-14T13:02:07.252 INFO:tasks.workunit.client.0.smithi111.stdout:8/188: creat d3/d1b/f3a x:0 0 0 2024-03-14T13:02:07.254 INFO:tasks.workunit.client.0.smithi111.stdout:8/189: chown d3/d22 4761179 1 2024-03-14T13:02:07.256 INFO:tasks.workunit.client.0.smithi111.stdout:1/111: rename l1d to d20/l27 0 2024-03-14T13:02:07.256 INFO:tasks.workunit.client.0.smithi111.stdout:2/83: link la dd/l16 0 2024-03-14T13:02:07.259 INFO:tasks.workunit.client.0.smithi111.stdout:2/84: mkdir dd/d17 0 2024-03-14T13:02:07.261 INFO:tasks.workunit.client.0.smithi111.stdout:7/101: link d0/d1/d5/de/d9/f12 d0/d1/d5/de/d1b/f1e 0 2024-03-14T13:02:07.262 INFO:tasks.workunit.client.0.smithi111.stdout:6/118: link d0/c5 d0/d17/d21/c25 0 2024-03-14T13:02:07.262 INFO:tasks.workunit.client.0.smithi111.stdout:7/102: dread - d0/d1/d5/f1a zero size 2024-03-14T13:02:07.263 INFO:tasks.workunit.client.0.smithi111.stdout:8/190: creat d3/d1b/d34/f3b x:0 0 0 2024-03-14T13:02:07.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/119: write d0/f14 [3680068,97651] 0 2024-03-14T13:02:07.265 INFO:tasks.workunit.client.0.smithi111.stdout:7/103: fsync d0/d1/d5/de/d9/f16 0 2024-03-14T13:02:07.265 INFO:tasks.workunit.client.0.smithi111.stdout:1/112: mkdir d20/d21/d28 0 2024-03-14T13:02:07.266 INFO:tasks.workunit.client.0.smithi111.stdout:1/113: chown f19 5 1 2024-03-14T13:02:07.267 INFO:tasks.workunit.client.0.smithi111.stdout:2/85: mkdir dd/d17/d18 0 2024-03-14T13:02:07.267 INFO:tasks.workunit.client.0.smithi111.stdout:7/104: chown d0/d1/d5/de/d14 174752 1 2024-03-14T13:02:07.269 INFO:tasks.workunit.client.0.smithi111.stdout:6/120: creat d0/d17/d20/f26 x:0 0 0 2024-03-14T13:02:07.269 INFO:tasks.workunit.client.0.smithi111.stdout:1/114: dread f9 [0,4194304] 0 2024-03-14T13:02:07.270 INFO:tasks.workunit.client.0.smithi111.stdout:8/191: fsync d3/d5/f1e 0 2024-03-14T13:02:07.271 INFO:tasks.workunit.client.0.smithi111.stdout:8/192: truncate d3/d5/f19 841316 0 2024-03-14T13:02:07.272 INFO:tasks.workunit.client.0.smithi111.stdout:8/193: stat d3/d1b/f2c 0 2024-03-14T13:02:07.273 INFO:tasks.workunit.client.0.smithi111.stdout:8/194: fsync d3/d1b/f24 0 2024-03-14T13:02:07.273 INFO:tasks.workunit.client.0.smithi111.stdout:6/121: symlink d0/l27 0 2024-03-14T13:02:07.274 INFO:tasks.workunit.client.0.smithi111.stdout:8/195: stat d3/d5/f1e 0 2024-03-14T13:02:07.275 INFO:tasks.workunit.client.0.smithi111.stdout:7/105: mknod d0/d11/c1f 0 2024-03-14T13:02:07.276 INFO:tasks.workunit.client.0.smithi111.stdout:7/106: write d0/d1/d5/f1a [167157,23235] 0 2024-03-14T13:02:07.277 INFO:tasks.workunit.client.0.smithi111.stdout:1/115: fdatasync d20/f24 0 2024-03-14T13:02:07.280 INFO:tasks.workunit.client.0.smithi111.stdout:6/122: creat d0/d17/d21/f28 x:0 0 0 2024-03-14T13:02:07.280 INFO:tasks.workunit.client.0.smithi111.stdout:6/123: dread - d0/d17/d20/f26 zero size 2024-03-14T13:02:07.281 INFO:tasks.workunit.client.0.smithi111.stdout:6/124: chown d0 434939579 1 2024-03-14T13:02:07.281 INFO:tasks.workunit.client.0.smithi111.stdout:8/196: creat d3/d1b/d34/f3c x:0 0 0 2024-03-14T13:02:07.282 INFO:tasks.workunit.client.0.smithi111.stdout:8/197: fsync d3/d5/fb 0 2024-03-14T13:02:07.282 INFO:tasks.workunit.client.0.smithi111.stdout:6/125: dread - d0/d17/d20/f26 zero size 2024-03-14T13:02:07.283 INFO:tasks.workunit.client.0.smithi111.stdout:6/126: read d0/f4 [2639668,104135] 0 2024-03-14T13:02:07.284 INFO:tasks.workunit.client.0.smithi111.stdout:6/127: dread - d0/d17/f1f zero size 2024-03-14T13:02:07.287 INFO:tasks.workunit.client.0.smithi111.stdout:1/116: mknod d20/d21/d28/c29 0 2024-03-14T13:02:07.288 INFO:tasks.workunit.client.0.smithi111.stdout:1/117: chown f3 57704195 1 2024-03-14T13:02:07.290 INFO:tasks.workunit.client.0.smithi111.stdout:1/118: write f9 [271670,39452] 0 2024-03-14T13:02:07.290 INFO:tasks.workunit.client.0.smithi111.stdout:8/198: dwrite d3/d1b/d34/f3b [0,4194304] 0 2024-03-14T13:02:07.292 INFO:tasks.workunit.client.0.smithi111.stdout:2/86: getdents dd/d17 0 2024-03-14T13:02:07.297 INFO:tasks.workunit.client.0.smithi111.stdout:2/87: mknod dd/d17/d18/c19 0 2024-03-14T13:02:07.300 INFO:tasks.workunit.client.0.smithi111.stdout:8/199: rename d3/l16 to d3/d1b/d20/l3d 0 2024-03-14T13:02:07.301 INFO:tasks.workunit.client.0.smithi111.stdout:1/119: dwrite f9 [0,4194304] 0 2024-03-14T13:02:07.301 INFO:tasks.workunit.client.0.smithi111.stdout:8/200: truncate d3/d1b/f2c 16616 0 2024-03-14T13:02:07.304 INFO:tasks.workunit.client.0.smithi111.stdout:1/120: creat d20/d21/f2a x:0 0 0 2024-03-14T13:02:07.307 INFO:tasks.workunit.client.0.smithi111.stdout:1/121: mknod d20/d25/c2b 0 2024-03-14T13:02:07.309 INFO:tasks.workunit.client.0.smithi111.stdout:8/201: dwrite d3/d1b/f2c [0,4194304] 0 2024-03-14T13:02:07.309 INFO:tasks.workunit.client.0.smithi111.stdout:1/122: unlink f17 0 2024-03-14T13:02:07.313 INFO:tasks.workunit.client.0.smithi111.stdout:8/202: unlink d3/d1b/f28 0 2024-03-14T13:02:07.314 INFO:tasks.workunit.client.0.smithi111.stdout:1/123: dread f3 [0,4194304] 0 2024-03-14T13:02:07.315 INFO:tasks.workunit.client.0.smithi111.stdout:8/203: fdatasync d3/dc/f11 0 2024-03-14T13:02:07.316 INFO:tasks.workunit.client.0.smithi111.stdout:8/204: read d3/d5/fb [3610162,21713] 0 2024-03-14T13:02:07.323 INFO:tasks.workunit.client.0.smithi111.stdout:8/205: dwrite d3/dc/f21 [0,4194304] 0 2024-03-14T13:02:07.333 INFO:tasks.workunit.client.0.smithi111.stdout:8/206: truncate d3/d1b/d20/d35/f39 2433468 0 2024-03-14T13:02:07.333 INFO:tasks.workunit.client.0.smithi111.stdout:8/207: readlink d3/d5/l15 0 2024-03-14T13:02:07.334 INFO:tasks.workunit.client.0.smithi111.stdout:8/208: fsync d3/d1b/f2c 0 2024-03-14T13:02:07.337 INFO:tasks.workunit.client.0.smithi111.stdout:8/209: chown d3/d1b/d20/l3d 15071405 1 2024-03-14T13:02:07.337 INFO:tasks.workunit.client.0.smithi111.stdout:8/210: readlink d3/d5/l36 0 2024-03-14T13:02:07.345 INFO:tasks.workunit.client.0.smithi111.stdout:8/211: dwrite d3/d5/fb [4194304,4194304] 0 2024-03-14T13:02:07.346 INFO:tasks.workunit.client.0.smithi111.stdout:8/212: chown d3/dc/l17 193 1 2024-03-14T13:02:07.346 INFO:tasks.workunit.client.0.smithi111.stdout:8/213: chown d3/d1b 7142809 1 2024-03-14T13:02:07.347 INFO:tasks.workunit.client.0.smithi111.stdout:8/214: chown d3/dc/d31/f38 20 1 2024-03-14T13:02:07.350 INFO:tasks.workunit.client.0.smithi111.stdout:8/215: truncate d3/d5/f10 3538024 0 2024-03-14T13:02:07.354 INFO:tasks.workunit.client.0.smithi111.stdout:8/216: rename d3/dc/f18 to d3/f3e 0 2024-03-14T13:02:07.358 INFO:tasks.workunit.client.0.smithi111.stdout:8/217: link d3/d1b/d34/f3b d3/d22/f3f 0 2024-03-14T13:02:07.364 INFO:tasks.workunit.client.0.smithi111.stdout:8/218: getdents d3/dc 0 2024-03-14T13:02:07.364 INFO:tasks.workunit.client.0.smithi111.stdout:8/219: readlink d3/d5/l36 0 2024-03-14T13:02:07.365 INFO:tasks.workunit.client.0.smithi111.stdout:8/220: write d3/d1b/f2c [4216206,104728] 0 2024-03-14T13:02:07.366 INFO:tasks.workunit.client.0.smithi111.stdout:8/221: chown d3/d5/l36 8 1 2024-03-14T13:02:07.371 INFO:tasks.workunit.client.0.smithi111.stdout:8/222: write d3/d5/f1e [477485,16461] 0 2024-03-14T13:02:07.372 INFO:tasks.workunit.client.0.smithi111.stdout:8/223: read d3/d5/fb [2456514,90888] 0 2024-03-14T13:02:07.375 INFO:tasks.workunit.client.0.smithi111.stdout:8/224: mkdir d3/d1b/d20/d35/d40 0 2024-03-14T13:02:07.375 INFO:tasks.workunit.client.0.smithi111.stdout:8/225: fdatasync d3/f3e 0 2024-03-14T13:02:07.377 INFO:tasks.workunit.client.0.smithi111.stdout:8/226: symlink d3/d22/d30/l41 0 2024-03-14T13:02:07.378 INFO:tasks.workunit.client.0.smithi111.stdout:8/227: symlink d3/dc/l42 0 2024-03-14T13:02:07.380 INFO:tasks.workunit.client.0.smithi111.stdout:8/228: creat d3/d22/d30/f43 x:0 0 0 2024-03-14T13:02:07.381 INFO:tasks.workunit.client.0.smithi111.stdout:8/229: fsync d3/d1b/d20/d35/f39 0 2024-03-14T13:02:07.383 INFO:tasks.workunit.client.0.smithi111.stdout:8/230: symlink d3/d1b/d34/l44 0 2024-03-14T13:02:07.385 INFO:tasks.workunit.client.0.smithi111.stdout:8/231: mkdir d3/dc/d45 0 2024-03-14T13:02:07.387 INFO:tasks.workunit.client.0.smithi111.stdout:8/232: symlink d3/d1b/d20/d35/d40/l46 0 2024-03-14T13:02:07.388 INFO:tasks.workunit.client.0.smithi111.stdout:8/233: truncate d3/d1b/d20/f2b 426224 0 2024-03-14T13:02:07.388 INFO:tasks.workunit.client.0.smithi111.stdout:8/234: fdatasync d3/d1b/f3a 0 2024-03-14T13:02:07.391 INFO:tasks.workunit.client.0.smithi111.stdout:8/235: rename d3/dc/d31/f38 to d3/d22/d30/f47 0 2024-03-14T13:02:07.392 INFO:tasks.workunit.client.0.smithi111.stdout:8/236: unlink d3/d5/f26 0 2024-03-14T13:02:07.393 INFO:tasks.workunit.client.0.smithi111.stdout:8/237: chown d3/d22/d30 3554051 1 2024-03-14T13:02:07.396 INFO:tasks.workunit.client.0.smithi111.stdout:8/238: dread f0 [0,4194304] 0 2024-03-14T13:02:07.398 INFO:tasks.workunit.client.0.smithi111.stdout:8/239: mkdir d3/d1b/d20/d35/d40/d48 0 2024-03-14T13:02:07.400 INFO:tasks.workunit.client.0.smithi111.stdout:8/240: fdatasync d3/d5/f12 0 2024-03-14T13:02:07.402 INFO:tasks.workunit.client.0.smithi111.stdout:8/241: mkdir d3/d49 0 2024-03-14T13:02:07.404 INFO:tasks.workunit.client.0.smithi111.stdout:8/242: mkdir d3/d22/d4a 0 2024-03-14T13:02:07.405 INFO:tasks.workunit.client.0.smithi111.stdout:8/243: creat d3/dc/d31/f4b x:0 0 0 2024-03-14T13:02:07.406 INFO:tasks.workunit.client.0.smithi111.stdout:8/244: readlink d3/dc/l42 0 2024-03-14T13:02:07.410 INFO:tasks.workunit.client.0.smithi111.stdout:8/245: dread d3/dc/f21 [0,4194304] 0 2024-03-14T13:02:07.412 INFO:tasks.workunit.client.0.smithi111.stdout:8/246: creat d3/dc/d31/f4c x:0 0 0 2024-03-14T13:02:07.417 INFO:tasks.workunit.client.0.smithi111.stdout:8/247: dwrite d3/d5/fd [4194304,4194304] 0 2024-03-14T13:02:07.419 INFO:tasks.workunit.client.0.smithi111.stdout:8/248: fdatasync d3/d22/d30/f47 0 2024-03-14T13:02:07.420 INFO:tasks.workunit.client.0.smithi111.stdout:8/249: dread - d3/d1b/f3a zero size 2024-03-14T13:02:07.422 INFO:tasks.workunit.client.0.smithi111.stdout:8/250: fsync d3/d1b/f1d 0 2024-03-14T13:02:07.425 INFO:tasks.workunit.client.0.smithi111.stdout:8/251: stat d3/d5/fd 0 2024-03-14T13:02:07.428 INFO:tasks.workunit.client.0.smithi111.stdout:8/252: getdents d3/d5 0 2024-03-14T13:02:07.435 INFO:tasks.workunit.client.0.smithi111.stdout:8/253: dwrite d3/d5/fb [4194304,4194304] 0 2024-03-14T13:02:07.442 INFO:tasks.workunit.client.0.smithi111.stdout:8/254: dwrite d3/dc/d31/f4c [0,4194304] 0 2024-03-14T13:02:07.443 INFO:tasks.workunit.client.0.smithi111.stdout:0/75: sync 2024-03-14T13:02:07.444 INFO:tasks.workunit.client.0.smithi111.stdout:4/165: sync 2024-03-14T13:02:07.446 INFO:tasks.workunit.client.0.smithi111.stdout:8/255: rename d3/d1b/d20/d35/f39 to d3/d1b/f4d 0 2024-03-14T13:02:07.452 INFO:tasks.workunit.client.0.smithi111.stdout:8/256: rename d3/d1b/f2c to d3/dc/f4e 0 2024-03-14T13:02:07.453 INFO:tasks.workunit.client.0.smithi111.stdout:4/166: dwrite fb [0,4194304] 0 2024-03-14T13:02:07.454 INFO:tasks.workunit.client.0.smithi111.stdout:8/257: write d3/d1b/d34/f3c [139939,1033] 0 2024-03-14T13:02:07.458 INFO:tasks.workunit.client.0.smithi111.stdout:4/167: write d1d/d22/d30/f32 [803315,99340] 0 2024-03-14T13:02:07.460 INFO:tasks.workunit.client.0.smithi111.stdout:4/168: chown d1d/l1e 85 1 2024-03-14T13:02:07.462 INFO:tasks.workunit.client.0.smithi111.stdout:8/258: symlink d3/d1b/d20/d35/l4f 0 2024-03-14T13:02:07.462 INFO:tasks.workunit.client.0.smithi111.stdout:0/76: dwrite d6/d9/f12 [0,4194304] 0 2024-03-14T13:02:07.463 INFO:tasks.workunit.client.0.smithi111.stdout:0/77: dread - d6/f13 zero size 2024-03-14T13:02:07.469 INFO:tasks.workunit.client.0.smithi111.stdout:8/259: rename d3/d1b/f4d to d3/d1b/d20/d35/d40/d48/f50 0 2024-03-14T13:02:07.472 INFO:tasks.workunit.client.0.smithi111.stdout:4/169: dread ff [0,4194304] 0 2024-03-14T13:02:07.474 INFO:tasks.workunit.client.0.smithi111.stdout:8/260: creat d3/d5/f51 x:0 0 0 2024-03-14T13:02:07.475 INFO:tasks.workunit.client.0.smithi111.stdout:0/78: dwrite d6/fd [0,4194304] 0 2024-03-14T13:02:07.477 INFO:tasks.workunit.client.0.smithi111.stdout:4/170: unlink f1b 0 2024-03-14T13:02:07.479 INFO:tasks.workunit.client.0.smithi111.stdout:8/261: mknod d3/d1b/d20/d35/d40/c52 0 2024-03-14T13:02:07.480 INFO:tasks.workunit.client.0.smithi111.stdout:0/79: symlink d6/d9/de/l17 0 2024-03-14T13:02:07.480 INFO:tasks.workunit.client.0.smithi111.stdout:8/262: dread - d3/d1b/f3a zero size 2024-03-14T13:02:07.482 INFO:tasks.workunit.client.0.smithi111.stdout:0/80: chown d6/fb 493018323 1 2024-03-14T13:02:07.484 INFO:tasks.workunit.client.0.smithi111.stdout:8/263: creat d3/d49/f53 x:0 0 0 2024-03-14T13:02:07.485 INFO:tasks.workunit.client.0.smithi111.stdout:0/81: write d6/fd [1434020,54934] 0 2024-03-14T13:02:07.488 INFO:tasks.workunit.client.0.smithi111.stdout:4/171: dwrite d1d/d22/d23/f34 [0,4194304] 0 2024-03-14T13:02:07.488 INFO:tasks.workunit.client.0.smithi111.stdout:8/264: symlink d3/l54 0 2024-03-14T13:02:07.489 INFO:tasks.workunit.client.0.smithi111.stdout:0/82: write d6/fb [4186663,96655] 0 2024-03-14T13:02:07.490 INFO:tasks.workunit.client.0.smithi111.stdout:0/83: fdatasync d6/f10 0 2024-03-14T13:02:07.492 INFO:tasks.workunit.client.0.smithi111.stdout:8/265: rename d3/d5/l36 to d3/d22/l55 0 2024-03-14T13:02:07.494 INFO:tasks.workunit.client.0.smithi111.stdout:4/172: dread f11 [0,4194304] 0 2024-03-14T13:02:07.498 INFO:tasks.workunit.client.0.smithi111.stdout:4/173: mknod d1d/d22/d30/c39 0 2024-03-14T13:02:07.500 INFO:tasks.workunit.client.0.smithi111.stdout:4/174: fdatasync d1d/d22/f37 0 2024-03-14T13:02:07.501 INFO:tasks.workunit.client.0.smithi111.stdout:0/84: getdents d6 0 2024-03-14T13:02:07.502 INFO:tasks.workunit.client.0.smithi111.stdout:8/266: dwrite d3/d1b/d34/f3c [0,4194304] 0 2024-03-14T13:02:07.503 INFO:tasks.workunit.client.0.smithi111.stdout:4/175: read fb [3349825,124803] 0 2024-03-14T13:02:07.509 INFO:tasks.workunit.client.0.smithi111.stdout:8/267: creat d3/dc/d45/f56 x:0 0 0 2024-03-14T13:02:07.514 INFO:tasks.workunit.client.0.smithi111.stdout:0/85: dwrite d6/d9/f12 [4194304,4194304] 0 2024-03-14T13:02:07.516 INFO:tasks.workunit.client.0.smithi111.stdout:4/176: truncate f12 3951930 0 2024-03-14T13:02:07.517 INFO:tasks.workunit.client.0.smithi111.stdout:4/177: chown d1d/f20 422844393 1 2024-03-14T13:02:07.520 INFO:tasks.workunit.client.0.smithi111.stdout:4/178: mknod d1d/d22/d30/c3a 0 2024-03-14T13:02:07.522 INFO:tasks.workunit.client.0.smithi111.stdout:8/268: dwrite d3/d1b/d20/d35/d40/d48/f50 [0,4194304] 0 2024-03-14T13:02:07.528 INFO:tasks.workunit.client.0.smithi111.stdout:8/269: dread d3/d1b/d34/f3c [0,4194304] 0 2024-03-14T13:02:07.530 INFO:tasks.workunit.client.0.smithi111.stdout:8/270: mknod d3/d1b/d34/c57 0 2024-03-14T13:02:07.531 INFO:tasks.workunit.client.0.smithi111.stdout:4/179: write ff [1519856,106637] 0 2024-03-14T13:02:07.532 INFO:tasks.workunit.client.0.smithi111.stdout:8/271: mknod d3/dc/d45/c58 0 2024-03-14T13:02:07.534 INFO:tasks.workunit.client.0.smithi111.stdout:8/272: mkdir d3/dc/d59 0 2024-03-14T13:02:07.535 INFO:tasks.workunit.client.0.smithi111.stdout:8/273: fdatasync d3/d22/d30/f47 0 2024-03-14T13:02:07.536 INFO:tasks.workunit.client.0.smithi111.stdout:4/180: creat d1d/d22/f3b x:0 0 0 2024-03-14T13:02:07.536 INFO:tasks.workunit.client.0.smithi111.stdout:8/274: chown d3/d1b/d20/d35/d40/d48 1831833 1 2024-03-14T13:02:07.537 INFO:tasks.workunit.client.0.smithi111.stdout:4/181: truncate d1d/d22/f35 748085 0 2024-03-14T13:02:07.538 INFO:tasks.workunit.client.0.smithi111.stdout:8/275: mknod d3/dc/c5a 0 2024-03-14T13:02:07.539 INFO:tasks.workunit.client.0.smithi111.stdout:4/182: creat d1d/d22/d28/f3c x:0 0 0 2024-03-14T13:02:07.540 INFO:tasks.workunit.client.0.smithi111.stdout:8/276: symlink d3/d1b/d20/l5b 0 2024-03-14T13:02:07.542 INFO:tasks.workunit.client.0.smithi111.stdout:8/277: link d3/dc/d45/f56 d3/d22/d4a/f5c 0 2024-03-14T13:02:07.543 INFO:tasks.workunit.client.0.smithi111.stdout:8/278: truncate d3/dc/d45/f56 47438 0 2024-03-14T13:02:07.546 INFO:tasks.workunit.client.0.smithi111.stdout:8/279: truncate d3/d1b/d34/f3b 1038795 0 2024-03-14T13:02:07.546 INFO:tasks.workunit.client.0.smithi111.stdout:4/183: dread fe [0,4194304] 0 2024-03-14T13:02:07.547 INFO:tasks.workunit.client.0.smithi111.stdout:8/280: fsync d3/dc/f21 0 2024-03-14T13:02:07.547 INFO:tasks.workunit.client.0.smithi111.stdout:4/184: creat d1d/d22/d30/f3d x:0 0 0 2024-03-14T13:02:07.549 INFO:tasks.workunit.client.0.smithi111.stdout:4/185: creat d1d/d22/d28/f3e x:0 0 0 2024-03-14T13:02:07.551 INFO:tasks.workunit.client.0.smithi111.stdout:8/281: link d3/d1b/d20/l33 d3/d1b/d20/d35/d40/d48/l5d 0 2024-03-14T13:02:07.553 INFO:tasks.workunit.client.0.smithi111.stdout:8/282: symlink d3/d1b/d20/l5e 0 2024-03-14T13:02:07.554 INFO:tasks.workunit.client.0.smithi111.stdout:4/186: link c16 d1d/d22/d23/c3f 0 2024-03-14T13:02:07.554 INFO:tasks.workunit.client.0.smithi111.stdout:8/283: chown d3/d1b/d34/l44 59497 1 2024-03-14T13:02:07.556 INFO:tasks.workunit.client.0.smithi111.stdout:4/187: mkdir d1d/d22/d40 0 2024-03-14T13:02:07.556 INFO:tasks.workunit.client.0.smithi111.stdout:8/284: dread d3/d1b/d20/f2b [0,4194304] 0 2024-03-14T13:02:07.557 INFO:tasks.workunit.client.0.smithi111.stdout:4/188: write f11 [4519569,108751] 0 2024-03-14T13:02:07.559 INFO:tasks.workunit.client.0.smithi111.stdout:4/189: write d1d/d22/d30/f3d [455772,81717] 0 2024-03-14T13:02:07.560 INFO:tasks.workunit.client.0.smithi111.stdout:4/190: stat d1d/d22/d40 0 2024-03-14T13:02:07.561 INFO:tasks.workunit.client.0.smithi111.stdout:4/191: write d1d/d22/f31 [1794322,62827] 0 2024-03-14T13:02:07.562 INFO:tasks.workunit.client.0.smithi111.stdout:4/192: truncate d1d/f2c 4253534 0 2024-03-14T13:02:07.566 INFO:tasks.workunit.client.0.smithi111.stdout:4/193: dread d1d/f26 [0,4194304] 0 2024-03-14T13:02:07.567 INFO:tasks.workunit.client.0.smithi111.stdout:4/194: chown d1d/d22/f31 15 1 2024-03-14T13:02:07.572 INFO:tasks.workunit.client.0.smithi111.stdout:4/195: write d1d/d22/f37 [568584,116825] 0 2024-03-14T13:02:07.573 INFO:tasks.workunit.client.0.smithi111.stdout:8/285: dread d3/d1b/d34/f3b [0,4194304] 0 2024-03-14T13:02:07.574 INFO:tasks.workunit.client.0.smithi111.stdout:8/286: readlink d3/dc/l17 0 2024-03-14T13:02:07.574 INFO:tasks.workunit.client.0.smithi111.stdout:4/196: read d1d/d22/d30/f3d [73868,62773] 0 2024-03-14T13:02:07.576 INFO:tasks.workunit.client.0.smithi111.stdout:8/287: chown d3/d1b/d20/d35/d40/d48/l5d 235 1 2024-03-14T13:02:07.577 INFO:tasks.workunit.client.0.smithi111.stdout:4/197: fdatasync d1d/d22/d30/f3d 0 2024-03-14T13:02:07.578 INFO:tasks.workunit.client.0.smithi111.stdout:8/288: mknod d3/dc/d31/c5f 0 2024-03-14T13:02:07.579 INFO:tasks.workunit.client.0.smithi111.stdout:8/289: chown d3/d1b/d34/f3c 457 1 2024-03-14T13:02:07.581 INFO:tasks.workunit.client.0.smithi111.stdout:4/198: dread f11 [0,4194304] 0 2024-03-14T13:02:07.582 INFO:tasks.workunit.client.0.smithi111.stdout:4/199: truncate f11 5353666 0 2024-03-14T13:02:07.582 INFO:tasks.workunit.client.0.smithi111.stdout:8/290: rename d3/d1b/d20/d35/d40/c52 to d3/d49/c60 0 2024-03-14T13:02:07.584 INFO:tasks.workunit.client.0.smithi111.stdout:8/291: mknod d3/d1b/d34/c61 0 2024-03-14T13:02:07.584 INFO:tasks.workunit.client.0.smithi111.stdout:4/200: creat d1d/d22/f41 x:0 0 0 2024-03-14T13:02:07.585 INFO:tasks.workunit.client.0.smithi111.stdout:8/292: write d3/d5/f19 [598849,78586] 0 2024-03-14T13:02:07.588 INFO:tasks.workunit.client.0.smithi111.stdout:4/201: rename d1d/d22/d28/c2d to d1d/d22/d40/c42 0 2024-03-14T13:02:07.589 INFO:tasks.workunit.client.0.smithi111.stdout:4/202: truncate d1d/f2e 565094 0 2024-03-14T13:02:07.590 INFO:tasks.workunit.client.0.smithi111.stdout:4/203: stat d1d/d22/f37 0 2024-03-14T13:02:07.592 INFO:tasks.workunit.client.0.smithi111.stdout:8/293: dwrite d3/d49/f53 [0,4194304] 0 2024-03-14T13:02:07.592 INFO:tasks.workunit.client.0.smithi111.stdout:4/204: creat d1d/f43 x:0 0 0 2024-03-14T13:02:07.593 INFO:tasks.workunit.client.0.smithi111.stdout:8/294: chown d3/dc/d45/c58 5808 1 2024-03-14T13:02:07.599 INFO:tasks.workunit.client.0.smithi111.stdout:4/205: creat d1d/d22/f44 x:0 0 0 2024-03-14T13:02:07.602 INFO:tasks.workunit.client.0.smithi111.stdout:4/206: mknod d1d/c45 0 2024-03-14T13:02:07.602 INFO:tasks.workunit.client.0.smithi111.stdout:8/295: link d3/dc/l17 d3/d1b/d20/d35/d40/d48/l62 0 2024-03-14T13:02:07.602 INFO:tasks.workunit.client.0.smithi111.stdout:8/296: stat f0 0 2024-03-14T13:02:07.605 INFO:tasks.workunit.client.0.smithi111.stdout:8/297: rename d3/d1b/f24 to d3/dc/d59/f63 0 2024-03-14T13:02:07.606 INFO:tasks.workunit.client.0.smithi111.stdout:4/207: link d1d/d22/d28/f29 d1d/d22/d23/f46 0 2024-03-14T13:02:07.607 INFO:tasks.workunit.client.0.smithi111.stdout:4/208: read fe [160742,47858] 0 2024-03-14T13:02:07.607 INFO:tasks.workunit.client.0.smithi111.stdout:4/209: chown c10 87701 1 2024-03-14T13:02:07.609 INFO:tasks.workunit.client.0.smithi111.stdout:8/298: rename d3/d1b/d20/d35/d40/d48/l62 to d3/d1b/d34/l64 0 2024-03-14T13:02:07.611 INFO:tasks.workunit.client.0.smithi111.stdout:8/299: write d3/dc/d31/f4b [692994,13395] 0 2024-03-14T13:02:07.611 INFO:tasks.workunit.client.0.smithi111.stdout:4/210: symlink d1d/d22/l47 0 2024-03-14T13:02:07.615 INFO:tasks.workunit.client.0.smithi111.stdout:4/211: mkdir d1d/d48 0 2024-03-14T13:02:07.617 INFO:tasks.workunit.client.0.smithi111.stdout:8/300: link d3/d22/d30/f47 d3/f65 0 2024-03-14T13:02:07.618 INFO:tasks.workunit.client.0.smithi111.stdout:4/212: dread d1d/d22/d30/f3d [0,4194304] 0 2024-03-14T13:02:07.619 INFO:tasks.workunit.client.0.smithi111.stdout:8/301: mkdir d3/d1b/d66 0 2024-03-14T13:02:07.620 INFO:tasks.workunit.client.0.smithi111.stdout:4/213: symlink d1d/d22/l49 0 2024-03-14T13:02:07.621 INFO:tasks.workunit.client.0.smithi111.stdout:8/302: mknod d3/dc/d45/c67 0 2024-03-14T13:02:07.622 INFO:tasks.workunit.client.0.smithi111.stdout:4/214: unlink d1d/c45 0 2024-03-14T13:02:07.623 INFO:tasks.workunit.client.0.smithi111.stdout:4/215: write d1d/d22/f41 [771843,129809] 0 2024-03-14T13:02:07.623 INFO:tasks.workunit.client.0.smithi111.stdout:8/303: creat d3/d1b/d20/d35/d40/d48/f68 x:0 0 0 2024-03-14T13:02:07.624 INFO:tasks.workunit.client.0.smithi111.stdout:8/304: write d3/d5/f51 [67691,107115] 0 2024-03-14T13:02:07.626 INFO:tasks.workunit.client.0.smithi111.stdout:8/305: write d3/d1b/d20/f37 [517798,55505] 0 2024-03-14T13:02:07.628 INFO:tasks.workunit.client.0.smithi111.stdout:4/216: dread d1d/f26 [0,4194304] 0 2024-03-14T13:02:07.629 INFO:tasks.workunit.client.0.smithi111.stdout:8/306: symlink d3/d1b/d20/d35/l69 0 2024-03-14T13:02:07.629 INFO:tasks.workunit.client.0.smithi111.stdout:4/217: write d1d/d22/f37 [1097877,113982] 0 2024-03-14T13:02:07.631 INFO:tasks.workunit.client.0.smithi111.stdout:4/218: readlink l18 0 2024-03-14T13:02:07.632 INFO:tasks.workunit.client.0.smithi111.stdout:8/307: rename d3/dc/f21 to d3/dc/f6a 0 2024-03-14T13:02:07.633 INFO:tasks.workunit.client.0.smithi111.stdout:4/219: dread fe [0,4194304] 0 2024-03-14T13:02:07.633 INFO:tasks.workunit.client.0.smithi111.stdout:4/220: chown d1d/d22/l49 3 1 2024-03-14T13:02:07.635 INFO:tasks.workunit.client.0.smithi111.stdout:8/308: creat d3/f6b x:0 0 0 2024-03-14T13:02:07.635 INFO:tasks.workunit.client.0.smithi111.stdout:4/221: mknod d1d/c4a 0 2024-03-14T13:02:07.636 INFO:tasks.workunit.client.0.smithi111.stdout:8/309: stat d3/d1b/d20/d35/d40/d48/l5d 0 2024-03-14T13:02:07.637 INFO:tasks.workunit.client.0.smithi111.stdout:8/310: dread - d3/d1b/d20/d35/d40/d48/f68 zero size 2024-03-14T13:02:07.638 INFO:tasks.workunit.client.0.smithi111.stdout:4/222: link d1d/d22/f35 d1d/d22/d28/f4b 0 2024-03-14T13:02:07.640 INFO:tasks.workunit.client.0.smithi111.stdout:4/223: creat d1d/d22/d28/f4c x:0 0 0 2024-03-14T13:02:07.641 INFO:tasks.workunit.client.0.smithi111.stdout:8/311: dread d3/d1b/d34/f3c [0,4194304] 0 2024-03-14T13:02:07.642 INFO:tasks.workunit.client.0.smithi111.stdout:4/224: creat d1d/d22/d30/f4d x:0 0 0 2024-03-14T13:02:07.645 INFO:tasks.workunit.client.0.smithi111.stdout:4/225: symlink d1d/d22/d23/l4e 0 2024-03-14T13:02:07.651 INFO:tasks.workunit.client.0.smithi111.stdout:4/226: dwrite d1d/d22/f37 [0,4194304] 0 2024-03-14T13:02:07.654 INFO:tasks.workunit.client.0.smithi111.stdout:4/227: mknod d1d/d22/d30/c4f 0 2024-03-14T13:02:07.658 INFO:tasks.workunit.client.0.smithi111.stdout:4/228: symlink d1d/d48/l50 0 2024-03-14T13:02:07.660 INFO:tasks.workunit.client.0.smithi111.stdout:4/229: chown d1d/d22/d23/f34 362196709 1 2024-03-14T13:02:07.663 INFO:tasks.workunit.client.0.smithi111.stdout:4/230: mkdir d1d/d22/d40/d51 0 2024-03-14T13:02:07.664 INFO:tasks.workunit.client.0.smithi111.stdout:8/312: dwrite d3/d22/d4a/f5c [0,4194304] 0 2024-03-14T13:02:07.669 INFO:tasks.workunit.client.0.smithi111.stdout:4/231: rename d1d/d22/l2b to d1d/l52 0 2024-03-14T13:02:07.670 INFO:tasks.workunit.client.0.smithi111.stdout:4/232: write ff [1444766,107365] 0 2024-03-14T13:02:07.673 INFO:tasks.workunit.client.0.smithi111.stdout:4/233: symlink d1d/d22/d30/l53 0 2024-03-14T13:02:07.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/88: sync 2024-03-14T13:02:07.677 INFO:tasks.workunit.client.0.smithi111.stdout:3/115: sync 2024-03-14T13:02:07.677 INFO:tasks.workunit.client.0.smithi111.stdout:5/191: sync 2024-03-14T13:02:07.677 INFO:tasks.workunit.client.0.smithi111.stdout:8/313: dwrite d3/f14 [0,4194304] 0 2024-03-14T13:02:07.679 INFO:tasks.workunit.client.0.smithi111.stdout:8/314: truncate d3/d1b/f3a 195116 0 2024-03-14T13:02:07.681 INFO:tasks.workunit.client.0.smithi111.stdout:3/116: truncate d0/f20 648342 0 2024-03-14T13:02:07.683 INFO:tasks.workunit.client.0.smithi111.stdout:9/89: getdents d5/d6/d14 0 2024-03-14T13:02:07.685 INFO:tasks.workunit.client.0.smithi111.stdout:3/117: stat d0/de/d19/f28 0 2024-03-14T13:02:07.686 INFO:tasks.workunit.client.0.smithi111.stdout:9/90: mkdir d5/d6/d1e 0 2024-03-14T13:02:07.686 INFO:tasks.workunit.client.0.smithi111.stdout:5/192: dwrite d2/f35 [0,4194304] 0 2024-03-14T13:02:07.687 INFO:tasks.workunit.client.0.smithi111.stdout:3/118: dread - d0/de/d19/f1a zero size 2024-03-14T13:02:07.689 INFO:tasks.workunit.client.0.smithi111.stdout:9/91: write d5/d6/ff [863170,5264] 0 2024-03-14T13:02:07.689 INFO:tasks.workunit.client.0.smithi111.stdout:5/193: write d2/f6 [3600683,42940] 0 2024-03-14T13:02:07.693 INFO:tasks.workunit.client.0.smithi111.stdout:4/234: truncate d1d/d22/f37 2338774 0 2024-03-14T13:02:07.697 INFO:tasks.workunit.client.0.smithi111.stdout:3/119: rename d0/de/d19/c2a to d0/de/c2c 0 2024-03-14T13:02:07.698 INFO:tasks.workunit.client.0.smithi111.stdout:5/194: link d2/d1f/d36/f3c d2/d19/d33/f40 0 2024-03-14T13:02:07.701 INFO:tasks.workunit.client.0.smithi111.stdout:4/235: dwrite d1d/f43 [0,4194304] 0 2024-03-14T13:02:07.701 INFO:tasks.workunit.client.0.smithi111.stdout:9/92: rename d5/d6/l1b to d5/d6/d14/l1f 0 2024-03-14T13:02:07.703 INFO:tasks.workunit.client.0.smithi111.stdout:3/120: mkdir d0/d2d 0 2024-03-14T13:02:07.703 INFO:tasks.workunit.client.0.smithi111.stdout:5/195: fdatasync d2/d19/d33/f40 0 2024-03-14T13:02:07.706 INFO:tasks.workunit.client.0.smithi111.stdout:5/196: dread - d2/d19/f1e zero size 2024-03-14T13:02:07.707 INFO:tasks.workunit.client.0.smithi111.stdout:5/197: symlink d2/d16/l41 0 2024-03-14T13:02:07.708 INFO:tasks.workunit.client.0.smithi111.stdout:9/93: creat d5/f20 x:0 0 0 2024-03-14T13:02:07.710 INFO:tasks.workunit.client.0.smithi111.stdout:5/198: creat d2/d16/f42 x:0 0 0 2024-03-14T13:02:07.711 INFO:tasks.workunit.client.0.smithi111.stdout:5/199: fsync d2/d19/f12 0 2024-03-14T13:02:07.711 INFO:tasks.workunit.client.0.smithi111.stdout:9/94: mknod d5/d6/c21 0 2024-03-14T13:02:07.715 INFO:tasks.workunit.client.0.smithi111.stdout:5/200: creat d2/d19/d33/f43 x:0 0 0 2024-03-14T13:02:07.717 INFO:tasks.workunit.client.0.smithi111.stdout:9/95: rename d5/d6/f7 to d5/d6/f22 0 2024-03-14T13:02:07.718 INFO:tasks.workunit.client.0.smithi111.stdout:5/201: mkdir d2/d19/d44 0 2024-03-14T13:02:07.722 INFO:tasks.workunit.client.0.smithi111.stdout:5/202: rename d2/d1c to d2/d45 0 2024-03-14T13:02:07.723 INFO:tasks.workunit.client.0.smithi111.stdout:5/203: write d2/d13/f29 [545154,48866] 0 2024-03-14T13:02:07.724 INFO:tasks.workunit.client.0.smithi111.stdout:4/236: truncate d1d/f43 3360682 0 2024-03-14T13:02:07.726 INFO:tasks.workunit.client.0.smithi111.stdout:5/204: dread - d2/d16/f38 zero size 2024-03-14T13:02:07.726 INFO:tasks.workunit.client.0.smithi111.stdout:4/237: chown d1d/d22/l49 9254963 1 2024-03-14T13:02:07.727 INFO:tasks.workunit.client.0.smithi111.stdout:3/121: dread d0/fd [0,4194304] 0 2024-03-14T13:02:07.728 INFO:tasks.workunit.client.0.smithi111.stdout:5/205: write d2/d1f/f28 [56632,104636] 0 2024-03-14T13:02:07.732 INFO:tasks.workunit.client.0.smithi111.stdout:3/122: getdents d0/de 0 2024-03-14T13:02:07.732 INFO:tasks.workunit.client.0.smithi111.stdout:4/238: dread f11 [0,4194304] 0 2024-03-14T13:02:07.733 INFO:tasks.workunit.client.0.smithi111.stdout:5/206: getdents d2/d1f/d36 0 2024-03-14T13:02:07.735 INFO:tasks.workunit.client.0.smithi111.stdout:3/123: symlink d0/de/d19/l2e 0 2024-03-14T13:02:07.742 INFO:tasks.workunit.client.0.smithi111.stdout:4/239: creat d1d/d22/f54 x:0 0 0 2024-03-14T13:02:07.743 INFO:tasks.workunit.client.0.smithi111.stdout:5/207: link d2/d13/l30 d2/d13/l46 0 2024-03-14T13:02:07.743 INFO:tasks.workunit.client.0.smithi111.stdout:5/208: dread - d2/d19/f1e zero size 2024-03-14T13:02:07.744 INFO:tasks.workunit.client.0.smithi111.stdout:5/209: readlink d2/d45/l2a 0 2024-03-14T13:02:07.745 INFO:tasks.workunit.client.0.smithi111.stdout:5/210: write d2/d13/f29 [213313,7132] 0 2024-03-14T13:02:07.747 INFO:tasks.workunit.client.0.smithi111.stdout:4/240: creat d1d/d22/f55 x:0 0 0 2024-03-14T13:02:07.747 INFO:tasks.workunit.client.0.smithi111.stdout:5/211: mknod d2/d16/c47 0 2024-03-14T13:02:07.750 INFO:tasks.workunit.client.0.smithi111.stdout:4/241: mkdir d1d/d22/d56 0 2024-03-14T13:02:07.751 INFO:tasks.workunit.client.0.smithi111.stdout:3/124: dwrite d0/de/f27 [0,4194304] 0 2024-03-14T13:02:07.752 INFO:tasks.workunit.client.0.smithi111.stdout:5/212: rename d2/d16/d2c/f3b to d2/d16/f48 0 2024-03-14T13:02:07.753 INFO:tasks.workunit.client.0.smithi111.stdout:4/242: mknod d1d/d22/d40/c57 0 2024-03-14T13:02:07.754 INFO:tasks.workunit.client.0.smithi111.stdout:4/243: chown d1d/d22/c24 818799 1 2024-03-14T13:02:07.757 INFO:tasks.workunit.client.0.smithi111.stdout:5/213: rename d2/d19 to d2/d16/d49 0 2024-03-14T13:02:07.758 INFO:tasks.workunit.client.0.smithi111.stdout:3/125: rename d0/de/l13 to d0/de/l2f 0 2024-03-14T13:02:07.759 INFO:tasks.workunit.client.0.smithi111.stdout:3/126: dread - d0/de/f2b zero size 2024-03-14T13:02:07.760 INFO:tasks.workunit.client.0.smithi111.stdout:4/244: link d1d/d22/d30/f38 d1d/d22/d23/f58 0 2024-03-14T13:02:07.760 INFO:tasks.workunit.client.0.smithi111.stdout:4/245: stat d1d/d22/l49 0 2024-03-14T13:02:07.761 INFO:tasks.workunit.client.0.smithi111.stdout:3/127: mkdir d0/d30 0 2024-03-14T13:02:07.762 INFO:tasks.workunit.client.0.smithi111.stdout:4/246: mkdir d1d/d22/d40/d51/d59 0 2024-03-14T13:02:07.764 INFO:tasks.workunit.client.0.smithi111.stdout:3/128: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:07.765 INFO:tasks.workunit.client.0.smithi111.stdout:4/247: mkdir d1d/d22/d5a 0 2024-03-14T13:02:07.766 INFO:tasks.workunit.client.0.smithi111.stdout:4/248: write d1d/d22/d23/f34 [2774015,76737] 0 2024-03-14T13:02:07.766 INFO:tasks.workunit.client.0.smithi111.stdout:3/129: creat d0/d30/f31 x:0 0 0 2024-03-14T13:02:07.766 INFO:tasks.workunit.client.0.smithi111.stdout:3/130: readlink d0/l16 0 2024-03-14T13:02:07.768 INFO:tasks.workunit.client.0.smithi111.stdout:4/249: dread d1d/d22/d30/f3d [0,4194304] 0 2024-03-14T13:02:07.768 INFO:tasks.workunit.client.0.smithi111.stdout:3/131: mknod d0/de/d19/c32 0 2024-03-14T13:02:07.769 INFO:tasks.workunit.client.0.smithi111.stdout:4/250: dread - d1d/d22/f55 zero size 2024-03-14T13:02:07.771 INFO:tasks.workunit.client.0.smithi111.stdout:4/251: read d1d/d22/f35 [598733,58139] 0 2024-03-14T13:02:07.772 INFO:tasks.workunit.client.0.smithi111.stdout:3/132: rename d0/l1e to d0/d30/l33 0 2024-03-14T13:02:07.778 INFO:tasks.workunit.client.0.smithi111.stdout:3/133: creat d0/f34 x:0 0 0 2024-03-14T13:02:07.780 INFO:tasks.workunit.client.0.smithi111.stdout:3/134: mkdir d0/d2d/d35 0 2024-03-14T13:02:07.785 INFO:tasks.workunit.client.0.smithi111.stdout:4/252: write f12 [947555,29675] 0 2024-03-14T13:02:07.786 INFO:tasks.workunit.client.0.smithi111.stdout:4/253: read f12 [1273846,34627] 0 2024-03-14T13:02:07.788 INFO:tasks.workunit.client.0.smithi111.stdout:3/135: dwrite d0/f8 [0,4194304] 0 2024-03-14T13:02:07.789 INFO:tasks.workunit.client.0.smithi111.stdout:4/254: mknod d1d/d22/d40/d51/d59/c5b 0 2024-03-14T13:02:07.790 INFO:tasks.workunit.client.0.smithi111.stdout:3/136: mknod d0/de/d19/c36 0 2024-03-14T13:02:07.794 INFO:tasks.workunit.client.0.smithi111.stdout:3/137: creat d0/de/f37 x:0 0 0 2024-03-14T13:02:07.796 INFO:tasks.workunit.client.0.smithi111.stdout:3/138: creat d0/d30/f38 x:0 0 0 2024-03-14T13:02:07.800 INFO:tasks.workunit.client.0.smithi111.stdout:3/139: dread d0/de/f27 [0,4194304] 0 2024-03-14T13:02:07.802 INFO:tasks.workunit.client.0.smithi111.stdout:3/140: symlink d0/l39 0 2024-03-14T13:02:07.803 INFO:tasks.workunit.client.0.smithi111.stdout:3/141: chown d0/f7 21693990 1 2024-03-14T13:02:07.809 INFO:tasks.workunit.client.0.smithi111.stdout:3/142: dwrite d0/fd [0,4194304] 0 2024-03-14T13:02:07.811 INFO:tasks.workunit.client.0.smithi111.stdout:3/143: chown d0/de/l2f 0 1 2024-03-14T13:02:07.811 INFO:tasks.workunit.client.0.smithi111.stdout:3/144: fsync d0/d30/f31 0 2024-03-14T13:02:07.812 INFO:tasks.workunit.client.0.smithi111.stdout:3/145: rename d0/de to d0/de/d3a 22 2024-03-14T13:02:07.815 INFO:tasks.workunit.client.0.smithi111.stdout:3/146: mknod d0/d30/c3b 0 2024-03-14T13:02:07.816 INFO:tasks.workunit.client.0.smithi111.stdout:4/255: dread d1d/d22/d28/f29 [0,4194304] 0 2024-03-14T13:02:07.818 INFO:tasks.workunit.client.0.smithi111.stdout:3/147: symlink d0/de/l3c 0 2024-03-14T13:02:07.819 INFO:tasks.workunit.client.0.smithi111.stdout:4/256: dread d1d/d22/d28/f4b [0,4194304] 0 2024-03-14T13:02:07.822 INFO:tasks.workunit.client.0.smithi111.stdout:3/148: rename d0/de/f2b to d0/d30/f3d 0 2024-03-14T13:02:07.826 INFO:tasks.workunit.client.0.smithi111.stdout:3/149: rename d0/f22 to d0/de/d19/f3e 0 2024-03-14T13:02:07.826 INFO:tasks.workunit.client.0.smithi111.stdout:4/257: dwrite d1d/f25 [0,4194304] 0 2024-03-14T13:02:07.827 INFO:tasks.workunit.client.0.smithi111.stdout:4/258: stat d1d/f20 0 2024-03-14T13:02:07.828 INFO:tasks.workunit.client.0.smithi111.stdout:4/259: truncate ff 1824520 0 2024-03-14T13:02:07.829 INFO:tasks.workunit.client.0.smithi111.stdout:3/150: symlink d0/de/d19/l3f 0 2024-03-14T13:02:07.830 INFO:tasks.workunit.client.0.smithi111.stdout:4/260: creat d1d/d22/d40/d51/f5c x:0 0 0 2024-03-14T13:02:07.831 INFO:tasks.workunit.client.0.smithi111.stdout:3/151: mkdir d0/de/d40 0 2024-03-14T13:02:07.836 INFO:tasks.workunit.client.0.smithi111.stdout:3/152: unlink d0/ff 0 2024-03-14T13:02:07.840 INFO:tasks.workunit.client.0.smithi111.stdout:4/261: rename d1d/d22/l47 to d1d/d22/l5d 0 2024-03-14T13:02:07.841 INFO:tasks.workunit.client.0.smithi111.stdout:4/262: fsync d1d/d22/d28/f3e 0 2024-03-14T13:02:07.844 INFO:tasks.workunit.client.0.smithi111.stdout:3/153: symlink d0/de/d40/l41 0 2024-03-14T13:02:07.845 INFO:tasks.workunit.client.0.smithi111.stdout:4/263: mkdir d1d/d22/d30/d5e 0 2024-03-14T13:02:07.848 INFO:tasks.workunit.client.0.smithi111.stdout:4/264: creat d1d/d22/d56/f5f x:0 0 0 2024-03-14T13:02:07.849 INFO:tasks.workunit.client.0.smithi111.stdout:4/265: write d1d/f25 [4288505,72607] 0 2024-03-14T13:02:07.851 INFO:tasks.workunit.client.0.smithi111.stdout:4/266: mknod d1d/d22/d40/c60 0 2024-03-14T13:02:07.853 INFO:tasks.workunit.client.0.smithi111.stdout:4/267: mknod d1d/d22/d40/c61 0 2024-03-14T13:02:07.855 INFO:tasks.workunit.client.0.smithi111.stdout:4/268: mknod d1d/d48/c62 0 2024-03-14T13:02:07.856 INFO:tasks.workunit.client.0.smithi111.stdout:3/154: truncate d0/f8 827399 0 2024-03-14T13:02:07.856 INFO:tasks.workunit.client.0.smithi111.stdout:4/269: write d1d/d22/f41 [51575,39102] 0 2024-03-14T13:02:07.857 INFO:tasks.workunit.client.0.smithi111.stdout:3/155: chown d0/de/d40/l41 1004439 1 2024-03-14T13:02:07.858 INFO:tasks.workunit.client.0.smithi111.stdout:4/270: dread d1d/d22/f35 [0,4194304] 0 2024-03-14T13:02:07.859 INFO:tasks.workunit.client.0.smithi111.stdout:3/156: symlink d0/de/d40/l42 0 2024-03-14T13:02:07.861 INFO:tasks.workunit.client.0.smithi111.stdout:4/271: rename d1d/d22/f31 to d1d/d22/d40/f63 0 2024-03-14T13:02:07.861 INFO:tasks.workunit.client.0.smithi111.stdout:4/272: fdatasync d1d/d22/f41 0 2024-03-14T13:02:07.862 INFO:tasks.workunit.client.0.smithi111.stdout:3/157: creat d0/d2d/f43 x:0 0 0 2024-03-14T13:02:07.863 INFO:tasks.workunit.client.0.smithi111.stdout:3/158: dread - d0/de/d19/f3e zero size 2024-03-14T13:02:07.870 INFO:tasks.workunit.client.0.smithi111.stdout:4/273: dwrite f12 [0,4194304] 0 2024-03-14T13:02:07.871 INFO:tasks.workunit.client.0.smithi111.stdout:4/274: readlink d1d/d48/l50 0 2024-03-14T13:02:07.878 INFO:tasks.workunit.client.0.smithi111.stdout:7/107: sync 2024-03-14T13:02:07.881 INFO:tasks.workunit.client.0.smithi111.stdout:3/159: dwrite d0/f20 [0,4194304] 0 2024-03-14T13:02:07.882 INFO:tasks.workunit.client.0.smithi111.stdout:3/160: chown d0/f20 256778701 1 2024-03-14T13:02:07.883 INFO:tasks.workunit.client.0.smithi111.stdout:7/108: dread d0/d1/d5/f1a [0,4194304] 0 2024-03-14T13:02:07.883 INFO:tasks.workunit.client.0.smithi111.stdout:3/161: fsync d0/d30/f31 0 2024-03-14T13:02:07.884 INFO:tasks.workunit.client.0.smithi111.stdout:4/275: dwrite d1d/d22/d28/f29 [4194304,4194304] 0 2024-03-14T13:02:07.888 INFO:tasks.workunit.client.0.smithi111.stdout:7/109: chown d0/d1/d5/de/d9/f16 793 1 2024-03-14T13:02:07.892 INFO:tasks.workunit.client.0.smithi111.stdout:4/276: link l6 d1d/d22/d56/l64 0 2024-03-14T13:02:07.894 INFO:tasks.workunit.client.0.smithi111.stdout:3/162: getdents d0/d30 0 2024-03-14T13:02:07.895 INFO:tasks.workunit.client.0.smithi111.stdout:4/277: write d1d/d22/d23/f34 [3237337,48998] 0 2024-03-14T13:02:07.896 INFO:tasks.workunit.client.0.smithi111.stdout:3/163: write d0/de/f27 [3815898,27692] 0 2024-03-14T13:02:07.897 INFO:tasks.workunit.client.0.smithi111.stdout:4/278: creat d1d/d22/d5a/f65 x:0 0 0 2024-03-14T13:02:07.899 INFO:tasks.workunit.client.0.smithi111.stdout:4/279: write d1d/d22/d28/f29 [4917741,90396] 0 2024-03-14T13:02:07.902 INFO:tasks.workunit.client.0.smithi111.stdout:4/280: chown c1c 9 1 2024-03-14T13:02:07.903 INFO:tasks.workunit.client.0.smithi111.stdout:4/281: mkdir d1d/d22/d5a/d66 0 2024-03-14T13:02:07.904 INFO:tasks.workunit.client.0.smithi111.stdout:3/164: dwrite d0/de/d19/f28 [0,4194304] 0 2024-03-14T13:02:07.906 INFO:tasks.workunit.client.0.smithi111.stdout:3/165: write d0/de/f12 [5147747,28534] 0 2024-03-14T13:02:07.909 INFO:tasks.workunit.client.0.smithi111.stdout:4/282: rename d1d/d22/d23/f27 to d1d/d22/d23/f67 0 2024-03-14T13:02:07.911 INFO:tasks.workunit.client.0.smithi111.stdout:3/166: readlink d0/de/d19/l29 0 2024-03-14T13:02:07.912 INFO:tasks.workunit.client.0.smithi111.stdout:3/167: truncate d0/de/f12 5778312 0 2024-03-14T13:02:07.914 INFO:tasks.workunit.client.0.smithi111.stdout:4/283: dread f11 [0,4194304] 0 2024-03-14T13:02:07.915 INFO:tasks.workunit.client.0.smithi111.stdout:4/284: chown ff 321 1 2024-03-14T13:02:07.919 INFO:tasks.workunit.client.0.smithi111.stdout:4/285: creat d1d/d22/d28/f68 x:0 0 0 2024-03-14T13:02:07.921 INFO:tasks.workunit.client.0.smithi111.stdout:3/168: dwrite d0/f1c [0,4194304] 0 2024-03-14T13:02:07.921 INFO:tasks.workunit.client.0.smithi111.stdout:4/286: mkdir d1d/d22/d40/d69 0 2024-03-14T13:02:07.922 INFO:tasks.workunit.client.0.smithi111.stdout:4/287: fdatasync d1d/d22/d28/f68 0 2024-03-14T13:02:07.926 INFO:tasks.workunit.client.0.smithi111.stdout:4/288: read d1d/d22/f37 [1837312,128107] 0 2024-03-14T13:02:07.927 INFO:tasks.workunit.client.0.smithi111.stdout:7/110: dread d0/d1/d5/de/d9/f12 [0,4194304] 0 2024-03-14T13:02:07.927 INFO:tasks.workunit.client.0.smithi111.stdout:4/289: dread - d1d/d22/d28/f4c zero size 2024-03-14T13:02:07.928 INFO:tasks.workunit.client.0.smithi111.stdout:4/290: chown d1d/d22/d30/d5e 1524847 1 2024-03-14T13:02:07.930 INFO:tasks.workunit.client.0.smithi111.stdout:7/111: symlink d0/d1/d5/de/d9/l20 0 2024-03-14T13:02:07.930 INFO:tasks.workunit.client.0.smithi111.stdout:4/291: write d1d/d22/d30/f32 [1284170,74268] 0 2024-03-14T13:02:07.936 INFO:tasks.workunit.client.0.smithi111.stdout:7/112: rename d0/d1/d5/de/d14 to d0/d1/d21 0 2024-03-14T13:02:07.990 INFO:tasks.workunit.client.0.smithi111.stdout:3/169: write d0/fd [823338,74150] 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/113: fdatasync d0/d1/d5/f1a 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:4/292: dwrite ff [0,4194304] 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/170: rmdir d0/d30 39 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/114: mkdir d0/d1/d22 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/115: unlink d0/c1c 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:4/293: write d1d/f43 [2626525,79993] 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/171: truncate d0/d30/f38 827754 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/116: symlink d0/l23 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/172: fsync d0/de/d19/f1a 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/117: mknod d0/d1/d22/c24 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/173: rmdir d0/de 39 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:3/174: dread d0/f8 [0,4194304] 0 2024-03-14T13:02:07.991 INFO:tasks.workunit.client.0.smithi111.stdout:4/294: dwrite d1d/d22/d28/f3e [0,4194304] 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:7/118: write d0/d1/d5/de/d9/f12 [2230040,114891] 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/175: chown d0/l16 869249463 1 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/176: unlink d0/d2d/f43 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/177: truncate d0/de/d19/f3e 760926 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/178: rename d0/f34 to d0/de/d19/f44 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/179: fdatasync d0/de/f12 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:4/295: write fe [1396191,93734] 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:7/119: write d0/d1/d5/de/d9/fb [1435491,36654] 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:4/296: write fe [1608133,102128] 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/180: unlink d0/de/d19/f3e 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:4/297: unlink d1d/f25 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/181: symlink d0/de/d19/l45 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:4/298: creat d1d/d22/d40/d51/f6a x:0 0 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:7/120: link d0/d1/d5/de/d9/f16 d0/d1/f25 0 2024-03-14T13:02:07.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/182: link d0/de/c2c d0/d30/c46 0 2024-03-14T13:02:07.993 INFO:tasks.workunit.client.0.smithi111.stdout:7/121: symlink d0/l26 0 2024-03-14T13:02:07.993 INFO:tasks.workunit.client.0.smithi111.stdout:3/183: chown d0/de/d19/l1d 7023 1 2024-03-14T13:02:07.993 INFO:tasks.workunit.client.0.smithi111.stdout:4/299: dwrite d1d/d22/d28/f29 [0,4194304] 0 2024-03-14T13:02:07.993 INFO:tasks.workunit.client.0.smithi111.stdout:3/184: link d0/f20 d0/de/d40/f47 0 2024-03-14T13:02:07.993 INFO:tasks.workunit.client.0.smithi111.stdout:3/185: chown d0/c21 1 1 2024-03-14T13:02:07.994 INFO:tasks.workunit.client.0.smithi111.stdout:3/186: read d0/f8 [70955,13548] 0 2024-03-14T13:02:07.998 INFO:tasks.workunit.client.0.smithi111.stdout:3/187: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:08.003 INFO:tasks.workunit.client.0.smithi111.stdout:3/188: link d0/fc d0/d30/f48 0 2024-03-14T13:02:08.005 INFO:tasks.workunit.client.0.smithi111.stdout:3/189: mknod d0/d30/c49 0 2024-03-14T13:02:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:07 smithi137 ceph-mon[50336]: pgmap v18: 65 pgs: 65 active+clean; 307 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 6.4 MiB/s wr, 130 op/s 2024-03-14T13:02:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:07 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:08.007 INFO:tasks.workunit.client.0.smithi111.stdout:7/122: dread d0/d1/fd [0,4194304] 0 2024-03-14T13:02:08.008 INFO:tasks.workunit.client.0.smithi111.stdout:3/190: creat d0/d30/f4a x:0 0 0 2024-03-14T13:02:08.014 INFO:tasks.workunit.client.0.smithi111.stdout:4/300: write d1d/d22/d28/f4b [740520,103802] 0 2024-03-14T13:02:08.015 INFO:tasks.workunit.client.0.smithi111.stdout:3/191: fsync d0/d30/f48 0 2024-03-14T13:02:08.017 INFO:tasks.workunit.client.0.smithi111.stdout:3/192: chown d0/de/d19/l3f 28263 1 2024-03-14T13:02:08.017 INFO:tasks.workunit.client.0.smithi111.stdout:7/123: dwrite d0/d1/d5/de/d1b/f1e [0,4194304] 0 2024-03-14T13:02:08.018 INFO:tasks.workunit.client.0.smithi111.stdout:4/301: symlink d1d/d22/d5a/d66/l6b 0 2024-03-14T13:02:08.019 INFO:tasks.workunit.client.0.smithi111.stdout:3/193: dread d0/fd [0,4194304] 0 2024-03-14T13:02:08.021 INFO:tasks.workunit.client.0.smithi111.stdout:4/302: mknod d1d/d22/d5a/c6c 0 2024-03-14T13:02:08.021 INFO:tasks.workunit.client.0.smithi111.stdout:3/194: read - d0/de/d19/f1a zero size 2024-03-14T13:02:08.024 INFO:tasks.workunit.client.0.smithi111.stdout:3/195: mknod d0/de/c4b 0 2024-03-14T13:02:08.025 INFO:tasks.workunit.client.0.smithi111.stdout:7/124: mknod d0/d1/d5/de/c27 0 2024-03-14T13:02:08.029 INFO:tasks.workunit.client.0.smithi111.stdout:7/125: link d0/d1/d5/de/d9/f19 d0/d1/f28 0 2024-03-14T13:02:08.030 INFO:tasks.workunit.client.0.smithi111.stdout:7/126: creat d0/d11/f29 x:0 0 0 2024-03-14T13:02:08.031 INFO:tasks.workunit.client.0.smithi111.stdout:3/196: dread d0/fc [0,4194304] 0 2024-03-14T13:02:08.039 INFO:tasks.workunit.client.0.smithi111.stdout:4/303: dwrite fb [4194304,4194304] 0 2024-03-14T13:02:08.041 INFO:tasks.workunit.client.0.smithi111.stdout:3/197: dwrite d0/f8 [0,4194304] 0 2024-03-14T13:02:08.045 INFO:tasks.workunit.client.0.smithi111.stdout:4/304: dread fe [0,4194304] 0 2024-03-14T13:02:08.048 INFO:tasks.workunit.client.0.smithi111.stdout:4/305: fsync fe 0 2024-03-14T13:02:08.050 INFO:tasks.workunit.client.0.smithi111.stdout:4/306: dread - d1d/d22/d30/f4d zero size 2024-03-14T13:02:08.051 INFO:tasks.workunit.client.0.smithi111.stdout:3/198: dread d0/f1c [0,4194304] 0 2024-03-14T13:02:08.051 INFO:tasks.workunit.client.0.smithi111.stdout:4/307: rmdir d1d/d22/d56 39 2024-03-14T13:02:08.053 INFO:tasks.workunit.client.0.smithi111.stdout:3/199: mkdir d0/d2d/d35/d4c 0 2024-03-14T13:02:08.054 INFO:tasks.workunit.client.0.smithi111.stdout:4/308: symlink d1d/d22/d5a/d66/l6d 0 2024-03-14T13:02:08.055 INFO:tasks.workunit.client.0.smithi111.stdout:3/200: rmdir d0/de 39 2024-03-14T13:02:08.057 INFO:tasks.workunit.client.0.smithi111.stdout:4/309: mknod d1d/d22/d30/c6e 0 2024-03-14T13:02:08.057 INFO:tasks.workunit.client.0.smithi111.stdout:3/201: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:08.059 INFO:tasks.workunit.client.0.smithi111.stdout:4/310: dread d1d/f2e [0,4194304] 0 2024-03-14T13:02:08.061 INFO:tasks.workunit.client.0.smithi111.stdout:4/311: dread d1d/f20 [4194304,4194304] 0 2024-03-14T13:02:08.064 INFO:tasks.workunit.client.0.smithi111.stdout:3/202: dwrite d0/de/d19/f26 [0,4194304] 0 2024-03-14T13:02:08.068 INFO:tasks.workunit.client.0.smithi111.stdout:3/203: link d0/de/d19/l2e d0/d2d/l4d 0 2024-03-14T13:02:08.070 INFO:tasks.workunit.client.0.smithi111.stdout:4/312: dwrite d1d/f20 [0,4194304] 0 2024-03-14T13:02:08.073 INFO:tasks.workunit.client.0.smithi111.stdout:3/204: getdents d0/de 0 2024-03-14T13:02:08.074 INFO:tasks.workunit.client.0.smithi111.stdout:4/313: mkdir d1d/d22/d30/d5e/d6f 0 2024-03-14T13:02:08.074 INFO:tasks.workunit.client.0.smithi111.stdout:4/314: chown d1d/d22/d40/f63 1 1 2024-03-14T13:02:08.075 INFO:tasks.workunit.client.0.smithi111.stdout:3/205: symlink d0/de/l4e 0 2024-03-14T13:02:08.077 INFO:tasks.workunit.client.0.smithi111.stdout:4/315: symlink d1d/d22/d5a/l70 0 2024-03-14T13:02:08.077 INFO:tasks.workunit.client.0.smithi111.stdout:3/206: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:08.077 INFO:tasks.workunit.client.0.smithi111.stdout:3/207: readlink d0/de/d40/l41 0 2024-03-14T13:02:08.078 INFO:tasks.workunit.client.0.smithi111.stdout:4/316: dread f11 [4194304,4194304] 0 2024-03-14T13:02:08.083 INFO:tasks.workunit.client.0.smithi111.stdout:4/317: rename d1d/d22/d28/f3c to d1d/d22/d56/f71 0 2024-03-14T13:02:08.085 INFO:tasks.workunit.client.0.smithi111.stdout:3/208: dwrite d0/d30/f31 [0,4194304] 0 2024-03-14T13:02:08.085 INFO:tasks.workunit.client.0.smithi111.stdout:4/318: write d1d/d22/f41 [76613,35190] 0 2024-03-14T13:02:08.086 INFO:tasks.workunit.client.0.smithi111.stdout:4/319: chown d1d/d22/f37 53726658 1 2024-03-14T13:02:08.088 INFO:tasks.workunit.client.0.smithi111.stdout:3/209: rename d0/de/l4e to d0/d30/l4f 0 2024-03-14T13:02:08.090 INFO:tasks.workunit.client.0.smithi111.stdout:4/320: symlink d1d/d22/d40/d69/l72 0 2024-03-14T13:02:08.091 INFO:tasks.workunit.client.0.smithi111.stdout:4/321: dread - d1d/d22/d5a/f65 zero size 2024-03-14T13:02:08.092 INFO:tasks.workunit.client.0.smithi111.stdout:3/210: mkdir d0/d2d/d35/d50 0 2024-03-14T13:02:08.096 INFO:tasks.workunit.client.0.smithi111.stdout:4/322: dread d1d/d22/d30/f32 [0,4194304] 0 2024-03-14T13:02:08.096 INFO:tasks.workunit.client.0.smithi111.stdout:4/323: chown d1d/d22/d40/d51 11103009 1 2024-03-14T13:02:08.098 INFO:tasks.workunit.client.0.smithi111.stdout:3/211: rename d0/de/d19/l2e to d0/de/l51 0 2024-03-14T13:02:08.099 INFO:tasks.workunit.client.0.smithi111.stdout:4/324: symlink d1d/l73 0 2024-03-14T13:02:08.101 INFO:tasks.workunit.client.0.smithi111.stdout:3/212: mknod d0/d30/c52 0 2024-03-14T13:02:08.102 INFO:tasks.workunit.client.0.smithi111.stdout:3/213: readlink d0/de/d19/l29 0 2024-03-14T13:02:08.104 INFO:tasks.workunit.client.0.smithi111.stdout:3/214: symlink d0/d30/l53 0 2024-03-14T13:02:08.106 INFO:tasks.workunit.client.0.smithi111.stdout:4/325: dwrite d1d/d22/f44 [0,4194304] 0 2024-03-14T13:02:08.107 INFO:tasks.workunit.client.0.smithi111.stdout:4/326: write d1d/d22/f35 [994326,73599] 0 2024-03-14T13:02:08.108 INFO:tasks.workunit.client.0.smithi111.stdout:3/215: creat d0/d2d/d35/d50/f54 x:0 0 0 2024-03-14T13:02:08.109 INFO:tasks.workunit.client.0.smithi111.stdout:4/327: write d1d/f36 [305283,67414] 0 2024-03-14T13:02:08.110 INFO:tasks.workunit.client.0.smithi111.stdout:3/216: unlink d0/d30/l33 0 2024-03-14T13:02:08.111 INFO:tasks.workunit.client.0.smithi111.stdout:3/217: dread - d0/d30/f3d zero size 2024-03-14T13:02:08.111 INFO:tasks.workunit.client.0.smithi111.stdout:4/328: symlink d1d/d22/d5a/d66/l74 0 2024-03-14T13:02:08.113 INFO:tasks.workunit.client.0.smithi111.stdout:3/218: dread d0/fc [0,4194304] 0 2024-03-14T13:02:08.113 INFO:tasks.workunit.client.0.smithi111.stdout:4/329: truncate d1d/d22/d28/f68 459342 0 2024-03-14T13:02:08.113 INFO:tasks.workunit.client.0.smithi111.stdout:3/219: chown d0/d30/c3b 220281237 1 2024-03-14T13:02:08.114 INFO:tasks.workunit.client.0.smithi111.stdout:4/330: write d1d/d22/f3b [404159,103568] 0 2024-03-14T13:02:08.116 INFO:tasks.workunit.client.0.smithi111.stdout:3/220: getdents d0/d2d/d35/d4c 0 2024-03-14T13:02:08.116 INFO:tasks.workunit.client.0.smithi111.stdout:4/331: chown d1d/d22/d5a/f65 1593529 1 2024-03-14T13:02:08.117 INFO:tasks.workunit.client.0.smithi111.stdout:3/221: fsync d0/f23 0 2024-03-14T13:02:08.122 INFO:tasks.workunit.client.0.smithi111.stdout:3/222: mkdir d0/de/d40/d55 0 2024-03-14T13:02:08.124 INFO:tasks.workunit.client.0.smithi111.stdout:4/332: dwrite d1d/d22/d56/f5f [0,4194304] 0 2024-03-14T13:02:08.127 INFO:tasks.workunit.client.0.smithi111.stdout:4/333: mkdir d1d/d22/d30/d5e/d6f/d75 0 2024-03-14T13:02:08.128 INFO:tasks.workunit.client.0.smithi111.stdout:3/223: dread d0/f1c [0,4194304] 0 2024-03-14T13:02:08.129 INFO:tasks.workunit.client.0.smithi111.stdout:4/334: mknod d1d/d22/d30/c76 0 2024-03-14T13:02:08.129 INFO:tasks.workunit.client.0.smithi111.stdout:3/224: truncate d0/de/d19/f1a 692740 0 2024-03-14T13:02:08.130 INFO:tasks.workunit.client.0.smithi111.stdout:3/225: stat d0/de/f37 0 2024-03-14T13:02:08.132 INFO:tasks.workunit.client.0.smithi111.stdout:4/335: unlink d1d/d22/f35 0 2024-03-14T13:02:08.134 INFO:tasks.workunit.client.0.smithi111.stdout:4/336: symlink d1d/d22/d5a/d66/l77 0 2024-03-14T13:02:08.136 INFO:tasks.workunit.client.0.smithi111.stdout:4/337: symlink d1d/d22/d5a/l78 0 2024-03-14T13:02:08.137 INFO:tasks.workunit.client.0.smithi111.stdout:4/338: write d1d/f20 [2171441,119911] 0 2024-03-14T13:02:08.138 INFO:tasks.workunit.client.0.smithi111.stdout:4/339: write fb [4622151,17843] 0 2024-03-14T13:02:08.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:07 smithi111 ceph-mon[39688]: pgmap v18: 65 pgs: 65 active+clean; 307 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 6.4 MiB/s wr, 130 op/s 2024-03-14T13:02:08.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:07 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:08.141 INFO:tasks.workunit.client.0.smithi111.stdout:4/340: rename d1d/d22/d28/f29 to d1d/d22/d5a/d66/f79 0 2024-03-14T13:02:08.144 INFO:tasks.workunit.client.0.smithi111.stdout:4/341: creat d1d/d22/d30/d5e/f7a x:0 0 0 2024-03-14T13:02:08.147 INFO:tasks.workunit.client.0.smithi111.stdout:4/342: dread d1d/d22/f3b [0,4194304] 0 2024-03-14T13:02:08.158 INFO:tasks.workunit.client.0.smithi111.stdout:4/343: write d1d/f26 [209338,78844] 0 2024-03-14T13:02:08.165 INFO:tasks.workunit.client.0.smithi111.stdout:2/88: sync 2024-03-14T13:02:08.165 INFO:tasks.workunit.client.0.smithi111.stdout:6/128: sync 2024-03-14T13:02:08.166 INFO:tasks.workunit.client.0.smithi111.stdout:2/89: chown c8 1 1 2024-03-14T13:02:08.166 INFO:tasks.workunit.client.0.smithi111.stdout:6/129: truncate d0/f1c 857993 0 2024-03-14T13:02:08.166 INFO:tasks.workunit.client.0.smithi111.stdout:2/90: chown f4 26693981 1 2024-03-14T13:02:08.167 INFO:tasks.workunit.client.0.smithi111.stdout:6/130: chown d0/d17/d21/f28 5371 1 2024-03-14T13:02:08.167 INFO:tasks.workunit.client.0.smithi111.stdout:2/91: write f1 [7101527,35155] 0 2024-03-14T13:02:08.168 INFO:tasks.workunit.client.0.smithi111.stdout:6/131: write d0/d17/f1e [524849,3843] 0 2024-03-14T13:02:08.170 INFO:tasks.workunit.client.0.smithi111.stdout:6/132: write d0/f1c [10182,51285] 0 2024-03-14T13:02:08.174 INFO:tasks.workunit.client.0.smithi111.stdout:4/344: write d1d/d22/d30/f38 [191579,94625] 0 2024-03-14T13:02:08.176 INFO:tasks.workunit.client.0.smithi111.stdout:4/345: fsync d1d/d22/f55 0 2024-03-14T13:02:08.183 INFO:tasks.workunit.client.0.smithi111.stdout:4/346: dwrite d1d/d22/d23/f46 [0,4194304] 0 2024-03-14T13:02:08.184 INFO:tasks.workunit.client.0.smithi111.stdout:2/92: dread f1 [0,4194304] 0 2024-03-14T13:02:08.185 INFO:tasks.workunit.client.0.smithi111.stdout:6/133: dread d0/d17/d21/f22 [0,4194304] 0 2024-03-14T13:02:08.190 INFO:tasks.workunit.client.0.smithi111.stdout:6/134: mknod d0/d17/d24/c29 0 2024-03-14T13:02:08.190 INFO:tasks.workunit.client.0.smithi111.stdout:4/347: dread d1d/f26 [0,4194304] 0 2024-03-14T13:02:08.191 INFO:tasks.workunit.client.0.smithi111.stdout:2/93: dread f1 [0,4194304] 0 2024-03-14T13:02:08.192 INFO:tasks.workunit.client.0.smithi111.stdout:2/94: mkdir dd/d11/d1a 0 2024-03-14T13:02:08.199 INFO:tasks.workunit.client.0.smithi111.stdout:4/348: dwrite d1d/d22/d40/f63 [4194304,4194304] 0 2024-03-14T13:02:08.202 INFO:tasks.workunit.client.0.smithi111.stdout:2/95: truncate f1 3076414 0 2024-03-14T13:02:08.203 INFO:tasks.workunit.client.0.smithi111.stdout:2/96: stat dd/l14 0 2024-03-14T13:02:08.206 INFO:tasks.workunit.client.0.smithi111.stdout:2/97: creat dd/d11/d1a/f1b x:0 0 0 2024-03-14T13:02:08.207 INFO:tasks.workunit.client.0.smithi111.stdout:4/349: dwrite f11 [0,4194304] 0 2024-03-14T13:02:08.208 INFO:tasks.workunit.client.0.smithi111.stdout:4/350: chown d1d/d22/d5a/d66/l77 11627986 1 2024-03-14T13:02:08.210 INFO:tasks.workunit.client.0.smithi111.stdout:4/351: creat d1d/d22/d56/f7b x:0 0 0 2024-03-14T13:02:08.215 INFO:tasks.workunit.client.0.smithi111.stdout:4/352: dwrite fe [0,4194304] 0 2024-03-14T13:02:08.216 INFO:tasks.workunit.client.0.smithi111.stdout:4/353: chown d1d/d22/d30/c3a 0 1 2024-03-14T13:02:08.220 INFO:tasks.workunit.client.0.smithi111.stdout:4/354: rename d1d/d22/d40/d51 to d1d/d22/d30/d7c 0 2024-03-14T13:02:08.221 INFO:tasks.workunit.client.0.smithi111.stdout:4/355: read - d1d/d22/f55 zero size 2024-03-14T13:02:08.225 INFO:tasks.workunit.client.0.smithi111.stdout:4/356: symlink d1d/d48/l7d 0 2024-03-14T13:02:08.229 INFO:tasks.workunit.client.0.smithi111.stdout:4/357: rename d1d/d22/d5a to d1d/d22/d30/d5e/d6f/d7e 0 2024-03-14T13:02:08.232 INFO:tasks.workunit.client.0.smithi111.stdout:4/358: rmdir d1d/d22/d30/d5e/d6f/d75 0 2024-03-14T13:02:08.233 INFO:tasks.workunit.client.0.smithi111.stdout:4/359: truncate d1d/d22/d30/f4d 113324 0 2024-03-14T13:02:08.235 INFO:tasks.workunit.client.0.smithi111.stdout:4/360: creat d1d/d22/d56/f7f x:0 0 0 2024-03-14T13:02:08.235 INFO:tasks.workunit.client.0.smithi111.stdout:4/361: stat d1d/f43 0 2024-03-14T13:02:08.236 INFO:tasks.workunit.client.0.smithi111.stdout:4/362: stat d1d/d22/d30/d5e/d6f 0 2024-03-14T13:02:08.238 INFO:tasks.workunit.client.0.smithi111.stdout:4/363: symlink d1d/d22/d28/l80 0 2024-03-14T13:02:08.239 INFO:tasks.workunit.client.0.smithi111.stdout:4/364: creat d1d/d22/d30/f81 x:0 0 0 2024-03-14T13:02:08.240 INFO:tasks.workunit.client.0.smithi111.stdout:4/365: dread - d1d/d22/d30/d5e/f7a zero size 2024-03-14T13:02:08.242 INFO:tasks.workunit.client.0.smithi111.stdout:4/366: mknod d1d/d22/d30/d5e/d6f/d7e/c82 0 2024-03-14T13:02:08.242 INFO:tasks.workunit.client.0.smithi111.stdout:4/367: stat d1d/d48/l7d 0 2024-03-14T13:02:08.244 INFO:tasks.workunit.client.0.smithi111.stdout:4/368: unlink d1d/f2c 0 2024-03-14T13:02:08.250 INFO:tasks.workunit.client.0.smithi111.stdout:4/369: symlink d1d/d22/d28/l83 0 2024-03-14T13:02:08.252 INFO:tasks.workunit.client.0.smithi111.stdout:4/370: creat d1d/d22/d30/d5e/f84 x:0 0 0 2024-03-14T13:02:08.254 INFO:tasks.workunit.client.0.smithi111.stdout:4/371: creat d1d/d22/d30/d5e/d6f/d7e/d66/f85 x:0 0 0 2024-03-14T13:02:08.255 INFO:tasks.workunit.client.0.smithi111.stdout:4/372: readlink d1d/d22/d30/d5e/d6f/d7e/d66/l74 0 2024-03-14T13:02:08.257 INFO:tasks.workunit.client.0.smithi111.stdout:4/373: mkdir d1d/d22/d40/d69/d86 0 2024-03-14T13:02:08.258 INFO:tasks.workunit.client.0.smithi111.stdout:4/374: write d1d/d22/d40/f63 [3984216,49688] 0 2024-03-14T13:02:08.261 INFO:tasks.workunit.client.0.smithi111.stdout:4/375: mkdir d1d/d22/d30/d87 0 2024-03-14T13:02:08.261 INFO:tasks.workunit.client.0.smithi111.stdout:4/376: write d1d/f43 [2628054,116691] 0 2024-03-14T13:02:08.262 INFO:tasks.workunit.client.0.smithi111.stdout:4/377: stat d1d/d22/d23/c2a 0 2024-03-14T13:02:08.263 INFO:tasks.workunit.client.0.smithi111.stdout:4/378: rmdir d1d/d22/d30/d5e/d6f 39 2024-03-14T13:02:08.263 INFO:tasks.workunit.client.0.smithi111.stdout:4/379: chown d1d/d22/d30/f81 56495637 1 2024-03-14T13:02:08.264 INFO:tasks.workunit.client.0.smithi111.stdout:4/380: dread - d1d/d22/d56/f71 zero size 2024-03-14T13:02:08.270 INFO:tasks.workunit.client.0.smithi111.stdout:4/381: truncate f11 47437 0 2024-03-14T13:02:08.271 INFO:tasks.workunit.client.0.smithi111.stdout:4/382: unlink d1d/d22/d23/l4e 0 2024-03-14T13:02:08.278 INFO:tasks.workunit.client.0.smithi111.stdout:4/383: dwrite d1d/d22/d30/d5e/d6f/d7e/d66/f85 [0,4194304] 0 2024-03-14T13:02:08.280 INFO:tasks.workunit.client.0.smithi111.stdout:4/384: write d1d/d22/d30/d5e/f84 [818607,14762] 0 2024-03-14T13:02:08.281 INFO:tasks.workunit.client.0.smithi111.stdout:4/385: read d1d/f36 [351203,53153] 0 2024-03-14T13:02:08.283 INFO:tasks.workunit.client.0.smithi111.stdout:4/386: symlink d1d/d22/d30/d7c/d59/l88 0 2024-03-14T13:02:08.285 INFO:tasks.workunit.client.0.smithi111.stdout:4/387: symlink d1d/d48/l89 0 2024-03-14T13:02:08.290 INFO:tasks.workunit.client.0.smithi111.stdout:4/388: rename d1d/d22/d30/d5e/d6f/d7e/l70 to d1d/d22/d28/l8a 0 2024-03-14T13:02:08.292 INFO:tasks.workunit.client.0.smithi111.stdout:4/389: mknod d1d/d22/d30/d5e/d6f/d7e/c8b 0 2024-03-14T13:02:08.293 INFO:tasks.workunit.client.0.smithi111.stdout:4/390: chown d1d/d22/d30/f32 47 1 2024-03-14T13:02:08.293 INFO:tasks.workunit.client.0.smithi111.stdout:4/391: chown d1d/d22/d30/f38 7863922 1 2024-03-14T13:02:08.294 INFO:tasks.workunit.client.0.smithi111.stdout:4/392: write d1d/d22/d30/d7c/f5c [674347,16585] 0 2024-03-14T13:02:08.297 INFO:tasks.workunit.client.0.smithi111.stdout:4/393: rename d1d/d22/d30/c6e to d1d/d22/d30/d5e/d6f/c8c 0 2024-03-14T13:02:08.297 INFO:tasks.workunit.client.0.smithi111.stdout:4/394: dread - d1d/d22/d56/f7b zero size 2024-03-14T13:02:08.300 INFO:tasks.workunit.client.0.smithi111.stdout:4/395: creat d1d/d22/d30/d5e/f8d x:0 0 0 2024-03-14T13:02:08.302 INFO:tasks.workunit.client.0.smithi111.stdout:4/396: creat d1d/d22/d30/d5e/d6f/f8e x:0 0 0 2024-03-14T13:02:08.303 INFO:tasks.workunit.client.0.smithi111.stdout:4/397: dread d1d/d22/d30/f32 [0,4194304] 0 2024-03-14T13:02:08.304 INFO:tasks.workunit.client.0.smithi111.stdout:4/398: fdatasync d1d/d22/d56/f7b 0 2024-03-14T13:02:08.305 INFO:tasks.workunit.client.0.smithi111.stdout:4/399: write d1d/d22/d23/f58 [52416,33840] 0 2024-03-14T13:02:08.307 INFO:tasks.workunit.client.0.smithi111.stdout:4/400: symlink d1d/d22/d30/d5e/d6f/d7e/d66/l8f 0 2024-03-14T13:02:08.310 INFO:tasks.workunit.client.0.smithi111.stdout:4/401: dread fb [4194304,4194304] 0 2024-03-14T13:02:08.312 INFO:tasks.workunit.client.0.smithi111.stdout:4/402: mknod d1d/d22/d30/d5e/d6f/c90 0 2024-03-14T13:02:08.314 INFO:tasks.workunit.client.0.smithi111.stdout:4/403: unlink d1d/d22/d30/d5e/d6f/d7e/d66/l74 0 2024-03-14T13:02:08.317 INFO:tasks.workunit.client.0.smithi111.stdout:4/404: dread d1d/d22/d30/d7c/f5c [0,4194304] 0 2024-03-14T13:02:08.318 INFO:tasks.workunit.client.0.smithi111.stdout:4/405: symlink d1d/d22/d30/l91 0 2024-03-14T13:02:08.319 INFO:tasks.workunit.client.0.smithi111.stdout:4/406: fsync d1d/f36 0 2024-03-14T13:02:08.328 INFO:tasks.workunit.client.0.smithi111.stdout:4/407: write d1d/f2e [1361424,98113] 0 2024-03-14T13:02:08.329 INFO:tasks.workunit.client.0.smithi111.stdout:4/408: rmdir d1d/d22/d40/d69 39 2024-03-14T13:02:08.334 INFO:tasks.workunit.client.0.smithi111.stdout:4/409: dread d1d/d22/d28/f3e [0,4194304] 0 2024-03-14T13:02:08.336 INFO:tasks.workunit.client.0.smithi111.stdout:4/410: mkdir d1d/d48/d92 0 2024-03-14T13:02:08.338 INFO:tasks.workunit.client.0.smithi111.stdout:4/411: creat d1d/d22/d30/d7c/f93 x:0 0 0 2024-03-14T13:02:08.340 INFO:tasks.workunit.client.0.smithi111.stdout:4/412: rename d1d/d22/d30/d5e/d6f/f8e to d1d/d22/d30/d5e/d6f/d7e/d66/f94 0 2024-03-14T13:02:08.341 INFO:tasks.workunit.client.0.smithi111.stdout:4/413: mknod d1d/d22/d30/d7c/d59/c95 0 2024-03-14T13:02:08.342 INFO:tasks.workunit.client.0.smithi111.stdout:4/414: chown d1d/d22/d30/d5e/d6f/d7e/l78 1753408 1 2024-03-14T13:02:08.348 INFO:tasks.workunit.client.0.smithi111.stdout:4/415: dwrite d1d/d22/f54 [0,4194304] 0 2024-03-14T13:02:08.351 INFO:tasks.workunit.client.0.smithi111.stdout:4/416: symlink d1d/d22/d30/d5e/d6f/d7e/l96 0 2024-03-14T13:02:08.353 INFO:tasks.workunit.client.0.smithi111.stdout:4/417: creat d1d/d22/d30/d87/f97 x:0 0 0 2024-03-14T13:02:08.355 INFO:tasks.workunit.client.0.smithi111.stdout:4/418: read d1d/f2e [514468,80517] 0 2024-03-14T13:02:08.355 INFO:tasks.workunit.client.0.smithi111.stdout:0/86: sync 2024-03-14T13:02:08.355 INFO:tasks.workunit.client.0.smithi111.stdout:1/124: sync 2024-03-14T13:02:08.355 INFO:tasks.workunit.client.0.smithi111.stdout:8/315: sync 2024-03-14T13:02:08.356 INFO:tasks.workunit.client.0.smithi111.stdout:1/125: dread - f19 zero size 2024-03-14T13:02:08.363 INFO:tasks.workunit.client.0.smithi111.stdout:1/126: mknod d20/c2c 0 2024-03-14T13:02:08.364 INFO:tasks.workunit.client.0.smithi111.stdout:4/419: rename d1d/d22/d23/c3f to d1d/d22/d30/d5e/c98 0 2024-03-14T13:02:08.367 INFO:tasks.workunit.client.0.smithi111.stdout:4/420: write d1d/d22/d30/f32 [199384,87790] 0 2024-03-14T13:02:08.374 INFO:tasks.workunit.client.0.smithi111.stdout:1/127: rename cb to d20/d21/d28/c2d 0 2024-03-14T13:02:08.378 INFO:tasks.workunit.client.0.smithi111.stdout:8/316: dwrite d3/f65 [0,4194304] 0 2024-03-14T13:02:08.379 INFO:tasks.workunit.client.0.smithi111.stdout:0/87: dwrite f2 [0,4194304] 0 2024-03-14T13:02:08.384 INFO:tasks.workunit.client.1.smithi137.stdout:0/1: sync 2024-03-14T13:02:08.385 INFO:tasks.workunit.client.0.smithi111.stdout:1/128: creat d20/d25/f2e x:0 0 0 2024-03-14T13:02:08.385 INFO:tasks.workunit.client.0.smithi111.stdout:1/129: fdatasync ff 0 2024-03-14T13:02:08.385 INFO:tasks.workunit.client.0.smithi111.stdout:0/88: chown d6 34925 1 2024-03-14T13:02:08.386 INFO:tasks.workunit.client.0.smithi111.stdout:4/421: dwrite d1d/d22/d30/d5e/f84 [0,4194304] 0 2024-03-14T13:02:08.386 INFO:tasks.workunit.client.0.smithi111.stdout:0/89: fsync d6/fb 0 2024-03-14T13:02:08.387 INFO:tasks.workunit.client.1.smithi137.stdout:0/2: symlink l0 0 2024-03-14T13:02:08.387 INFO:tasks.workunit.client.1.smithi137.stdout:0/3: write - no filename 2024-03-14T13:02:08.390 INFO:tasks.workunit.client.0.smithi111.stdout:0/90: dread - d6/f10 zero size 2024-03-14T13:02:08.390 INFO:tasks.workunit.client.0.smithi111.stdout:1/130: symlink d20/d21/l2f 0 2024-03-14T13:02:08.393 INFO:tasks.workunit.client.0.smithi111.stdout:0/91: read d6/d9/f12 [7013726,7698] 0 2024-03-14T13:02:08.395 INFO:tasks.workunit.client.0.smithi111.stdout:4/422: symlink d1d/d22/d40/l99 0 2024-03-14T13:02:08.397 INFO:tasks.workunit.client.0.smithi111.stdout:1/131: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:08.400 INFO:tasks.workunit.client.0.smithi111.stdout:0/92: link d6/f10 d6/d9/de/f18 0 2024-03-14T13:02:08.401 INFO:tasks.workunit.client.0.smithi111.stdout:1/132: fdatasync fa 0 2024-03-14T13:02:08.404 INFO:tasks.workunit.client.0.smithi111.stdout:8/317: truncate d3/dc/f4e 1543897 0 2024-03-14T13:02:08.405 INFO:tasks.workunit.client.0.smithi111.stdout:0/93: dread f1 [0,4194304] 0 2024-03-14T13:02:08.406 INFO:tasks.workunit.client.0.smithi111.stdout:8/318: creat d3/dc/f6c x:0 0 0 2024-03-14T13:02:08.415 INFO:tasks.workunit.client.0.smithi111.stdout:0/94: dwrite f1 [4194304,4194304] 0 2024-03-14T13:02:08.416 INFO:tasks.workunit.client.0.smithi111.stdout:8/319: dwrite d3/f3e [0,4194304] 0 2024-03-14T13:02:08.426 INFO:tasks.workunit.client.0.smithi111.stdout:8/320: dread d3/dc/d45/f56 [0,4194304] 0 2024-03-14T13:02:08.427 INFO:tasks.workunit.client.0.smithi111.stdout:1/133: dread ff [0,4194304] 0 2024-03-14T13:02:08.431 INFO:tasks.workunit.client.0.smithi111.stdout:1/134: creat d20/d25/f30 x:0 0 0 2024-03-14T13:02:08.432 INFO:tasks.workunit.client.0.smithi111.stdout:1/135: read f3 [1488452,90244] 0 2024-03-14T13:02:08.436 INFO:tasks.workunit.client.0.smithi111.stdout:8/321: dwrite d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:08.440 INFO:tasks.workunit.client.0.smithi111.stdout:8/322: creat d3/f6d x:0 0 0 2024-03-14T13:02:08.443 INFO:tasks.workunit.client.0.smithi111.stdout:8/323: symlink d3/d1b/d20/d35/d40/l6e 0 2024-03-14T13:02:08.445 INFO:tasks.workunit.client.0.smithi111.stdout:0/95: dwrite f3 [0,4194304] 0 2024-03-14T13:02:08.449 INFO:tasks.workunit.client.0.smithi111.stdout:0/96: dread d6/d9/f12 [0,4194304] 0 2024-03-14T13:02:08.451 INFO:tasks.workunit.client.0.smithi111.stdout:0/97: mknod d6/d9/d15/c19 0 2024-03-14T13:02:08.452 INFO:tasks.workunit.client.0.smithi111.stdout:0/98: truncate d6/fb 4730540 0 2024-03-14T13:02:08.455 INFO:tasks.workunit.client.0.smithi111.stdout:0/99: rename d6/f13 to d6/f1a 0 2024-03-14T13:02:08.461 INFO:tasks.workunit.client.0.smithi111.stdout:0/100: dwrite d6/fd [4194304,4194304] 0 2024-03-14T13:02:08.462 INFO:tasks.workunit.client.0.smithi111.stdout:0/101: readlink d6/d9/l11 0 2024-03-14T13:02:08.468 INFO:tasks.workunit.client.0.smithi111.stdout:1/136: dread f9 [0,4194304] 0 2024-03-14T13:02:08.470 INFO:tasks.workunit.client.0.smithi111.stdout:1/137: mkdir d20/d25/d31 0 2024-03-14T13:02:08.470 INFO:tasks.workunit.client.0.smithi111.stdout:1/138: fdatasync fa 0 2024-03-14T13:02:08.473 INFO:tasks.workunit.client.0.smithi111.stdout:0/102: dwrite d6/d9/f12 [0,4194304] 0 2024-03-14T13:02:08.473 INFO:tasks.workunit.client.0.smithi111.stdout:1/139: rename l12 to d20/d21/l32 0 2024-03-14T13:02:08.475 INFO:tasks.workunit.client.0.smithi111.stdout:1/140: mknod d20/d25/c33 0 2024-03-14T13:02:08.477 INFO:tasks.workunit.client.0.smithi111.stdout:0/103: creat d6/d9/d15/f1b x:0 0 0 2024-03-14T13:02:08.477 INFO:tasks.workunit.client.0.smithi111.stdout:1/141: creat d20/d25/f34 x:0 0 0 2024-03-14T13:02:08.479 INFO:tasks.workunit.client.0.smithi111.stdout:1/142: symlink d20/d21/d28/l35 0 2024-03-14T13:02:08.480 INFO:tasks.workunit.client.0.smithi111.stdout:1/143: chown f1a 22686103 1 2024-03-14T13:02:08.480 INFO:tasks.workunit.client.0.smithi111.stdout:1/144: chown f9 29636 1 2024-03-14T13:02:08.480 INFO:tasks.workunit.client.0.smithi111.stdout:1/145: chown d20 34163 1 2024-03-14T13:02:08.481 INFO:tasks.workunit.client.0.smithi111.stdout:1/146: chown fa 35579 1 2024-03-14T13:02:08.485 INFO:tasks.workunit.client.0.smithi111.stdout:1/147: dread f11 [0,4194304] 0 2024-03-14T13:02:08.485 INFO:tasks.workunit.client.0.smithi111.stdout:1/148: chown f1a 3 1 2024-03-14T13:02:08.488 INFO:tasks.workunit.client.0.smithi111.stdout:1/149: fsync d20/f24 0 2024-03-14T13:02:08.490 INFO:tasks.workunit.client.0.smithi111.stdout:1/150: unlink d20/d21/d28/c29 0 2024-03-14T13:02:08.492 INFO:tasks.workunit.client.0.smithi111.stdout:1/151: symlink d20/d25/d31/l36 0 2024-03-14T13:02:08.493 INFO:tasks.workunit.client.0.smithi111.stdout:1/152: chown l1e 1137 1 2024-03-14T13:02:08.493 INFO:tasks.workunit.client.0.smithi111.stdout:1/153: truncate f11 2097415 0 2024-03-14T13:02:08.495 INFO:tasks.workunit.client.0.smithi111.stdout:1/154: mknod d20/d25/c37 0 2024-03-14T13:02:08.496 INFO:tasks.workunit.client.0.smithi111.stdout:1/155: creat d20/d25/f38 x:0 0 0 2024-03-14T13:02:08.501 INFO:tasks.workunit.client.0.smithi111.stdout:1/156: dwrite f19 [0,4194304] 0 2024-03-14T13:02:08.503 INFO:tasks.workunit.client.0.smithi111.stdout:1/157: creat d20/d25/f39 x:0 0 0 2024-03-14T13:02:08.505 INFO:tasks.workunit.client.0.smithi111.stdout:1/158: creat d20/d25/d31/f3a x:0 0 0 2024-03-14T13:02:08.506 INFO:tasks.workunit.client.0.smithi111.stdout:1/159: creat d20/d25/d31/f3b x:0 0 0 2024-03-14T13:02:08.507 INFO:tasks.workunit.client.0.smithi111.stdout:1/160: dread - f1a zero size 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:6/9: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:4/22: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:2/1: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:3/8: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:2/2: truncate - no filename 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:7/15: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:5/21: sync 2024-03-14T13:02:08.509 INFO:tasks.workunit.client.1.smithi137.stdout:5/22: rmdir - no directory 2024-03-14T13:02:08.510 INFO:tasks.workunit.client.1.smithi137.stdout:8/13: sync 2024-03-14T13:02:08.510 INFO:tasks.workunit.client.1.smithi137.stdout:1/45: sync 2024-03-14T13:02:08.510 INFO:tasks.workunit.client.1.smithi137.stdout:9/40: sync 2024-03-14T13:02:08.510 INFO:tasks.workunit.client.1.smithi137.stdout:1/46: chown d7/l8 6 1 2024-03-14T13:02:08.511 INFO:tasks.workunit.client.1.smithi137.stdout:5/23: creat f6 x:0 0 0 2024-03-14T13:02:08.513 INFO:tasks.workunit.client.1.smithi137.stdout:3/9: fdatasync f1 0 2024-03-14T13:02:08.513 INFO:tasks.workunit.client.1.smithi137.stdout:3/10: dread - f1 zero size 2024-03-14T13:02:08.513 INFO:tasks.workunit.client.0.smithi111.stdout:1/161: dwrite d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:08.516 INFO:tasks.workunit.client.1.smithi137.stdout:8/14: dread d1/f2 [0,4194304] 0 2024-03-14T13:02:08.516 INFO:tasks.workunit.client.1.smithi137.stdout:1/47: rename d7/fa to d7/fb 0 2024-03-14T13:02:08.516 INFO:tasks.workunit.client.1.smithi137.stdout:8/15: readlink d1/l4 0 2024-03-14T13:02:08.517 INFO:tasks.workunit.client.1.smithi137.stdout:8/16: read d1/f2 [381383,28803] 0 2024-03-14T13:02:08.520 INFO:tasks.workunit.client.1.smithi137.stdout:8/17: truncate d1/f2 1492468 0 2024-03-14T13:02:08.520 INFO:tasks.workunit.client.1.smithi137.stdout:8/18: chown d1/l4 112320841 1 2024-03-14T13:02:08.521 INFO:tasks.workunit.client.0.smithi111.stdout:1/162: link ld d20/d25/d31/l3c 0 2024-03-14T13:02:08.522 INFO:tasks.workunit.client.1.smithi137.stdout:8/19: fsync f0 0 2024-03-14T13:02:08.523 INFO:tasks.workunit.client.1.smithi137.stdout:7/16: write f2 [493374,94679] 0 2024-03-14T13:02:08.523 INFO:tasks.workunit.client.0.smithi111.stdout:0/104: getdents d6 0 2024-03-14T13:02:08.524 INFO:tasks.workunit.client.1.smithi137.stdout:4/23: write f2 [989482,48708] 0 2024-03-14T13:02:08.524 INFO:tasks.workunit.client.0.smithi111.stdout:1/163: creat d20/d25/f3d x:0 0 0 2024-03-14T13:02:08.525 INFO:tasks.workunit.client.1.smithi137.stdout:3/11: write f1 [642191,31247] 0 2024-03-14T13:02:08.526 INFO:tasks.workunit.client.0.smithi111.stdout:1/164: write f3 [3364495,93427] 0 2024-03-14T13:02:08.526 INFO:tasks.workunit.client.0.smithi111.stdout:0/105: rmdir d6/d9/de 39 2024-03-14T13:02:08.527 INFO:tasks.workunit.client.1.smithi137.stdout:7/17: mkdir d3 0 2024-03-14T13:02:08.527 INFO:tasks.workunit.client.1.smithi137.stdout:4/24: write f2 [3826572,82561] 0 2024-03-14T13:02:08.527 INFO:tasks.workunit.client.0.smithi111.stdout:0/106: read f3 [8232911,38583] 0 2024-03-14T13:02:08.529 INFO:tasks.workunit.client.1.smithi137.stdout:4/25: write f2 [2759599,8993] 0 2024-03-14T13:02:08.530 INFO:tasks.workunit.client.0.smithi111.stdout:1/165: link d20/d25/d31/f3a d20/d25/f3e 0 2024-03-14T13:02:08.531 INFO:tasks.workunit.client.1.smithi137.stdout:9/41: write f1 [287643,112815] 0 2024-03-14T13:02:08.532 INFO:tasks.workunit.client.1.smithi137.stdout:7/18: link f1 d3/f4 0 2024-03-14T13:02:08.533 INFO:tasks.workunit.client.1.smithi137.stdout:1/48: fsync d7/fb 0 2024-03-14T13:02:08.534 INFO:tasks.workunit.client.1.smithi137.stdout:4/26: fdatasync f2 0 2024-03-14T13:02:08.535 INFO:tasks.workunit.client.0.smithi111.stdout:1/166: mkdir d20/d21/d3f 0 2024-03-14T13:02:08.535 INFO:tasks.workunit.client.1.smithi137.stdout:3/12: dread f1 [0,4194304] 0 2024-03-14T13:02:08.536 INFO:tasks.workunit.client.0.smithi111.stdout:0/107: symlink d6/d9/de/l1c 0 2024-03-14T13:02:08.536 INFO:tasks.workunit.client.0.smithi111.stdout:0/108: dread - d6/f1a zero size 2024-03-14T13:02:08.536 INFO:tasks.workunit.client.1.smithi137.stdout:1/49: rmdir d7 39 2024-03-14T13:02:08.537 INFO:tasks.workunit.client.0.smithi111.stdout:1/167: write f9 [4725177,30288] 0 2024-03-14T13:02:08.537 INFO:tasks.workunit.client.1.smithi137.stdout:7/19: dread f2 [0,4194304] 0 2024-03-14T13:02:08.538 INFO:tasks.workunit.client.0.smithi111.stdout:0/109: dread - d6/f10 zero size 2024-03-14T13:02:08.538 INFO:tasks.workunit.client.1.smithi137.stdout:4/27: creat f3 x:0 0 0 2024-03-14T13:02:08.539 INFO:tasks.workunit.client.1.smithi137.stdout:3/13: dread f1 [0,4194304] 0 2024-03-14T13:02:08.539 INFO:tasks.workunit.client.1.smithi137.stdout:8/20: dread d1/f2 [0,4194304] 0 2024-03-14T13:02:08.540 INFO:tasks.workunit.client.1.smithi137.stdout:4/28: fsync f2 0 2024-03-14T13:02:08.540 INFO:tasks.workunit.client.0.smithi111.stdout:9/96: sync 2024-03-14T13:02:08.541 INFO:tasks.workunit.client.1.smithi137.stdout:9/42: dread d6/fa [0,4194304] 0 2024-03-14T13:02:08.542 INFO:tasks.workunit.client.1.smithi137.stdout:4/29: mkdir d4 0 2024-03-14T13:02:08.543 INFO:tasks.workunit.client.1.smithi137.stdout:1/50: mknod d7/cc 0 2024-03-14T13:02:08.543 INFO:tasks.workunit.client.1.smithi137.stdout:8/21: dread f0 [0,4194304] 0 2024-03-14T13:02:08.544 INFO:tasks.workunit.client.1.smithi137.stdout:9/43: write f1 [5136446,23794] 0 2024-03-14T13:02:08.545 INFO:tasks.workunit.client.1.smithi137.stdout:3/14: rename l2 to l3 0 2024-03-14T13:02:08.546 INFO:tasks.workunit.client.1.smithi137.stdout:8/22: readlink d1/l4 0 2024-03-14T13:02:08.546 INFO:tasks.workunit.client.1.smithi137.stdout:1/51: write f5 [4021875,98900] 0 2024-03-14T13:02:08.547 INFO:tasks.workunit.client.0.smithi111.stdout:1/168: dwrite d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:08.549 INFO:tasks.workunit.client.1.smithi137.stdout:9/44: mknod d6/cd 0 2024-03-14T13:02:08.549 INFO:tasks.workunit.client.0.smithi111.stdout:9/97: truncate d5/d6/d8/f12 1441465 0 2024-03-14T13:02:08.551 INFO:tasks.workunit.client.1.smithi137.stdout:1/52: creat d7/d9/fd x:0 0 0 2024-03-14T13:02:08.552 INFO:tasks.workunit.client.0.smithi111.stdout:1/169: write d20/d25/f30 [77376,130261] 0 2024-03-14T13:02:08.553 INFO:tasks.workunit.client.1.smithi137.stdout:7/20: dwrite f2 [0,4194304] 0 2024-03-14T13:02:08.553 INFO:tasks.workunit.client.0.smithi111.stdout:0/110: dwrite d6/f1a [0,4194304] 0 2024-03-14T13:02:08.554 INFO:tasks.workunit.client.0.smithi111.stdout:1/170: chown d20/d25/f3e 0 1 2024-03-14T13:02:08.554 INFO:tasks.workunit.client.1.smithi137.stdout:9/45: rename d6/f7 to d6/fe 0 2024-03-14T13:02:08.556 INFO:tasks.workunit.client.1.smithi137.stdout:1/53: creat d7/d9/fe x:0 0 0 2024-03-14T13:02:08.559 INFO:tasks.workunit.client.0.smithi111.stdout:0/111: creat d6/d9/de/f1d x:0 0 0 2024-03-14T13:02:08.560 INFO:tasks.workunit.client.1.smithi137.stdout:1/54: chown f6 26653192 1 2024-03-14T13:02:08.560 INFO:tasks.workunit.client.1.smithi137.stdout:4/30: truncate f2 3767836 0 2024-03-14T13:02:08.561 INFO:tasks.workunit.client.1.smithi137.stdout:4/31: dread - f3 zero size 2024-03-14T13:02:08.562 INFO:tasks.workunit.client.1.smithi137.stdout:9/46: truncate d6/fe 4708160 0 2024-03-14T13:02:08.562 INFO:tasks.workunit.client.0.smithi111.stdout:0/112: creat d6/f1e x:0 0 0 2024-03-14T13:02:08.563 INFO:tasks.workunit.client.1.smithi137.stdout:4/32: dread - f3 zero size 2024-03-14T13:02:08.564 INFO:tasks.workunit.client.0.smithi111.stdout:9/98: truncate d5/fa 824782 0 2024-03-14T13:02:08.566 INFO:tasks.workunit.client.1.smithi137.stdout:3/15: dwrite f1 [0,4194304] 0 2024-03-14T13:02:08.567 INFO:tasks.workunit.client.0.smithi111.stdout:9/99: write d5/f20 [556035,44167] 0 2024-03-14T13:02:08.568 INFO:tasks.workunit.client.1.smithi137.stdout:9/47: mkdir d6/df 0 2024-03-14T13:02:08.569 INFO:tasks.workunit.client.0.smithi111.stdout:0/113: dread f3 [0,4194304] 0 2024-03-14T13:02:08.570 INFO:tasks.workunit.client.0.smithi111.stdout:9/100: write d5/d6/d14/f1c [423205,98182] 0 2024-03-14T13:02:08.572 INFO:tasks.workunit.client.0.smithi111.stdout:1/171: dwrite d20/d25/d31/f3a [0,4194304] 0 2024-03-14T13:02:08.573 INFO:tasks.workunit.client.1.smithi137.stdout:3/16: mkdir d4 0 2024-03-14T13:02:08.575 INFO:tasks.workunit.client.1.smithi137.stdout:9/48: creat d6/df/f10 x:0 0 0 2024-03-14T13:02:08.575 INFO:tasks.workunit.client.1.smithi137.stdout:9/49: stat d6/fa 0 2024-03-14T13:02:08.576 INFO:tasks.workunit.client.1.smithi137.stdout:3/17: creat d4/f5 x:0 0 0 2024-03-14T13:02:08.576 INFO:tasks.workunit.client.0.smithi111.stdout:0/114: creat d6/f1f x:0 0 0 2024-03-14T13:02:08.577 INFO:tasks.workunit.client.1.smithi137.stdout:3/18: write f1 [2399288,74780] 0 2024-03-14T13:02:08.580 INFO:tasks.workunit.client.1.smithi137.stdout:3/19: symlink d4/l6 0 2024-03-14T13:02:08.581 INFO:tasks.workunit.client.0.smithi111.stdout:1/172: mkdir d20/d40 0 2024-03-14T13:02:08.581 INFO:tasks.workunit.client.1.smithi137.stdout:3/20: readlink l3 0 2024-03-14T13:02:08.582 INFO:tasks.workunit.client.1.smithi137.stdout:3/21: chown d4 6 1 2024-03-14T13:02:08.582 INFO:tasks.workunit.client.0.smithi111.stdout:9/101: link d5/d6/c18 d5/d6/d8/d1a/c23 0 2024-03-14T13:02:08.582 INFO:tasks.workunit.client.0.smithi111.stdout:9/102: chown d5/d6 28048047 1 2024-03-14T13:02:08.585 INFO:tasks.workunit.client.0.smithi111.stdout:1/173: creat d20/d21/f41 x:0 0 0 2024-03-14T13:02:08.586 INFO:tasks.workunit.client.0.smithi111.stdout:1/174: creat d20/d21/f42 x:0 0 0 2024-03-14T13:02:08.589 INFO:tasks.workunit.client.1.smithi137.stdout:3/22: dwrite f1 [0,4194304] 0 2024-03-14T13:02:08.589 INFO:tasks.workunit.client.1.smithi137.stdout:3/23: chown d4/f5 19113609 1 2024-03-14T13:02:08.590 INFO:tasks.workunit.client.0.smithi111.stdout:1/175: getdents d20/d25/d31 0 2024-03-14T13:02:08.590 INFO:tasks.workunit.client.0.smithi111.stdout:1/176: dread - f1a zero size 2024-03-14T13:02:08.591 INFO:tasks.workunit.client.0.smithi111.stdout:1/177: write d20/d21/f2a [381906,26306] 0 2024-03-14T13:02:08.592 INFO:tasks.workunit.client.0.smithi111.stdout:1/178: chown f11 4748 1 2024-03-14T13:02:08.593 INFO:tasks.workunit.client.1.smithi137.stdout:3/24: mknod d4/c7 0 2024-03-14T13:02:08.593 INFO:tasks.workunit.client.0.smithi111.stdout:1/179: mknod d20/c43 0 2024-03-14T13:02:08.598 INFO:tasks.workunit.client.1.smithi137.stdout:3/25: unlink d4/c7 0 2024-03-14T13:02:08.598 INFO:tasks.workunit.client.0.smithi111.stdout:1/180: creat d20/f44 x:0 0 0 2024-03-14T13:02:08.598 INFO:tasks.workunit.client.0.smithi111.stdout:1/181: truncate f1a 766855 0 2024-03-14T13:02:08.599 INFO:tasks.workunit.client.0.smithi111.stdout:1/182: dread - d20/f44 zero size 2024-03-14T13:02:08.600 INFO:tasks.workunit.client.1.smithi137.stdout:9/50: getdents d6 0 2024-03-14T13:02:08.602 INFO:tasks.workunit.client.0.smithi111.stdout:9/103: rmdir d5/d6/d8 39 2024-03-14T13:02:08.604 INFO:tasks.workunit.client.1.smithi137.stdout:9/51: creat d6/df/f11 x:0 0 0 2024-03-14T13:02:08.607 INFO:tasks.workunit.client.0.smithi111.stdout:0/115: dwrite d6/f10 [0,4194304] 0 2024-03-14T13:02:08.607 INFO:tasks.workunit.client.1.smithi137.stdout:7/21: dwrite d3/f4 [0,4194304] 0 2024-03-14T13:02:08.608 INFO:tasks.workunit.client.1.smithi137.stdout:3/26: dread f1 [0,4194304] 0 2024-03-14T13:02:08.609 INFO:tasks.workunit.client.1.smithi137.stdout:1/55: dwrite f3 [0,4194304] 0 2024-03-14T13:02:08.610 INFO:tasks.workunit.client.0.smithi111.stdout:0/116: symlink d6/d9/de/l20 0 2024-03-14T13:02:08.611 INFO:tasks.workunit.client.1.smithi137.stdout:9/52: mknod d6/df/c12 0 2024-03-14T13:02:08.611 INFO:tasks.workunit.client.0.smithi111.stdout:0/117: mkdir d6/d21 0 2024-03-14T13:02:08.613 INFO:tasks.workunit.client.1.smithi137.stdout:3/27: symlink d4/l8 0 2024-03-14T13:02:08.614 INFO:tasks.workunit.client.1.smithi137.stdout:1/56: dread - f4 zero size 2024-03-14T13:02:08.614 INFO:tasks.workunit.client.1.smithi137.stdout:3/28: write d4/f5 [809000,30707] 0 2024-03-14T13:02:08.615 INFO:tasks.workunit.client.1.smithi137.stdout:7/22: mkdir d3/d5 0 2024-03-14T13:02:08.615 INFO:tasks.workunit.client.1.smithi137.stdout:1/57: read - f4 zero size 2024-03-14T13:02:08.616 INFO:tasks.workunit.client.0.smithi111.stdout:0/118: mknod d6/d9/c22 0 2024-03-14T13:02:08.617 INFO:tasks.workunit.client.1.smithi137.stdout:1/58: chown d7 1244 1 2024-03-14T13:02:08.617 INFO:tasks.workunit.client.1.smithi137.stdout:0/4: sync 2024-03-14T13:02:08.617 INFO:tasks.workunit.client.1.smithi137.stdout:3/29: unlink d4/l6 0 2024-03-14T13:02:08.618 INFO:tasks.workunit.client.1.smithi137.stdout:7/23: mkdir d3/d6 0 2024-03-14T13:02:08.622 INFO:tasks.workunit.client.1.smithi137.stdout:0/5: mkdir d1 0 2024-03-14T13:02:08.622 INFO:tasks.workunit.client.1.smithi137.stdout:0/6: chown d1 105 1 2024-03-14T13:02:08.622 INFO:tasks.workunit.client.1.smithi137.stdout:0/7: write - no filename 2024-03-14T13:02:08.623 INFO:tasks.workunit.client.1.smithi137.stdout:0/8: dwrite - no filename 2024-03-14T13:02:08.623 INFO:tasks.workunit.client.1.smithi137.stdout:0/9: write - no filename 2024-03-14T13:02:08.624 INFO:tasks.workunit.client.1.smithi137.stdout:3/30: creat d4/f9 x:0 0 0 2024-03-14T13:02:08.624 INFO:tasks.workunit.client.1.smithi137.stdout:1/59: dread d7/fb [0,4194304] 0 2024-03-14T13:02:08.625 INFO:tasks.workunit.client.1.smithi137.stdout:3/31: chown f1 111124398 1 2024-03-14T13:02:08.625 INFO:tasks.workunit.client.1.smithi137.stdout:1/60: readlink d7/l8 0 2024-03-14T13:02:08.626 INFO:tasks.workunit.client.1.smithi137.stdout:3/32: stat d4/l8 0 2024-03-14T13:02:08.627 INFO:tasks.workunit.client.1.smithi137.stdout:0/10: symlink d1/l2 0 2024-03-14T13:02:08.627 INFO:tasks.workunit.client.1.smithi137.stdout:0/11: write - no filename 2024-03-14T13:02:08.627 INFO:tasks.workunit.client.1.smithi137.stdout:0/12: dread - no filename 2024-03-14T13:02:08.627 INFO:tasks.workunit.client.1.smithi137.stdout:0/13: truncate - no filename 2024-03-14T13:02:08.627 INFO:tasks.workunit.client.1.smithi137.stdout:1/61: creat d7/d9/ff x:0 0 0 2024-03-14T13:02:08.631 INFO:tasks.workunit.client.1.smithi137.stdout:0/14: rename l0 to d1/l3 0 2024-03-14T13:02:08.632 INFO:tasks.workunit.client.1.smithi137.stdout:3/33: dread d4/f5 [0,4194304] 0 2024-03-14T13:02:08.634 INFO:tasks.workunit.client.1.smithi137.stdout:0/15: creat d1/f4 x:0 0 0 2024-03-14T13:02:08.634 INFO:tasks.workunit.client.1.smithi137.stdout:7/24: dwrite f2 [0,4194304] 0 2024-03-14T13:02:08.634 INFO:tasks.workunit.client.0.smithi111.stdout:9/104: dread d5/d6/d8/f12 [0,4194304] 0 2024-03-14T13:02:08.635 INFO:tasks.workunit.client.0.smithi111.stdout:9/105: read d5/d6/d8/f12 [288894,51348] 0 2024-03-14T13:02:08.636 INFO:tasks.workunit.client.1.smithi137.stdout:3/34: dread d4/f5 [0,4194304] 0 2024-03-14T13:02:08.636 INFO:tasks.workunit.client.0.smithi111.stdout:9/106: read f4 [5383505,42104] 0 2024-03-14T13:02:08.638 INFO:tasks.workunit.client.0.smithi111.stdout:9/107: fdatasync d5/d6/d8/f16 0 2024-03-14T13:02:08.640 INFO:tasks.workunit.client.1.smithi137.stdout:3/35: rename l0 to d4/la 0 2024-03-14T13:02:08.641 INFO:tasks.workunit.client.1.smithi137.stdout:7/25: symlink d3/d6/l7 0 2024-03-14T13:02:08.642 INFO:tasks.workunit.client.0.smithi111.stdout:9/108: rmdir d5/d6/d1e 0 2024-03-14T13:02:08.643 INFO:tasks.workunit.client.1.smithi137.stdout:4/33: truncate f2 2103976 0 2024-03-14T13:02:08.643 INFO:tasks.workunit.client.1.smithi137.stdout:7/26: write f1 [4707519,65525] 0 2024-03-14T13:02:08.643 INFO:tasks.workunit.client.1.smithi137.stdout:3/36: creat d4/fb x:0 0 0 2024-03-14T13:02:08.643 INFO:tasks.workunit.client.1.smithi137.stdout:7/27: readlink d3/d6/l7 0 2024-03-14T13:02:08.644 INFO:tasks.workunit.client.1.smithi137.stdout:4/34: truncate f3 662372 0 2024-03-14T13:02:08.644 INFO:tasks.workunit.client.0.smithi111.stdout:9/109: write d5/d6/f22 [802863,40127] 0 2024-03-14T13:02:08.644 INFO:tasks.workunit.client.1.smithi137.stdout:3/37: truncate d4/f5 924470 0 2024-03-14T13:02:08.645 INFO:tasks.workunit.client.1.smithi137.stdout:4/35: read f3 [474824,40900] 0 2024-03-14T13:02:08.647 INFO:tasks.workunit.client.1.smithi137.stdout:9/53: truncate d6/fe 2647213 0 2024-03-14T13:02:08.650 INFO:tasks.workunit.client.1.smithi137.stdout:1/62: truncate d7/fb 140849 0 2024-03-14T13:02:08.653 INFO:tasks.workunit.client.1.smithi137.stdout:7/28: dread d3/f4 [4194304,4194304] 0 2024-03-14T13:02:08.653 INFO:tasks.workunit.client.0.smithi111.stdout:0/119: dread d6/fd [0,4194304] 0 2024-03-14T13:02:08.654 INFO:tasks.workunit.client.1.smithi137.stdout:1/63: creat d7/f10 x:0 0 0 2024-03-14T13:02:08.655 INFO:tasks.workunit.client.1.smithi137.stdout:9/54: dwrite f1 [0,4194304] 0 2024-03-14T13:02:08.655 INFO:tasks.workunit.client.1.smithi137.stdout:4/36: fdatasync f2 0 2024-03-14T13:02:08.655 INFO:tasks.workunit.client.1.smithi137.stdout:4/37: chown f2 67765860 1 2024-03-14T13:02:08.656 INFO:tasks.workunit.client.1.smithi137.stdout:1/64: truncate d7/d9/fd 12296 0 2024-03-14T13:02:08.658 INFO:tasks.workunit.client.0.smithi111.stdout:9/110: dwrite d5/fa [0,4194304] 0 2024-03-14T13:02:08.659 INFO:tasks.workunit.client.1.smithi137.stdout:7/29: write f1 [1395723,31030] 0 2024-03-14T13:02:08.659 INFO:tasks.workunit.client.1.smithi137.stdout:1/65: rename d7 to d7/d9/d11 22 2024-03-14T13:02:08.659 INFO:tasks.workunit.client.1.smithi137.stdout:9/55: fdatasync d6/fa 0 2024-03-14T13:02:08.659 INFO:tasks.workunit.client.0.smithi111.stdout:9/111: chown d5/fe 0 1 2024-03-14T13:02:08.661 INFO:tasks.workunit.client.1.smithi137.stdout:4/38: mknod d4/c5 0 2024-03-14T13:02:08.661 INFO:tasks.workunit.client.1.smithi137.stdout:0/16: getdents d1 0 2024-03-14T13:02:08.661 INFO:tasks.workunit.client.0.smithi111.stdout:9/112: rmdir d5 39 2024-03-14T13:02:08.664 INFO:tasks.workunit.client.0.smithi111.stdout:9/113: fsync d5/d6/f22 0 2024-03-14T13:02:08.665 INFO:tasks.workunit.client.0.smithi111.stdout:0/120: truncate f1 4299460 0 2024-03-14T13:02:08.665 INFO:tasks.workunit.client.0.smithi111.stdout:0/121: fdatasync d6/f1e 0 2024-03-14T13:02:08.666 INFO:tasks.workunit.client.1.smithi137.stdout:0/17: dread - d1/f4 zero size 2024-03-14T13:02:08.666 INFO:tasks.workunit.client.1.smithi137.stdout:3/38: rmdir d4 39 2024-03-14T13:02:08.666 INFO:tasks.workunit.client.1.smithi137.stdout:7/30: dread f2 [0,4194304] 0 2024-03-14T13:02:08.667 INFO:tasks.workunit.client.1.smithi137.stdout:1/66: dread f5 [0,4194304] 0 2024-03-14T13:02:08.668 INFO:tasks.workunit.client.0.smithi111.stdout:0/122: creat d6/d9/de/f23 x:0 0 0 2024-03-14T13:02:08.668 INFO:tasks.workunit.client.0.smithi111.stdout:0/123: stat d6/fb 0 2024-03-14T13:02:08.669 INFO:tasks.workunit.client.0.smithi111.stdout:9/114: rename d5/d6/c13 to d5/d6/c24 0 2024-03-14T13:02:08.670 INFO:tasks.workunit.client.1.smithi137.stdout:4/39: rmdir d4 39 2024-03-14T13:02:08.670 INFO:tasks.workunit.client.0.smithi111.stdout:0/124: creat d6/d9/d15/f24 x:0 0 0 2024-03-14T13:02:08.674 INFO:tasks.workunit.client.0.smithi111.stdout:9/115: fdatasync f4 0 2024-03-14T13:02:08.674 INFO:tasks.workunit.client.1.smithi137.stdout:9/56: creat d6/df/f13 x:0 0 0 2024-03-14T13:02:08.674 INFO:tasks.workunit.client.1.smithi137.stdout:3/39: readlink l3 0 2024-03-14T13:02:08.676 INFO:tasks.workunit.client.0.smithi111.stdout:9/116: mknod d5/d6/d8/d1a/c25 0 2024-03-14T13:02:08.676 INFO:tasks.workunit.client.1.smithi137.stdout:7/31: fdatasync f1 0 2024-03-14T13:02:08.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/117: chown c3 264262565 1 2024-03-14T13:02:08.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/118: chown d5/d6/d8 774 1 2024-03-14T13:02:08.678 INFO:tasks.workunit.client.0.smithi111.stdout:0/125: dwrite d6/fb [0,4194304] 0 2024-03-14T13:02:08.679 INFO:tasks.workunit.client.1.smithi137.stdout:1/67: symlink d7/l12 0 2024-03-14T13:02:08.679 INFO:tasks.workunit.client.0.smithi111.stdout:0/126: fsync d6/f10 0 2024-03-14T13:02:08.679 INFO:tasks.workunit.client.0.smithi111.stdout:0/127: stat d6/d9/de/f1d 0 2024-03-14T13:02:08.680 INFO:tasks.workunit.client.1.smithi137.stdout:0/18: dwrite d1/f4 [0,4194304] 0 2024-03-14T13:02:08.682 INFO:tasks.workunit.client.0.smithi111.stdout:0/128: mkdir d6/d9/d15/d25 0 2024-03-14T13:02:08.684 INFO:tasks.workunit.client.1.smithi137.stdout:3/40: chown d4 2062653 1 2024-03-14T13:02:08.684 INFO:tasks.workunit.client.0.smithi111.stdout:0/129: creat d6/d9/de/f26 x:0 0 0 2024-03-14T13:02:08.685 INFO:tasks.workunit.client.1.smithi137.stdout:7/32: rename f2 to d3/d5/f8 0 2024-03-14T13:02:08.686 INFO:tasks.workunit.client.0.smithi111.stdout:0/130: creat d6/d9/de/f27 x:0 0 0 2024-03-14T13:02:08.686 INFO:tasks.workunit.client.0.smithi111.stdout:0/131: read d6/f1a [238570,65534] 0 2024-03-14T13:02:08.688 INFO:tasks.workunit.client.1.smithi137.stdout:1/68: write f3 [57289,125856] 0 2024-03-14T13:02:08.690 INFO:tasks.workunit.client.0.smithi111.stdout:0/132: link d6/f10 d6/d9/f28 0 2024-03-14T13:02:08.692 INFO:tasks.workunit.client.1.smithi137.stdout:1/69: write d7/fb [748591,104725] 0 2024-03-14T13:02:08.693 INFO:tasks.workunit.client.1.smithi137.stdout:1/70: fsync d7/d9/fd 0 2024-03-14T13:02:08.693 INFO:tasks.workunit.client.1.smithi137.stdout:4/40: creat d4/f6 x:0 0 0 2024-03-14T13:02:08.693 INFO:tasks.workunit.client.0.smithi111.stdout:0/133: truncate d6/d9/d15/f24 97855 0 2024-03-14T13:02:08.696 INFO:tasks.workunit.client.0.smithi111.stdout:0/134: fdatasync f3 0 2024-03-14T13:02:08.697 INFO:tasks.workunit.client.1.smithi137.stdout:0/19: dread d1/f4 [0,4194304] 0 2024-03-14T13:02:08.698 INFO:tasks.workunit.client.0.smithi111.stdout:0/135: write d6/d9/d15/f24 [464045,130213] 0 2024-03-14T13:02:08.699 INFO:tasks.workunit.client.1.smithi137.stdout:7/33: dwrite d3/f4 [0,4194304] 0 2024-03-14T13:02:08.700 INFO:tasks.workunit.client.1.smithi137.stdout:1/71: read f6 [421940,59313] 0 2024-03-14T13:02:08.700 INFO:tasks.workunit.client.0.smithi111.stdout:0/136: fsync f3 0 2024-03-14T13:02:08.701 INFO:tasks.workunit.client.1.smithi137.stdout:1/72: write d7/d9/ff [723269,27434] 0 2024-03-14T13:02:08.702 INFO:tasks.workunit.client.0.smithi111.stdout:0/137: readlink d6/d9/de/l20 0 2024-03-14T13:02:08.702 INFO:tasks.workunit.client.1.smithi137.stdout:9/57: getdents d6/df 0 2024-03-14T13:02:08.703 INFO:tasks.workunit.client.1.smithi137.stdout:4/41: mknod d4/c7 0 2024-03-14T13:02:08.703 INFO:tasks.workunit.client.1.smithi137.stdout:9/58: read - d6/df/f11 zero size 2024-03-14T13:02:08.705 INFO:tasks.workunit.client.0.smithi111.stdout:0/138: link d6/d9/c22 d6/d9/c29 0 2024-03-14T13:02:08.706 INFO:tasks.workunit.client.1.smithi137.stdout:0/20: mkdir d1/d5 0 2024-03-14T13:02:08.707 INFO:tasks.workunit.client.1.smithi137.stdout:4/42: unlink d4/f6 0 2024-03-14T13:02:08.710 INFO:tasks.workunit.client.1.smithi137.stdout:7/34: rename d3/d5/f8 to d3/d6/f9 0 2024-03-14T13:02:08.711 INFO:tasks.workunit.client.1.smithi137.stdout:7/35: readlink d3/d6/l7 0 2024-03-14T13:02:08.712 INFO:tasks.workunit.client.1.smithi137.stdout:1/73: dread d7/d9/ff [0,4194304] 0 2024-03-14T13:02:08.713 INFO:tasks.workunit.client.0.smithi111.stdout:0/139: getdents d6/d9/de 0 2024-03-14T13:02:08.713 INFO:tasks.workunit.client.1.smithi137.stdout:7/36: write d3/f4 [2698850,129218] 0 2024-03-14T13:02:08.716 INFO:tasks.workunit.client.0.smithi111.stdout:0/140: link d6/d9/de/l17 d6/d21/l2a 0 2024-03-14T13:02:08.717 INFO:tasks.workunit.client.1.smithi137.stdout:3/41: truncate f1 3348918 0 2024-03-14T13:02:08.718 INFO:tasks.workunit.client.1.smithi137.stdout:1/74: symlink d7/d9/l13 0 2024-03-14T13:02:08.718 INFO:tasks.workunit.client.0.smithi111.stdout:0/141: symlink d6/d9/d15/d25/l2b 0 2024-03-14T13:02:08.719 INFO:tasks.workunit.client.1.smithi137.stdout:3/42: dread - d4/f9 zero size 2024-03-14T13:02:08.719 INFO:tasks.workunit.client.1.smithi137.stdout:0/21: rename d1/f4 to d1/d5/f6 0 2024-03-14T13:02:08.720 INFO:tasks.workunit.client.1.smithi137.stdout:3/43: chown d4/l8 409689 1 2024-03-14T13:02:08.720 INFO:tasks.workunit.client.1.smithi137.stdout:7/37: symlink d3/d5/la 0 2024-03-14T13:02:08.720 INFO:tasks.workunit.client.0.smithi111.stdout:0/142: creat d6/d9/de/f2c x:0 0 0 2024-03-14T13:02:08.721 INFO:tasks.workunit.client.0.smithi111.stdout:9/119: dread f2 [0,4194304] 0 2024-03-14T13:02:08.722 INFO:tasks.workunit.client.1.smithi137.stdout:1/75: write f5 [4580829,49587] 0 2024-03-14T13:02:08.723 INFO:tasks.workunit.client.1.smithi137.stdout:1/76: write f5 [67152,128341] 0 2024-03-14T13:02:08.723 INFO:tasks.workunit.client.1.smithi137.stdout:7/38: rmdir d3/d6 39 2024-03-14T13:02:08.724 INFO:tasks.workunit.client.0.smithi111.stdout:0/143: truncate d6/d9/f28 549335 0 2024-03-14T13:02:08.724 INFO:tasks.workunit.client.1.smithi137.stdout:3/44: fdatasync f1 0 2024-03-14T13:02:08.725 INFO:tasks.workunit.client.1.smithi137.stdout:1/77: fsync f6 0 2024-03-14T13:02:08.727 INFO:tasks.workunit.client.1.smithi137.stdout:7/39: readlink d3/d6/l7 0 2024-03-14T13:02:08.728 INFO:tasks.workunit.client.1.smithi137.stdout:3/45: unlink l3 0 2024-03-14T13:02:08.728 INFO:tasks.workunit.client.0.smithi111.stdout:9/120: dwrite d5/f20 [0,4194304] 0 2024-03-14T13:02:08.730 INFO:tasks.workunit.client.1.smithi137.stdout:3/46: dread - d4/fb zero size 2024-03-14T13:02:08.730 INFO:tasks.workunit.client.1.smithi137.stdout:0/22: dwrite d1/d5/f6 [0,4194304] 0 2024-03-14T13:02:08.732 INFO:tasks.workunit.client.0.smithi111.stdout:9/121: dread d5/d6/f9 [0,4194304] 0 2024-03-14T13:02:08.733 INFO:tasks.workunit.client.1.smithi137.stdout:3/47: write d4/fb [2616,121] 0 2024-03-14T13:02:08.733 INFO:tasks.workunit.client.1.smithi137.stdout:9/59: write d6/fa [3192216,15086] 0 2024-03-14T13:02:08.736 INFO:tasks.workunit.client.0.smithi111.stdout:9/122: dread d5/f20 [0,4194304] 0 2024-03-14T13:02:08.739 INFO:tasks.workunit.client.1.smithi137.stdout:4/43: getdents d4 0 2024-03-14T13:02:08.740 INFO:tasks.workunit.client.1.smithi137.stdout:7/40: dread d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:08.741 INFO:tasks.workunit.client.1.smithi137.stdout:7/41: fsync f1 0 2024-03-14T13:02:08.742 INFO:tasks.workunit.client.1.smithi137.stdout:3/48: creat d4/fc x:0 0 0 2024-03-14T13:02:08.744 INFO:tasks.workunit.client.1.smithi137.stdout:3/49: write d4/fc [843305,53313] 0 2024-03-14T13:02:08.744 INFO:tasks.workunit.client.1.smithi137.stdout:4/44: creat d4/f8 x:0 0 0 2024-03-14T13:02:08.748 INFO:tasks.workunit.client.1.smithi137.stdout:9/60: link d6/fe d6/df/f14 0 2024-03-14T13:02:08.749 INFO:tasks.workunit.client.1.smithi137.stdout:7/42: dread d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:08.749 INFO:tasks.workunit.client.1.smithi137.stdout:4/45: dread f2 [0,4194304] 0 2024-03-14T13:02:08.752 INFO:tasks.workunit.client.1.smithi137.stdout:7/43: creat d3/d6/fb x:0 0 0 2024-03-14T13:02:08.753 INFO:tasks.workunit.client.1.smithi137.stdout:9/61: rename d6/df/f11 to d6/df/f15 0 2024-03-14T13:02:08.754 INFO:tasks.workunit.client.1.smithi137.stdout:3/50: dwrite d4/fb [0,4194304] 0 2024-03-14T13:02:08.755 INFO:tasks.workunit.client.0.smithi111.stdout:9/123: dwrite f4 [0,4194304] 0 2024-03-14T13:02:08.757 INFO:tasks.workunit.client.0.smithi111.stdout:9/124: truncate d5/f20 1087679 0 2024-03-14T13:02:08.757 INFO:tasks.workunit.client.0.smithi111.stdout:9/125: chown d5/d6/d8/c1d 1 1 2024-03-14T13:02:08.758 INFO:tasks.workunit.client.1.smithi137.stdout:7/44: symlink d3/d6/lc 0 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:3/226: sync 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:5/214: sync 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:7/127: sync 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:6/135: sync 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:4/423: sync 2024-03-14T13:02:08.759 INFO:tasks.workunit.client.0.smithi111.stdout:2/98: sync 2024-03-14T13:02:08.760 INFO:tasks.workunit.client.1.smithi137.stdout:3/51: creat d4/fd x:0 0 0 2024-03-14T13:02:08.760 INFO:tasks.workunit.client.1.smithi137.stdout:4/46: dwrite f2 [0,4194304] 0 2024-03-14T13:02:08.762 INFO:tasks.workunit.client.0.smithi111.stdout:9/126: creat d5/f26 x:0 0 0 2024-03-14T13:02:08.763 INFO:tasks.workunit.client.0.smithi111.stdout:7/128: dread - d0/d1/d5/de/d9/f16 zero size 2024-03-14T13:02:08.764 INFO:tasks.workunit.client.0.smithi111.stdout:6/136: rmdir d0 39 2024-03-14T13:02:08.765 INFO:tasks.workunit.client.0.smithi111.stdout:7/129: dread - d0/d1/f25 zero size 2024-03-14T13:02:08.766 INFO:tasks.workunit.client.1.smithi137.stdout:7/45: creat d3/d5/fd x:0 0 0 2024-03-14T13:02:08.766 INFO:tasks.workunit.client.1.smithi137.stdout:3/52: unlink d4/fc 0 2024-03-14T13:02:08.766 INFO:tasks.workunit.client.0.smithi111.stdout:2/99: rmdir dd/d17/d18 39 2024-03-14T13:02:08.767 INFO:tasks.workunit.client.0.smithi111.stdout:2/100: fdatasync dd/fe 0 2024-03-14T13:02:08.767 INFO:tasks.workunit.client.1.smithi137.stdout:4/47: mkdir d4/d9 0 2024-03-14T13:02:08.768 INFO:tasks.workunit.client.0.smithi111.stdout:5/215: read d2/d13/f3d [3491367,13340] 0 2024-03-14T13:02:08.768 INFO:tasks.workunit.client.0.smithi111.stdout:2/101: chown l2 310 1 2024-03-14T13:02:08.769 INFO:tasks.workunit.client.1.smithi137.stdout:7/46: write d3/f4 [3535909,31046] 0 2024-03-14T13:02:08.769 INFO:tasks.workunit.client.1.smithi137.stdout:4/48: write f2 [3510427,111984] 0 2024-03-14T13:02:08.771 INFO:tasks.workunit.client.1.smithi137.stdout:6/10: sync 2024-03-14T13:02:08.772 INFO:tasks.workunit.client.0.smithi111.stdout:9/127: read d5/d6/f22 [3731869,113280] 0 2024-03-14T13:02:08.772 INFO:tasks.workunit.client.1.smithi137.stdout:9/62: dwrite d6/df/f15 [0,4194304] 0 2024-03-14T13:02:08.775 INFO:tasks.workunit.client.0.smithi111.stdout:4/424: mknod d1d/d22/d30/d5e/d6f/d7e/d66/c9a 0 2024-03-14T13:02:08.776 INFO:tasks.workunit.client.0.smithi111.stdout:6/137: mknod d0/d17/d24/c2a 0 2024-03-14T13:02:08.777 INFO:tasks.workunit.client.1.smithi137.stdout:6/11: mknod d2/c3 0 2024-03-14T13:02:08.778 INFO:tasks.workunit.client.0.smithi111.stdout:9/128: write d5/d6/f22 [5070685,35805] 0 2024-03-14T13:02:08.778 INFO:tasks.workunit.client.0.smithi111.stdout:7/130: mkdir d0/d1/d21/d2a 0 2024-03-14T13:02:08.779 INFO:tasks.workunit.client.1.smithi137.stdout:9/63: mknod d6/c16 0 2024-03-14T13:02:08.779 INFO:tasks.workunit.client.0.smithi111.stdout:3/227: dwrite d0/d30/f3d [0,4194304] 0 2024-03-14T13:02:08.779 INFO:tasks.workunit.client.0.smithi111.stdout:2/102: read dd/f10 [1960053,46439] 0 2024-03-14T13:02:08.781 INFO:tasks.workunit.client.1.smithi137.stdout:9/64: fdatasync d6/df/f10 0 2024-03-14T13:02:08.781 INFO:tasks.workunit.client.1.smithi137.stdout:7/47: rename d3/d6/fb to d3/fe 0 2024-03-14T13:02:08.781 INFO:tasks.workunit.client.1.smithi137.stdout:6/12: rmdir d2 39 2024-03-14T13:02:08.781 INFO:tasks.workunit.client.0.smithi111.stdout:5/216: mkdir d2/d16/d49/d44/d4a 0 2024-03-14T13:02:08.782 INFO:tasks.workunit.client.0.smithi111.stdout:2/103: stat lc 0 2024-03-14T13:02:08.783 INFO:tasks.workunit.client.0.smithi111.stdout:6/138: creat d0/d17/f2b x:0 0 0 2024-03-14T13:02:08.785 INFO:tasks.workunit.client.0.smithi111.stdout:4/425: mknod d1d/d22/d30/d7c/c9b 0 2024-03-14T13:02:08.785 INFO:tasks.workunit.client.1.smithi137.stdout:4/49: rename d4/c5 to d4/d9/ca 0 2024-03-14T13:02:08.787 INFO:tasks.workunit.client.0.smithi111.stdout:2/104: rmdir dd 39 2024-03-14T13:02:08.788 INFO:tasks.workunit.client.1.smithi137.stdout:4/50: chown l1 106 1 2024-03-14T13:02:08.788 INFO:tasks.workunit.client.0.smithi111.stdout:3/228: stat d0/de/l51 0 2024-03-14T13:02:08.788 INFO:tasks.workunit.client.0.smithi111.stdout:7/131: fdatasync d0/d1/d21/f15 0 2024-03-14T13:02:08.790 INFO:tasks.workunit.client.0.smithi111.stdout:5/217: unlink d2/d16/f48 0 2024-03-14T13:02:08.791 INFO:tasks.workunit.client.0.smithi111.stdout:6/139: creat d0/d17/f2c x:0 0 0 2024-03-14T13:02:08.792 INFO:tasks.workunit.client.0.smithi111.stdout:4/426: dread d1d/d22/d30/f38 [0,4194304] 0 2024-03-14T13:02:08.793 INFO:tasks.workunit.client.0.smithi111.stdout:5/218: chown d2/d16/d49/d44/d4a 21623113 1 2024-03-14T13:02:08.794 INFO:tasks.workunit.client.1.smithi137.stdout:6/13: link d2/c3 d2/c4 0 2024-03-14T13:02:08.794 INFO:tasks.workunit.client.1.smithi137.stdout:9/65: dread d6/fc [0,4194304] 0 2024-03-14T13:02:08.795 INFO:tasks.workunit.client.0.smithi111.stdout:2/105: fdatasync dd/d11/d1a/f1b 0 2024-03-14T13:02:08.795 INFO:tasks.workunit.client.1.smithi137.stdout:7/48: dwrite d3/d5/fd [0,4194304] 0 2024-03-14T13:02:08.796 INFO:tasks.workunit.client.0.smithi111.stdout:4/427: chown d1d/f43 128144321 1 2024-03-14T13:02:08.796 INFO:tasks.workunit.client.0.smithi111.stdout:9/129: dwrite f4 [4194304,4194304] 0 2024-03-14T13:02:08.797 INFO:tasks.workunit.client.1.smithi137.stdout:4/51: symlink d4/d9/lb 0 2024-03-14T13:02:08.797 INFO:tasks.workunit.client.0.smithi111.stdout:6/140: creat d0/d17/d21/f2d x:0 0 0 2024-03-14T13:02:08.800 INFO:tasks.workunit.client.0.smithi111.stdout:5/219: creat d2/d13/f4b x:0 0 0 2024-03-14T13:02:08.800 INFO:tasks.workunit.client.0.smithi111.stdout:9/130: chown d5/d6/d8/d1a 92285204 1 2024-03-14T13:02:08.800 INFO:tasks.workunit.client.0.smithi111.stdout:6/141: stat d0/l23 0 2024-03-14T13:02:08.802 INFO:tasks.workunit.client.0.smithi111.stdout:5/220: chown d2/d1f/f28 922 1 2024-03-14T13:02:08.802 INFO:tasks.workunit.client.1.smithi137.stdout:3/53: getdents d4 0 2024-03-14T13:02:08.803 INFO:tasks.workunit.client.0.smithi111.stdout:4/428: read d1d/d22/d30/f4d [62113,14402] 0 2024-03-14T13:02:08.804 INFO:tasks.workunit.client.0.smithi111.stdout:3/229: dwrite d0/d30/f3d [0,4194304] 0 2024-03-14T13:02:08.805 INFO:tasks.workunit.client.1.smithi137.stdout:3/54: stat d4/f9 0 2024-03-14T13:02:08.805 INFO:tasks.workunit.client.1.smithi137.stdout:6/14: rmdir d2 39 2024-03-14T13:02:08.807 INFO:tasks.workunit.client.0.smithi111.stdout:5/221: unlink d2/d13/f3d 0 2024-03-14T13:02:08.807 INFO:tasks.workunit.client.0.smithi111.stdout:4/429: fdatasync d1d/f36 0 2024-03-14T13:02:08.811 INFO:tasks.workunit.client.0.smithi111.stdout:5/222: write d2/f6 [594693,72677] 0 2024-03-14T13:02:08.811 INFO:tasks.workunit.client.1.smithi137.stdout:6/15: dread - f0 zero size 2024-03-14T13:02:08.811 INFO:tasks.workunit.client.1.smithi137.stdout:4/52: dread f3 [0,4194304] 0 2024-03-14T13:02:08.812 INFO:tasks.workunit.client.1.smithi137.stdout:3/55: mkdir d4/de 0 2024-03-14T13:02:08.814 INFO:tasks.workunit.client.0.smithi111.stdout:5/223: dread - d2/d16/f42 zero size 2024-03-14T13:02:08.815 INFO:tasks.workunit.client.1.smithi137.stdout:3/56: chown d4/la 10 1 2024-03-14T13:02:08.816 INFO:tasks.workunit.client.1.smithi137.stdout:4/53: creat d4/d9/fc x:0 0 0 2024-03-14T13:02:08.817 INFO:tasks.workunit.client.1.smithi137.stdout:4/54: chown d4 381688 1 2024-03-14T13:02:08.817 INFO:tasks.workunit.client.1.smithi137.stdout:9/66: dwrite d6/df/f13 [0,4194304] 0 2024-03-14T13:02:08.818 INFO:tasks.workunit.client.0.smithi111.stdout:2/106: dwrite dd/f10 [4194304,4194304] 0 2024-03-14T13:02:08.818 INFO:tasks.workunit.client.0.smithi111.stdout:9/131: link d5/d6/d14/l1f d5/d6/d14/l27 0 2024-03-14T13:02:08.819 INFO:tasks.workunit.client.1.smithi137.stdout:4/55: dread - d4/f8 zero size 2024-03-14T13:02:08.819 INFO:tasks.workunit.client.1.smithi137.stdout:4/56: read - d4/d9/fc zero size 2024-03-14T13:02:08.822 INFO:tasks.workunit.client.1.smithi137.stdout:3/57: mkdir d4/df 0 2024-03-14T13:02:08.822 INFO:tasks.workunit.client.0.smithi111.stdout:6/142: dwrite d0/d17/f2c [0,4194304] 0 2024-03-14T13:02:08.823 INFO:tasks.workunit.client.1.smithi137.stdout:6/16: symlink d2/l5 0 2024-03-14T13:02:08.826 INFO:tasks.workunit.client.1.smithi137.stdout:4/57: rename l1 to d4/ld 0 2024-03-14T13:02:08.826 INFO:tasks.workunit.client.0.smithi111.stdout:5/224: truncate d2/d45/f34 575204 0 2024-03-14T13:02:08.827 INFO:tasks.workunit.client.1.smithi137.stdout:3/58: write d4/f5 [1377597,23189] 0 2024-03-14T13:02:08.827 INFO:tasks.workunit.client.1.smithi137.stdout:6/17: mkdir d2/d6 0 2024-03-14T13:02:08.828 INFO:tasks.workunit.client.1.smithi137.stdout:6/18: dread - f0 zero size 2024-03-14T13:02:08.829 INFO:tasks.workunit.client.0.smithi111.stdout:6/143: chown d0/c10 467 1 2024-03-14T13:02:08.829 INFO:tasks.workunit.client.0.smithi111.stdout:7/132: write d0/d1/d21/f15 [1031349,39751] 0 2024-03-14T13:02:08.830 INFO:tasks.workunit.client.1.smithi137.stdout:4/58: chown d4/d9/ca 1 1 2024-03-14T13:02:08.830 INFO:tasks.workunit.client.1.smithi137.stdout:6/19: chown f0 86 1 2024-03-14T13:02:08.830 INFO:tasks.workunit.client.0.smithi111.stdout:2/107: creat dd/d11/d1a/f1c x:0 0 0 2024-03-14T13:02:08.831 INFO:tasks.workunit.client.0.smithi111.stdout:6/144: stat d0/ce 0 2024-03-14T13:02:08.833 INFO:tasks.workunit.client.1.smithi137.stdout:4/59: symlink d4/d9/le 0 2024-03-14T13:02:08.834 INFO:tasks.workunit.client.0.smithi111.stdout:9/132: mknod d5/c28 0 2024-03-14T13:02:08.834 INFO:tasks.workunit.client.0.smithi111.stdout:3/230: dwrite d0/d30/f31 [0,4194304] 0 2024-03-14T13:02:08.834 INFO:tasks.workunit.client.0.smithi111.stdout:6/145: chown d0/f6 0 1 2024-03-14T13:02:08.834 INFO:tasks.workunit.client.0.smithi111.stdout:4/430: dwrite d1d/d22/d30/d5e/f8d [0,4194304] 0 2024-03-14T13:02:08.834 INFO:tasks.workunit.client.0.smithi111.stdout:2/108: symlink dd/l1d 0 2024-03-14T13:02:08.837 INFO:tasks.workunit.client.0.smithi111.stdout:9/133: fdatasync d5/d6/f22 0 2024-03-14T13:02:08.837 INFO:tasks.workunit.client.0.smithi111.stdout:5/225: write d2/d13/f4b [794085,40178] 0 2024-03-14T13:02:08.843 INFO:tasks.workunit.client.1.smithi137.stdout:4/60: dwrite d4/d9/fc [0,4194304] 0 2024-03-14T13:02:08.844 INFO:tasks.workunit.client.0.smithi111.stdout:5/226: chown d2/c9 86644355 1 2024-03-14T13:02:08.844 INFO:tasks.workunit.client.1.smithi137.stdout:4/61: read f2 [2897761,38668] 0 2024-03-14T13:02:08.846 INFO:tasks.workunit.client.1.smithi137.stdout:4/62: dread f3 [0,4194304] 0 2024-03-14T13:02:08.847 INFO:tasks.workunit.client.0.smithi111.stdout:3/231: creat d0/d2d/d35/d50/f56 x:0 0 0 2024-03-14T13:02:08.847 INFO:tasks.workunit.client.0.smithi111.stdout:4/431: write d1d/d22/d30/f4d [450939,9203] 0 2024-03-14T13:02:08.849 INFO:tasks.workunit.client.0.smithi111.stdout:9/134: symlink d5/d6/d14/l29 0 2024-03-14T13:02:08.850 INFO:tasks.workunit.client.1.smithi137.stdout:4/63: creat d4/ff x:0 0 0 2024-03-14T13:02:08.850 INFO:tasks.workunit.client.1.smithi137.stdout:9/67: fdatasync d6/df/f15 0 2024-03-14T13:02:08.854 INFO:tasks.workunit.client.0.smithi111.stdout:9/135: write d5/d6/fc [1047735,120165] 0 2024-03-14T13:02:08.855 INFO:tasks.workunit.client.0.smithi111.stdout:5/227: mkdir d2/d16/d49/d4c 0 2024-03-14T13:02:08.856 INFO:tasks.workunit.client.0.smithi111.stdout:2/109: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:08.856 INFO:tasks.workunit.client.0.smithi111.stdout:6/146: dread d0/d17/d21/f22 [0,4194304] 0 2024-03-14T13:02:08.859 INFO:tasks.workunit.client.0.smithi111.stdout:7/133: dwrite d0/d1/fd [0,4194304] 0 2024-03-14T13:02:08.861 INFO:tasks.workunit.client.0.smithi111.stdout:2/110: write dd/d11/d1a/f1c [3437433,130196] 0 2024-03-14T13:02:08.862 INFO:tasks.workunit.client.0.smithi111.stdout:3/232: link d0/fd d0/d30/f57 0 2024-03-14T13:02:08.862 INFO:tasks.workunit.client.0.smithi111.stdout:5/228: symlink d2/d1f/d36/l4d 0 2024-03-14T13:02:08.864 INFO:tasks.workunit.client.0.smithi111.stdout:3/233: fsync d0/de/d19/f26 0 2024-03-14T13:02:08.866 INFO:tasks.workunit.client.1.smithi137.stdout:6/20: write f0 [77483,39382] 0 2024-03-14T13:02:08.867 INFO:tasks.workunit.client.0.smithi111.stdout:7/134: unlink d0/d1/fd 0 2024-03-14T13:02:08.868 INFO:tasks.workunit.client.0.smithi111.stdout:9/136: dread f2 [4194304,4194304] 0 2024-03-14T13:02:08.868 INFO:tasks.workunit.client.1.smithi137.stdout:6/21: chown d2/c3 7916 1 2024-03-14T13:02:08.872 INFO:tasks.workunit.client.1.smithi137.stdout:6/22: creat d2/f7 x:0 0 0 2024-03-14T13:02:08.874 INFO:tasks.workunit.client.0.smithi111.stdout:6/147: rmdir d0/d17/d21 39 2024-03-14T13:02:08.876 INFO:tasks.workunit.client.0.smithi111.stdout:5/229: symlink d2/d16/d2c/l4e 0 2024-03-14T13:02:08.879 INFO:tasks.workunit.client.0.smithi111.stdout:3/234: rename d0/de/d19/f28 to d0/de/d40/d55/f58 0 2024-03-14T13:02:08.881 INFO:tasks.workunit.client.0.smithi111.stdout:2/111: symlink dd/d17/d18/l1e 0 2024-03-14T13:02:08.883 INFO:tasks.workunit.client.0.smithi111.stdout:6/148: dread - d0/d17/d21/f28 zero size 2024-03-14T13:02:08.883 INFO:tasks.workunit.client.0.smithi111.stdout:5/230: creat d2/d13/f4f x:0 0 0 2024-03-14T13:02:08.884 INFO:tasks.workunit.client.0.smithi111.stdout:9/137: unlink d5/d6/d14/l27 0 2024-03-14T13:02:08.885 INFO:tasks.workunit.client.1.smithi137.stdout:6/23: dread f0 [0,4194304] 0 2024-03-14T13:02:08.886 INFO:tasks.workunit.client.0.smithi111.stdout:2/112: symlink dd/l1f 0 2024-03-14T13:02:08.886 INFO:tasks.workunit.client.1.smithi137.stdout:6/24: dread f0 [0,4194304] 0 2024-03-14T13:02:08.889 INFO:tasks.workunit.client.0.smithi111.stdout:5/231: write d2/d1f/d36/f3c [244452,42554] 0 2024-03-14T13:02:08.889 INFO:tasks.workunit.client.0.smithi111.stdout:2/113: stat dd/c12 0 2024-03-14T13:02:08.892 INFO:tasks.workunit.client.0.smithi111.stdout:4/432: dwrite d1d/d22/f37 [0,4194304] 0 2024-03-14T13:02:08.894 INFO:tasks.workunit.client.0.smithi111.stdout:3/235: dwrite d0/d30/f3d [0,4194304] 0 2024-03-14T13:02:08.895 INFO:tasks.workunit.client.0.smithi111.stdout:5/232: chown d2/d13/f15 84389 1 2024-03-14T13:02:08.896 INFO:tasks.workunit.client.0.smithi111.stdout:2/114: symlink dd/d17/d18/l20 0 2024-03-14T13:02:08.896 INFO:tasks.workunit.client.0.smithi111.stdout:5/233: chown d2 5 1 2024-03-14T13:02:08.897 INFO:tasks.workunit.client.1.smithi137.stdout:4/64: dwrite f3 [0,4194304] 0 2024-03-14T13:02:08.898 INFO:tasks.workunit.client.1.smithi137.stdout:9/68: dwrite d6/df/f14 [0,4194304] 0 2024-03-14T13:02:08.901 INFO:tasks.workunit.client.0.smithi111.stdout:4/433: rmdir d1d/d48/d92 0 2024-03-14T13:02:08.903 INFO:tasks.workunit.client.0.smithi111.stdout:2/115: chown dd/d17/d18/c19 257865890 1 2024-03-14T13:02:08.904 INFO:tasks.workunit.client.0.smithi111.stdout:3/236: write d0/f1c [90762,66564] 0 2024-03-14T13:02:08.905 INFO:tasks.workunit.client.0.smithi111.stdout:6/149: dwrite d0/d17/d21/f28 [0,4194304] 0 2024-03-14T13:02:08.906 INFO:tasks.workunit.client.0.smithi111.stdout:9/138: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:08.906 INFO:tasks.workunit.client.1.smithi137.stdout:4/65: dread - d4/ff zero size 2024-03-14T13:02:08.906 INFO:tasks.workunit.client.1.smithi137.stdout:4/66: chown d4 1914531 1 2024-03-14T13:02:08.907 INFO:tasks.workunit.client.1.smithi137.stdout:9/69: creat d6/df/f17 x:0 0 0 2024-03-14T13:02:08.907 INFO:tasks.workunit.client.1.smithi137.stdout:4/67: mkdir d4/d10 0 2024-03-14T13:02:08.908 INFO:tasks.workunit.client.0.smithi111.stdout:3/237: chown d0/d30/f4a 64 1 2024-03-14T13:02:08.908 INFO:tasks.workunit.client.1.smithi137.stdout:6/25: dwrite f0 [0,4194304] 0 2024-03-14T13:02:08.909 INFO:tasks.workunit.client.1.smithi137.stdout:9/70: write d6/df/f15 [2378470,109907] 0 2024-03-14T13:02:08.912 INFO:tasks.workunit.client.0.smithi111.stdout:3/238: readlink d0/de/d19/l3f 0 2024-03-14T13:02:08.912 INFO:tasks.workunit.client.0.smithi111.stdout:6/150: mknod d0/d17/d20/c2e 0 2024-03-14T13:02:08.913 INFO:tasks.workunit.client.1.smithi137.stdout:4/68: creat d4/f11 x:0 0 0 2024-03-14T13:02:08.915 INFO:tasks.workunit.client.0.smithi111.stdout:9/139: unlink d5/d6/d14/f1c 0 2024-03-14T13:02:08.915 INFO:tasks.workunit.client.1.smithi137.stdout:4/69: truncate d4/f8 861456 0 2024-03-14T13:02:08.915 INFO:tasks.workunit.client.1.smithi137.stdout:6/26: mkdir d2/d6/d8 0 2024-03-14T13:02:08.916 INFO:tasks.workunit.client.0.smithi111.stdout:4/434: symlink d1d/d22/d30/d5e/d6f/l9c 0 2024-03-14T13:02:08.916 INFO:tasks.workunit.client.1.smithi137.stdout:9/71: dread d6/df/f14 [0,4194304] 0 2024-03-14T13:02:08.916 INFO:tasks.workunit.client.1.smithi137.stdout:6/27: read - d2/f7 zero size 2024-03-14T13:02:08.917 INFO:tasks.workunit.client.1.smithi137.stdout:9/72: chown d6/cd 3578239 1 2024-03-14T13:02:08.917 INFO:tasks.workunit.client.1.smithi137.stdout:4/70: creat d4/f12 x:0 0 0 2024-03-14T13:02:08.920 INFO:tasks.workunit.client.1.smithi137.stdout:4/71: creat d4/f13 x:0 0 0 2024-03-14T13:02:08.921 INFO:tasks.workunit.client.0.smithi111.stdout:6/151: link d0/d17/d20/f26 d0/f2f 0 2024-03-14T13:02:08.922 INFO:tasks.workunit.client.0.smithi111.stdout:2/116: dwrite dd/fe [0,4194304] 0 2024-03-14T13:02:08.923 INFO:tasks.workunit.client.0.smithi111.stdout:9/140: mkdir d5/d6/d2a 0 2024-03-14T13:02:08.924 INFO:tasks.workunit.client.1.smithi137.stdout:9/73: rename d6/df/f15 to d6/f18 0 2024-03-14T13:02:08.924 INFO:tasks.workunit.client.0.smithi111.stdout:4/435: mknod d1d/d22/d30/d5e/d6f/d7e/d66/c9d 0 2024-03-14T13:02:08.925 INFO:tasks.workunit.client.0.smithi111.stdout:6/152: fdatasync d0/f2f 0 2024-03-14T13:02:08.925 INFO:tasks.workunit.client.0.smithi111.stdout:4/436: chown d1d/d22/d30/d5e/d6f/d7e 3288 1 2024-03-14T13:02:08.926 INFO:tasks.workunit.client.1.smithi137.stdout:9/74: chown d6/df/f14 61642 1 2024-03-14T13:02:08.927 INFO:tasks.workunit.client.0.smithi111.stdout:4/437: readlink d1d/d22/d28/l83 0 2024-03-14T13:02:08.928 INFO:tasks.workunit.client.0.smithi111.stdout:9/141: mknod d5/d6/c2b 0 2024-03-14T13:02:08.929 INFO:tasks.workunit.client.1.smithi137.stdout:9/75: mkdir d6/df/d19 0 2024-03-14T13:02:08.929 INFO:tasks.workunit.client.1.smithi137.stdout:6/28: dwrite f0 [0,4194304] 0 2024-03-14T13:02:08.930 INFO:tasks.workunit.client.0.smithi111.stdout:6/153: creat d0/d17/d20/f30 x:0 0 0 2024-03-14T13:02:08.931 INFO:tasks.workunit.client.0.smithi111.stdout:4/438: symlink d1d/d22/d28/l9e 0 2024-03-14T13:02:08.933 INFO:tasks.workunit.client.0.smithi111.stdout:6/154: chown d0/d17/d21/f2d 0 1 2024-03-14T13:02:08.933 INFO:tasks.workunit.client.0.smithi111.stdout:2/117: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:08.934 INFO:tasks.workunit.client.1.smithi137.stdout:6/29: rename d2 to d2/d6/d9 22 2024-03-14T13:02:08.936 INFO:tasks.workunit.client.0.smithi111.stdout:2/118: unlink f4 0 2024-03-14T13:02:08.939 INFO:tasks.workunit.client.1.smithi137.stdout:6/30: stat d2/l5 0 2024-03-14T13:02:08.940 INFO:tasks.workunit.client.0.smithi111.stdout:2/119: creat dd/d11/d1a/f21 x:0 0 0 2024-03-14T13:02:08.940 INFO:tasks.workunit.client.0.smithi111.stdout:4/439: creat d1d/d22/d40/d69/d86/f9f x:0 0 0 2024-03-14T13:02:08.941 INFO:tasks.workunit.client.0.smithi111.stdout:6/155: mknod d0/d17/c31 0 2024-03-14T13:02:08.941 INFO:tasks.workunit.client.0.smithi111.stdout:9/142: dwrite d5/fa [0,4194304] 0 2024-03-14T13:02:08.942 INFO:tasks.workunit.client.1.smithi137.stdout:9/76: rename d6/cd to d6/df/d19/c1a 0 2024-03-14T13:02:08.943 INFO:tasks.workunit.client.0.smithi111.stdout:2/120: mknod dd/c22 0 2024-03-14T13:02:08.947 INFO:tasks.workunit.client.0.smithi111.stdout:4/440: creat d1d/d22/d40/fa0 x:0 0 0 2024-03-14T13:02:08.948 INFO:tasks.workunit.client.0.smithi111.stdout:3/239: rename d0/de/d40/d55/f58 to d0/f59 0 2024-03-14T13:02:08.950 INFO:tasks.workunit.client.1.smithi137.stdout:4/72: truncate f3 1259976 0 2024-03-14T13:02:08.951 INFO:tasks.workunit.client.1.smithi137.stdout:4/73: write d4/ff [907497,101449] 0 2024-03-14T13:02:08.951 INFO:tasks.workunit.client.0.smithi111.stdout:2/121: symlink dd/d17/d18/l23 0 2024-03-14T13:02:08.952 INFO:tasks.workunit.client.0.smithi111.stdout:9/143: mknod d5/d6/d2a/c2c 0 2024-03-14T13:02:08.954 INFO:tasks.workunit.client.0.smithi111.stdout:4/441: unlink c1c 0 2024-03-14T13:02:08.954 INFO:tasks.workunit.client.0.smithi111.stdout:2/122: dread - dd/d11/d1a/f21 zero size 2024-03-14T13:02:08.954 INFO:tasks.workunit.client.1.smithi137.stdout:4/74: creat d4/d10/f14 x:0 0 0 2024-03-14T13:02:08.956 INFO:tasks.workunit.client.0.smithi111.stdout:9/144: mkdir d5/d6/d2a/d2d 0 2024-03-14T13:02:08.959 INFO:tasks.workunit.client.0.smithi111.stdout:3/240: dwrite d0/de/f12 [4194304,4194304] 0 2024-03-14T13:02:08.961 INFO:tasks.workunit.client.0.smithi111.stdout:4/442: unlink c7 0 2024-03-14T13:02:08.965 INFO:tasks.workunit.client.0.smithi111.stdout:9/145: mknod d5/d6/d2a/d2d/c2e 0 2024-03-14T13:02:08.967 INFO:tasks.workunit.client.0.smithi111.stdout:4/443: creat d1d/d22/d30/d7c/fa1 x:0 0 0 2024-03-14T13:02:08.968 INFO:tasks.workunit.client.0.smithi111.stdout:3/241: truncate d0/de/d40/f47 307514 0 2024-03-14T13:02:08.969 INFO:tasks.workunit.client.0.smithi111.stdout:4/444: readlink d1d/d22/d30/d5e/d6f/d7e/d66/l6b 0 2024-03-14T13:02:08.976 INFO:tasks.workunit.client.0.smithi111.stdout:6/156: rmdir d0/d17/d20 39 2024-03-14T13:02:08.979 INFO:tasks.workunit.client.0.smithi111.stdout:6/157: creat d0/d17/d20/f32 x:0 0 0 2024-03-14T13:02:08.980 INFO:tasks.workunit.client.0.smithi111.stdout:6/158: rmdir d0/d17/d24 39 2024-03-14T13:02:08.982 INFO:tasks.workunit.client.0.smithi111.stdout:6/159: mknod d0/d17/d21/c33 0 2024-03-14T13:02:08.982 INFO:tasks.workunit.client.0.smithi111.stdout:6/160: stat d0/d17/d20/c2e 0 2024-03-14T13:02:08.984 INFO:tasks.workunit.client.0.smithi111.stdout:6/161: write d0/d17/f2c [4730299,51342] 0 2024-03-14T13:02:08.985 INFO:tasks.workunit.client.1.smithi137.stdout:6/31: truncate f0 1330505 0 2024-03-14T13:02:08.986 INFO:tasks.workunit.client.1.smithi137.stdout:9/77: truncate d6/fa 2486800 0 2024-03-14T13:02:08.987 INFO:tasks.workunit.client.0.smithi111.stdout:6/162: write d0/d17/d20/f32 [563617,17397] 0 2024-03-14T13:02:08.987 INFO:tasks.workunit.client.1.smithi137.stdout:9/78: symlink d6/df/l1b 0 2024-03-14T13:02:08.988 INFO:tasks.workunit.client.0.smithi111.stdout:2/123: rename dd/d17/d18 to dd/d24 0 2024-03-14T13:02:08.989 INFO:tasks.workunit.client.1.smithi137.stdout:9/79: creat d6/f1c x:0 0 0 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/3: sync 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/4: fsync - no filename 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/5: stat c0 0 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/6: read - no filename 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/7: dwrite - no filename 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/8: fdatasync - no filename 2024-03-14T13:02:08.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/9: dwrite - no filename 2024-03-14T13:02:08.991 INFO:tasks.workunit.client.0.smithi111.stdout:6/163: mknod d0/d17/d21/c34 0 2024-03-14T13:02:08.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/10: chown c0 825159 1 2024-03-14T13:02:08.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/11: dwrite - no filename 2024-03-14T13:02:08.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/12: fsync - no filename 2024-03-14T13:02:08.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/13: truncate - no filename 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:2/14: dread - no filename 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:5/24: sync 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:9/80: creat d6/f1d x:0 0 0 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:5/25: dread - f6 zero size 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:2/15: symlink l1 0 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:2/16: read - no filename 2024-03-14T13:02:08.992 INFO:tasks.workunit.client.1.smithi137.stdout:2/17: dread - no filename 2024-03-14T13:02:08.993 INFO:tasks.workunit.client.0.smithi111.stdout:2/124: creat dd/d11/f25 x:0 0 0 2024-03-14T13:02:08.993 INFO:tasks.workunit.client.1.smithi137.stdout:5/26: dread - f5 zero size 2024-03-14T13:02:08.994 INFO:tasks.workunit.client.0.smithi111.stdout:6/164: dread d0/ff [0,4194304] 0 2024-03-14T13:02:08.994 INFO:tasks.workunit.client.1.smithi137.stdout:9/81: mknod d6/df/d19/c1e 0 2024-03-14T13:02:08.996 INFO:tasks.workunit.client.1.smithi137.stdout:2/18: mkdir d2 0 2024-03-14T13:02:08.996 INFO:tasks.workunit.client.0.smithi111.stdout:6/165: chown d0/f6 2888524 1 2024-03-14T13:02:08.996 INFO:tasks.workunit.client.0.smithi111.stdout:9/146: rename d5/f10 to d5/d6/f2f 0 2024-03-14T13:02:08.996 INFO:tasks.workunit.client.1.smithi137.stdout:2/19: chown d2 15250 1 2024-03-14T13:02:08.997 INFO:tasks.workunit.client.1.smithi137.stdout:2/20: dread - no filename 2024-03-14T13:02:08.997 INFO:tasks.workunit.client.0.smithi111.stdout:6/166: fdatasync d0/d17/d21/f2d 0 2024-03-14T13:02:08.998 INFO:tasks.workunit.client.1.smithi137.stdout:5/27: fsync f5 0 2024-03-14T13:02:08.999 INFO:tasks.workunit.client.1.smithi137.stdout:9/82: write d6/df/f10 [662790,98813] 0 2024-03-14T13:02:08.999 INFO:tasks.workunit.client.0.smithi111.stdout:4/445: getdents d1d/d22/d30/d7c 0 2024-03-14T13:02:09.000 INFO:tasks.workunit.client.1.smithi137.stdout:5/28: read - f6 zero size 2024-03-14T13:02:09.000 INFO:tasks.workunit.client.0.smithi111.stdout:2/125: dread dd/fe [0,4194304] 0 2024-03-14T13:02:09.001 INFO:tasks.workunit.client.0.smithi111.stdout:4/446: fsync d1d/d22/d40/f63 0 2024-03-14T13:02:09.001 INFO:tasks.workunit.client.1.smithi137.stdout:2/21: symlink d2/l3 0 2024-03-14T13:02:09.001 INFO:tasks.workunit.client.1.smithi137.stdout:2/22: write - no filename 2024-03-14T13:02:09.002 INFO:tasks.workunit.client.0.smithi111.stdout:3/242: rename d0/f59 to d0/de/d19/f5a 0 2024-03-14T13:02:09.002 INFO:tasks.workunit.client.0.smithi111.stdout:6/167: chown d0/d17/d21 88096531 1 2024-03-14T13:02:09.003 INFO:tasks.workunit.client.1.smithi137.stdout:6/32: dwrite f0 [0,4194304] 0 2024-03-14T13:02:09.003 INFO:tasks.workunit.client.1.smithi137.stdout:5/29: truncate f4 735823 0 2024-03-14T13:02:09.004 INFO:tasks.workunit.client.0.smithi111.stdout:3/243: stat d0/d30/f38 0 2024-03-14T13:02:09.004 INFO:tasks.workunit.client.0.smithi111.stdout:9/147: creat d5/d6/d8/f30 x:0 0 0 2024-03-14T13:02:09.005 INFO:tasks.workunit.client.0.smithi111.stdout:9/148: stat d5/d6/f9 0 2024-03-14T13:02:09.005 INFO:tasks.workunit.client.1.smithi137.stdout:2/23: creat d2/f4 x:0 0 0 2024-03-14T13:02:09.005 INFO:tasks.workunit.client.1.smithi137.stdout:2/24: fdatasync d2/f4 0 2024-03-14T13:02:09.006 INFO:tasks.workunit.client.0.smithi111.stdout:2/126: symlink dd/d11/l26 0 2024-03-14T13:02:09.006 INFO:tasks.workunit.client.0.smithi111.stdout:9/149: chown c1 897514 1 2024-03-14T13:02:09.006 INFO:tasks.workunit.client.1.smithi137.stdout:5/30: mkdir d7 0 2024-03-14T13:02:09.007 INFO:tasks.workunit.client.0.smithi111.stdout:6/168: creat d0/f35 x:0 0 0 2024-03-14T13:02:09.010 INFO:tasks.workunit.client.1.smithi137.stdout:2/25: creat d2/f5 x:0 0 0 2024-03-14T13:02:09.010 INFO:tasks.workunit.client.1.smithi137.stdout:2/26: stat d2 0 2024-03-14T13:02:09.010 INFO:tasks.workunit.client.0.smithi111.stdout:2/127: mknod dd/d11/d1a/c27 0 2024-03-14T13:02:09.010 INFO:tasks.workunit.client.1.smithi137.stdout:2/27: chown d2/f5 1479 1 2024-03-14T13:02:09.011 INFO:tasks.workunit.client.1.smithi137.stdout:2/28: chown d2/f4 224 1 2024-03-14T13:02:09.011 INFO:tasks.workunit.client.1.smithi137.stdout:5/31: rename f6 to d7/f8 0 2024-03-14T13:02:09.012 INFO:tasks.workunit.client.0.smithi111.stdout:4/447: dwrite d1d/d22/d30/d5e/d6f/d7e/f65 [0,4194304] 0 2024-03-14T13:02:09.012 INFO:tasks.workunit.client.1.smithi137.stdout:5/32: dread - d7/f8 zero size 2024-03-14T13:02:09.014 INFO:tasks.workunit.client.0.smithi111.stdout:3/244: symlink d0/d2d/d35/d4c/l5b 0 2024-03-14T13:02:09.016 INFO:tasks.workunit.client.0.smithi111.stdout:2/128: write dd/d11/d1a/f1b [311645,80378] 0 2024-03-14T13:02:09.017 INFO:tasks.workunit.client.1.smithi137.stdout:6/33: getdents d2/d6 0 2024-03-14T13:02:09.018 INFO:tasks.workunit.client.0.smithi111.stdout:9/150: creat d5/f31 x:0 0 0 2024-03-14T13:02:09.019 INFO:tasks.workunit.client.1.smithi137.stdout:6/34: mknod d2/d6/d8/ca 0 2024-03-14T13:02:09.020 INFO:tasks.workunit.client.0.smithi111.stdout:4/448: creat d1d/d22/d23/fa2 x:0 0 0 2024-03-14T13:02:09.020 INFO:tasks.workunit.client.1.smithi137.stdout:6/35: write f0 [2953962,37390] 0 2024-03-14T13:02:09.021 INFO:tasks.workunit.client.0.smithi111.stdout:6/169: dwrite d0/d17/d21/f28 [0,4194304] 0 2024-03-14T13:02:09.021 INFO:tasks.workunit.client.1.smithi137.stdout:6/36: write d2/f7 [851301,64112] 0 2024-03-14T13:02:09.022 INFO:tasks.workunit.client.0.smithi111.stdout:2/129: chown la 32 1 2024-03-14T13:02:09.024 INFO:tasks.workunit.client.1.smithi137.stdout:6/37: mknod d2/d6/cb 0 2024-03-14T13:02:09.024 INFO:tasks.workunit.client.1.smithi137.stdout:2/29: fsync d2/f4 0 2024-03-14T13:02:09.025 INFO:tasks.workunit.client.0.smithi111.stdout:2/130: fdatasync dd/d11/d1a/f1c 0 2024-03-14T13:02:09.025 INFO:tasks.workunit.client.1.smithi137.stdout:6/38: chown d2/d6 1 1 2024-03-14T13:02:09.027 INFO:tasks.workunit.client.1.smithi137.stdout:6/39: rmdir d2/d6 39 2024-03-14T13:02:09.027 INFO:tasks.workunit.client.0.smithi111.stdout:6/170: mknod d0/d17/d20/c36 0 2024-03-14T13:02:09.030 INFO:tasks.workunit.client.1.smithi137.stdout:9/83: truncate d6/fa 1987232 0 2024-03-14T13:02:09.031 INFO:tasks.workunit.client.0.smithi111.stdout:9/151: dread d5/d6/ff [0,4194304] 0 2024-03-14T13:02:09.031 INFO:tasks.workunit.client.0.smithi111.stdout:3/245: link d0/d30/l4f d0/de/d19/l5c 0 2024-03-14T13:02:09.033 INFO:tasks.workunit.client.0.smithi111.stdout:2/131: mknod dd/d24/c28 0 2024-03-14T13:02:09.033 INFO:tasks.workunit.client.0.smithi111.stdout:3/246: chown d0/d30/f57 912 1 2024-03-14T13:02:09.034 INFO:tasks.workunit.client.0.smithi111.stdout:3/247: dread - d0/de/d19/f44 zero size 2024-03-14T13:02:09.034 INFO:tasks.workunit.client.0.smithi111.stdout:6/171: mkdir d0/d17/d21/d37 0 2024-03-14T13:02:09.035 INFO:tasks.workunit.client.1.smithi137.stdout:9/84: mkdir d6/df/d19/d1f 0 2024-03-14T13:02:09.035 INFO:tasks.workunit.client.1.smithi137.stdout:2/30: dwrite d2/f5 [0,4194304] 0 2024-03-14T13:02:09.035 INFO:tasks.workunit.client.0.smithi111.stdout:6/172: fdatasync d0/f14 0 2024-03-14T13:02:09.035 INFO:tasks.workunit.client.0.smithi111.stdout:3/248: read d0/d30/f31 [153896,50153] 0 2024-03-14T13:02:09.036 INFO:tasks.workunit.client.0.smithi111.stdout:9/152: creat d5/d6/d2a/d2d/f32 x:0 0 0 2024-03-14T13:02:09.036 INFO:tasks.workunit.client.0.smithi111.stdout:2/132: symlink dd/l29 0 2024-03-14T13:02:09.036 INFO:tasks.workunit.client.0.smithi111.stdout:9/153: chown d5/d6/d8 3344 1 2024-03-14T13:02:09.038 INFO:tasks.workunit.client.0.smithi111.stdout:4/449: rename d1d/d22/d30/d5e/d6f/d7e/c82 to d1d/d22/d30/d5e/ca3 0 2024-03-14T13:02:09.039 INFO:tasks.workunit.client.0.smithi111.stdout:4/450: write d1d/f36 [226220,90556] 0 2024-03-14T13:02:09.040 INFO:tasks.workunit.client.1.smithi137.stdout:2/31: symlink d2/l6 0 2024-03-14T13:02:09.042 INFO:tasks.workunit.client.1.smithi137.stdout:6/40: dwrite f0 [0,4194304] 0 2024-03-14T13:02:09.045 INFO:tasks.workunit.client.1.smithi137.stdout:9/85: link f1 d6/df/f20 0 2024-03-14T13:02:09.045 INFO:tasks.workunit.client.1.smithi137.stdout:5/33: dwrite f4 [0,4194304] 0 2024-03-14T13:02:09.046 INFO:tasks.workunit.client.0.smithi111.stdout:2/133: getdents dd/d11/d1a 0 2024-03-14T13:02:09.046 INFO:tasks.workunit.client.0.smithi111.stdout:9/154: dwrite d5/d6/f22 [4194304,4194304] 0 2024-03-14T13:02:09.049 INFO:tasks.workunit.client.0.smithi111.stdout:2/134: write dd/d11/d1a/f1c [2233846,78513] 0 2024-03-14T13:02:09.051 INFO:tasks.workunit.client.1.smithi137.stdout:6/41: creat d2/d6/fc x:0 0 0 2024-03-14T13:02:09.053 INFO:tasks.workunit.client.1.smithi137.stdout:2/32: dread d2/f5 [0,4194304] 0 2024-03-14T13:02:09.053 INFO:tasks.workunit.client.1.smithi137.stdout:9/86: symlink d6/df/d19/d1f/l21 0 2024-03-14T13:02:09.054 INFO:tasks.workunit.client.1.smithi137.stdout:2/33: chown d2/f5 144569902 1 2024-03-14T13:02:09.055 INFO:tasks.workunit.client.1.smithi137.stdout:6/42: mknod d2/cd 0 2024-03-14T13:02:09.056 INFO:tasks.workunit.client.0.smithi111.stdout:2/135: mknod dd/c2a 0 2024-03-14T13:02:09.059 INFO:tasks.workunit.client.1.smithi137.stdout:6/43: mkdir d2/d6/de 0 2024-03-14T13:02:09.060 INFO:tasks.workunit.client.0.smithi111.stdout:2/136: read dd/f10 [4106736,90136] 0 2024-03-14T13:02:09.060 INFO:tasks.workunit.client.0.smithi111.stdout:2/137: chown dd/f10 20939633 1 2024-03-14T13:02:09.061 INFO:tasks.workunit.client.1.smithi137.stdout:6/44: fdatasync d2/f7 0 2024-03-14T13:02:09.061 INFO:tasks.workunit.client.0.smithi111.stdout:2/138: dread - dd/d11/f25 zero size 2024-03-14T13:02:09.062 INFO:tasks.workunit.client.0.smithi111.stdout:2/139: write dd/d11/d1a/f1c [2529877,19078] 0 2024-03-14T13:02:09.063 INFO:tasks.workunit.client.0.smithi111.stdout:2/140: chown c0 56639566 1 2024-03-14T13:02:09.065 INFO:tasks.workunit.client.1.smithi137.stdout:2/34: dwrite d2/f4 [0,4194304] 0 2024-03-14T13:02:09.065 INFO:tasks.workunit.client.1.smithi137.stdout:6/45: symlink d2/d6/de/lf 0 2024-03-14T13:02:09.066 INFO:tasks.workunit.client.1.smithi137.stdout:7/49: sync 2024-03-14T13:02:09.067 INFO:tasks.workunit.client.1.smithi137.stdout:1/78: sync 2024-03-14T13:02:09.067 INFO:tasks.workunit.client.1.smithi137.stdout:8/23: sync 2024-03-14T13:02:09.067 INFO:tasks.workunit.client.1.smithi137.stdout:3/59: sync 2024-03-14T13:02:09.067 INFO:tasks.workunit.client.1.smithi137.stdout:0/23: sync 2024-03-14T13:02:09.068 INFO:tasks.workunit.client.1.smithi137.stdout:2/35: write d2/f4 [973682,105614] 0 2024-03-14T13:02:09.068 INFO:tasks.workunit.client.1.smithi137.stdout:1/79: chown c1 260869 1 2024-03-14T13:02:09.068 INFO:tasks.workunit.client.1.smithi137.stdout:7/50: chown d3/d6/l7 366333 1 2024-03-14T13:02:09.069 INFO:tasks.workunit.client.1.smithi137.stdout:6/46: mknod d2/c10 0 2024-03-14T13:02:09.072 INFO:tasks.workunit.client.1.smithi137.stdout:8/24: unlink f0 0 2024-03-14T13:02:09.072 INFO:tasks.workunit.client.1.smithi137.stdout:3/60: write d4/fb [777414,117725] 0 2024-03-14T13:02:09.076 INFO:tasks.workunit.client.1.smithi137.stdout:2/36: chown d2/l6 2 1 2024-03-14T13:02:09.076 INFO:tasks.workunit.client.1.smithi137.stdout:8/25: read d1/f2 [563427,60917] 0 2024-03-14T13:02:09.076 INFO:tasks.workunit.client.1.smithi137.stdout:3/61: readlink d4/la 0 2024-03-14T13:02:09.076 INFO:tasks.workunit.client.1.smithi137.stdout:1/80: fdatasync d7/d9/ff 0 2024-03-14T13:02:09.077 INFO:tasks.workunit.client.1.smithi137.stdout:3/62: dread - d4/f9 zero size 2024-03-14T13:02:09.077 INFO:tasks.workunit.client.1.smithi137.stdout:7/51: rmdir d3/d5 39 2024-03-14T13:02:09.077 INFO:tasks.workunit.client.1.smithi137.stdout:5/34: truncate f4 3660355 0 2024-03-14T13:02:09.079 INFO:tasks.workunit.client.1.smithi137.stdout:0/24: symlink d1/d5/l7 0 2024-03-14T13:02:09.081 INFO:tasks.workunit.client.1.smithi137.stdout:3/63: dread f1 [0,4194304] 0 2024-03-14T13:02:09.082 INFO:tasks.workunit.client.1.smithi137.stdout:9/87: truncate d6/f18 3812065 0 2024-03-14T13:02:09.084 INFO:tasks.workunit.client.1.smithi137.stdout:0/25: rename d1/d5/f6 to d1/f8 0 2024-03-14T13:02:09.084 INFO:tasks.workunit.client.1.smithi137.stdout:3/64: rename d4 to d4/df/d10 22 2024-03-14T13:02:09.085 INFO:tasks.workunit.client.1.smithi137.stdout:3/65: write f1 [3451234,62472] 0 2024-03-14T13:02:09.087 INFO:tasks.workunit.client.1.smithi137.stdout:9/88: symlink d6/df/d19/l22 0 2024-03-14T13:02:09.088 INFO:tasks.workunit.client.1.smithi137.stdout:7/52: creat d3/ff x:0 0 0 2024-03-14T13:02:09.089 INFO:tasks.workunit.client.1.smithi137.stdout:0/26: mkdir d1/d5/d9 0 2024-03-14T13:02:09.092 INFO:tasks.workunit.client.1.smithi137.stdout:0/27: write d1/f8 [3327623,23883] 0 2024-03-14T13:02:09.094 INFO:tasks.workunit.client.1.smithi137.stdout:3/66: mknod d4/df/c11 0 2024-03-14T13:02:09.094 INFO:tasks.workunit.client.1.smithi137.stdout:6/47: rmdir d2 39 2024-03-14T13:02:09.097 INFO:tasks.workunit.client.1.smithi137.stdout:5/35: write f4 [4154667,36576] 0 2024-03-14T13:02:09.098 INFO:tasks.workunit.client.0.smithi111.stdout:9/155: mknod d5/c33 0 2024-03-14T13:02:09.099 INFO:tasks.workunit.client.1.smithi137.stdout:8/26: write d1/f2 [1032331,15945] 0 2024-03-14T13:02:09.102 INFO:tasks.workunit.client.0.smithi111.stdout:9/156: write f4 [8790065,99475] 0 2024-03-14T13:02:09.105 INFO:tasks.workunit.client.1.smithi137.stdout:1/81: truncate f5 4385282 0 2024-03-14T13:02:09.106 INFO:tasks.workunit.client.0.smithi111.stdout:9/157: mknod d5/d6/d2a/d2d/c34 0 2024-03-14T13:02:09.106 INFO:tasks.workunit.client.1.smithi137.stdout:5/36: symlink d7/l9 0 2024-03-14T13:02:09.106 INFO:tasks.workunit.client.1.smithi137.stdout:9/89: dread d6/f18 [0,4194304] 0 2024-03-14T13:02:09.107 INFO:tasks.workunit.client.0.smithi111.stdout:9/158: chown d5/d6/d2a/d2d/f32 662898710 1 2024-03-14T13:02:09.107 INFO:tasks.workunit.client.1.smithi137.stdout:5/37: truncate d7/f8 273524 0 2024-03-14T13:02:09.107 INFO:tasks.workunit.client.0.smithi111.stdout:9/159: stat d5/f31 0 2024-03-14T13:02:09.109 INFO:tasks.workunit.client.1.smithi137.stdout:3/67: symlink d4/de/l12 0 2024-03-14T13:02:09.111 INFO:tasks.workunit.client.0.smithi111.stdout:9/160: rename d5/d6/d2a/d2d/c34 to d5/d6/d2a/c35 0 2024-03-14T13:02:09.112 INFO:tasks.workunit.client.1.smithi137.stdout:8/27: rename d1/l4 to d1/l5 0 2024-03-14T13:02:09.112 INFO:tasks.workunit.client.1.smithi137.stdout:8/28: chown d1 248 1 2024-03-14T13:02:09.112 INFO:tasks.workunit.client.1.smithi137.stdout:0/28: truncate d1/f8 2151899 0 2024-03-14T13:02:09.113 INFO:tasks.workunit.client.0.smithi111.stdout:9/161: chown d5/d6/d2a/d2d/f32 211987739 1 2024-03-14T13:02:09.113 INFO:tasks.workunit.client.1.smithi137.stdout:1/82: rmdir d7 39 2024-03-14T13:02:09.115 INFO:tasks.workunit.client.0.smithi111.stdout:9/162: write d5/d6/fc [1368037,26276] 0 2024-03-14T13:02:09.116 INFO:tasks.workunit.client.1.smithi137.stdout:9/90: symlink d6/df/d19/l23 0 2024-03-14T13:02:09.117 INFO:tasks.workunit.client.0.smithi111.stdout:9/163: chown d5/d6/d14/l29 1 1 2024-03-14T13:02:09.117 INFO:tasks.workunit.client.0.smithi111.stdout:3/249: dwrite d0/f8 [0,4194304] 0 2024-03-14T13:02:09.118 INFO:tasks.workunit.client.1.smithi137.stdout:9/91: read d6/df/f10 [50244,16313] 0 2024-03-14T13:02:09.118 INFO:tasks.workunit.client.1.smithi137.stdout:5/38: creat d7/fa x:0 0 0 2024-03-14T13:02:09.118 INFO:tasks.workunit.client.1.smithi137.stdout:9/92: read - d6/df/f17 zero size 2024-03-14T13:02:09.119 INFO:tasks.workunit.client.0.smithi111.stdout:9/164: rmdir d5/d6/d14 39 2024-03-14T13:02:09.120 INFO:tasks.workunit.client.1.smithi137.stdout:3/68: mkdir d4/de/d13 0 2024-03-14T13:02:09.121 INFO:tasks.workunit.client.0.smithi111.stdout:3/250: rmdir d0/d2d/d35/d4c 39 2024-03-14T13:02:09.121 INFO:tasks.workunit.client.0.smithi111.stdout:3/251: chown d0/de/d19/f1a 11 1 2024-03-14T13:02:09.122 INFO:tasks.workunit.client.1.smithi137.stdout:7/53: getdents d3/d5 0 2024-03-14T13:02:09.123 INFO:tasks.workunit.client.0.smithi111.stdout:9/165: write d5/d6/d8/f30 [562485,108080] 0 2024-03-14T13:02:09.123 INFO:tasks.workunit.client.0.smithi111.stdout:3/252: readlink d0/d30/l53 0 2024-03-14T13:02:09.124 INFO:tasks.workunit.client.1.smithi137.stdout:8/29: rmdir d1 39 2024-03-14T13:02:09.125 INFO:tasks.workunit.client.1.smithi137.stdout:1/83: stat d7/cc 0 2024-03-14T13:02:09.126 INFO:tasks.workunit.client.1.smithi137.stdout:6/48: creat d2/f11 x:0 0 0 2024-03-14T13:02:09.127 INFO:tasks.workunit.client.0.smithi111.stdout:3/253: symlink d0/l5d 0 2024-03-14T13:02:09.128 INFO:tasks.workunit.client.1.smithi137.stdout:5/39: creat d7/fb x:0 0 0 2024-03-14T13:02:09.128 INFO:tasks.workunit.client.0.smithi111.stdout:3/254: stat d0/fc 0 2024-03-14T13:02:09.128 INFO:tasks.workunit.client.0.smithi111.stdout:3/255: chown d0/d2d/d35/d50/f56 6300 1 2024-03-14T13:02:09.128 INFO:tasks.workunit.client.1.smithi137.stdout:6/49: readlink d2/d6/de/lf 0 2024-03-14T13:02:09.129 INFO:tasks.workunit.client.1.smithi137.stdout:3/69: dwrite d4/fb [4194304,4194304] 0 2024-03-14T13:02:09.131 INFO:tasks.workunit.client.1.smithi137.stdout:3/70: write d4/fb [7256874,24458] 0 2024-03-14T13:02:09.131 INFO:tasks.workunit.client.0.smithi111.stdout:3/256: rename d0/de to d0/d2d/d35/d50/d5e 0 2024-03-14T13:02:09.131 INFO:tasks.workunit.client.0.smithi111.stdout:3/257: chown d0/d2d/d35/d50/f56 1337444 1 2024-03-14T13:02:09.132 INFO:tasks.workunit.client.1.smithi137.stdout:7/54: symlink d3/d6/l10 0 2024-03-14T13:02:09.135 INFO:tasks.workunit.client.1.smithi137.stdout:1/84: creat d7/d9/f14 x:0 0 0 2024-03-14T13:02:09.137 INFO:tasks.workunit.client.1.smithi137.stdout:5/40: mkdir d7/dc 0 2024-03-14T13:02:09.137 INFO:tasks.workunit.client.1.smithi137.stdout:1/85: write d7/d9/f14 [475916,87768] 0 2024-03-14T13:02:09.137 INFO:tasks.workunit.client.0.smithi111.stdout:9/166: truncate f4 3716067 0 2024-03-14T13:02:09.138 INFO:tasks.workunit.client.1.smithi137.stdout:6/50: mknod d2/d6/c12 0 2024-03-14T13:02:09.138 INFO:tasks.workunit.client.1.smithi137.stdout:6/51: stat d2/d6/d8/ca 0 2024-03-14T13:02:09.139 INFO:tasks.workunit.client.1.smithi137.stdout:8/30: creat d1/f6 x:0 0 0 2024-03-14T13:02:09.141 INFO:tasks.workunit.client.1.smithi137.stdout:3/71: rename d4/f9 to d4/df/f14 0 2024-03-14T13:02:09.142 INFO:tasks.workunit.client.1.smithi137.stdout:3/72: read - d4/df/f14 zero size 2024-03-14T13:02:09.143 INFO:tasks.workunit.client.1.smithi137.stdout:5/41: symlink d7/ld 0 2024-03-14T13:02:09.144 INFO:tasks.workunit.client.0.smithi111.stdout:9/167: write d5/d6/d8/f12 [552559,23792] 0 2024-03-14T13:02:09.147 INFO:tasks.workunit.client.0.smithi111.stdout:9/168: readlink d5/d6/d14/l29 0 2024-03-14T13:02:09.147 INFO:tasks.workunit.client.0.smithi111.stdout:9/169: chown d5/d6/ff 46090 1 2024-03-14T13:02:09.148 INFO:tasks.workunit.client.1.smithi137.stdout:6/52: creat d2/d6/de/f13 x:0 0 0 2024-03-14T13:02:09.149 INFO:tasks.workunit.client.0.smithi111.stdout:9/170: mknod d5/d6/c36 0 2024-03-14T13:02:09.149 INFO:tasks.workunit.client.1.smithi137.stdout:7/55: getdents d3/d5 0 2024-03-14T13:02:09.150 INFO:tasks.workunit.client.0.smithi111.stdout:9/171: symlink d5/d6/l37 0 2024-03-14T13:02:09.151 INFO:tasks.workunit.client.1.smithi137.stdout:5/42: creat d7/fe x:0 0 0 2024-03-14T13:02:09.151 INFO:tasks.workunit.client.1.smithi137.stdout:6/53: chown d2/f7 7397 1 2024-03-14T13:02:09.151 INFO:tasks.workunit.client.1.smithi137.stdout:7/56: write f1 [553312,67882] 0 2024-03-14T13:02:09.152 INFO:tasks.workunit.client.1.smithi137.stdout:8/31: dwrite d1/f6 [0,4194304] 0 2024-03-14T13:02:09.152 INFO:tasks.workunit.client.0.smithi111.stdout:9/172: truncate d5/d6/fc 1554090 0 2024-03-14T13:02:09.152 INFO:tasks.workunit.client.0.smithi111.stdout:9/173: truncate d5/d6/f9 1755641 0 2024-03-14T13:02:09.153 INFO:tasks.workunit.client.1.smithi137.stdout:3/73: mknod d4/de/d13/c15 0 2024-03-14T13:02:09.154 INFO:tasks.workunit.client.1.smithi137.stdout:1/86: rmdir d7/d9 39 2024-03-14T13:02:09.156 INFO:tasks.workunit.client.1.smithi137.stdout:9/93: truncate d6/df/f13 1658371 0 2024-03-14T13:02:09.159 INFO:tasks.workunit.client.1.smithi137.stdout:9/94: readlink d6/df/d19/l23 0 2024-03-14T13:02:09.159 INFO:tasks.workunit.client.1.smithi137.stdout:1/87: dread f3 [0,4194304] 0 2024-03-14T13:02:09.159 INFO:tasks.workunit.client.1.smithi137.stdout:3/74: mkdir d4/de/d16 0 2024-03-14T13:02:09.160 INFO:tasks.workunit.client.0.smithi111.stdout:9/174: dread f2 [0,4194304] 0 2024-03-14T13:02:09.160 INFO:tasks.workunit.client.1.smithi137.stdout:3/75: write d4/fb [8315924,11529] 0 2024-03-14T13:02:09.163 INFO:tasks.workunit.client.1.smithi137.stdout:7/57: rename d3/f4 to d3/d5/f11 0 2024-03-14T13:02:09.163 INFO:tasks.workunit.client.0.smithi111.stdout:1/183: sync 2024-03-14T13:02:09.163 INFO:tasks.workunit.client.0.smithi111.stdout:8/324: sync 2024-03-14T13:02:09.163 INFO:tasks.workunit.client.0.smithi111.stdout:0/144: sync 2024-03-14T13:02:09.164 INFO:tasks.workunit.client.0.smithi111.stdout:8/325: chown d3/d5/c2f 168 1 2024-03-14T13:02:09.164 INFO:tasks.workunit.client.1.smithi137.stdout:8/32: rmdir d1 39 2024-03-14T13:02:09.166 INFO:tasks.workunit.client.0.smithi111.stdout:1/184: creat d20/d25/d31/f45 x:0 0 0 2024-03-14T13:02:09.166 INFO:tasks.workunit.client.1.smithi137.stdout:4/75: sync 2024-03-14T13:02:09.166 INFO:tasks.workunit.client.1.smithi137.stdout:0/29: chown d1/f8 868 1 2024-03-14T13:02:09.167 INFO:tasks.workunit.client.0.smithi111.stdout:9/175: dread d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:09.169 INFO:tasks.workunit.client.0.smithi111.stdout:1/185: creat d20/d25/f46 x:0 0 0 2024-03-14T13:02:09.171 INFO:tasks.workunit.client.0.smithi111.stdout:9/176: mknod d5/d6/d8/c38 0 2024-03-14T13:02:09.171 INFO:tasks.workunit.client.1.smithi137.stdout:3/76: rename d4/f5 to d4/de/d16/f17 0 2024-03-14T13:02:09.172 INFO:tasks.workunit.client.0.smithi111.stdout:1/186: fdatasync d20/d25/f38 0 2024-03-14T13:02:09.172 INFO:tasks.workunit.client.1.smithi137.stdout:7/58: write d3/fe [288211,17496] 0 2024-03-14T13:02:09.175 INFO:tasks.workunit.client.0.smithi111.stdout:0/145: dwrite d6/d9/d15/f1b [0,4194304] 0 2024-03-14T13:02:09.175 INFO:tasks.workunit.client.0.smithi111.stdout:8/326: dwrite d3/f6b [0,4194304] 0 2024-03-14T13:02:09.176 INFO:tasks.workunit.client.1.smithi137.stdout:4/76: symlink d4/d10/l15 0 2024-03-14T13:02:09.176 INFO:tasks.workunit.client.0.smithi111.stdout:9/177: write d5/d6/ff [97467,58418] 0 2024-03-14T13:02:09.176 INFO:tasks.workunit.client.1.smithi137.stdout:7/59: dread d3/d5/fd [0,4194304] 0 2024-03-14T13:02:09.178 INFO:tasks.workunit.client.0.smithi111.stdout:0/146: dread - d6/d9/de/f27 zero size 2024-03-14T13:02:09.180 INFO:tasks.workunit.client.1.smithi137.stdout:3/77: creat d4/f18 x:0 0 0 2024-03-14T13:02:09.181 INFO:tasks.workunit.client.0.smithi111.stdout:1/187: creat d20/d21/d3f/f47 x:0 0 0 2024-03-14T13:02:09.181 INFO:tasks.workunit.client.0.smithi111.stdout:0/147: symlink d6/d9/de/l2d 0 2024-03-14T13:02:09.181 INFO:tasks.workunit.client.1.smithi137.stdout:3/78: read - d4/f18 zero size 2024-03-14T13:02:09.182 INFO:tasks.workunit.client.1.smithi137.stdout:0/30: rename d1/l2 to d1/d5/d9/la 0 2024-03-14T13:02:09.184 INFO:tasks.workunit.client.1.smithi137.stdout:8/33: rename d1 to d1/d7 22 2024-03-14T13:02:09.185 INFO:tasks.workunit.client.1.smithi137.stdout:6/54: link d2/d6/d8/ca d2/d6/c14 0 2024-03-14T13:02:09.185 INFO:tasks.workunit.client.1.smithi137.stdout:8/34: write d1/f2 [1971372,90307] 0 2024-03-14T13:02:09.186 INFO:tasks.workunit.client.0.smithi111.stdout:9/178: chown d5/d6/d8/d1a/c25 0 1 2024-03-14T13:02:09.186 INFO:tasks.workunit.client.0.smithi111.stdout:0/148: creat d6/d9/d15/f2e x:0 0 0 2024-03-14T13:02:09.186 INFO:tasks.workunit.client.0.smithi111.stdout:1/188: symlink d20/d25/l48 0 2024-03-14T13:02:09.187 INFO:tasks.workunit.client.1.smithi137.stdout:5/43: getdents d7 0 2024-03-14T13:02:09.188 INFO:tasks.workunit.client.1.smithi137.stdout:6/55: truncate d2/f7 1138092 0 2024-03-14T13:02:09.188 INFO:tasks.workunit.client.1.smithi137.stdout:5/44: chown d7/ld 102873 1 2024-03-14T13:02:09.189 INFO:tasks.workunit.client.1.smithi137.stdout:6/56: write d2/f11 [877969,80908] 0 2024-03-14T13:02:09.190 INFO:tasks.workunit.client.0.smithi111.stdout:0/149: truncate d6/d9/de/f1d 908476 0 2024-03-14T13:02:09.190 INFO:tasks.workunit.client.0.smithi111.stdout:8/327: dwrite d3/f6d [0,4194304] 0 2024-03-14T13:02:09.192 INFO:tasks.workunit.client.0.smithi111.stdout:1/189: unlink ld 0 2024-03-14T13:02:09.193 INFO:tasks.workunit.client.1.smithi137.stdout:7/60: mkdir d3/d5/d12 0 2024-03-14T13:02:09.194 INFO:tasks.workunit.client.0.smithi111.stdout:8/328: creat d3/d1b/f6f x:0 0 0 2024-03-14T13:02:09.194 INFO:tasks.workunit.client.0.smithi111.stdout:0/150: rmdir d6/d21 39 2024-03-14T13:02:09.194 INFO:tasks.workunit.client.1.smithi137.stdout:7/61: truncate d3/fe 944531 0 2024-03-14T13:02:09.195 INFO:tasks.workunit.client.1.smithi137.stdout:7/62: stat d3/d6/l10 0 2024-03-14T13:02:09.195 INFO:tasks.workunit.client.0.smithi111.stdout:9/179: getdents d5/d6/d2a/d2d 0 2024-03-14T13:02:09.196 INFO:tasks.workunit.client.0.smithi111.stdout:9/180: fdatasync d5/d6/fc 0 2024-03-14T13:02:09.196 INFO:tasks.workunit.client.1.smithi137.stdout:3/79: fdatasync d4/df/f14 0 2024-03-14T13:02:09.197 INFO:tasks.workunit.client.0.smithi111.stdout:1/190: mknod d20/d25/c49 0 2024-03-14T13:02:09.199 INFO:tasks.workunit.client.1.smithi137.stdout:1/88: write f5 [107044,74247] 0 2024-03-14T13:02:09.202 INFO:tasks.workunit.client.0.smithi111.stdout:1/191: symlink d20/d21/l4a 0 2024-03-14T13:02:09.203 INFO:tasks.workunit.client.0.smithi111.stdout:1/192: fdatasync f1a 0 2024-03-14T13:02:09.203 INFO:tasks.workunit.client.0.smithi111.stdout:9/181: mknod d5/d6/c39 0 2024-03-14T13:02:09.205 INFO:tasks.workunit.client.1.smithi137.stdout:8/35: creat d1/f8 x:0 0 0 2024-03-14T13:02:09.206 INFO:tasks.workunit.client.1.smithi137.stdout:8/36: chown d1/f8 3763294 1 2024-03-14T13:02:09.206 INFO:tasks.workunit.client.0.smithi111.stdout:9/182: creat d5/d6/d8/f3a x:0 0 0 2024-03-14T13:02:09.207 INFO:tasks.workunit.client.0.smithi111.stdout:8/329: dwrite f0 [0,4194304] 0 2024-03-14T13:02:09.208 INFO:tasks.workunit.client.1.smithi137.stdout:6/57: creat d2/d6/d8/f15 x:0 0 0 2024-03-14T13:02:09.208 INFO:tasks.workunit.client.0.smithi111.stdout:1/193: symlink d20/d21/d3f/l4b 0 2024-03-14T13:02:09.209 INFO:tasks.workunit.client.0.smithi111.stdout:1/194: read - d20/d21/f41 zero size 2024-03-14T13:02:09.210 INFO:tasks.workunit.client.0.smithi111.stdout:9/183: symlink d5/d6/d14/l3b 0 2024-03-14T13:02:09.211 INFO:tasks.workunit.client.1.smithi137.stdout:3/80: mkdir d4/d19 0 2024-03-14T13:02:09.212 INFO:tasks.workunit.client.0.smithi111.stdout:9/184: chown d5/d6/d2a/d2d/f32 1146 1 2024-03-14T13:02:09.212 INFO:tasks.workunit.client.1.smithi137.stdout:0/31: write d1/f8 [2365425,30275] 0 2024-03-14T13:02:09.217 INFO:tasks.workunit.client.1.smithi137.stdout:8/37: mknod d1/c9 0 2024-03-14T13:02:09.217 INFO:tasks.workunit.client.1.smithi137.stdout:3/81: dread d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:09.218 INFO:tasks.workunit.client.1.smithi137.stdout:9/95: link d6/df/f13 d6/df/d19/d1f/f24 0 2024-03-14T13:02:09.219 INFO:tasks.workunit.client.0.smithi111.stdout:8/330: rename d3/c1a to d3/d1b/d20/d35/d40/d48/c70 0 2024-03-14T13:02:09.219 INFO:tasks.workunit.client.1.smithi137.stdout:3/82: write f1 [2651111,63136] 0 2024-03-14T13:02:09.220 INFO:tasks.workunit.client.0.smithi111.stdout:1/195: link d20/d25/d31/f3b d20/d40/f4c 0 2024-03-14T13:02:09.220 INFO:tasks.workunit.client.1.smithi137.stdout:6/58: mknod d2/d6/de/c16 0 2024-03-14T13:02:09.221 INFO:tasks.workunit.client.0.smithi111.stdout:0/151: dwrite f3 [4194304,4194304] 0 2024-03-14T13:02:09.221 INFO:tasks.workunit.client.1.smithi137.stdout:4/77: link f2 d4/d9/f16 0 2024-03-14T13:02:09.221 INFO:tasks.workunit.client.1.smithi137.stdout:7/63: mknod d3/c13 0 2024-03-14T13:02:09.222 INFO:tasks.workunit.client.0.smithi111.stdout:9/185: dwrite d5/d6/d8/f30 [0,4194304] 0 2024-03-14T13:02:09.222 INFO:tasks.workunit.client.0.smithi111.stdout:1/196: chown c10 5521 1 2024-03-14T13:02:09.223 INFO:tasks.workunit.client.1.smithi137.stdout:5/45: rmdir d7/dc 0 2024-03-14T13:02:09.223 INFO:tasks.workunit.client.0.smithi111.stdout:8/331: symlink d3/d1b/d20/l71 0 2024-03-14T13:02:09.223 INFO:tasks.workunit.client.1.smithi137.stdout:4/78: write d4/f13 [876908,113410] 0 2024-03-14T13:02:09.225 INFO:tasks.workunit.client.1.smithi137.stdout:8/38: symlink d1/la 0 2024-03-14T13:02:09.225 INFO:tasks.workunit.client.1.smithi137.stdout:5/46: stat f5 0 2024-03-14T13:02:09.227 INFO:tasks.workunit.client.0.smithi111.stdout:0/152: mknod d6/d9/d15/d25/c2f 0 2024-03-14T13:02:09.229 INFO:tasks.workunit.client.0.smithi111.stdout:1/197: mknod d20/d25/c4d 0 2024-03-14T13:02:09.230 INFO:tasks.workunit.client.0.smithi111.stdout:8/332: creat d3/d22/f72 x:0 0 0 2024-03-14T13:02:09.231 INFO:tasks.workunit.client.0.smithi111.stdout:9/186: dread d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:09.231 INFO:tasks.workunit.client.1.smithi137.stdout:0/32: dwrite d1/f8 [0,4194304] 0 2024-03-14T13:02:09.231 INFO:tasks.workunit.client.0.smithi111.stdout:0/153: truncate d6/f1a 2412785 0 2024-03-14T13:02:09.232 INFO:tasks.workunit.client.1.smithi137.stdout:1/89: getdents d7 0 2024-03-14T13:02:09.234 INFO:tasks.workunit.client.1.smithi137.stdout:8/39: truncate d1/f8 269877 0 2024-03-14T13:02:09.234 INFO:tasks.workunit.client.0.smithi111.stdout:9/187: dread d5/d6/ff [0,4194304] 0 2024-03-14T13:02:09.235 INFO:tasks.workunit.client.0.smithi111.stdout:1/198: mkdir d20/d21/d4e 0 2024-03-14T13:02:09.236 INFO:tasks.workunit.client.0.smithi111.stdout:8/333: dread d3/f6b [0,4194304] 0 2024-03-14T13:02:09.239 INFO:tasks.workunit.client.0.smithi111.stdout:0/154: unlink f2 0 2024-03-14T13:02:09.240 INFO:tasks.workunit.client.1.smithi137.stdout:5/47: write d7/fa [455335,97197] 0 2024-03-14T13:02:09.241 INFO:tasks.workunit.client.1.smithi137.stdout:8/40: write d1/f6 [1727152,307] 0 2024-03-14T13:02:09.241 INFO:tasks.workunit.client.0.smithi111.stdout:8/334: fdatasync d3/d1b/d34/f3c 0 2024-03-14T13:02:09.244 INFO:tasks.workunit.client.0.smithi111.stdout:1/199: creat d20/d40/f4f x:0 0 0 2024-03-14T13:02:09.244 INFO:tasks.workunit.client.0.smithi111.stdout:8/335: dread - d3/dc/f6c zero size 2024-03-14T13:02:09.244 INFO:tasks.workunit.client.0.smithi111.stdout:9/188: mknod d5/c3c 0 2024-03-14T13:02:09.246 INFO:tasks.workunit.client.1.smithi137.stdout:6/59: dwrite d2/d6/fc [0,4194304] 0 2024-03-14T13:02:09.248 INFO:tasks.workunit.client.0.smithi111.stdout:9/189: fsync d5/fe 0 2024-03-14T13:02:09.249 INFO:tasks.workunit.client.0.smithi111.stdout:0/155: dwrite d6/f1f [0,4194304] 0 2024-03-14T13:02:09.251 INFO:tasks.workunit.client.0.smithi111.stdout:1/200: truncate d20/d25/d31/f45 209291 0 2024-03-14T13:02:09.251 INFO:tasks.workunit.client.0.smithi111.stdout:0/156: dread - d6/d9/de/f27 zero size 2024-03-14T13:02:09.252 INFO:tasks.workunit.client.0.smithi111.stdout:8/336: rename d3/d1b/d20/l33 to d3/dc/d45/l73 0 2024-03-14T13:02:09.253 INFO:tasks.workunit.client.0.smithi111.stdout:9/190: fsync f4 0 2024-03-14T13:02:09.255 INFO:tasks.workunit.client.0.smithi111.stdout:8/337: truncate d3/d22/d30/f43 951079 0 2024-03-14T13:02:09.255 INFO:tasks.workunit.client.0.smithi111.stdout:1/201: symlink d20/d40/l50 0 2024-03-14T13:02:09.256 INFO:tasks.workunit.client.0.smithi111.stdout:0/157: dread d6/f10 [0,4194304] 0 2024-03-14T13:02:09.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/338: creat d3/d22/d30/f74 x:0 0 0 2024-03-14T13:02:09.260 INFO:tasks.workunit.client.0.smithi111.stdout:0/158: write f5 [438402,130231] 0 2024-03-14T13:02:09.260 INFO:tasks.workunit.client.0.smithi111.stdout:1/202: mknod d20/d25/d31/c51 0 2024-03-14T13:02:09.260 INFO:tasks.workunit.client.1.smithi137.stdout:4/79: dwrite d4/f13 [0,4194304] 0 2024-03-14T13:02:09.262 INFO:tasks.workunit.client.1.smithi137.stdout:0/33: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:09.262 INFO:tasks.workunit.client.1.smithi137.stdout:3/83: truncate d4/fb 7118340 0 2024-03-14T13:02:09.263 INFO:tasks.workunit.client.0.smithi111.stdout:9/191: rename d5/d6/f9 to d5/d6/f3d 0 2024-03-14T13:02:09.263 INFO:tasks.workunit.client.0.smithi111.stdout:1/203: write d20/d40/f4f [807855,65201] 0 2024-03-14T13:02:09.266 INFO:tasks.workunit.client.1.smithi137.stdout:7/64: dwrite d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:09.266 INFO:tasks.workunit.client.1.smithi137.stdout:7/65: dread - d3/ff zero size 2024-03-14T13:02:09.267 INFO:tasks.workunit.client.1.smithi137.stdout:6/60: link d2/d6/fc d2/f17 0 2024-03-14T13:02:09.268 INFO:tasks.workunit.client.0.smithi111.stdout:0/159: creat d6/d9/de/f30 x:0 0 0 2024-03-14T13:02:09.270 INFO:tasks.workunit.client.1.smithi137.stdout:8/41: dwrite d1/f2 [0,4194304] 0 2024-03-14T13:02:09.271 INFO:tasks.workunit.client.1.smithi137.stdout:4/80: readlink d4/d10/l15 0 2024-03-14T13:02:09.272 INFO:tasks.workunit.client.1.smithi137.stdout:6/61: write d2/f17 [1858561,110280] 0 2024-03-14T13:02:09.273 INFO:tasks.workunit.client.1.smithi137.stdout:0/34: mknod d1/cb 0 2024-03-14T13:02:09.274 INFO:tasks.workunit.client.0.smithi111.stdout:1/204: readlink d20/d25/d31/l36 0 2024-03-14T13:02:09.275 INFO:tasks.workunit.client.0.smithi111.stdout:8/339: dwrite d3/dc/d31/f4c [0,4194304] 0 2024-03-14T13:02:09.276 INFO:tasks.workunit.client.0.smithi111.stdout:0/160: rename d6/d9/d15/f24 to d6/d9/d15/d25/f31 0 2024-03-14T13:02:09.276 INFO:tasks.workunit.client.0.smithi111.stdout:8/340: chown d3/d1b/d20/d35/d40 14080 1 2024-03-14T13:02:09.277 INFO:tasks.workunit.client.0.smithi111.stdout:8/341: write d3/d1b/f6f [663598,36538] 0 2024-03-14T13:02:09.278 INFO:tasks.workunit.client.1.smithi137.stdout:9/96: dwrite d6/df/f20 [4194304,4194304] 0 2024-03-14T13:02:09.278 INFO:tasks.workunit.client.1.smithi137.stdout:7/66: rename d3/d5/d12 to d3/d6/d14 0 2024-03-14T13:02:09.285 INFO:tasks.workunit.client.0.smithi111.stdout:9/192: getdents d5/d6/d8/d1a 0 2024-03-14T13:02:09.287 INFO:tasks.workunit.client.0.smithi111.stdout:1/205: link c14 d20/d25/d31/c52 0 2024-03-14T13:02:09.287 INFO:tasks.workunit.client.0.smithi111.stdout:9/193: chown d5/d6/fc 78359208 1 2024-03-14T13:02:09.288 INFO:tasks.workunit.client.1.smithi137.stdout:9/97: mknod d6/df/d19/c25 0 2024-03-14T13:02:09.288 INFO:tasks.workunit.client.1.smithi137.stdout:0/35: write d1/f8 [164587,97307] 0 2024-03-14T13:02:09.288 INFO:tasks.workunit.client.1.smithi137.stdout:4/81: mknod d4/d10/c17 0 2024-03-14T13:02:09.289 INFO:tasks.workunit.client.0.smithi111.stdout:8/342: dwrite d3/d1b/d34/f3c [0,4194304] 0 2024-03-14T13:02:09.290 INFO:tasks.workunit.client.0.smithi111.stdout:8/343: chown d3/dc/d45 1782030 1 2024-03-14T13:02:09.294 INFO:tasks.workunit.client.1.smithi137.stdout:6/62: rename d2/c4 to d2/d6/de/c18 0 2024-03-14T13:02:09.295 INFO:tasks.workunit.client.0.smithi111.stdout:9/194: dread d5/d6/f2f [0,4194304] 0 2024-03-14T13:02:09.302 INFO:tasks.workunit.client.0.smithi111.stdout:8/344: dwrite d3/dc/d31/f4c [0,4194304] 0 2024-03-14T13:02:09.302 INFO:tasks.workunit.client.0.smithi111.stdout:9/195: dread d5/fe [0,4194304] 0 2024-03-14T13:02:09.302 INFO:tasks.workunit.client.1.smithi137.stdout:0/36: dwrite d1/f8 [0,4194304] 0 2024-03-14T13:02:09.303 INFO:tasks.workunit.client.1.smithi137.stdout:3/84: unlink d4/fb 0 2024-03-14T13:02:09.304 INFO:tasks.workunit.client.1.smithi137.stdout:4/82: creat d4/d9/f18 x:0 0 0 2024-03-14T13:02:09.309 INFO:tasks.workunit.client.1.smithi137.stdout:6/63: mknod d2/d6/c19 0 2024-03-14T13:02:09.316 INFO:tasks.workunit.client.1.smithi137.stdout:6/64: symlink d2/d6/d8/l1a 0 2024-03-14T13:02:09.317 INFO:tasks.workunit.client.1.smithi137.stdout:8/42: dwrite d1/f6 [0,4194304] 0 2024-03-14T13:02:09.318 INFO:tasks.workunit.client.0.smithi111.stdout:0/161: rename d6/d9/d15/d25 to d6/d9/de/d32 0 2024-03-14T13:02:09.321 INFO:tasks.workunit.client.1.smithi137.stdout:8/43: creat d1/fb x:0 0 0 2024-03-14T13:02:09.322 INFO:tasks.workunit.client.0.smithi111.stdout:0/162: dread d6/f1f [0,4194304] 0 2024-03-14T13:02:09.322 INFO:tasks.workunit.client.1.smithi137.stdout:4/83: dwrite d4/f12 [0,4194304] 0 2024-03-14T13:02:09.326 INFO:tasks.workunit.client.0.smithi111.stdout:0/163: rename d6/d9/f28 to d6/d9/de/d32/f33 0 2024-03-14T13:02:09.326 INFO:tasks.workunit.client.0.smithi111.stdout:0/164: stat d6/d9/l11 0 2024-03-14T13:02:09.327 INFO:tasks.workunit.client.0.smithi111.stdout:1/206: rmdir d20 39 2024-03-14T13:02:09.327 INFO:tasks.workunit.client.1.smithi137.stdout:8/44: write d1/fb [978522,66944] 0 2024-03-14T13:02:09.328 INFO:tasks.workunit.client.1.smithi137.stdout:8/45: stat d1/fb 0 2024-03-14T13:02:09.329 INFO:tasks.workunit.client.1.smithi137.stdout:7/67: write d3/d5/fd [3529683,6125] 0 2024-03-14T13:02:09.331 INFO:tasks.workunit.client.0.smithi111.stdout:0/165: mknod d6/d9/de/c34 0 2024-03-14T13:02:09.335 INFO:tasks.workunit.client.0.smithi111.stdout:7/135: sync 2024-03-14T13:02:09.336 INFO:tasks.workunit.client.0.smithi111.stdout:5/234: sync 2024-03-14T13:02:09.336 INFO:tasks.workunit.client.1.smithi137.stdout:4/84: link d4/d10/l15 d4/l19 0 2024-03-14T13:02:09.336 INFO:tasks.workunit.client.0.smithi111.stdout:0/166: link d6/d9/cf d6/d9/d15/c35 0 2024-03-14T13:02:09.337 INFO:tasks.workunit.client.0.smithi111.stdout:9/196: dwrite f2 [0,4194304] 0 2024-03-14T13:02:09.337 INFO:tasks.workunit.client.1.smithi137.stdout:3/85: truncate f1 1943576 0 2024-03-14T13:02:09.338 INFO:tasks.workunit.client.0.smithi111.stdout:5/235: write d2/f6 [1328363,120935] 0 2024-03-14T13:02:09.340 INFO:tasks.workunit.client.1.smithi137.stdout:3/86: readlink d4/l8 0 2024-03-14T13:02:09.340 INFO:tasks.workunit.client.0.smithi111.stdout:9/197: stat d5/d6/c18 0 2024-03-14T13:02:09.340 INFO:tasks.workunit.client.1.smithi137.stdout:0/37: truncate d1/f8 3136852 0 2024-03-14T13:02:09.341 INFO:tasks.workunit.client.1.smithi137.stdout:2/37: sync 2024-03-14T13:02:09.341 INFO:tasks.workunit.client.0.smithi111.stdout:8/345: truncate d3/dc/f6a 611933 0 2024-03-14T13:02:09.342 INFO:tasks.workunit.client.1.smithi137.stdout:8/46: dwrite d1/fb [0,4194304] 0 2024-03-14T13:02:09.345 INFO:tasks.workunit.client.0.smithi111.stdout:5/236: creat d2/d16/d49/d44/f50 x:0 0 0 2024-03-14T13:02:09.346 INFO:tasks.workunit.client.1.smithi137.stdout:0/38: stat d1/d5/d9 0 2024-03-14T13:02:09.346 INFO:tasks.workunit.client.1.smithi137.stdout:7/68: dwrite d3/d5/fd [0,4194304] 0 2024-03-14T13:02:09.347 INFO:tasks.workunit.client.0.smithi111.stdout:7/136: dwrite d0/d1/d5/f1a [0,4194304] 0 2024-03-14T13:02:09.348 INFO:tasks.workunit.client.1.smithi137.stdout:2/38: mkdir d2/d7 0 2024-03-14T13:02:09.348 INFO:tasks.workunit.client.0.smithi111.stdout:7/137: stat d0/d1 0 2024-03-14T13:02:09.351 INFO:tasks.workunit.client.0.smithi111.stdout:8/346: write d3/f6b [2236912,125131] 0 2024-03-14T13:02:09.352 INFO:tasks.workunit.client.0.smithi111.stdout:5/237: symlink d2/d13/l51 0 2024-03-14T13:02:09.353 INFO:tasks.workunit.client.0.smithi111.stdout:7/138: read - d0/d1/f28 zero size 2024-03-14T13:02:09.355 INFO:tasks.workunit.client.0.smithi111.stdout:0/167: truncate f3 2260531 0 2024-03-14T13:02:09.355 INFO:tasks.workunit.client.0.smithi111.stdout:8/347: fdatasync d3/f3e 0 2024-03-14T13:02:09.355 INFO:tasks.workunit.client.1.smithi137.stdout:9/98: dwrite d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:09.356 INFO:tasks.workunit.client.0.smithi111.stdout:5/238: dread - d2/d16/d49/d44/f50 zero size 2024-03-14T13:02:09.356 INFO:tasks.workunit.client.1.smithi137.stdout:7/69: chown d3/d6/l7 5 1 2024-03-14T13:02:09.361 INFO:tasks.workunit.client.0.smithi111.stdout:8/348: creat d3/d1b/f75 x:0 0 0 2024-03-14T13:02:09.363 INFO:tasks.workunit.client.0.smithi111.stdout:1/207: dwrite f11 [0,4194304] 0 2024-03-14T13:02:09.363 INFO:tasks.workunit.client.1.smithi137.stdout:7/70: creat d3/d6/f15 x:0 0 0 2024-03-14T13:02:09.366 INFO:tasks.workunit.client.1.smithi137.stdout:4/85: dwrite d4/ff [0,4194304] 0 2024-03-14T13:02:09.366 INFO:tasks.workunit.client.0.smithi111.stdout:1/208: truncate f9 4847048 0 2024-03-14T13:02:09.367 INFO:tasks.workunit.client.1.smithi137.stdout:7/71: readlink d3/d6/l10 0 2024-03-14T13:02:09.368 INFO:tasks.workunit.client.1.smithi137.stdout:8/47: dwrite d1/f6 [0,4194304] 0 2024-03-14T13:02:09.368 INFO:tasks.workunit.client.0.smithi111.stdout:7/139: rmdir d0/d1/d5/de/d9 39 2024-03-14T13:02:09.370 INFO:tasks.workunit.client.0.smithi111.stdout:9/198: dwrite d5/d6/f3d [0,4194304] 0 2024-03-14T13:02:09.371 INFO:tasks.workunit.client.1.smithi137.stdout:8/48: chown d1/f2 389 1 2024-03-14T13:02:09.371 INFO:tasks.workunit.client.1.smithi137.stdout:7/72: write d3/d5/f11 [925515,24672] 0 2024-03-14T13:02:09.372 INFO:tasks.workunit.client.1.smithi137.stdout:9/99: dread d6/f18 [0,4194304] 0 2024-03-14T13:02:09.373 INFO:tasks.workunit.client.1.smithi137.stdout:4/86: unlink d4/d10/l15 0 2024-03-14T13:02:09.373 INFO:tasks.workunit.client.0.smithi111.stdout:5/239: rename d2/d45/c21 to d2/d16/d49/d44/c52 0 2024-03-14T13:02:09.374 INFO:tasks.workunit.client.0.smithi111.stdout:7/140: symlink d0/d1/d22/l2b 0 2024-03-14T13:02:09.375 INFO:tasks.workunit.client.0.smithi111.stdout:9/199: rmdir d5/d6 39 2024-03-14T13:02:09.375 INFO:tasks.workunit.client.1.smithi137.stdout:8/49: readlink d1/la 0 2024-03-14T13:02:09.375 INFO:tasks.workunit.client.1.smithi137.stdout:7/73: write d3/d6/f15 [465743,120846] 0 2024-03-14T13:02:09.375 INFO:tasks.workunit.client.0.smithi111.stdout:5/240: fsync d2/d16/f38 0 2024-03-14T13:02:09.376 INFO:tasks.workunit.client.1.smithi137.stdout:9/100: mkdir d6/d26 0 2024-03-14T13:02:09.378 INFO:tasks.workunit.client.1.smithi137.stdout:9/101: chown d6/df/d19/d1f 122485595 1 2024-03-14T13:02:09.379 INFO:tasks.workunit.client.0.smithi111.stdout:8/349: dread d3/f65 [0,4194304] 0 2024-03-14T13:02:09.382 INFO:tasks.workunit.client.0.smithi111.stdout:8/350: truncate d3/d1b/f75 79065 0 2024-03-14T13:02:09.382 INFO:tasks.workunit.client.1.smithi137.stdout:8/50: dread d1/f2 [0,4194304] 0 2024-03-14T13:02:09.384 INFO:tasks.workunit.client.0.smithi111.stdout:8/351: chown d3/d1b/d20/l5e 487 1 2024-03-14T13:02:09.384 INFO:tasks.workunit.client.0.smithi111.stdout:9/200: creat d5/d6/d2a/f3e x:0 0 0 2024-03-14T13:02:09.385 INFO:tasks.workunit.client.1.smithi137.stdout:9/102: unlink d6/df/d19/c25 0 2024-03-14T13:02:09.385 INFO:tasks.workunit.client.1.smithi137.stdout:8/51: chown d1/c9 225497708 1 2024-03-14T13:02:09.385 INFO:tasks.workunit.client.1.smithi137.stdout:7/74: dwrite d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:09.386 INFO:tasks.workunit.client.0.smithi111.stdout:7/141: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:09.387 INFO:tasks.workunit.client.0.smithi111.stdout:5/241: creat d2/f53 x:0 0 0 2024-03-14T13:02:09.388 INFO:tasks.workunit.client.1.smithi137.stdout:8/52: mkdir d1/dc 0 2024-03-14T13:02:09.389 INFO:tasks.workunit.client.1.smithi137.stdout:9/103: read d6/f18 [2391792,87714] 0 2024-03-14T13:02:09.390 INFO:tasks.workunit.client.1.smithi137.stdout:8/53: creat d1/fd x:0 0 0 2024-03-14T13:02:09.390 INFO:tasks.workunit.client.0.smithi111.stdout:7/142: write d0/d1/d21/f15 [494159,124762] 0 2024-03-14T13:02:09.391 INFO:tasks.workunit.client.1.smithi137.stdout:7/75: dread f1 [0,4194304] 0 2024-03-14T13:02:09.392 INFO:tasks.workunit.client.1.smithi137.stdout:8/54: write d1/f8 [939331,62921] 0 2024-03-14T13:02:09.393 INFO:tasks.workunit.client.1.smithi137.stdout:8/55: chown d1/fb 1094887 1 2024-03-14T13:02:09.396 INFO:tasks.workunit.client.1.smithi137.stdout:7/76: creat d3/d5/f16 x:0 0 0 2024-03-14T13:02:09.396 INFO:tasks.workunit.client.0.smithi111.stdout:8/352: creat d3/d1b/d20/d35/d40/f76 x:0 0 0 2024-03-14T13:02:09.397 INFO:tasks.workunit.client.1.smithi137.stdout:9/104: getdents d6/df 0 2024-03-14T13:02:09.399 INFO:tasks.workunit.client.1.smithi137.stdout:9/105: rename d6/df/d19 to d6/df/d19/d27 22 2024-03-14T13:02:09.402 INFO:tasks.workunit.client.0.smithi111.stdout:9/201: getdents d5 0 2024-03-14T13:02:09.404 INFO:tasks.workunit.client.1.smithi137.stdout:8/56: mknod d1/ce 0 2024-03-14T13:02:09.405 INFO:tasks.workunit.client.0.smithi111.stdout:7/143: dread - d0/d1/d5/de/d9/f19 zero size 2024-03-14T13:02:09.406 INFO:tasks.workunit.client.0.smithi111.stdout:0/168: read f3 [432742,67491] 0 2024-03-14T13:02:09.406 INFO:tasks.workunit.client.1.smithi137.stdout:9/106: symlink d6/d26/l28 0 2024-03-14T13:02:09.408 INFO:tasks.workunit.client.0.smithi111.stdout:0/169: stat d6/d9/de/l2d 0 2024-03-14T13:02:09.411 INFO:tasks.workunit.client.0.smithi111.stdout:7/144: symlink d0/d1/d5/de/d1b/l2c 0 2024-03-14T13:02:09.412 INFO:tasks.workunit.client.1.smithi137.stdout:0/39: dwrite d1/f8 [0,4194304] 0 2024-03-14T13:02:09.412 INFO:tasks.workunit.client.1.smithi137.stdout:9/107: truncate d6/fc 1441412 0 2024-03-14T13:02:09.413 INFO:tasks.workunit.client.0.smithi111.stdout:9/202: dwrite d5/d6/f3d [0,4194304] 0 2024-03-14T13:02:09.414 INFO:tasks.workunit.client.1.smithi137.stdout:9/108: read - d6/f1d zero size 2024-03-14T13:02:09.414 INFO:tasks.workunit.client.0.smithi111.stdout:0/170: link f5 d6/d9/de/f36 0 2024-03-14T13:02:09.415 INFO:tasks.workunit.client.0.smithi111.stdout:7/145: mknod d0/d1/d21/d2a/c2d 0 2024-03-14T13:02:09.416 INFO:tasks.workunit.client.0.smithi111.stdout:7/146: read - d0/d1/f25 zero size 2024-03-14T13:02:09.417 INFO:tasks.workunit.client.0.smithi111.stdout:9/203: dread d5/d6/ff [0,4194304] 0 2024-03-14T13:02:09.417 INFO:tasks.workunit.client.1.smithi137.stdout:8/57: dwrite d1/f2 [0,4194304] 0 2024-03-14T13:02:09.419 INFO:tasks.workunit.client.0.smithi111.stdout:9/204: dread - d5/d6/d2a/d2d/f32 zero size 2024-03-14T13:02:09.422 INFO:tasks.workunit.client.0.smithi111.stdout:7/147: creat d0/f2e x:0 0 0 2024-03-14T13:02:09.423 INFO:tasks.workunit.client.0.smithi111.stdout:0/171: link d6/f1a d6/d9/d15/f37 0 2024-03-14T13:02:09.424 INFO:tasks.workunit.client.0.smithi111.stdout:7/148: rmdir d0/d1/d5/de/d9 39 2024-03-14T13:02:09.425 INFO:tasks.workunit.client.0.smithi111.stdout:0/172: truncate d6/d9/d15/f16 296764 0 2024-03-14T13:02:09.426 INFO:tasks.workunit.client.1.smithi137.stdout:9/109: rename d6/d26/l28 to d6/df/l29 0 2024-03-14T13:02:09.426 INFO:tasks.workunit.client.1.smithi137.stdout:2/39: truncate d2/f4 2693737 0 2024-03-14T13:02:09.428 INFO:tasks.workunit.client.1.smithi137.stdout:7/77: rmdir d3/d6 39 2024-03-14T13:02:09.430 INFO:tasks.workunit.client.0.smithi111.stdout:9/205: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:09.432 INFO:tasks.workunit.client.0.smithi111.stdout:9/206: write d5/d6/d2a/d2d/f32 [1042621,81024] 0 2024-03-14T13:02:09.432 INFO:tasks.workunit.client.0.smithi111.stdout:0/173: link d6/d9/de/l20 d6/l38 0 2024-03-14T13:02:09.432 INFO:tasks.workunit.client.0.smithi111.stdout:0/174: dread - d6/f1e zero size 2024-03-14T13:02:09.433 INFO:tasks.workunit.client.0.smithi111.stdout:9/207: chown d5/f26 819683231 1 2024-03-14T13:02:09.435 INFO:tasks.workunit.client.1.smithi137.stdout:0/40: dwrite d1/f8 [0,4194304] 0 2024-03-14T13:02:09.435 INFO:tasks.workunit.client.0.smithi111.stdout:9/208: mknod d5/d6/d2a/d2d/c3f 0 2024-03-14T13:02:09.438 INFO:tasks.workunit.client.0.smithi111.stdout:9/209: mknod d5/d6/d8/c40 0 2024-03-14T13:02:09.439 INFO:tasks.workunit.client.1.smithi137.stdout:9/110: dread d6/fe [0,4194304] 0 2024-03-14T13:02:09.441 INFO:tasks.workunit.client.0.smithi111.stdout:1/209: write d20/d25/d31/f3b [322848,91349] 0 2024-03-14T13:02:09.442 INFO:tasks.workunit.client.1.smithi137.stdout:9/111: stat d6/d26 0 2024-03-14T13:02:09.444 INFO:tasks.workunit.client.1.smithi137.stdout:7/78: dread d3/fe [0,4194304] 0 2024-03-14T13:02:09.444 INFO:tasks.workunit.client.0.smithi111.stdout:5/242: truncate d2/d13/f4b 162463 0 2024-03-14T13:02:09.444 INFO:tasks.workunit.client.0.smithi111.stdout:1/210: truncate d20/f44 307980 0 2024-03-14T13:02:09.448 INFO:tasks.workunit.client.0.smithi111.stdout:9/210: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:09.448 INFO:tasks.workunit.client.0.smithi111.stdout:5/243: write d2/d13/f29 [1693438,115496] 0 2024-03-14T13:02:09.449 INFO:tasks.workunit.client.1.smithi137.stdout:4/87: dwrite f2 [0,4194304] 0 2024-03-14T13:02:09.450 INFO:tasks.workunit.client.1.smithi137.stdout:3/87: write f1 [1237845,63699] 0 2024-03-14T13:02:09.451 INFO:tasks.workunit.client.1.smithi137.stdout:8/58: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:09.451 INFO:tasks.workunit.client.0.smithi111.stdout:5/244: dread - d2/d16/d49/d44/f50 zero size 2024-03-14T13:02:09.453 INFO:tasks.workunit.client.1.smithi137.stdout:9/112: creat d6/df/d19/d1f/f2a x:0 0 0 2024-03-14T13:02:09.453 INFO:tasks.workunit.client.0.smithi111.stdout:1/211: link d20/c2c d20/c53 0 2024-03-14T13:02:09.454 INFO:tasks.workunit.client.0.smithi111.stdout:8/353: dwrite d3/d1b/d20/f2b [0,4194304] 0 2024-03-14T13:02:09.454 INFO:tasks.workunit.client.0.smithi111.stdout:9/211: creat d5/d6/d8/f41 x:0 0 0 2024-03-14T13:02:09.456 INFO:tasks.workunit.client.0.smithi111.stdout:1/212: read - d20/d25/f46 zero size 2024-03-14T13:02:09.457 INFO:tasks.workunit.client.0.smithi111.stdout:9/212: mkdir d5/d42 0 2024-03-14T13:02:09.457 INFO:tasks.workunit.client.1.smithi137.stdout:9/113: unlink f1 0 2024-03-14T13:02:09.460 INFO:tasks.workunit.client.1.smithi137.stdout:9/114: creat d6/f2b x:0 0 0 2024-03-14T13:02:09.460 INFO:tasks.workunit.client.0.smithi111.stdout:1/213: readlink d20/l22 0 2024-03-14T13:02:09.460 INFO:tasks.workunit.client.0.smithi111.stdout:5/245: rename d2/d16/d2c/c3a to d2/d1f/c54 0 2024-03-14T13:02:09.461 INFO:tasks.workunit.client.1.smithi137.stdout:8/59: dread d1/fb [0,4194304] 0 2024-03-14T13:02:09.462 INFO:tasks.workunit.client.0.smithi111.stdout:8/354: mkdir d3/d1b/d20/d35/d40/d48/d77 0 2024-03-14T13:02:09.462 INFO:tasks.workunit.client.0.smithi111.stdout:5/246: chown d2/d1f/d36/f3c 0 1 2024-03-14T13:02:09.462 INFO:tasks.workunit.client.1.smithi137.stdout:3/88: dread f1 [0,4194304] 0 2024-03-14T13:02:09.462 INFO:tasks.workunit.client.1.smithi137.stdout:0/41: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:09.464 INFO:tasks.workunit.client.1.smithi137.stdout:9/115: creat d6/f2c x:0 0 0 2024-03-14T13:02:09.465 INFO:tasks.workunit.client.0.smithi111.stdout:8/355: creat d3/f78 x:0 0 0 2024-03-14T13:02:09.467 INFO:tasks.workunit.client.1.smithi137.stdout:8/60: symlink d1/dc/lf 0 2024-03-14T13:02:09.467 INFO:tasks.workunit.client.0.smithi111.stdout:8/356: write d3/d1b/d20/d35/d40/d48/f68 [543607,82768] 0 2024-03-14T13:02:09.467 INFO:tasks.workunit.client.0.smithi111.stdout:5/247: dread d2/d13/f1d [0,4194304] 0 2024-03-14T13:02:09.468 INFO:tasks.workunit.client.1.smithi137.stdout:7/79: dwrite d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:09.470 INFO:tasks.workunit.client.0.smithi111.stdout:5/248: creat d2/d16/d49/d4c/f55 x:0 0 0 2024-03-14T13:02:09.471 INFO:tasks.workunit.client.1.smithi137.stdout:7/80: fdatasync d3/d5/f16 0 2024-03-14T13:02:09.471 INFO:tasks.workunit.client.1.smithi137.stdout:8/61: read d1/f8 [968583,108602] 0 2024-03-14T13:02:09.472 INFO:tasks.workunit.client.1.smithi137.stdout:3/89: creat d4/de/d16/f1a x:0 0 0 2024-03-14T13:02:09.473 INFO:tasks.workunit.client.0.smithi111.stdout:8/357: rename d3/dc/f6a to d3/f79 0 2024-03-14T13:02:09.474 INFO:tasks.workunit.client.1.smithi137.stdout:9/116: dread d6/df/f10 [0,4194304] 0 2024-03-14T13:02:09.475 INFO:tasks.workunit.client.1.smithi137.stdout:8/62: mkdir d1/dc/d10 0 2024-03-14T13:02:09.476 INFO:tasks.workunit.client.0.smithi111.stdout:5/249: rename d2/d1f/l26 to d2/d16/d49/d44/l56 0 2024-03-14T13:02:09.477 INFO:tasks.workunit.client.1.smithi137.stdout:9/117: symlink d6/l2d 0 2024-03-14T13:02:09.478 INFO:tasks.workunit.client.1.smithi137.stdout:8/63: mkdir d1/dc/d11 0 2024-03-14T13:02:09.479 INFO:tasks.workunit.client.0.smithi111.stdout:8/358: link d3/d1b/d20/l5e d3/d49/l7a 0 2024-03-14T13:02:09.479 INFO:tasks.workunit.client.1.smithi137.stdout:0/42: dwrite d1/f8 [0,4194304] 0 2024-03-14T13:02:09.481 INFO:tasks.workunit.client.1.smithi137.stdout:9/118: dread d6/df/f10 [0,4194304] 0 2024-03-14T13:02:09.484 INFO:tasks.workunit.client.1.smithi137.stdout:0/43: symlink d1/d5/lc 0 2024-03-14T13:02:09.484 INFO:tasks.workunit.client.0.smithi111.stdout:5/250: dwrite d2/d1f/f24 [0,4194304] 0 2024-03-14T13:02:09.485 INFO:tasks.workunit.client.0.smithi111.stdout:8/359: dread d3/d5/f12 [0,4194304] 0 2024-03-14T13:02:09.485 INFO:tasks.workunit.client.0.smithi111.stdout:8/360: read d3/f6b [1780853,51122] 0 2024-03-14T13:02:09.485 INFO:tasks.workunit.client.1.smithi137.stdout:3/90: getdents d4/de 0 2024-03-14T13:02:09.485 INFO:tasks.workunit.client.1.smithi137.stdout:9/119: write d6/df/f14 [1372469,97438] 0 2024-03-14T13:02:09.486 INFO:tasks.workunit.client.1.smithi137.stdout:3/91: fsync d4/f18 0 2024-03-14T13:02:09.487 INFO:tasks.workunit.client.1.smithi137.stdout:8/64: link d1/f2 d1/dc/d10/f12 0 2024-03-14T13:02:09.488 INFO:tasks.workunit.client.1.smithi137.stdout:8/65: fsync d1/f6 0 2024-03-14T13:02:09.488 INFO:tasks.workunit.client.1.smithi137.stdout:0/44: creat d1/fd x:0 0 0 2024-03-14T13:02:09.488 INFO:tasks.workunit.client.0.smithi111.stdout:5/251: rename d2/d1f/f37 to d2/d1f/f57 0 2024-03-14T13:02:09.488 INFO:tasks.workunit.client.1.smithi137.stdout:3/92: truncate d4/f18 799586 0 2024-03-14T13:02:09.489 INFO:tasks.workunit.client.1.smithi137.stdout:8/66: read d1/f2 [1365666,62355] 0 2024-03-14T13:02:09.489 INFO:tasks.workunit.client.0.smithi111.stdout:8/361: write d3/d22/f3f [215970,124072] 0 2024-03-14T13:02:09.490 INFO:tasks.workunit.client.0.smithi111.stdout:5/252: write d2/d45/f34 [484342,103873] 0 2024-03-14T13:02:09.491 INFO:tasks.workunit.client.0.smithi111.stdout:8/362: chown d3/d22/d30/l41 14635546 1 2024-03-14T13:02:09.492 INFO:tasks.workunit.client.1.smithi137.stdout:9/120: unlink d6/df/d19/c1a 0 2024-03-14T13:02:09.492 INFO:tasks.workunit.client.0.smithi111.stdout:5/253: mknod d2/d16/d49/d33/c58 0 2024-03-14T13:02:09.492 INFO:tasks.workunit.client.1.smithi137.stdout:0/45: chown d1/d5/d9/la 54917 1 2024-03-14T13:02:09.494 INFO:tasks.workunit.client.1.smithi137.stdout:9/121: symlink d6/df/l2e 0 2024-03-14T13:02:09.495 INFO:tasks.workunit.client.0.smithi111.stdout:5/254: symlink d2/d16/d49/d44/d4a/l59 0 2024-03-14T13:02:09.495 INFO:tasks.workunit.client.1.smithi137.stdout:9/122: read - d6/f2c zero size 2024-03-14T13:02:09.497 INFO:tasks.workunit.client.0.smithi111.stdout:5/255: mknod d2/d16/d49/d44/d4a/c5a 0 2024-03-14T13:02:09.497 INFO:tasks.workunit.client.0.smithi111.stdout:8/363: read d3/f14 [110525,42324] 0 2024-03-14T13:02:09.498 INFO:tasks.workunit.client.1.smithi137.stdout:3/93: dwrite d4/df/f14 [0,4194304] 0 2024-03-14T13:02:09.499 INFO:tasks.workunit.client.0.smithi111.stdout:5/256: creat d2/d16/d49/d33/f5b x:0 0 0 2024-03-14T13:02:09.500 INFO:tasks.workunit.client.0.smithi111.stdout:8/364: creat d3/d1b/d66/f7b x:0 0 0 2024-03-14T13:02:09.501 INFO:tasks.workunit.client.0.smithi111.stdout:5/257: chown d2/d16/d49/f27 1949167 1 2024-03-14T13:02:09.501 INFO:tasks.workunit.client.1.smithi137.stdout:3/94: mknod d4/de/d13/c1b 0 2024-03-14T13:02:09.501 INFO:tasks.workunit.client.0.smithi111.stdout:5/258: fsync d2/d16/d2c/f3e 0 2024-03-14T13:02:09.502 INFO:tasks.workunit.client.0.smithi111.stdout:8/365: write d3/dc/d59/f63 [144135,71091] 0 2024-03-14T13:02:09.502 INFO:tasks.workunit.client.1.smithi137.stdout:0/46: dwrite d1/fd [0,4194304] 0 2024-03-14T13:02:09.503 INFO:tasks.workunit.client.1.smithi137.stdout:3/95: fdatasync d4/de/d16/f1a 0 2024-03-14T13:02:09.503 INFO:tasks.workunit.client.0.smithi111.stdout:5/259: write d2/d16/f38 [395405,36251] 0 2024-03-14T13:02:09.503 INFO:tasks.workunit.client.0.smithi111.stdout:8/366: chown d3/dc/d31/f4c 10 1 2024-03-14T13:02:09.506 INFO:tasks.workunit.client.1.smithi137.stdout:3/96: dread f1 [0,4194304] 0 2024-03-14T13:02:09.507 INFO:tasks.workunit.client.0.smithi111.stdout:8/367: creat d3/dc/d45/f7c x:0 0 0 2024-03-14T13:02:09.507 INFO:tasks.workunit.client.1.smithi137.stdout:3/97: stat d4 0 2024-03-14T13:02:09.508 INFO:tasks.workunit.client.0.smithi111.stdout:8/368: creat d3/d22/d30/f7d x:0 0 0 2024-03-14T13:02:09.511 INFO:tasks.workunit.client.0.smithi111.stdout:8/369: mkdir d3/d1b/d20/d35/d40/d48/d7e 0 2024-03-14T13:02:09.514 INFO:tasks.workunit.client.1.smithi137.stdout:3/98: dwrite d4/de/d16/f1a [0,4194304] 0 2024-03-14T13:02:09.541 INFO:tasks.workunit.client.1.smithi137.stdout:4/88: read d4/d9/fc [4021529,124967] 0 2024-03-14T13:02:09.542 INFO:tasks.workunit.client.1.smithi137.stdout:2/40: dread d2/f4 [0,4194304] 0 2024-03-14T13:02:09.542 INFO:tasks.workunit.client.0.smithi111.stdout:7/149: truncate d0/d1/f2 2685078 0 2024-03-14T13:02:09.543 INFO:tasks.workunit.client.0.smithi111.stdout:7/150: symlink d0/l2f 0 2024-03-14T13:02:09.544 INFO:tasks.workunit.client.0.smithi111.stdout:7/151: chown d0/d1/f25 0 1 2024-03-14T13:02:09.549 INFO:tasks.workunit.client.0.smithi111.stdout:5/260: rmdir d2/d13 39 2024-03-14T13:02:09.550 INFO:tasks.workunit.client.0.smithi111.stdout:5/261: fdatasync d2/d45/f34 0 2024-03-14T13:02:09.551 INFO:tasks.workunit.client.1.smithi137.stdout:9/123: chown d6/df/f20 550184686 1 2024-03-14T13:02:09.551 INFO:tasks.workunit.client.0.smithi111.stdout:5/262: fsync d2/d1f/d36/f3c 0 2024-03-14T13:02:09.552 INFO:tasks.workunit.client.0.smithi111.stdout:5/263: chown d2/d16/d49/d33/f43 448 1 2024-03-14T13:02:09.553 INFO:tasks.workunit.client.0.smithi111.stdout:7/152: link d0/d1/d5/de/d9/l20 d0/d1/d22/l30 0 2024-03-14T13:02:09.556 INFO:tasks.workunit.client.1.smithi137.stdout:9/124: symlink d6/d26/l2f 0 2024-03-14T13:02:09.559 INFO:tasks.workunit.client.1.smithi137.stdout:4/89: unlink d4/ff 0 2024-03-14T13:02:09.559 INFO:tasks.workunit.client.0.smithi111.stdout:8/370: dread d3/d5/f1e [0,4194304] 0 2024-03-14T13:02:09.561 INFO:tasks.workunit.client.0.smithi111.stdout:8/371: chown d3/d1b/d20/d35/d40/d48 253 1 2024-03-14T13:02:09.561 INFO:tasks.workunit.client.0.smithi111.stdout:9/213: truncate d5/fa 4875500 0 2024-03-14T13:02:09.562 INFO:tasks.workunit.client.0.smithi111.stdout:7/153: rename d0/d1/d5/de/d9/l13 to d0/d1/d5/de/d9/l31 0 2024-03-14T13:02:09.563 INFO:tasks.workunit.client.1.smithi137.stdout:2/41: getdents d2 0 2024-03-14T13:02:09.565 INFO:tasks.workunit.client.0.smithi111.stdout:9/214: stat d5/d6/d2a/c35 0 2024-03-14T13:02:09.566 INFO:tasks.workunit.client.0.smithi111.stdout:8/372: dread d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:09.567 INFO:tasks.workunit.client.0.smithi111.stdout:7/154: mknod d0/d1/d5/de/d9/c32 0 2024-03-14T13:02:09.567 INFO:tasks.workunit.client.1.smithi137.stdout:2/42: unlink c0 0 2024-03-14T13:02:09.568 INFO:tasks.workunit.client.1.smithi137.stdout:3/99: rmdir d4 39 2024-03-14T13:02:09.570 INFO:tasks.workunit.client.0.smithi111.stdout:9/215: creat d5/d6/d8/d1a/f43 x:0 0 0 2024-03-14T13:02:09.570 INFO:tasks.workunit.client.1.smithi137.stdout:0/47: truncate d1/fd 2005561 0 2024-03-14T13:02:09.571 INFO:tasks.workunit.client.1.smithi137.stdout:0/48: readlink d1/d5/l7 0 2024-03-14T13:02:09.572 INFO:tasks.workunit.client.1.smithi137.stdout:4/90: link f2 d4/f1a 0 2024-03-14T13:02:09.573 INFO:tasks.workunit.client.0.smithi111.stdout:9/216: read d5/f20 [425220,1936] 0 2024-03-14T13:02:09.575 INFO:tasks.workunit.client.1.smithi137.stdout:2/43: write d2/f4 [945533,9117] 0 2024-03-14T13:02:09.575 INFO:tasks.workunit.client.1.smithi137.stdout:1/90: sync 2024-03-14T13:02:09.576 INFO:tasks.workunit.client.1.smithi137.stdout:1/91: readlink d7/l8 0 2024-03-14T13:02:09.577 INFO:tasks.workunit.client.1.smithi137.stdout:0/49: mknod d1/d5/ce 0 2024-03-14T13:02:09.577 INFO:tasks.workunit.client.1.smithi137.stdout:1/92: write f5 [5237938,83769] 0 2024-03-14T13:02:09.579 INFO:tasks.workunit.client.0.smithi111.stdout:8/373: rename d3/d1b/d20/d35/d40/d48/c70 to d3/d5/c7f 0 2024-03-14T13:02:09.579 INFO:tasks.workunit.client.0.smithi111.stdout:9/217: symlink d5/d6/d8/l44 0 2024-03-14T13:02:09.581 INFO:tasks.workunit.client.0.smithi111.stdout:8/374: write d3/d1b/d20/f2b [1753153,104338] 0 2024-03-14T13:02:09.582 INFO:tasks.workunit.client.1.smithi137.stdout:3/100: mkdir d4/de/d16/d1c 0 2024-03-14T13:02:09.582 INFO:tasks.workunit.client.1.smithi137.stdout:2/44: dread d2/f5 [0,4194304] 0 2024-03-14T13:02:09.583 INFO:tasks.workunit.client.0.smithi111.stdout:8/375: dread - d3/d22/d30/f7d zero size 2024-03-14T13:02:09.586 INFO:tasks.workunit.client.0.smithi111.stdout:8/376: mknod d3/dc/d31/c80 0 2024-03-14T13:02:09.596 INFO:tasks.workunit.client.0.smithi111.stdout:9/218: truncate f2 2242112 0 2024-03-14T13:02:09.597 INFO:tasks.workunit.client.0.smithi111.stdout:7/155: write d0/f7 [3001103,75029] 0 2024-03-14T13:02:09.597 INFO:tasks.workunit.client.0.smithi111.stdout:5/264: chown d2/d13/f4b 123996 1 2024-03-14T13:02:09.600 INFO:tasks.workunit.client.0.smithi111.stdout:5/265: creat d2/d45/f5c x:0 0 0 2024-03-14T13:02:09.602 INFO:tasks.workunit.client.0.smithi111.stdout:9/219: mknod d5/d42/c45 0 2024-03-14T13:02:09.603 INFO:tasks.workunit.client.0.smithi111.stdout:5/266: creat d2/d16/d49/f5d x:0 0 0 2024-03-14T13:02:09.603 INFO:tasks.workunit.client.0.smithi111.stdout:9/220: fdatasync d5/d6/f22 0 2024-03-14T13:02:09.607 INFO:tasks.workunit.client.0.smithi111.stdout:5/267: readlink d2/l1a 0 2024-03-14T13:02:09.609 INFO:tasks.workunit.client.0.smithi111.stdout:5/268: fsync d2/d45/f5c 0 2024-03-14T13:02:09.609 INFO:tasks.workunit.client.0.smithi111.stdout:9/221: dread - d5/f31 zero size 2024-03-14T13:02:09.613 INFO:tasks.workunit.client.0.smithi111.stdout:9/222: write d5/f26 [148045,3787] 0 2024-03-14T13:02:09.613 INFO:tasks.workunit.client.0.smithi111.stdout:5/269: creat d2/d1f/f5e x:0 0 0 2024-03-14T13:02:09.613 INFO:tasks.workunit.client.0.smithi111.stdout:7/156: dwrite d0/d1/d5/de/d9/f19 [0,4194304] 0 2024-03-14T13:02:09.615 INFO:tasks.workunit.client.0.smithi111.stdout:7/157: dread - d0/d1/d5/de/d9/f16 zero size 2024-03-14T13:02:09.615 INFO:tasks.workunit.client.0.smithi111.stdout:9/223: write d5/d6/d8/f41 [584307,46781] 0 2024-03-14T13:02:09.620 INFO:tasks.workunit.client.0.smithi111.stdout:9/224: creat d5/d42/f46 x:0 0 0 2024-03-14T13:02:09.623 INFO:tasks.workunit.client.0.smithi111.stdout:9/225: symlink d5/d42/l47 0 2024-03-14T13:02:09.624 INFO:tasks.workunit.client.0.smithi111.stdout:6/173: sync 2024-03-14T13:02:09.626 INFO:tasks.workunit.client.0.smithi111.stdout:9/226: dread - d5/d6/d2a/f3e zero size 2024-03-14T13:02:09.626 INFO:tasks.workunit.client.0.smithi111.stdout:5/270: dwrite d2/d16/d49/f1e [0,4194304] 0 2024-03-14T13:02:09.628 INFO:tasks.workunit.client.1.smithi137.stdout:1/93: creat d7/d9/f15 x:0 0 0 2024-03-14T13:02:09.629 INFO:tasks.workunit.client.0.smithi111.stdout:5/271: symlink d2/d16/d49/d33/l5f 0 2024-03-14T13:02:09.630 INFO:tasks.workunit.client.0.smithi111.stdout:6/174: creat d0/d17/d24/f38 x:0 0 0 2024-03-14T13:02:09.631 INFO:tasks.workunit.client.1.smithi137.stdout:2/45: creat d2/f8 x:0 0 0 2024-03-14T13:02:09.631 INFO:tasks.workunit.client.1.smithi137.stdout:1/94: chown d7/d9 1 1 2024-03-14T13:02:09.631 INFO:tasks.workunit.client.1.smithi137.stdout:4/91: dwrite d4/d10/f14 [0,4194304] 0 2024-03-14T13:02:09.631 INFO:tasks.workunit.client.0.smithi111.stdout:6/175: chown d0/l15 210948184 1 2024-03-14T13:02:09.633 INFO:tasks.workunit.client.0.smithi111.stdout:9/227: link d5/d6/d8/c40 d5/d6/d14/c48 0 2024-03-14T13:02:09.636 INFO:tasks.workunit.client.0.smithi111.stdout:5/272: link d2/d1f/f28 d2/d16/d49/d44/d4a/f60 0 2024-03-14T13:02:09.636 INFO:tasks.workunit.client.0.smithi111.stdout:7/158: dread d0/d1/d5/de/d1b/f1e [0,4194304] 0 2024-03-14T13:02:09.638 INFO:tasks.workunit.client.0.smithi111.stdout:5/273: write d2/f7 [2232653,88600] 0 2024-03-14T13:02:09.639 INFO:tasks.workunit.client.0.smithi111.stdout:7/159: mknod d0/d1/d5/de/c33 0 2024-03-14T13:02:09.640 INFO:tasks.workunit.client.0.smithi111.stdout:6/176: dread d0/d17/f1e [0,4194304] 0 2024-03-14T13:02:09.641 INFO:tasks.workunit.client.0.smithi111.stdout:5/274: symlink d2/d16/d49/d44/l61 0 2024-03-14T13:02:09.641 INFO:tasks.workunit.client.1.smithi137.stdout:3/101: getdents d4 0 2024-03-14T13:02:09.644 INFO:tasks.workunit.client.0.smithi111.stdout:7/160: getdents d0/d1 0 2024-03-14T13:02:09.645 INFO:tasks.workunit.client.0.smithi111.stdout:7/161: readlink d0/d1/d22/l2b 0 2024-03-14T13:02:09.645 INFO:tasks.workunit.client.0.smithi111.stdout:5/275: creat d2/d16/d49/d44/d4a/f62 x:0 0 0 2024-03-14T13:02:09.646 INFO:tasks.workunit.client.0.smithi111.stdout:6/177: creat d0/d17/d21/d37/f39 x:0 0 0 2024-03-14T13:02:09.648 INFO:tasks.workunit.client.0.smithi111.stdout:6/178: rmdir d0/d17/d21 39 2024-03-14T13:02:09.649 INFO:tasks.workunit.client.0.smithi111.stdout:5/276: creat d2/d16/d49/d44/d4a/f63 x:0 0 0 2024-03-14T13:02:09.654 INFO:tasks.workunit.client.0.smithi111.stdout:7/162: dwrite d0/d1/d5/f1a [0,4194304] 0 2024-03-14T13:02:09.654 INFO:tasks.workunit.client.0.smithi111.stdout:6/179: link d0/l13 d0/d17/d20/l3a 0 2024-03-14T13:02:09.654 INFO:tasks.workunit.client.1.smithi137.stdout:2/46: dwrite d2/f4 [0,4194304] 0 2024-03-14T13:02:09.654 INFO:tasks.workunit.client.1.smithi137.stdout:9/125: dwrite d6/fa [0,4194304] 0 2024-03-14T13:02:09.657 INFO:tasks.workunit.client.1.smithi137.stdout:3/102: dread d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:09.659 INFO:tasks.workunit.client.0.smithi111.stdout:7/163: unlink d0/d1/d5/de/d1b/f1e 0 2024-03-14T13:02:09.659 INFO:tasks.workunit.client.1.smithi137.stdout:2/47: write d2/f8 [38157,69865] 0 2024-03-14T13:02:09.660 INFO:tasks.workunit.client.0.smithi111.stdout:5/277: dwrite d2/d16/f42 [0,4194304] 0 2024-03-14T13:02:09.660 INFO:tasks.workunit.client.1.smithi137.stdout:4/92: dwrite d4/f11 [0,4194304] 0 2024-03-14T13:02:09.661 INFO:tasks.workunit.client.1.smithi137.stdout:2/48: chown d2/d7 4055201 1 2024-03-14T13:02:09.667 INFO:tasks.workunit.client.1.smithi137.stdout:2/49: write d2/f8 [423857,110801] 0 2024-03-14T13:02:09.667 INFO:tasks.workunit.client.1.smithi137.stdout:1/95: dwrite d7/fb [0,4194304] 0 2024-03-14T13:02:09.668 INFO:tasks.workunit.client.1.smithi137.stdout:1/96: write f3 [1996873,78061] 0 2024-03-14T13:02:09.669 INFO:tasks.workunit.client.0.smithi111.stdout:5/278: mknod d2/d16/d49/d33/c64 0 2024-03-14T13:02:09.669 INFO:tasks.workunit.client.1.smithi137.stdout:4/93: fsync d4/f1a 0 2024-03-14T13:02:09.669 INFO:tasks.workunit.client.1.smithi137.stdout:9/126: dread d6/fe [0,4194304] 0 2024-03-14T13:02:09.670 INFO:tasks.workunit.client.0.smithi111.stdout:6/180: link d0/c1b d0/c3b 0 2024-03-14T13:02:09.672 INFO:tasks.workunit.client.0.smithi111.stdout:5/279: rename d2/d13/c3f to d2/d16/d49/d44/c65 0 2024-03-14T13:02:09.674 INFO:tasks.workunit.client.0.smithi111.stdout:7/164: creat d0/d1/d21/f34 x:0 0 0 2024-03-14T13:02:09.676 INFO:tasks.workunit.client.1.smithi137.stdout:2/50: mknod d2/d7/c9 0 2024-03-14T13:02:09.677 INFO:tasks.workunit.client.1.smithi137.stdout:4/94: chown d4/f8 228274 1 2024-03-14T13:02:09.677 INFO:tasks.workunit.client.0.smithi111.stdout:5/280: creat d2/d45/f66 x:0 0 0 2024-03-14T13:02:09.678 INFO:tasks.workunit.client.0.smithi111.stdout:6/181: mknod d0/d17/d24/c3c 0 2024-03-14T13:02:09.682 INFO:tasks.workunit.client.1.smithi137.stdout:9/127: stat d6/df/d19/d1f/f24 0 2024-03-14T13:02:09.683 INFO:tasks.workunit.client.1.smithi137.stdout:3/103: dwrite d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:09.684 INFO:tasks.workunit.client.1.smithi137.stdout:4/95: chown d4/d9/lb 25029 1 2024-03-14T13:02:09.685 INFO:tasks.workunit.client.1.smithi137.stdout:3/104: readlink d4/la 0 2024-03-14T13:02:09.686 INFO:tasks.workunit.client.0.smithi111.stdout:6/182: mkdir d0/d17/d21/d37/d3d 0 2024-03-14T13:02:09.687 INFO:tasks.workunit.client.0.smithi111.stdout:9/228: write d5/fe [631765,72937] 0 2024-03-14T13:02:09.688 INFO:tasks.workunit.client.1.smithi137.stdout:1/97: dwrite d7/d9/fe [0,4194304] 0 2024-03-14T13:02:09.689 INFO:tasks.workunit.client.1.smithi137.stdout:9/128: chown d6/df/d19/l22 7373 1 2024-03-14T13:02:09.689 INFO:tasks.workunit.client.1.smithi137.stdout:3/105: chown d4/fd 60 1 2024-03-14T13:02:09.689 INFO:tasks.workunit.client.1.smithi137.stdout:4/96: dread - d4/d9/f18 zero size 2024-03-14T13:02:09.689 INFO:tasks.workunit.client.0.smithi111.stdout:6/183: creat d0/f3e x:0 0 0 2024-03-14T13:02:09.690 INFO:tasks.workunit.client.0.smithi111.stdout:7/165: dwrite d0/f2e [0,4194304] 0 2024-03-14T13:02:09.690 INFO:tasks.workunit.client.1.smithi137.stdout:3/106: rename d4/de to d4/de/d13/d1d 22 2024-03-14T13:02:09.690 INFO:tasks.workunit.client.1.smithi137.stdout:2/51: dread d2/f5 [0,4194304] 0 2024-03-14T13:02:09.691 INFO:tasks.workunit.client.1.smithi137.stdout:2/52: chown d2/d7 1599887 1 2024-03-14T13:02:09.692 INFO:tasks.workunit.client.1.smithi137.stdout:4/97: mkdir d4/d9/d1b 0 2024-03-14T13:02:09.692 INFO:tasks.workunit.client.0.smithi111.stdout:6/184: mknod d0/d17/d21/c3f 0 2024-03-14T13:02:09.694 INFO:tasks.workunit.client.1.smithi137.stdout:9/129: rmdir d6 39 2024-03-14T13:02:09.695 INFO:tasks.workunit.client.1.smithi137.stdout:1/98: mknod d7/c16 0 2024-03-14T13:02:09.696 INFO:tasks.workunit.client.1.smithi137.stdout:3/107: dread d4/de/d16/f1a [0,4194304] 0 2024-03-14T13:02:09.696 INFO:tasks.workunit.client.1.smithi137.stdout:5/48: sync 2024-03-14T13:02:09.699 INFO:tasks.workunit.client.1.smithi137.stdout:5/49: readlink l2 0 2024-03-14T13:02:09.699 INFO:tasks.workunit.client.1.smithi137.stdout:4/98: dread f2 [0,4194304] 0 2024-03-14T13:02:09.704 INFO:tasks.workunit.client.0.smithi111.stdout:6/185: rename d0/f2f to d0/d17/f40 0 2024-03-14T13:02:09.706 INFO:tasks.workunit.client.1.smithi137.stdout:5/50: rename d7/f8 to d7/ff 0 2024-03-14T13:02:09.706 INFO:tasks.workunit.client.1.smithi137.stdout:2/53: dread d2/f8 [0,4194304] 0 2024-03-14T13:02:09.707 INFO:tasks.workunit.client.0.smithi111.stdout:6/186: creat d0/d17/d21/d37/d3d/f41 x:0 0 0 2024-03-14T13:02:09.708 INFO:tasks.workunit.client.0.smithi111.stdout:6/187: mkdir d0/d17/d21/d42 0 2024-03-14T13:02:09.713 INFO:tasks.workunit.client.1.smithi137.stdout:9/130: link d6/df/d19/c1e d6/df/c30 0 2024-03-14T13:02:09.713 INFO:tasks.workunit.client.1.smithi137.stdout:3/108: dread d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:09.714 INFO:tasks.workunit.client.1.smithi137.stdout:3/109: dread - d4/fd zero size 2024-03-14T13:02:09.715 INFO:tasks.workunit.client.1.smithi137.stdout:5/51: link d7/fe d7/f10 0 2024-03-14T13:02:09.716 INFO:tasks.workunit.client.1.smithi137.stdout:5/52: write d7/fb [346513,73707] 0 2024-03-14T13:02:09.717 INFO:tasks.workunit.client.1.smithi137.stdout:5/53: chown d7/fe 33385 1 2024-03-14T13:02:09.717 INFO:tasks.workunit.client.0.smithi111.stdout:7/166: write d0/d1/d5/de/d9/f16 [891802,97080] 0 2024-03-14T13:02:09.718 INFO:tasks.workunit.client.1.smithi137.stdout:9/131: read d6/df/f13 [3217034,90966] 0 2024-03-14T13:02:09.718 INFO:tasks.workunit.client.1.smithi137.stdout:5/54: chown f5 0 1 2024-03-14T13:02:09.720 INFO:tasks.workunit.client.1.smithi137.stdout:1/99: rmdir d7/d9 39 2024-03-14T13:02:09.721 INFO:tasks.workunit.client.0.smithi111.stdout:5/281: dwrite d2/d1f/f28 [0,4194304] 0 2024-03-14T13:02:09.723 INFO:tasks.workunit.client.0.smithi111.stdout:5/282: readlink d2/d16/d49/d33/l39 0 2024-03-14T13:02:09.723 INFO:tasks.workunit.client.1.smithi137.stdout:5/55: fsync d7/fa 0 2024-03-14T13:02:09.723 INFO:tasks.workunit.client.1.smithi137.stdout:3/110: symlink d4/d19/l1e 0 2024-03-14T13:02:09.724 INFO:tasks.workunit.client.1.smithi137.stdout:9/132: mknod d6/c31 0 2024-03-14T13:02:09.725 INFO:tasks.workunit.client.0.smithi111.stdout:6/188: truncate d0/ff 1900463 0 2024-03-14T13:02:09.726 INFO:tasks.workunit.client.0.smithi111.stdout:5/283: write d2/d16/d49/d33/f5b [494536,80261] 0 2024-03-14T13:02:09.727 INFO:tasks.workunit.client.1.smithi137.stdout:9/133: dread - d6/f2c zero size 2024-03-14T13:02:09.727 INFO:tasks.workunit.client.0.smithi111.stdout:7/167: dwrite d0/d1/d5/f1a [4194304,4194304] 0 2024-03-14T13:02:09.728 INFO:tasks.workunit.client.1.smithi137.stdout:4/99: dwrite d4/f1a [4194304,4194304] 0 2024-03-14T13:02:09.729 INFO:tasks.workunit.client.0.smithi111.stdout:6/189: mknod d0/d17/d21/d37/c43 0 2024-03-14T13:02:09.729 INFO:tasks.workunit.client.0.smithi111.stdout:7/168: read d0/d1/d5/de/d9/f19 [639098,73609] 0 2024-03-14T13:02:09.729 INFO:tasks.workunit.client.0.smithi111.stdout:6/190: stat d0/f1c 0 2024-03-14T13:02:09.730 INFO:tasks.workunit.client.0.smithi111.stdout:5/284: symlink d2/d16/d49/d44/l67 0 2024-03-14T13:02:09.731 INFO:tasks.workunit.client.0.smithi111.stdout:6/191: fsync d0/d17/d21/f2d 0 2024-03-14T13:02:09.734 INFO:tasks.workunit.client.0.smithi111.stdout:5/285: truncate d2/d16/d49/d44/f50 121607 0 2024-03-14T13:02:09.736 INFO:tasks.workunit.client.0.smithi111.stdout:6/192: rename d0/d17/f2c to d0/d17/d21/f44 0 2024-03-14T13:02:09.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/286: chown d2/d16/d49/d44/d4a/f63 275789725 1 2024-03-14T13:02:09.738 INFO:tasks.workunit.client.1.smithi137.stdout:2/54: truncate d2/f5 1456506 0 2024-03-14T13:02:09.740 INFO:tasks.workunit.client.1.smithi137.stdout:9/134: readlink d6/df/l29 0 2024-03-14T13:02:09.740 INFO:tasks.workunit.client.1.smithi137.stdout:1/100: dread - d7/d9/f15 zero size 2024-03-14T13:02:09.740 INFO:tasks.workunit.client.1.smithi137.stdout:3/111: dwrite d4/f18 [0,4194304] 0 2024-03-14T13:02:09.740 INFO:tasks.workunit.client.0.smithi111.stdout:7/169: dwrite d0/d1/d5/de/d9/fb [0,4194304] 0 2024-03-14T13:02:09.741 INFO:tasks.workunit.client.1.smithi137.stdout:2/55: creat d2/fa x:0 0 0 2024-03-14T13:02:09.743 INFO:tasks.workunit.client.0.smithi111.stdout:5/287: write d2/d16/d49/d33/f40 [24680,4205] 0 2024-03-14T13:02:09.745 INFO:tasks.workunit.client.0.smithi111.stdout:6/193: creat d0/d17/d21/d37/f45 x:0 0 0 2024-03-14T13:02:09.745 INFO:tasks.workunit.client.0.smithi111.stdout:5/288: write d2/f35 [2010837,120741] 0 2024-03-14T13:02:09.746 INFO:tasks.workunit.client.1.smithi137.stdout:9/135: mkdir d6/d32 0 2024-03-14T13:02:09.747 INFO:tasks.workunit.client.0.smithi111.stdout:5/289: stat d2/d16/d49/d44 0 2024-03-14T13:02:09.748 INFO:tasks.workunit.client.1.smithi137.stdout:4/100: dwrite d4/f1a [0,4194304] 0 2024-03-14T13:02:09.748 INFO:tasks.workunit.client.0.smithi111.stdout:5/290: write d2/d16/f42 [1933520,54558] 0 2024-03-14T13:02:09.751 INFO:tasks.workunit.client.1.smithi137.stdout:5/56: getdents d7 0 2024-03-14T13:02:09.751 INFO:tasks.workunit.client.1.smithi137.stdout:1/101: dread d7/d9/fd [0,4194304] 0 2024-03-14T13:02:09.751 INFO:tasks.workunit.client.1.smithi137.stdout:2/56: write d2/f4 [4801148,11768] 0 2024-03-14T13:02:09.751 INFO:tasks.workunit.client.0.smithi111.stdout:5/291: write d2/d16/d49/f1e [4932445,34482] 0 2024-03-14T13:02:09.752 INFO:tasks.workunit.client.1.smithi137.stdout:9/136: mkdir d6/df/d19/d33 0 2024-03-14T13:02:09.754 INFO:tasks.workunit.client.0.smithi111.stdout:6/194: creat d0/d17/d24/f46 x:0 0 0 2024-03-14T13:02:09.755 INFO:tasks.workunit.client.0.smithi111.stdout:7/170: getdents d0/d1/d5/de 0 2024-03-14T13:02:09.760 INFO:tasks.workunit.client.0.smithi111.stdout:6/195: truncate d0/d17/f1f 292757 0 2024-03-14T13:02:09.761 INFO:tasks.workunit.client.1.smithi137.stdout:1/102: readlink d7/d9/l13 0 2024-03-14T13:02:09.761 INFO:tasks.workunit.client.1.smithi137.stdout:5/57: creat d7/f11 x:0 0 0 2024-03-14T13:02:09.761 INFO:tasks.workunit.client.1.smithi137.stdout:2/57: mkdir d2/d7/db 0 2024-03-14T13:02:09.763 INFO:tasks.workunit.client.0.smithi111.stdout:6/196: mkdir d0/d17/d21/d47 0 2024-03-14T13:02:09.765 INFO:tasks.workunit.client.0.smithi111.stdout:5/292: dwrite d2/d45/f34 [0,4194304] 0 2024-03-14T13:02:09.768 INFO:tasks.workunit.client.0.smithi111.stdout:6/197: creat d0/d17/d20/f48 x:0 0 0 2024-03-14T13:02:09.768 INFO:tasks.workunit.client.0.smithi111.stdout:7/171: dread d0/d1/d5/de/d9/fb [0,4194304] 0 2024-03-14T13:02:09.769 INFO:tasks.workunit.client.1.smithi137.stdout:2/58: rmdir d2/d7 39 2024-03-14T13:02:09.769 INFO:tasks.workunit.client.1.smithi137.stdout:9/137: rmdir d6/d32 0 2024-03-14T13:02:09.769 INFO:tasks.workunit.client.1.smithi137.stdout:4/101: dread f2 [0,4194304] 0 2024-03-14T13:02:09.769 INFO:tasks.workunit.client.1.smithi137.stdout:1/103: truncate d7/f10 1025879 0 2024-03-14T13:02:09.769 INFO:tasks.workunit.client.0.smithi111.stdout:5/293: mkdir d2/d16/d49/d33/d68 0 2024-03-14T13:02:09.771 INFO:tasks.workunit.client.0.smithi111.stdout:7/172: unlink d0/d1/f25 0 2024-03-14T13:02:09.772 INFO:tasks.workunit.client.1.smithi137.stdout:1/104: dread d7/d9/fd [0,4194304] 0 2024-03-14T13:02:09.774 INFO:tasks.workunit.client.1.smithi137.stdout:3/112: dwrite d4/de/d16/f17 [4194304,4194304] 0 2024-03-14T13:02:09.776 INFO:tasks.workunit.client.1.smithi137.stdout:6/65: sync 2024-03-14T13:02:09.776 INFO:tasks.workunit.client.1.smithi137.stdout:8/67: sync 2024-03-14T13:02:09.776 INFO:tasks.workunit.client.1.smithi137.stdout:7/81: sync 2024-03-14T13:02:09.777 INFO:tasks.workunit.client.1.smithi137.stdout:1/105: mknod d7/d9/c17 0 2024-03-14T13:02:09.778 INFO:tasks.workunit.client.0.smithi111.stdout:7/173: rename d0/d1/d22/l30 to d0/d11/l35 0 2024-03-14T13:02:09.778 INFO:tasks.workunit.client.0.smithi111.stdout:6/198: dwrite d0/d17/d20/f26 [0,4194304] 0 2024-03-14T13:02:09.778 INFO:tasks.workunit.client.1.smithi137.stdout:8/68: stat d1/f2 0 2024-03-14T13:02:09.778 INFO:tasks.workunit.client.1.smithi137.stdout:2/59: dread d2/f4 [0,4194304] 0 2024-03-14T13:02:09.779 INFO:tasks.workunit.client.1.smithi137.stdout:9/138: dread d6/f18 [0,4194304] 0 2024-03-14T13:02:09.780 INFO:tasks.workunit.client.1.smithi137.stdout:7/82: symlink d3/d5/l17 0 2024-03-14T13:02:09.781 INFO:tasks.workunit.client.0.smithi111.stdout:7/174: symlink d0/d1/d5/de/l36 0 2024-03-14T13:02:09.783 INFO:tasks.workunit.client.1.smithi137.stdout:4/102: dwrite d4/f11 [0,4194304] 0 2024-03-14T13:02:09.784 INFO:tasks.workunit.client.1.smithi137.stdout:6/66: write d2/f7 [2182584,70821] 0 2024-03-14T13:02:09.784 INFO:tasks.workunit.client.0.smithi111.stdout:7/175: symlink d0/d1/d5/l37 0 2024-03-14T13:02:09.790 INFO:tasks.workunit.client.1.smithi137.stdout:9/139: creat d6/df/d19/d1f/f34 x:0 0 0 2024-03-14T13:02:09.793 INFO:tasks.workunit.client.1.smithi137.stdout:2/60: mknod d2/cc 0 2024-03-14T13:02:09.794 INFO:tasks.workunit.client.1.smithi137.stdout:6/67: rename d2/d6 to d2/d6/de/d1b 22 2024-03-14T13:02:09.796 INFO:tasks.workunit.client.1.smithi137.stdout:7/83: creat d3/f18 x:0 0 0 2024-03-14T13:02:09.796 INFO:tasks.workunit.client.1.smithi137.stdout:6/68: chown d2/f11 154 1 2024-03-14T13:02:09.797 INFO:tasks.workunit.client.1.smithi137.stdout:6/69: chown d2/d6/de/c16 4635 1 2024-03-14T13:02:09.800 INFO:tasks.workunit.client.0.smithi111.stdout:6/199: rename d0/d17/d21/d37 to d0/d17/d21/d42/d49 0 2024-03-14T13:02:09.801 INFO:tasks.workunit.client.1.smithi137.stdout:4/103: truncate d4/d9/f18 158992 0 2024-03-14T13:02:09.801 INFO:tasks.workunit.client.0.smithi111.stdout:6/200: write d0/d17/d20/f26 [1489195,91801] 0 2024-03-14T13:02:09.806 INFO:tasks.workunit.client.1.smithi137.stdout:5/58: rmdir d7 39 2024-03-14T13:02:09.808 INFO:tasks.workunit.client.1.smithi137.stdout:3/113: dwrite d4/de/d16/f17 [4194304,4194304] 0 2024-03-14T13:02:09.811 INFO:tasks.workunit.client.1.smithi137.stdout:7/84: unlink d3/d5/fd 0 2024-03-14T13:02:09.812 INFO:tasks.workunit.client.1.smithi137.stdout:4/104: readlink d4/l19 0 2024-03-14T13:02:09.813 INFO:tasks.workunit.client.0.smithi111.stdout:7/176: rename d0/d1/d5/de to d0/d1/d5/d38 0 2024-03-14T13:02:09.814 INFO:tasks.workunit.client.1.smithi137.stdout:1/106: dwrite d7/d9/f15 [0,4194304] 0 2024-03-14T13:02:09.814 INFO:tasks.workunit.client.1.smithi137.stdout:8/69: dwrite d1/fb [4194304,4194304] 0 2024-03-14T13:02:09.816 INFO:tasks.workunit.client.1.smithi137.stdout:6/70: dwrite d2/d6/fc [0,4194304] 0 2024-03-14T13:02:09.818 INFO:tasks.workunit.client.0.smithi111.stdout:7/177: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:09.818 INFO:tasks.workunit.client.1.smithi137.stdout:2/61: mknod d2/d7/db/cd 0 2024-03-14T13:02:09.818 INFO:tasks.workunit.client.1.smithi137.stdout:2/62: dread - d2/fa zero size 2024-03-14T13:02:09.820 INFO:tasks.workunit.client.1.smithi137.stdout:1/107: read f5 [4686121,94774] 0 2024-03-14T13:02:09.822 INFO:tasks.workunit.client.0.smithi111.stdout:7/178: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:09.822 INFO:tasks.workunit.client.1.smithi137.stdout:3/114: write d4/de/d16/f1a [1019727,95750] 0 2024-03-14T13:02:09.825 INFO:tasks.workunit.client.1.smithi137.stdout:1/108: chown d7/d9/f15 98790932 1 2024-03-14T13:02:09.825 INFO:tasks.workunit.client.1.smithi137.stdout:6/71: rmdir d2 39 2024-03-14T13:02:09.826 INFO:tasks.workunit.client.1.smithi137.stdout:1/109: stat d7/d9/f14 0 2024-03-14T13:02:09.827 INFO:tasks.workunit.client.0.smithi111.stdout:7/179: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:09.827 INFO:tasks.workunit.client.1.smithi137.stdout:5/59: rename d7/l9 to d7/l12 0 2024-03-14T13:02:09.827 INFO:tasks.workunit.client.0.smithi111.stdout:7/180: fdatasync d0/f2e 0 2024-03-14T13:02:09.829 INFO:tasks.workunit.client.1.smithi137.stdout:5/60: chown c1 464788902 1 2024-03-14T13:02:09.832 INFO:tasks.workunit.client.1.smithi137.stdout:7/85: dwrite d3/ff [0,4194304] 0 2024-03-14T13:02:09.840 INFO:tasks.workunit.client.1.smithi137.stdout:1/110: creat d7/d9/f18 x:0 0 0 2024-03-14T13:02:09.841 INFO:tasks.workunit.client.1.smithi137.stdout:6/72: symlink d2/d6/de/l1c 0 2024-03-14T13:02:09.842 INFO:tasks.workunit.client.1.smithi137.stdout:7/86: dwrite d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:09.842 INFO:tasks.workunit.client.1.smithi137.stdout:8/70: getdents d1/dc 0 2024-03-14T13:02:09.843 INFO:tasks.workunit.client.1.smithi137.stdout:4/105: link d4/ld d4/d9/l1c 0 2024-03-14T13:02:09.844 INFO:tasks.workunit.client.0.smithi111.stdout:7/181: truncate d0/d1/f2 1516829 0 2024-03-14T13:02:09.845 INFO:tasks.workunit.client.0.smithi111.stdout:7/182: chown d0/d1/d21 57974041 1 2024-03-14T13:02:09.848 INFO:tasks.workunit.client.0.smithi111.stdout:7/183: dread d0/f2e [0,4194304] 0 2024-03-14T13:02:09.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/73: mkdir d2/d6/d1d 0 2024-03-14T13:02:09.849 INFO:tasks.workunit.client.0.smithi111.stdout:7/184: creat d0/d1/d22/f39 x:0 0 0 2024-03-14T13:02:09.850 INFO:tasks.workunit.client.1.smithi137.stdout:7/87: creat d3/d5/f19 x:0 0 0 2024-03-14T13:02:09.850 INFO:tasks.workunit.client.1.smithi137.stdout:6/74: write d2/f11 [1616194,38845] 0 2024-03-14T13:02:09.851 INFO:tasks.workunit.client.0.smithi111.stdout:7/185: read d0/d1/d5/d38/d9/fb [767350,22534] 0 2024-03-14T13:02:09.852 INFO:tasks.workunit.client.1.smithi137.stdout:8/71: mkdir d1/dc/d11/d13 0 2024-03-14T13:02:09.852 INFO:tasks.workunit.client.0.smithi111.stdout:7/186: fdatasync d0/d1/d22/f39 0 2024-03-14T13:02:09.852 INFO:tasks.workunit.client.0.smithi111.stdout:4/451: sync 2024-03-14T13:02:09.852 INFO:tasks.workunit.client.0.smithi111.stdout:2/141: sync 2024-03-14T13:02:09.854 INFO:tasks.workunit.client.0.smithi111.stdout:4/452: write d1d/d22/d30/d5e/d6f/d7e/f65 [820260,110826] 0 2024-03-14T13:02:09.857 INFO:tasks.workunit.client.1.smithi137.stdout:7/88: unlink d3/c13 0 2024-03-14T13:02:09.857 INFO:tasks.workunit.client.0.smithi111.stdout:4/453: truncate d1d/d22/d30/f32 1399406 0 2024-03-14T13:02:09.857 INFO:tasks.workunit.client.0.smithi111.stdout:7/187: unlink d0/d1/d5/d38/d9/f16 0 2024-03-14T13:02:09.858 INFO:tasks.workunit.client.1.smithi137.stdout:1/111: dwrite d7/d9/f18 [0,4194304] 0 2024-03-14T13:02:09.858 INFO:tasks.workunit.client.0.smithi111.stdout:4/454: fsync d1d/d22/f41 0 2024-03-14T13:02:09.859 INFO:tasks.workunit.client.1.smithi137.stdout:4/106: dread d4/f12 [0,4194304] 0 2024-03-14T13:02:09.859 INFO:tasks.workunit.client.1.smithi137.stdout:7/89: chown d3 42 1 2024-03-14T13:02:09.860 INFO:tasks.workunit.client.1.smithi137.stdout:1/112: mkdir d7/d9/d19 0 2024-03-14T13:02:09.861 INFO:tasks.workunit.client.1.smithi137.stdout:6/75: dread d2/f17 [0,4194304] 0 2024-03-14T13:02:09.862 INFO:tasks.workunit.client.1.smithi137.stdout:6/76: stat d2/d6/d8/l1a 0 2024-03-14T13:02:09.864 INFO:tasks.workunit.client.0.smithi111.stdout:4/455: rename d1d/d22/d40/c42 to d1d/d22/d28/ca4 0 2024-03-14T13:02:09.866 INFO:tasks.workunit.client.0.smithi111.stdout:4/456: symlink d1d/d48/la5 0 2024-03-14T13:02:09.867 INFO:tasks.workunit.client.0.smithi111.stdout:4/457: fsync d1d/f2e 0 2024-03-14T13:02:09.867 INFO:tasks.workunit.client.1.smithi137.stdout:7/90: unlink d3/d6/l7 0 2024-03-14T13:02:09.868 INFO:tasks.workunit.client.0.smithi111.stdout:2/142: write dd/f10 [4577414,23232] 0 2024-03-14T13:02:09.870 INFO:tasks.workunit.client.1.smithi137.stdout:6/77: creat d2/d6/d8/f1e x:0 0 0 2024-03-14T13:02:09.871 INFO:tasks.workunit.client.1.smithi137.stdout:1/113: mknod d7/c1a 0 2024-03-14T13:02:09.872 INFO:tasks.workunit.client.1.smithi137.stdout:4/107: link d4/l19 d4/d9/l1d 0 2024-03-14T13:02:09.873 INFO:tasks.workunit.client.0.smithi111.stdout:2/143: rename dd/d24/l20 to dd/d11/d1a/l2b 0 2024-03-14T13:02:09.874 INFO:tasks.workunit.client.0.smithi111.stdout:2/144: write dd/d11/d1a/f1b [819428,66854] 0 2024-03-14T13:02:09.874 INFO:tasks.workunit.client.1.smithi137.stdout:7/91: creat d3/d6/d14/f1a x:0 0 0 2024-03-14T13:02:09.874 INFO:tasks.workunit.client.1.smithi137.stdout:4/108: stat d4/d10/c17 0 2024-03-14T13:02:09.877 INFO:tasks.workunit.client.0.smithi111.stdout:7/188: truncate d0/d1/f2 1178356 0 2024-03-14T13:02:09.877 INFO:tasks.workunit.client.0.smithi111.stdout:7/189: read - d0/d1/d22/f39 zero size 2024-03-14T13:02:09.879 INFO:tasks.workunit.client.0.smithi111.stdout:7/190: read d0/d1/d5/f1a [3965130,102190] 0 2024-03-14T13:02:09.880 INFO:tasks.workunit.client.0.smithi111.stdout:7/191: chown d0/d11/l35 52309062 1 2024-03-14T13:02:09.881 INFO:tasks.workunit.client.0.smithi111.stdout:7/192: chown d0/f2e 95269 1 2024-03-14T13:02:09.882 INFO:tasks.workunit.client.0.smithi111.stdout:7/193: dread - d0/d1/d21/f34 zero size 2024-03-14T13:02:09.885 INFO:tasks.workunit.client.0.smithi111.stdout:7/194: mkdir d0/d1/d21/d3a 0 2024-03-14T13:02:09.887 INFO:tasks.workunit.client.0.smithi111.stdout:4/458: dread d1d/d22/d30/d5e/f84 [0,4194304] 0 2024-03-14T13:02:09.888 INFO:tasks.workunit.client.0.smithi111.stdout:4/459: write d1d/d22/d23/f46 [8680035,2656] 0 2024-03-14T13:02:09.890 INFO:tasks.workunit.client.0.smithi111.stdout:7/195: write d0/d1/f28 [473663,10414] 0 2024-03-14T13:02:09.891 INFO:tasks.workunit.client.0.smithi111.stdout:7/196: fsync d0/d1/d21/f15 0 2024-03-14T13:02:09.895 INFO:tasks.workunit.client.0.smithi111.stdout:4/460: dwrite d1d/f2e [0,4194304] 0 2024-03-14T13:02:09.895 INFO:tasks.workunit.client.0.smithi111.stdout:7/197: symlink d0/d1/d5/d38/l3b 0 2024-03-14T13:02:09.898 INFO:tasks.workunit.client.0.smithi111.stdout:4/461: mknod d1d/d22/d23/ca6 0 2024-03-14T13:02:09.900 INFO:tasks.workunit.client.0.smithi111.stdout:7/198: rename d0/d1/d5/d38/d9/l20 to d0/d1/d21/d2a/l3c 0 2024-03-14T13:02:09.900 INFO:tasks.workunit.client.0.smithi111.stdout:7/199: chown d0/d1/d5/d38 2327867 1 2024-03-14T13:02:09.901 INFO:tasks.workunit.client.0.smithi111.stdout:4/462: write d1d/d22/d28/f4c [444296,34357] 0 2024-03-14T13:02:09.903 INFO:tasks.workunit.client.0.smithi111.stdout:7/200: creat d0/d1/d5/d38/d9/f3d x:0 0 0 2024-03-14T13:02:09.904 INFO:tasks.workunit.client.0.smithi111.stdout:4/463: mknod d1d/d22/d30/d7c/ca7 0 2024-03-14T13:02:09.911 INFO:tasks.workunit.client.0.smithi111.stdout:7/201: dwrite d0/d1/d5/d38/d9/f3d [0,4194304] 0 2024-03-14T13:02:09.913 INFO:tasks.workunit.client.0.smithi111.stdout:7/202: mkdir d0/d1/d22/d3e 0 2024-03-14T13:02:09.917 INFO:tasks.workunit.client.1.smithi137.stdout:0/50: sync 2024-03-14T13:02:09.917 INFO:tasks.workunit.client.0.smithi111.stdout:4/464: truncate d1d/d22/d30/d7c/f5c 192544 0 2024-03-14T13:02:09.917 INFO:tasks.workunit.client.0.smithi111.stdout:7/203: stat d0/d1/d5/d38/d9/f12 0 2024-03-14T13:02:09.919 INFO:tasks.workunit.client.0.smithi111.stdout:4/465: rmdir d1d/d22/d23 39 2024-03-14T13:02:09.921 INFO:tasks.workunit.client.0.smithi111.stdout:4/466: write d1d/d22/d28/f4c [175398,9670] 0 2024-03-14T13:02:09.922 INFO:tasks.workunit.client.0.smithi111.stdout:7/204: creat d0/d1/d21/f3f x:0 0 0 2024-03-14T13:02:09.924 INFO:tasks.workunit.client.0.smithi111.stdout:4/467: mknod d1d/d22/d23/ca8 0 2024-03-14T13:02:09.926 INFO:tasks.workunit.client.1.smithi137.stdout:4/109: fsync d4/d10/f14 0 2024-03-14T13:02:09.926 INFO:tasks.workunit.client.0.smithi111.stdout:4/468: mknod d1d/d22/d56/ca9 0 2024-03-14T13:02:09.926 INFO:tasks.workunit.client.1.smithi137.stdout:1/114: truncate d7/d9/ff 766824 0 2024-03-14T13:02:09.927 INFO:tasks.workunit.client.1.smithi137.stdout:6/78: creat d2/d6/d8/f1f x:0 0 0 2024-03-14T13:02:09.927 INFO:tasks.workunit.client.1.smithi137.stdout:0/51: chown d1/d5 12 1 2024-03-14T13:02:09.927 INFO:tasks.workunit.client.0.smithi111.stdout:4/469: creat d1d/d22/d40/d69/d86/faa x:0 0 0 2024-03-14T13:02:09.928 INFO:tasks.workunit.client.0.smithi111.stdout:4/470: fdatasync d1d/d22/d23/f34 0 2024-03-14T13:02:09.929 INFO:tasks.workunit.client.1.smithi137.stdout:4/110: rename d4/d9 to d4/d9/d1b/d1e 22 2024-03-14T13:02:09.931 INFO:tasks.workunit.client.1.smithi137.stdout:2/63: dwrite d2/f5 [0,4194304] 0 2024-03-14T13:02:09.931 INFO:tasks.workunit.client.1.smithi137.stdout:0/52: symlink d1/lf 0 2024-03-14T13:02:09.931 INFO:tasks.workunit.client.1.smithi137.stdout:6/79: fsync d2/d6/fc 0 2024-03-14T13:02:09.932 INFO:tasks.workunit.client.1.smithi137.stdout:2/64: chown l1 96353222 1 2024-03-14T13:02:09.933 INFO:tasks.workunit.client.1.smithi137.stdout:1/115: mknod d7/d9/d19/c1b 0 2024-03-14T13:02:09.934 INFO:tasks.workunit.client.1.smithi137.stdout:7/92: getdents d3 0 2024-03-14T13:02:09.935 INFO:tasks.workunit.client.1.smithi137.stdout:2/65: stat d2/d7/db/cd 0 2024-03-14T13:02:09.935 INFO:tasks.workunit.client.1.smithi137.stdout:4/111: creat d4/d9/f1f x:0 0 0 2024-03-14T13:02:09.936 INFO:tasks.workunit.client.1.smithi137.stdout:0/53: mkdir d1/d5/d10 0 2024-03-14T13:02:09.937 INFO:tasks.workunit.client.1.smithi137.stdout:4/112: chown d4/d9/fc 991828 1 2024-03-14T13:02:09.937 INFO:tasks.workunit.client.1.smithi137.stdout:6/80: truncate d2/d6/d8/f1f 4511 0 2024-03-14T13:02:09.940 INFO:tasks.workunit.client.1.smithi137.stdout:1/116: rename d7/d9/fe to d7/d9/d19/f1c 0 2024-03-14T13:02:09.940 INFO:tasks.workunit.client.0.smithi111.stdout:7/205: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:09.942 INFO:tasks.workunit.client.0.smithi111.stdout:7/206: readlink d0/d11/l35 0 2024-03-14T13:02:09.944 INFO:tasks.workunit.client.0.smithi111.stdout:4/471: dread d1d/d22/d28/f3e [0,4194304] 0 2024-03-14T13:02:09.945 INFO:tasks.workunit.client.1.smithi137.stdout:4/113: mknod d4/d9/c20 0 2024-03-14T13:02:09.945 INFO:tasks.workunit.client.1.smithi137.stdout:0/54: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:09.947 INFO:tasks.workunit.client.0.smithi111.stdout:4/472: creat d1d/d22/d40/fab x:0 0 0 2024-03-14T13:02:09.948 INFO:tasks.workunit.client.0.smithi111.stdout:4/473: chown d1d/d22/d40/d69/d86/faa 42451632 1 2024-03-14T13:02:09.948 INFO:tasks.workunit.client.0.smithi111.stdout:4/474: chown d1d/d22/d56/f7b 0 1 2024-03-14T13:02:09.950 INFO:tasks.workunit.client.0.smithi111.stdout:4/475: mknod d1d/d22/d30/cac 0 2024-03-14T13:02:09.952 INFO:tasks.workunit.client.1.smithi137.stdout:6/81: symlink d2/d6/l20 0 2024-03-14T13:02:09.953 INFO:tasks.workunit.client.0.smithi111.stdout:7/207: truncate d0/d1/d5/d38/d9/f12 2972346 0 2024-03-14T13:02:09.953 INFO:tasks.workunit.client.1.smithi137.stdout:7/93: link d3/d5/f16 d3/d6/d14/f1b 0 2024-03-14T13:02:09.954 INFO:tasks.workunit.client.1.smithi137.stdout:0/55: unlink d1/d5/lc 0 2024-03-14T13:02:09.954 INFO:tasks.workunit.client.0.smithi111.stdout:7/208: chown d0/d1 13106365 1 2024-03-14T13:02:09.955 INFO:tasks.workunit.client.0.smithi111.stdout:7/209: fdatasync d0/d1/d21/f3f 0 2024-03-14T13:02:09.955 INFO:tasks.workunit.client.1.smithi137.stdout:2/66: dwrite d2/f8 [0,4194304] 0 2024-03-14T13:02:09.957 INFO:tasks.workunit.client.0.smithi111.stdout:7/210: symlink d0/d1/d5/d38/l40 0 2024-03-14T13:02:09.958 INFO:tasks.workunit.client.1.smithi137.stdout:2/67: truncate d2/fa 763341 0 2024-03-14T13:02:09.958 INFO:tasks.workunit.client.0.smithi111.stdout:4/476: truncate fb 536908 0 2024-03-14T13:02:09.959 INFO:tasks.workunit.client.1.smithi137.stdout:0/56: write d1/fd [834712,1895] 0 2024-03-14T13:02:09.959 INFO:tasks.workunit.client.0.smithi111.stdout:7/211: truncate d0/d1/d5/f1a 6233274 0 2024-03-14T13:02:09.960 INFO:tasks.workunit.client.1.smithi137.stdout:6/82: mknod d2/c21 0 2024-03-14T13:02:09.962 INFO:tasks.workunit.client.1.smithi137.stdout:2/68: unlink d2/d7/c9 0 2024-03-14T13:02:09.963 INFO:tasks.workunit.client.1.smithi137.stdout:2/69: chown d2 1699 1 2024-03-14T13:02:09.964 INFO:tasks.workunit.client.1.smithi137.stdout:2/70: stat d2/d7/db/cd 0 2024-03-14T13:02:09.964 INFO:tasks.workunit.client.1.smithi137.stdout:0/57: write d1/f8 [3372389,110980] 0 2024-03-14T13:02:09.964 INFO:tasks.workunit.client.1.smithi137.stdout:2/71: readlink d2/l6 0 2024-03-14T13:02:09.965 INFO:tasks.workunit.client.0.smithi111.stdout:4/477: dread d1d/d22/d28/f3e [0,4194304] 0 2024-03-14T13:02:09.965 INFO:tasks.workunit.client.1.smithi137.stdout:6/83: mkdir d2/d6/d1d/d22 0 2024-03-14T13:02:09.967 INFO:tasks.workunit.client.1.smithi137.stdout:1/117: write f4 [177588,33359] 0 2024-03-14T13:02:09.967 INFO:tasks.workunit.client.0.smithi111.stdout:4/478: write d1d/d22/d30/d7c/f93 [871605,69180] 0 2024-03-14T13:02:09.968 INFO:tasks.workunit.client.0.smithi111.stdout:4/479: fsync fe 0 2024-03-14T13:02:09.969 INFO:tasks.workunit.client.1.smithi137.stdout:2/72: write d2/f4 [1852081,14192] 0 2024-03-14T13:02:09.969 INFO:tasks.workunit.client.0.smithi111.stdout:7/212: dwrite d0/d1/d5/d38/d9/fb [4194304,4194304] 0 2024-03-14T13:02:09.970 INFO:tasks.workunit.client.1.smithi137.stdout:1/118: fsync d7/f10 0 2024-03-14T13:02:09.970 INFO:tasks.workunit.client.1.smithi137.stdout:6/84: write d2/d6/d8/f15 [1047997,75613] 0 2024-03-14T13:02:09.971 INFO:tasks.workunit.client.1.smithi137.stdout:4/114: write d4/d9/fc [3213013,107371] 0 2024-03-14T13:02:09.972 INFO:tasks.workunit.client.1.smithi137.stdout:1/119: chown d7/c1a 23202523 1 2024-03-14T13:02:09.972 INFO:tasks.workunit.client.0.smithi111.stdout:4/480: truncate f11 256040 0 2024-03-14T13:02:09.973 INFO:tasks.workunit.client.1.smithi137.stdout:0/58: dread d1/fd [0,4194304] 0 2024-03-14T13:02:09.974 INFO:tasks.workunit.client.1.smithi137.stdout:0/59: readlink d1/l3 0 2024-03-14T13:02:09.976 INFO:tasks.workunit.client.0.smithi111.stdout:4/481: write d1d/d22/d30/d5e/d6f/d7e/d66/f79 [233952,65755] 0 2024-03-14T13:02:09.980 INFO:tasks.workunit.client.1.smithi137.stdout:2/73: read d2/fa [493444,93899] 0 2024-03-14T13:02:09.980 INFO:tasks.workunit.client.1.smithi137.stdout:6/85: rmdir d2/d6/d1d/d22 0 2024-03-14T13:02:09.983 INFO:tasks.workunit.client.0.smithi111.stdout:4/482: rmdir d1d/d22/d30 39 2024-03-14T13:02:09.985 INFO:tasks.workunit.client.1.smithi137.stdout:0/60: rmdir d1/d5 39 2024-03-14T13:02:09.986 INFO:tasks.workunit.client.0.smithi111.stdout:4/483: write d1d/d22/d56/f7b [797387,111711] 0 2024-03-14T13:02:09.987 INFO:tasks.workunit.client.0.smithi111.stdout:7/213: dwrite d0/d1/d5/d38/d9/f19 [4194304,4194304] 0 2024-03-14T13:02:09.989 INFO:tasks.workunit.client.1.smithi137.stdout:4/115: rename d4/d10/f14 to d4/d9/d1b/f21 0 2024-03-14T13:02:09.990 INFO:tasks.workunit.client.1.smithi137.stdout:7/94: dwrite d3/fe [0,4194304] 0 2024-03-14T13:02:09.994 INFO:tasks.workunit.client.0.smithi111.stdout:4/484: creat d1d/d22/fad x:0 0 0 2024-03-14T13:02:09.996 INFO:tasks.workunit.client.0.smithi111.stdout:4/485: mknod d1d/d22/d40/d69/cae 0 2024-03-14T13:02:09.996 INFO:tasks.workunit.client.1.smithi137.stdout:0/61: mkdir d1/d11 0 2024-03-14T13:02:09.996 INFO:tasks.workunit.client.1.smithi137.stdout:4/116: mknod d4/d9/d1b/c22 0 2024-03-14T13:02:09.998 INFO:tasks.workunit.client.1.smithi137.stdout:1/120: dread d7/d9/f18 [0,4194304] 0 2024-03-14T13:02:09.998 INFO:tasks.workunit.client.0.smithi111.stdout:4/486: rmdir d1d/d22/d30/d87 39 2024-03-14T13:02:09.998 INFO:tasks.workunit.client.0.smithi111.stdout:4/487: readlink d1d/d22/d40/l99 0 2024-03-14T13:02:10.003 INFO:tasks.workunit.client.1.smithi137.stdout:7/95: creat d3/f1c x:0 0 0 2024-03-14T13:02:10.003 INFO:tasks.workunit.client.0.smithi111.stdout:4/488: unlink d1d/d22/d30/d5e/d6f/d7e/l96 0 2024-03-14T13:02:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:09 smithi137 ceph-mon[50336]: pgmap v19: 65 pgs: 65 active+clean; 553 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.6 MiB/s rd, 42 MiB/s wr, 182 op/s 2024-03-14T13:02:10.006 INFO:tasks.workunit.client.1.smithi137.stdout:2/74: dwrite d2/fa [0,4194304] 0 2024-03-14T13:02:10.006 INFO:tasks.workunit.client.0.smithi111.stdout:4/489: rename d1d/d48/l89 to d1d/d22/d23/laf 0 2024-03-14T13:02:10.007 INFO:tasks.workunit.client.1.smithi137.stdout:2/75: write d2/f8 [3193797,21892] 0 2024-03-14T13:02:10.007 INFO:tasks.workunit.client.1.smithi137.stdout:4/117: symlink d4/d10/l23 0 2024-03-14T13:02:10.009 INFO:tasks.workunit.client.0.smithi111.stdout:4/490: creat d1d/d22/d28/fb0 x:0 0 0 2024-03-14T13:02:10.011 INFO:tasks.workunit.client.1.smithi137.stdout:0/62: unlink d1/d5/d9/la 0 2024-03-14T13:02:10.011 INFO:tasks.workunit.client.0.smithi111.stdout:4/491: mkdir d1d/d22/d40/d69/d86/db1 0 2024-03-14T13:02:10.013 INFO:tasks.workunit.client.1.smithi137.stdout:4/118: write d4/d9/f16 [2831908,81741] 0 2024-03-14T13:02:10.013 INFO:tasks.workunit.client.0.smithi111.stdout:4/492: creat d1d/d22/d23/fb2 x:0 0 0 2024-03-14T13:02:10.014 INFO:tasks.workunit.client.1.smithi137.stdout:1/121: creat d7/f1d x:0 0 0 2024-03-14T13:02:10.015 INFO:tasks.workunit.client.1.smithi137.stdout:4/119: chown d4/ld 1389175051 1 2024-03-14T13:02:10.016 INFO:tasks.workunit.client.1.smithi137.stdout:0/63: creat d1/f12 x:0 0 0 2024-03-14T13:02:10.017 INFO:tasks.workunit.client.1.smithi137.stdout:4/120: write d4/f8 [176081,20774] 0 2024-03-14T13:02:10.017 INFO:tasks.workunit.client.0.smithi111.stdout:4/493: rename ff to d1d/d22/d30/d5e/fb3 0 2024-03-14T13:02:10.017 INFO:tasks.workunit.client.0.smithi111.stdout:4/494: stat fe 0 2024-03-14T13:02:10.019 INFO:tasks.workunit.client.1.smithi137.stdout:1/122: symlink d7/l1e 0 2024-03-14T13:02:10.025 INFO:tasks.workunit.client.1.smithi137.stdout:4/121: rename d4/d9/c20 to d4/c24 0 2024-03-14T13:02:10.026 INFO:tasks.workunit.client.1.smithi137.stdout:6/86: truncate d2/d6/d8/f15 1093560 0 2024-03-14T13:02:10.028 INFO:tasks.workunit.client.1.smithi137.stdout:4/122: write d4/d9/f16 [868676,44239] 0 2024-03-14T13:02:10.028 INFO:tasks.workunit.client.1.smithi137.stdout:6/87: fdatasync d2/d6/fc 0 2024-03-14T13:02:10.032 INFO:tasks.workunit.client.1.smithi137.stdout:2/76: truncate d2/f4 467730 0 2024-03-14T13:02:10.032 INFO:tasks.workunit.client.1.smithi137.stdout:1/123: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:10.034 INFO:tasks.workunit.client.1.smithi137.stdout:0/64: truncate d1/f8 2226228 0 2024-03-14T13:02:10.035 INFO:tasks.workunit.client.1.smithi137.stdout:1/124: fsync f4 0 2024-03-14T13:02:10.036 INFO:tasks.workunit.client.1.smithi137.stdout:6/88: truncate d2/d6/de/f13 493426 0 2024-03-14T13:02:10.037 INFO:tasks.workunit.client.0.smithi111.stdout:4/495: dwrite d1d/d22/d30/d7c/f5c [0,4194304] 0 2024-03-14T13:02:10.039 INFO:tasks.workunit.client.1.smithi137.stdout:2/77: mkdir d2/d7/db/de 0 2024-03-14T13:02:10.041 INFO:tasks.workunit.client.1.smithi137.stdout:7/96: write d3/d5/f16 [289170,34913] 0 2024-03-14T13:02:10.043 INFO:tasks.workunit.client.1.smithi137.stdout:1/125: mkdir d7/d9/d19/d1f 0 2024-03-14T13:02:10.044 INFO:tasks.workunit.client.1.smithi137.stdout:4/123: link d4/d9/fc d4/d10/f25 0 2024-03-14T13:02:10.044 INFO:tasks.workunit.client.1.smithi137.stdout:0/65: getdents d1/d5/d10 0 2024-03-14T13:02:10.047 INFO:tasks.workunit.client.1.smithi137.stdout:2/78: creat d2/d7/db/ff x:0 0 0 2024-03-14T13:02:10.048 INFO:tasks.workunit.client.1.smithi137.stdout:4/124: fdatasync f2 0 2024-03-14T13:02:10.048 INFO:tasks.workunit.client.1.smithi137.stdout:6/89: dwrite f0 [4194304,4194304] 0 2024-03-14T13:02:10.049 INFO:tasks.workunit.client.0.smithi111.stdout:4/496: dwrite f11 [0,4194304] 0 2024-03-14T13:02:10.049 INFO:tasks.workunit.client.0.smithi111.stdout:4/497: chown d1d/d22 360958492 1 2024-03-14T13:02:10.051 INFO:tasks.workunit.client.1.smithi137.stdout:1/126: write d7/d9/f18 [4919903,79843] 0 2024-03-14T13:02:10.051 INFO:tasks.workunit.client.1.smithi137.stdout:1/127: readlink d7/l1e 0 2024-03-14T13:02:10.051 INFO:tasks.workunit.client.1.smithi137.stdout:4/125: mkdir d4/d10/d26 0 2024-03-14T13:02:10.053 INFO:tasks.workunit.client.1.smithi137.stdout:7/97: creat d3/d6/d14/f1d x:0 0 0 2024-03-14T13:02:10.053 INFO:tasks.workunit.client.0.smithi111.stdout:4/498: getdents d1d/d22/d56 0 2024-03-14T13:02:10.053 INFO:tasks.workunit.client.1.smithi137.stdout:6/90: symlink d2/d6/l23 0 2024-03-14T13:02:10.054 INFO:tasks.workunit.client.0.smithi111.stdout:4/499: stat d1d/d22/d23/fb2 0 2024-03-14T13:02:10.054 INFO:tasks.workunit.client.1.smithi137.stdout:7/98: write d3/d5/f16 [5948,44458] 0 2024-03-14T13:02:10.055 INFO:tasks.workunit.client.1.smithi137.stdout:2/79: dread d2/fa [0,4194304] 0 2024-03-14T13:02:10.056 INFO:tasks.workunit.client.0.smithi111.stdout:4/500: symlink d1d/d22/lb4 0 2024-03-14T13:02:10.056 INFO:tasks.workunit.client.0.smithi111.stdout:4/501: fdatasync d1d/d22/d23/f34 0 2024-03-14T13:02:10.058 INFO:tasks.workunit.client.1.smithi137.stdout:1/128: dread f3 [0,4194304] 0 2024-03-14T13:02:10.058 INFO:tasks.workunit.client.1.smithi137.stdout:9/140: sync 2024-03-14T13:02:10.058 INFO:tasks.workunit.client.1.smithi137.stdout:6/91: rename d2/cd to d2/d6/c24 0 2024-03-14T13:02:10.060 INFO:tasks.workunit.client.1.smithi137.stdout:4/126: mkdir d4/d27 0 2024-03-14T13:02:10.065 INFO:tasks.workunit.client.0.smithi111.stdout:4/502: dwrite d1d/d22/d28/f68 [0,4194304] 0 2024-03-14T13:02:10.066 INFO:tasks.workunit.client.1.smithi137.stdout:9/141: unlink d6/c16 0 2024-03-14T13:02:10.066 INFO:tasks.workunit.client.0.smithi111.stdout:4/503: fdatasync d1d/d22/d30/d5e/d6f/d7e/d66/f79 0 2024-03-14T13:02:10.067 INFO:tasks.workunit.client.0.smithi111.stdout:4/504: dread - d1d/d22/fad zero size 2024-03-14T13:02:10.070 INFO:tasks.workunit.client.1.smithi137.stdout:6/92: dwrite d2/f17 [0,4194304] 0 2024-03-14T13:02:10.070 INFO:tasks.workunit.client.0.smithi111.stdout:4/505: symlink d1d/d22/d30/d7c/lb5 0 2024-03-14T13:02:10.070 INFO:tasks.workunit.client.1.smithi137.stdout:2/80: dread d2/f8 [0,4194304] 0 2024-03-14T13:02:10.072 INFO:tasks.workunit.client.1.smithi137.stdout:1/129: symlink d7/l20 0 2024-03-14T13:02:10.074 INFO:tasks.workunit.client.0.smithi111.stdout:4/506: rename d1d/d22/f3b to d1d/d22/d28/fb6 0 2024-03-14T13:02:10.076 INFO:tasks.workunit.client.1.smithi137.stdout:4/127: creat d4/d27/f28 x:0 0 0 2024-03-14T13:02:10.079 INFO:tasks.workunit.client.0.smithi111.stdout:4/507: rename d1d/d22/d23/f46 to d1d/d22/d30/d7c/d59/fb7 0 2024-03-14T13:02:10.080 INFO:tasks.workunit.client.1.smithi137.stdout:2/81: fsync d2/fa 0 2024-03-14T13:02:10.081 INFO:tasks.workunit.client.1.smithi137.stdout:4/128: dread d4/d9/d1b/f21 [0,4194304] 0 2024-03-14T13:02:10.082 INFO:tasks.workunit.client.1.smithi137.stdout:7/99: rename d3/d5/f16 to d3/f1e 0 2024-03-14T13:02:10.084 INFO:tasks.workunit.client.0.smithi111.stdout:4/508: rmdir d1d/d22/d40/d69/d86 39 2024-03-14T13:02:10.085 INFO:tasks.workunit.client.1.smithi137.stdout:9/142: unlink d6/df/f17 0 2024-03-14T13:02:10.086 INFO:tasks.workunit.client.0.smithi111.stdout:4/509: unlink d1d/d22/d23/c2a 0 2024-03-14T13:02:10.086 INFO:tasks.workunit.client.0.smithi111.stdout:4/510: chown d1d/d22/d56/ca9 162834 1 2024-03-14T13:02:10.087 INFO:tasks.workunit.client.1.smithi137.stdout:9/143: write d6/f1d [1035316,53976] 0 2024-03-14T13:02:10.088 INFO:tasks.workunit.client.1.smithi137.stdout:2/82: creat d2/d7/f10 x:0 0 0 2024-03-14T13:02:10.089 INFO:tasks.workunit.client.1.smithi137.stdout:2/83: readlink l1 0 2024-03-14T13:02:10.089 INFO:tasks.workunit.client.1.smithi137.stdout:4/129: mknod d4/d9/d1b/c29 0 2024-03-14T13:02:10.090 INFO:tasks.workunit.client.1.smithi137.stdout:4/130: chown d4/d9/d1b 1065 1 2024-03-14T13:02:10.090 INFO:tasks.workunit.client.0.smithi111.stdout:4/511: dread d1d/d22/d30/d5e/f84 [0,4194304] 0 2024-03-14T13:02:10.092 INFO:tasks.workunit.client.1.smithi137.stdout:1/130: mknod d7/d9/d19/d1f/c21 0 2024-03-14T13:02:10.093 INFO:tasks.workunit.client.1.smithi137.stdout:6/93: creat d2/f25 x:0 0 0 2024-03-14T13:02:10.094 INFO:tasks.workunit.client.0.smithi111.stdout:4/512: unlink d1d/d22/d30/d5e/d6f/d7e/d66/f79 0 2024-03-14T13:02:10.094 INFO:tasks.workunit.client.0.smithi111.stdout:0/175: sync 2024-03-14T13:02:10.094 INFO:tasks.workunit.client.0.smithi111.stdout:3/258: sync 2024-03-14T13:02:10.094 INFO:tasks.workunit.client.0.smithi111.stdout:1/214: sync 2024-03-14T13:02:10.097 INFO:tasks.workunit.client.1.smithi137.stdout:2/84: rename d2/cc to d2/d7/db/de/c11 0 2024-03-14T13:02:10.098 INFO:tasks.workunit.client.0.smithi111.stdout:1/215: creat d20/d40/f54 x:0 0 0 2024-03-14T13:02:10.099 INFO:tasks.workunit.client.1.smithi137.stdout:7/100: mknod d3/d6/d14/c1f 0 2024-03-14T13:02:10.100 INFO:tasks.workunit.client.1.smithi137.stdout:4/131: rmdir d4/d9/d1b 39 2024-03-14T13:02:10.101 INFO:tasks.workunit.client.0.smithi111.stdout:3/259: creat d0/d2d/d35/d50/d5e/f5f x:0 0 0 2024-03-14T13:02:10.102 INFO:tasks.workunit.client.1.smithi137.stdout:1/131: unlink d7/d9/d19/f1c 0 2024-03-14T13:02:10.103 INFO:tasks.workunit.client.1.smithi137.stdout:7/101: dread d3/f1e [0,4194304] 0 2024-03-14T13:02:10.105 INFO:tasks.workunit.client.1.smithi137.stdout:7/102: fdatasync d3/d6/d14/f1a 0 2024-03-14T13:02:10.106 INFO:tasks.workunit.client.1.smithi137.stdout:1/132: mkdir d7/d9/d19/d1f/d22 0 2024-03-14T13:02:10.106 INFO:tasks.workunit.client.0.smithi111.stdout:3/260: mknod d0/d2d/d35/c60 0 2024-03-14T13:02:10.107 INFO:tasks.workunit.client.1.smithi137.stdout:4/132: mknod d4/c2a 0 2024-03-14T13:02:10.108 INFO:tasks.workunit.client.1.smithi137.stdout:2/85: rename d2/f8 to d2/f12 0 2024-03-14T13:02:10.109 INFO:tasks.workunit.client.0.smithi111.stdout:0/176: dwrite d6/d9/de/f36 [0,4194304] 0 2024-03-14T13:02:10.109 INFO:tasks.workunit.client.1.smithi137.stdout:7/103: stat d3/fe 0 2024-03-14T13:02:10.112 INFO:tasks.workunit.client.1.smithi137.stdout:1/133: dread d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:10.112 INFO:tasks.workunit.client.1.smithi137.stdout:9/144: dwrite d6/df/f14 [0,4194304] 0 2024-03-14T13:02:10.113 INFO:tasks.workunit.client.0.smithi111.stdout:1/216: dread fa [0,4194304] 0 2024-03-14T13:02:10.113 INFO:tasks.workunit.client.0.smithi111.stdout:0/177: rmdir d6/d9/de/d32 39 2024-03-14T13:02:10.114 INFO:tasks.workunit.client.1.smithi137.stdout:4/133: link d4/f8 d4/d27/f2b 0 2024-03-14T13:02:10.114 INFO:tasks.workunit.client.1.smithi137.stdout:2/86: creat d2/d7/f13 x:0 0 0 2024-03-14T13:02:10.114 INFO:tasks.workunit.client.0.smithi111.stdout:1/217: chown f19 3253267 1 2024-03-14T13:02:10.115 INFO:tasks.workunit.client.1.smithi137.stdout:2/87: chown l1 184 1 2024-03-14T13:02:10.117 INFO:tasks.workunit.client.0.smithi111.stdout:3/261: creat d0/d2d/d35/f61 x:0 0 0 2024-03-14T13:02:10.118 INFO:tasks.workunit.client.0.smithi111.stdout:0/178: mknod d6/d9/de/d32/c39 0 2024-03-14T13:02:10.118 INFO:tasks.workunit.client.1.smithi137.stdout:2/88: chown d2/d7/db 31 1 2024-03-14T13:02:10.119 INFO:tasks.workunit.client.1.smithi137.stdout:9/145: creat d6/df/d19/d1f/f35 x:0 0 0 2024-03-14T13:02:10.119 INFO:tasks.workunit.client.1.smithi137.stdout:4/134: mknod d4/d9/c2c 0 2024-03-14T13:02:10.120 INFO:tasks.workunit.client.0.smithi111.stdout:1/218: rename d20/d25/d31/l3c to d20/d21/l55 0 2024-03-14T13:02:10.121 INFO:tasks.workunit.client.1.smithi137.stdout:2/89: write d2/d7/f10 [53405,62793] 0 2024-03-14T13:02:10.122 INFO:tasks.workunit.client.0.smithi111.stdout:1/219: truncate d20/d25/d31/f45 690407 0 2024-03-14T13:02:10.122 INFO:tasks.workunit.client.1.smithi137.stdout:4/135: write d4/d9/f18 [92813,25283] 0 2024-03-14T13:02:10.122 INFO:tasks.workunit.client.1.smithi137.stdout:9/146: readlink d6/df/l2e 0 2024-03-14T13:02:10.122 INFO:tasks.workunit.client.1.smithi137.stdout:7/104: dread d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:10.123 INFO:tasks.workunit.client.0.smithi111.stdout:1/220: dread - d20/d25/f46 zero size 2024-03-14T13:02:10.123 INFO:tasks.workunit.client.0.smithi111.stdout:1/221: chown l1 445257673 1 2024-03-14T13:02:10.124 INFO:tasks.workunit.client.1.smithi137.stdout:2/90: write d2/fa [645787,37007] 0 2024-03-14T13:02:10.124 INFO:tasks.workunit.client.1.smithi137.stdout:4/136: write d4/f11 [2436937,95146] 0 2024-03-14T13:02:10.125 INFO:tasks.workunit.client.0.smithi111.stdout:0/179: mkdir d6/d9/de/d3a 0 2024-03-14T13:02:10.127 INFO:tasks.workunit.client.0.smithi111.stdout:0/180: creat d6/d9/de/f3b x:0 0 0 2024-03-14T13:02:10.130 INFO:tasks.workunit.client.0.smithi111.stdout:0/181: creat d6/d9/de/f3c x:0 0 0 2024-03-14T13:02:10.131 INFO:tasks.workunit.client.0.smithi111.stdout:0/182: mkdir d6/d3d 0 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.0.smithi111.stdout:0/183: symlink d6/d9/d15/l3e 0 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/134: dread d7/fb [0,4194304] 0 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/135: chown d7 1 1 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.1.smithi137.stdout:9/147: rename d6/f2c to d6/df/d19/d33/f36 0 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.1.smithi137.stdout:4/137: dread d4/f12 [0,4194304] 0 2024-03-14T13:02:10.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/136: creat d7/d9/d19/d1f/f23 x:0 0 0 2024-03-14T13:02:10.134 INFO:tasks.workunit.client.1.smithi137.stdout:2/91: link d2/d7/f10 d2/f14 0 2024-03-14T13:02:10.136 INFO:tasks.workunit.client.0.smithi111.stdout:3/262: write d0/d2d/d35/d50/d5e/d40/f47 [1178957,35239] 0 2024-03-14T13:02:10.137 INFO:tasks.workunit.client.1.smithi137.stdout:9/148: rename d6/f18 to d6/df/d19/f37 0 2024-03-14T13:02:10.137 INFO:tasks.workunit.client.1.smithi137.stdout:9/149: stat d6/l2d 0 2024-03-14T13:02:10.138 INFO:tasks.workunit.client.1.smithi137.stdout:9/150: fdatasync d6/df/f14 0 2024-03-14T13:02:10.139 INFO:tasks.workunit.client.1.smithi137.stdout:1/137: symlink d7/d9/d19/d1f/l24 0 2024-03-14T13:02:10.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:09 smithi111 ceph-mon[39688]: pgmap v19: 65 pgs: 65 active+clean; 553 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 2.6 MiB/s rd, 42 MiB/s wr, 182 op/s 2024-03-14T13:02:10.140 INFO:tasks.workunit.client.1.smithi137.stdout:9/151: write d6/fa [1713725,19550] 0 2024-03-14T13:02:10.144 INFO:tasks.workunit.client.1.smithi137.stdout:9/152: mknod d6/c38 0 2024-03-14T13:02:10.146 INFO:tasks.workunit.client.1.smithi137.stdout:1/138: dwrite d7/d9/f18 [0,4194304] 0 2024-03-14T13:02:10.147 INFO:tasks.workunit.client.1.smithi137.stdout:9/153: symlink d6/df/d19/d33/l39 0 2024-03-14T13:02:10.148 INFO:tasks.workunit.client.1.smithi137.stdout:9/154: chown d6/c38 353 1 2024-03-14T13:02:10.150 INFO:tasks.workunit.client.1.smithi137.stdout:1/139: rename f4 to d7/d9/d19/d1f/d22/f25 0 2024-03-14T13:02:10.152 INFO:tasks.workunit.client.1.smithi137.stdout:9/155: mkdir d6/df/d19/d3a 0 2024-03-14T13:02:10.153 INFO:tasks.workunit.client.1.smithi137.stdout:1/140: mkdir d7/d9/d19/d1f/d26 0 2024-03-14T13:02:10.154 INFO:tasks.workunit.client.1.smithi137.stdout:1/141: dread d7/d9/fd [0,4194304] 0 2024-03-14T13:02:10.154 INFO:tasks.workunit.client.1.smithi137.stdout:9/156: creat d6/df/d19/d1f/f3b x:0 0 0 2024-03-14T13:02:10.156 INFO:tasks.workunit.client.1.smithi137.stdout:9/157: read d6/df/f14 [3930892,33209] 0 2024-03-14T13:02:10.156 INFO:tasks.workunit.client.1.smithi137.stdout:1/142: read d7/d9/fd [7910,102112] 0 2024-03-14T13:02:10.167 INFO:tasks.workunit.client.1.smithi137.stdout:9/158: write d6/df/d19/d1f/f24 [2531451,114547] 0 2024-03-14T13:02:10.168 INFO:tasks.workunit.client.1.smithi137.stdout:4/138: dwrite d4/d9/fc [4194304,4194304] 0 2024-03-14T13:02:10.171 INFO:tasks.workunit.client.1.smithi137.stdout:4/139: readlink d4/d9/l1c 0 2024-03-14T13:02:10.177 INFO:tasks.workunit.client.1.smithi137.stdout:1/143: dwrite f6 [0,4194304] 0 2024-03-14T13:02:10.179 INFO:tasks.workunit.client.1.smithi137.stdout:9/159: getdents d6/df/d19/d33 0 2024-03-14T13:02:10.179 INFO:tasks.workunit.client.1.smithi137.stdout:4/140: truncate d4/f1a 7525004 0 2024-03-14T13:02:10.179 INFO:tasks.workunit.client.1.smithi137.stdout:1/144: write d7/d9/d19/d1f/f23 [403774,78924] 0 2024-03-14T13:02:10.182 INFO:tasks.workunit.client.1.smithi137.stdout:9/160: unlink d6/f1c 0 2024-03-14T13:02:10.182 INFO:tasks.workunit.client.1.smithi137.stdout:4/141: rename d4/d9 to d4/d9/d2d 22 2024-03-14T13:02:10.183 INFO:tasks.workunit.client.1.smithi137.stdout:9/161: chown d6/df/d19/d1f/f3b 32712 1 2024-03-14T13:02:10.184 INFO:tasks.workunit.client.1.smithi137.stdout:4/142: write d4/f12 [3649900,16115] 0 2024-03-14T13:02:10.184 INFO:tasks.workunit.client.1.smithi137.stdout:1/145: mknod d7/d9/d19/d1f/d26/c27 0 2024-03-14T13:02:10.189 INFO:tasks.workunit.client.1.smithi137.stdout:1/146: mknod d7/c28 0 2024-03-14T13:02:10.191 INFO:tasks.workunit.client.1.smithi137.stdout:4/143: creat d4/f2e x:0 0 0 2024-03-14T13:02:10.194 INFO:tasks.workunit.client.1.smithi137.stdout:9/162: dwrite d6/f2b [0,4194304] 0 2024-03-14T13:02:10.195 INFO:tasks.workunit.client.1.smithi137.stdout:4/144: mknod d4/d9/d1b/c2f 0 2024-03-14T13:02:10.196 INFO:tasks.workunit.client.1.smithi137.stdout:1/147: getdents d7/d9/d19/d1f 0 2024-03-14T13:02:10.200 INFO:tasks.workunit.client.1.smithi137.stdout:4/145: readlink d4/d9/l1d 0 2024-03-14T13:02:10.202 INFO:tasks.workunit.client.1.smithi137.stdout:1/148: mkdir d7/d29 0 2024-03-14T13:02:10.205 INFO:tasks.workunit.client.1.smithi137.stdout:1/149: write f3 [2228697,66877] 0 2024-03-14T13:02:10.205 INFO:tasks.workunit.client.1.smithi137.stdout:4/146: mknod d4/d9/c30 0 2024-03-14T13:02:10.206 INFO:tasks.workunit.client.1.smithi137.stdout:9/163: dwrite d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:10.209 INFO:tasks.workunit.client.0.smithi111.stdout:3/263: symlink d0/d2d/d35/d50/d5e/d19/l62 0 2024-03-14T13:02:10.210 INFO:tasks.workunit.client.1.smithi137.stdout:1/150: rename d7/d9/f18 to d7/d9/d19/d1f/f2a 0 2024-03-14T13:02:10.212 INFO:tasks.workunit.client.1.smithi137.stdout:1/151: dread d7/d9/fd [0,4194304] 0 2024-03-14T13:02:10.213 INFO:tasks.workunit.client.0.smithi111.stdout:0/184: dwrite d6/f1f [4194304,4194304] 0 2024-03-14T13:02:10.215 INFO:tasks.workunit.client.1.smithi137.stdout:4/147: getdents d4/d9/d1b 0 2024-03-14T13:02:10.215 INFO:tasks.workunit.client.1.smithi137.stdout:1/152: write d7/f10 [1864544,23303] 0 2024-03-14T13:02:10.215 INFO:tasks.workunit.client.0.smithi111.stdout:0/185: write f5 [481818,106368] 0 2024-03-14T13:02:10.215 INFO:tasks.workunit.client.0.smithi111.stdout:3/264: creat d0/d2d/d35/f63 x:0 0 0 2024-03-14T13:02:10.216 INFO:tasks.workunit.client.0.smithi111.stdout:0/186: chown d6/d9/d15/l3e 338056766 1 2024-03-14T13:02:10.218 INFO:tasks.workunit.client.0.smithi111.stdout:0/187: write d6/d9/de/f26 [955664,22085] 0 2024-03-14T13:02:10.219 INFO:tasks.workunit.client.1.smithi137.stdout:9/164: dwrite d6/df/d19/d1f/f2a [0,4194304] 0 2024-03-14T13:02:10.223 INFO:tasks.workunit.client.1.smithi137.stdout:1/153: dread f5 [0,4194304] 0 2024-03-14T13:02:10.227 INFO:tasks.workunit.client.0.smithi111.stdout:3/265: dwrite d0/d2d/d35/d50/d5e/f5f [0,4194304] 0 2024-03-14T13:02:10.231 INFO:tasks.workunit.client.0.smithi111.stdout:3/266: chown d0/d2d/d35/d50/d5e/d19/l45 23116390 1 2024-03-14T13:02:10.232 INFO:tasks.workunit.client.0.smithi111.stdout:0/188: dwrite d6/d9/de/f26 [0,4194304] 0 2024-03-14T13:02:10.234 INFO:tasks.workunit.client.0.smithi111.stdout:3/267: symlink d0/d30/l64 0 2024-03-14T13:02:10.235 INFO:tasks.workunit.client.0.smithi111.stdout:0/189: symlink d6/d21/l3f 0 2024-03-14T13:02:10.237 INFO:tasks.workunit.client.0.smithi111.stdout:3/268: mkdir d0/d2d/d35/d50/d5e/d19/d65 0 2024-03-14T13:02:10.238 INFO:tasks.workunit.client.0.smithi111.stdout:3/269: dread - d0/d30/f4a zero size 2024-03-14T13:02:10.238 INFO:tasks.workunit.client.0.smithi111.stdout:0/190: read d6/f10 [322274,61670] 0 2024-03-14T13:02:10.239 INFO:tasks.workunit.client.0.smithi111.stdout:3/270: chown d0/fd 20254494 1 2024-03-14T13:02:10.240 INFO:tasks.workunit.client.0.smithi111.stdout:0/191: truncate f3 2145398 0 2024-03-14T13:02:10.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/192: rename d6/d3d to d6/d3d/d40 22 2024-03-14T13:02:10.244 INFO:tasks.workunit.client.0.smithi111.stdout:3/271: link d0/d2d/d35/d50/d5e/d40/l42 d0/d2d/d35/d50/d5e/d40/l66 0 2024-03-14T13:02:10.246 INFO:tasks.workunit.client.0.smithi111.stdout:3/272: read d0/f23 [198075,106863] 0 2024-03-14T13:02:10.252 INFO:tasks.workunit.client.0.smithi111.stdout:3/273: dwrite d0/d2d/d35/f63 [0,4194304] 0 2024-03-14T13:02:10.254 INFO:tasks.workunit.client.0.smithi111.stdout:3/274: truncate d0/d30/f48 913217 0 2024-03-14T13:02:10.254 INFO:tasks.workunit.client.0.smithi111.stdout:3/275: chown d0/d2d/d35/d50/d5e/d19/l3f 3 1 2024-03-14T13:02:10.256 INFO:tasks.workunit.client.0.smithi111.stdout:3/276: mkdir d0/d2d/d35/d50/d5e/d40/d67 0 2024-03-14T13:02:10.260 INFO:tasks.workunit.client.0.smithi111.stdout:3/277: dread d0/d30/f57 [0,4194304] 0 2024-03-14T13:02:10.261 INFO:tasks.workunit.client.0.smithi111.stdout:0/193: fsync d6/d9/de/f26 0 2024-03-14T13:02:10.262 INFO:tasks.workunit.client.0.smithi111.stdout:3/278: rmdir d0/d2d/d35/d50/d5e/d19 39 2024-03-14T13:02:10.267 INFO:tasks.workunit.client.0.smithi111.stdout:3/279: symlink d0/d2d/d35/d50/d5e/d40/d67/l68 0 2024-03-14T13:02:10.267 INFO:tasks.workunit.client.0.smithi111.stdout:0/194: dread d6/fb [4194304,4194304] 0 2024-03-14T13:02:10.269 INFO:tasks.workunit.client.0.smithi111.stdout:3/280: fsync d0/f7 0 2024-03-14T13:02:10.270 INFO:tasks.workunit.client.0.smithi111.stdout:0/195: fsync d6/f10 0 2024-03-14T13:02:10.277 INFO:tasks.workunit.client.0.smithi111.stdout:8/377: sync 2024-03-14T13:02:10.277 INFO:tasks.workunit.client.0.smithi111.stdout:3/281: dwrite d0/d2d/d35/d50/f54 [0,4194304] 0 2024-03-14T13:02:10.279 INFO:tasks.workunit.client.0.smithi111.stdout:0/196: dwrite d6/d9/de/f27 [0,4194304] 0 2024-03-14T13:02:10.279 INFO:tasks.workunit.client.0.smithi111.stdout:8/378: readlink d3/d1b/d20/d35/l69 0 2024-03-14T13:02:10.282 INFO:tasks.workunit.client.0.smithi111.stdout:8/379: creat d3/dc/d45/f81 x:0 0 0 2024-03-14T13:02:10.284 INFO:tasks.workunit.client.0.smithi111.stdout:3/282: mkdir d0/d2d/d35/d69 0 2024-03-14T13:02:10.284 INFO:tasks.workunit.client.0.smithi111.stdout:8/380: write d3/d1b/d20/d35/d40/d48/f68 [77994,121452] 0 2024-03-14T13:02:10.287 INFO:tasks.workunit.client.1.smithi137.stdout:5/61: sync 2024-03-14T13:02:10.287 INFO:tasks.workunit.client.1.smithi137.stdout:8/72: sync 2024-03-14T13:02:10.287 INFO:tasks.workunit.client.1.smithi137.stdout:3/115: sync 2024-03-14T13:02:10.287 INFO:tasks.workunit.client.1.smithi137.stdout:8/73: fdatasync d1/fd 0 2024-03-14T13:02:10.289 INFO:tasks.workunit.client.0.smithi111.stdout:0/197: write d6/d9/de/d32/f31 [1040233,96081] 0 2024-03-14T13:02:10.290 INFO:tasks.workunit.client.0.smithi111.stdout:8/381: dread f0 [0,4194304] 0 2024-03-14T13:02:10.291 INFO:tasks.workunit.client.0.smithi111.stdout:0/198: write d6/d9/d15/f1b [1042693,130670] 0 2024-03-14T13:02:10.292 INFO:tasks.workunit.client.1.smithi137.stdout:8/74: dread d1/f6 [0,4194304] 0 2024-03-14T13:02:10.293 INFO:tasks.workunit.client.0.smithi111.stdout:8/382: dread - d3/d1b/d20/d35/d40/f76 zero size 2024-03-14T13:02:10.293 INFO:tasks.workunit.client.0.smithi111.stdout:8/383: chown d3 0 1 2024-03-14T13:02:10.293 INFO:tasks.workunit.client.0.smithi111.stdout:3/283: link d0/d2d/d35/d50/d5e/d40/l42 d0/d2d/d35/l6a 0 2024-03-14T13:02:10.294 INFO:tasks.workunit.client.0.smithi111.stdout:3/284: fsync d0/f20 0 2024-03-14T13:02:10.296 INFO:tasks.workunit.client.0.smithi111.stdout:0/199: creat d6/d9/de/f41 x:0 0 0 2024-03-14T13:02:10.296 INFO:tasks.workunit.client.0.smithi111.stdout:3/285: fsync d0/d30/f57 0 2024-03-14T13:02:10.297 INFO:tasks.workunit.client.0.smithi111.stdout:8/384: mknod d3/d22/d4a/c82 0 2024-03-14T13:02:10.298 INFO:tasks.workunit.client.0.smithi111.stdout:0/200: rmdir d6/d9/d15 39 2024-03-14T13:02:10.299 INFO:tasks.workunit.client.0.smithi111.stdout:8/385: unlink d3/d5/c29 0 2024-03-14T13:02:10.300 INFO:tasks.workunit.client.0.smithi111.stdout:8/386: chown d3/l54 8324888 1 2024-03-14T13:02:10.301 INFO:tasks.workunit.client.1.smithi137.stdout:9/165: creat d6/df/d19/d33/f3c x:0 0 0 2024-03-14T13:02:10.301 INFO:tasks.workunit.client.0.smithi111.stdout:3/286: mkdir d0/d2d/d35/d50/d5e/d40/d67/d6b 0 2024-03-14T13:02:10.303 INFO:tasks.workunit.client.1.smithi137.stdout:5/62: symlink d7/l13 0 2024-03-14T13:02:10.305 INFO:tasks.workunit.client.1.smithi137.stdout:3/116: rename d4/df/f14 to d4/de/f1f 0 2024-03-14T13:02:10.306 INFO:tasks.workunit.client.1.smithi137.stdout:1/154: creat d7/d29/f2b x:0 0 0 2024-03-14T13:02:10.307 INFO:tasks.workunit.client.0.smithi111.stdout:8/387: getdents d3/d1b/d20/d35 0 2024-03-14T13:02:10.308 INFO:tasks.workunit.client.0.smithi111.stdout:3/287: write d0/f7 [2618833,34414] 0 2024-03-14T13:02:10.308 INFO:tasks.workunit.client.1.smithi137.stdout:5/63: creat d7/f14 x:0 0 0 2024-03-14T13:02:10.308 INFO:tasks.workunit.client.1.smithi137.stdout:3/117: write d4/de/d16/f1a [2548421,78612] 0 2024-03-14T13:02:10.309 INFO:tasks.workunit.client.0.smithi111.stdout:8/388: write d3/d1b/d34/f3c [54895,121919] 0 2024-03-14T13:02:10.309 INFO:tasks.workunit.client.1.smithi137.stdout:4/148: creat d4/f31 x:0 0 0 2024-03-14T13:02:10.309 INFO:tasks.workunit.client.0.smithi111.stdout:8/389: stat d3/dc/d45/c67 0 2024-03-14T13:02:10.311 INFO:tasks.workunit.client.1.smithi137.stdout:8/75: rename d1/la to d1/dc/d11/d13/l14 0 2024-03-14T13:02:10.311 INFO:tasks.workunit.client.0.smithi111.stdout:3/288: mkdir d0/d2d/d35/d50/d6c 0 2024-03-14T13:02:10.311 INFO:tasks.workunit.client.0.smithi111.stdout:8/390: write d3/f6d [3196052,100281] 0 2024-03-14T13:02:10.313 INFO:tasks.workunit.client.1.smithi137.stdout:9/166: dwrite d6/df/f14 [0,4194304] 0 2024-03-14T13:02:10.313 INFO:tasks.workunit.client.1.smithi137.stdout:4/149: read d4/d9/fc [6650962,15136] 0 2024-03-14T13:02:10.315 INFO:tasks.workunit.client.0.smithi111.stdout:3/289: unlink d0/d2d/d35/d4c/l5b 0 2024-03-14T13:02:10.315 INFO:tasks.workunit.client.0.smithi111.stdout:3/290: stat d0/f7 0 2024-03-14T13:02:10.316 INFO:tasks.workunit.client.1.smithi137.stdout:1/155: chown d7/l8 52884 1 2024-03-14T13:02:10.316 INFO:tasks.workunit.client.1.smithi137.stdout:4/150: chown d4/f12 65 1 2024-03-14T13:02:10.317 INFO:tasks.workunit.client.1.smithi137.stdout:3/118: creat d4/de/d16/f20 x:0 0 0 2024-03-14T13:02:10.318 INFO:tasks.workunit.client.1.smithi137.stdout:3/119: truncate d4/fd 12641 0 2024-03-14T13:02:10.318 INFO:tasks.workunit.client.1.smithi137.stdout:5/64: rename f5 to d7/f15 0 2024-03-14T13:02:10.319 INFO:tasks.workunit.client.1.smithi137.stdout:8/76: mknod d1/dc/d11/c15 0 2024-03-14T13:02:10.320 INFO:tasks.workunit.client.1.smithi137.stdout:8/77: chown d1/fd 59 1 2024-03-14T13:02:10.322 INFO:tasks.workunit.client.1.smithi137.stdout:1/156: symlink d7/d29/l2c 0 2024-03-14T13:02:10.327 INFO:tasks.workunit.client.1.smithi137.stdout:3/120: dwrite d4/de/d16/f17 [4194304,4194304] 0 2024-03-14T13:02:10.329 INFO:tasks.workunit.client.1.smithi137.stdout:5/65: unlink d7/f14 0 2024-03-14T13:02:10.329 INFO:tasks.workunit.client.1.smithi137.stdout:9/167: mknod d6/df/d19/d3a/c3d 0 2024-03-14T13:02:10.331 INFO:tasks.workunit.client.1.smithi137.stdout:8/78: rename d1/fb to d1/dc/d11/d13/f16 0 2024-03-14T13:02:10.333 INFO:tasks.workunit.client.1.smithi137.stdout:5/66: dread d7/ff [0,4194304] 0 2024-03-14T13:02:10.334 INFO:tasks.workunit.client.1.smithi137.stdout:3/121: creat d4/de/d13/f21 x:0 0 0 2024-03-14T13:02:10.334 INFO:tasks.workunit.client.0.smithi111.stdout:8/391: truncate d3/d5/fb 6210880 0 2024-03-14T13:02:10.334 INFO:tasks.workunit.client.1.smithi137.stdout:8/79: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:10.336 INFO:tasks.workunit.client.1.smithi137.stdout:9/168: creat d6/df/d19/d33/f3e x:0 0 0 2024-03-14T13:02:10.338 INFO:tasks.workunit.client.0.smithi111.stdout:8/392: rename d3/dc/d59 to d3/d1b/d20/d83 0 2024-03-14T13:02:10.338 INFO:tasks.workunit.client.0.smithi111.stdout:8/393: fdatasync d3/d1b/f3a 0 2024-03-14T13:02:10.338 INFO:tasks.workunit.client.0.smithi111.stdout:8/394: chown d3/f3e 424587 1 2024-03-14T13:02:10.339 INFO:tasks.workunit.client.1.smithi137.stdout:1/157: mknod d7/d29/c2d 0 2024-03-14T13:02:10.339 INFO:tasks.workunit.client.0.smithi111.stdout:8/395: chown d3/d22/d4a/c82 3 1 2024-03-14T13:02:10.340 INFO:tasks.workunit.client.0.smithi111.stdout:8/396: write d3/d22/d30/f74 [350128,17928] 0 2024-03-14T13:02:10.340 INFO:tasks.workunit.client.1.smithi137.stdout:3/122: write d4/de/f1f [953417,86298] 0 2024-03-14T13:02:10.341 INFO:tasks.workunit.client.0.smithi111.stdout:8/397: truncate d3/dc/d45/f56 4755028 0 2024-03-14T13:02:10.342 INFO:tasks.workunit.client.1.smithi137.stdout:8/80: unlink d1/c9 0 2024-03-14T13:02:10.343 INFO:tasks.workunit.client.1.smithi137.stdout:9/169: mknod d6/c3f 0 2024-03-14T13:02:10.344 INFO:tasks.workunit.client.0.smithi111.stdout:8/398: rename d3/dc/d45/c58 to d3/d1b/d20/d35/d40/d48/d7e/c84 0 2024-03-14T13:02:10.354 INFO:tasks.workunit.client.1.smithi137.stdout:4/151: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:10.354 INFO:tasks.workunit.client.1.smithi137.stdout:4/152: read d4/f12 [3809351,41230] 0 2024-03-14T13:02:10.355 INFO:tasks.workunit.client.1.smithi137.stdout:9/170: dread d6/df/f20 [0,4194304] 0 2024-03-14T13:02:10.363 INFO:tasks.workunit.client.1.smithi137.stdout:1/158: truncate d7/d9/d19/d1f/f2a 1710235 0 2024-03-14T13:02:10.363 INFO:tasks.workunit.client.1.smithi137.stdout:9/171: mkdir d6/df/d19/d3a/d40 0 2024-03-14T13:02:10.366 INFO:tasks.workunit.client.1.smithi137.stdout:9/172: chown d6/df/d19/d1f/f35 12 1 2024-03-14T13:02:10.366 INFO:tasks.workunit.client.1.smithi137.stdout:4/153: link d4/f13 d4/f32 0 2024-03-14T13:02:10.369 INFO:tasks.workunit.client.1.smithi137.stdout:9/173: readlink d6/df/d19/l23 0 2024-03-14T13:02:10.372 INFO:tasks.workunit.client.1.smithi137.stdout:9/174: write d6/df/f14 [3610708,12532] 0 2024-03-14T13:02:10.374 INFO:tasks.workunit.client.1.smithi137.stdout:8/81: dwrite d1/dc/d10/f12 [0,4194304] 0 2024-03-14T13:02:10.377 INFO:tasks.workunit.client.0.smithi111.stdout:9/229: sync 2024-03-14T13:02:10.377 INFO:tasks.workunit.client.0.smithi111.stdout:5/294: sync 2024-03-14T13:02:10.377 INFO:tasks.workunit.client.1.smithi137.stdout:1/159: dwrite f3 [0,4194304] 0 2024-03-14T13:02:10.377 INFO:tasks.workunit.client.1.smithi137.stdout:9/175: write d6/df/d19/d1f/f3b [166160,70912] 0 2024-03-14T13:02:10.378 INFO:tasks.workunit.client.1.smithi137.stdout:4/154: dread d4/f13 [0,4194304] 0 2024-03-14T13:02:10.379 INFO:tasks.workunit.client.1.smithi137.stdout:9/176: chown d6/df/d19/d33/l39 24 1 2024-03-14T13:02:10.379 INFO:tasks.workunit.client.1.smithi137.stdout:8/82: mkdir d1/d17 0 2024-03-14T13:02:10.383 INFO:tasks.workunit.client.1.smithi137.stdout:1/160: mkdir d7/d9/d19/d1f/d2e 0 2024-03-14T13:02:10.384 INFO:tasks.workunit.client.1.smithi137.stdout:4/155: rmdir d4 39 2024-03-14T13:02:10.386 INFO:tasks.workunit.client.0.smithi111.stdout:8/399: truncate d3/d1b/d20/d35/d40/d48/f50 1731591 0 2024-03-14T13:02:10.387 INFO:tasks.workunit.client.0.smithi111.stdout:9/230: chown d5/d6/d8/f3a 8 1 2024-03-14T13:02:10.387 INFO:tasks.workunit.client.0.smithi111.stdout:8/400: chown d3/f6b 3263535 1 2024-03-14T13:02:10.387 INFO:tasks.workunit.client.1.smithi137.stdout:4/156: readlink d4/d10/l23 0 2024-03-14T13:02:10.389 INFO:tasks.workunit.client.0.smithi111.stdout:8/401: fsync d3/d5/f1e 0 2024-03-14T13:02:10.389 INFO:tasks.workunit.client.1.smithi137.stdout:4/157: rename d4/d9 to d4/d9/d1b/d33 22 2024-03-14T13:02:10.389 INFO:tasks.workunit.client.1.smithi137.stdout:8/83: dwrite d1/fd [0,4194304] 0 2024-03-14T13:02:10.390 INFO:tasks.workunit.client.0.smithi111.stdout:8/402: write d3/d5/f1e [4513030,15822] 0 2024-03-14T13:02:10.390 INFO:tasks.workunit.client.1.smithi137.stdout:4/158: fdatasync d4/d27/f28 0 2024-03-14T13:02:10.391 INFO:tasks.workunit.client.0.smithi111.stdout:5/295: link d2/d13/l46 d2/d45/l69 0 2024-03-14T13:02:10.391 INFO:tasks.workunit.client.1.smithi137.stdout:8/84: truncate d1/f2 5105919 0 2024-03-14T13:02:10.391 INFO:tasks.workunit.client.0.smithi111.stdout:8/403: write d3/f3e [4419613,28976] 0 2024-03-14T13:02:10.393 INFO:tasks.workunit.client.0.smithi111.stdout:5/296: chown d2/d1f/d36/f3c 14276 1 2024-03-14T13:02:10.393 INFO:tasks.workunit.client.1.smithi137.stdout:4/159: write d4/f32 [319465,120159] 0 2024-03-14T13:02:10.393 INFO:tasks.workunit.client.0.smithi111.stdout:8/404: dread - d3/f78 zero size 2024-03-14T13:02:10.393 INFO:tasks.workunit.client.0.smithi111.stdout:9/231: dread d5/d6/f22 [4194304,4194304] 0 2024-03-14T13:02:10.394 INFO:tasks.workunit.client.1.smithi137.stdout:4/160: write d4/f2e [1025151,109975] 0 2024-03-14T13:02:10.395 INFO:tasks.workunit.client.0.smithi111.stdout:9/232: read d5/fe [1940130,68853] 0 2024-03-14T13:02:10.395 INFO:tasks.workunit.client.0.smithi111.stdout:5/297: write d2/d16/d49/f11 [2651599,72974] 0 2024-03-14T13:02:10.395 INFO:tasks.workunit.client.1.smithi137.stdout:4/161: truncate d4/f2e 1540146 0 2024-03-14T13:02:10.396 INFO:tasks.workunit.client.1.smithi137.stdout:8/85: unlink d1/dc/d10/f12 0 2024-03-14T13:02:10.400 INFO:tasks.workunit.client.1.smithi137.stdout:8/86: chown d1/dc/d11/d13/l14 158558703 1 2024-03-14T13:02:10.401 INFO:tasks.workunit.client.0.smithi111.stdout:8/405: rename d3/l8 to d3/d1b/d20/l85 0 2024-03-14T13:02:10.401 INFO:tasks.workunit.client.1.smithi137.stdout:4/162: truncate d4/f1a 8257344 0 2024-03-14T13:02:10.402 INFO:tasks.workunit.client.1.smithi137.stdout:8/87: symlink d1/dc/d10/l18 0 2024-03-14T13:02:10.403 INFO:tasks.workunit.client.0.smithi111.stdout:8/406: fdatasync d3/dc/d31/f4b 0 2024-03-14T13:02:10.404 INFO:tasks.workunit.client.1.smithi137.stdout:4/163: creat d4/d10/f34 x:0 0 0 2024-03-14T13:02:10.404 INFO:tasks.workunit.client.0.smithi111.stdout:9/233: dwrite f2 [0,4194304] 0 2024-03-14T13:02:10.405 INFO:tasks.workunit.client.1.smithi137.stdout:8/88: write d1/f8 [1126733,116926] 0 2024-03-14T13:02:10.407 INFO:tasks.workunit.client.1.smithi137.stdout:8/89: chown d1/l5 25 1 2024-03-14T13:02:10.409 INFO:tasks.workunit.client.1.smithi137.stdout:4/164: rename d4/d9/le to d4/d9/d1b/l35 0 2024-03-14T13:02:10.409 INFO:tasks.workunit.client.1.smithi137.stdout:8/90: mknod d1/dc/d11/c19 0 2024-03-14T13:02:10.414 INFO:tasks.workunit.client.0.smithi111.stdout:8/407: dwrite d3/d5/f1e [0,4194304] 0 2024-03-14T13:02:10.416 INFO:tasks.workunit.client.1.smithi137.stdout:8/91: creat d1/dc/d10/f1a x:0 0 0 2024-03-14T13:02:10.416 INFO:tasks.workunit.client.0.smithi111.stdout:9/234: dwrite d5/d6/fc [0,4194304] 0 2024-03-14T13:02:10.418 INFO:tasks.workunit.client.1.smithi137.stdout:4/165: link d4/d9/l1d d4/d10/l36 0 2024-03-14T13:02:10.420 INFO:tasks.workunit.client.1.smithi137.stdout:4/166: creat d4/f37 x:0 0 0 2024-03-14T13:02:10.427 INFO:tasks.workunit.client.0.smithi111.stdout:5/298: dread d2/f35 [0,4194304] 0 2024-03-14T13:02:10.428 INFO:tasks.workunit.client.0.smithi111.stdout:5/299: chown d2/d16/d2c/f3e 40 1 2024-03-14T13:02:10.428 INFO:tasks.workunit.client.0.smithi111.stdout:5/300: dread - d2/d45/f5c zero size 2024-03-14T13:02:10.430 INFO:tasks.workunit.client.0.smithi111.stdout:5/301: truncate d2/d13/f4b 230458 0 2024-03-14T13:02:10.432 INFO:tasks.workunit.client.0.smithi111.stdout:5/302: mkdir d2/d1f/d6a 0 2024-03-14T13:02:10.434 INFO:tasks.workunit.client.0.smithi111.stdout:5/303: mknod d2/d16/d49/c6b 0 2024-03-14T13:02:10.434 INFO:tasks.workunit.client.0.smithi111.stdout:5/304: read d2/d45/f34 [218115,80661] 0 2024-03-14T13:02:10.435 INFO:tasks.workunit.client.1.smithi137.stdout:4/167: dwrite d4/f12 [0,4194304] 0 2024-03-14T13:02:10.435 INFO:tasks.workunit.client.0.smithi111.stdout:5/305: read - d2/f53 zero size 2024-03-14T13:02:10.436 INFO:tasks.workunit.client.1.smithi137.stdout:4/168: rmdir d4/d27 39 2024-03-14T13:02:10.438 INFO:tasks.workunit.client.0.smithi111.stdout:5/306: unlink d2/d13/f1d 0 2024-03-14T13:02:10.439 INFO:tasks.workunit.client.1.smithi137.stdout:4/169: mkdir d4/d9/d1b/d38 0 2024-03-14T13:02:10.441 INFO:tasks.workunit.client.1.smithi137.stdout:4/170: chown d4/d27/f2b 235302 1 2024-03-14T13:02:10.442 INFO:tasks.workunit.client.1.smithi137.stdout:4/171: write d4/d10/f34 [695141,115704] 0 2024-03-14T13:02:10.445 INFO:tasks.workunit.client.1.smithi137.stdout:4/172: rename d4/c24 to d4/c39 0 2024-03-14T13:02:10.446 INFO:tasks.workunit.client.1.smithi137.stdout:4/173: truncate d4/d10/f34 1850786 0 2024-03-14T13:02:10.447 INFO:tasks.workunit.client.1.smithi137.stdout:4/174: chown d4/d9/f1f 18888759 1 2024-03-14T13:02:10.447 INFO:tasks.workunit.client.1.smithi137.stdout:4/175: write d4/f32 [51945,37497] 0 2024-03-14T13:02:10.448 INFO:tasks.workunit.client.1.smithi137.stdout:4/176: fsync d4/f8 0 2024-03-14T13:02:10.453 INFO:tasks.workunit.client.1.smithi137.stdout:4/177: dread f2 [0,4194304] 0 2024-03-14T13:02:10.455 INFO:tasks.workunit.client.1.smithi137.stdout:4/178: mkdir d4/d9/d3a 0 2024-03-14T13:02:10.460 INFO:tasks.workunit.client.1.smithi137.stdout:4/179: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:10.464 INFO:tasks.workunit.client.1.smithi137.stdout:4/180: creat d4/d9/d1b/d38/f3b x:0 0 0 2024-03-14T13:02:10.481 INFO:tasks.workunit.client.1.smithi137.stdout:6/94: sync 2024-03-14T13:02:10.481 INFO:tasks.workunit.client.1.smithi137.stdout:0/66: sync 2024-03-14T13:02:10.482 INFO:tasks.workunit.client.1.smithi137.stdout:4/181: dwrite d4/d9/f16 [0,4194304] 0 2024-03-14T13:02:10.483 INFO:tasks.workunit.client.1.smithi137.stdout:0/67: read d1/f8 [1432883,13586] 0 2024-03-14T13:02:10.484 INFO:tasks.workunit.client.1.smithi137.stdout:0/68: chown d1/f12 4 1 2024-03-14T13:02:10.484 INFO:tasks.workunit.client.1.smithi137.stdout:6/95: unlink d2/d6/l20 0 2024-03-14T13:02:10.485 INFO:tasks.workunit.client.1.smithi137.stdout:6/96: stat d2/l5 0 2024-03-14T13:02:10.487 INFO:tasks.workunit.client.1.smithi137.stdout:0/69: write d1/f12 [737713,22989] 0 2024-03-14T13:02:10.487 INFO:tasks.workunit.client.1.smithi137.stdout:4/182: creat d4/f3c x:0 0 0 2024-03-14T13:02:10.490 INFO:tasks.workunit.client.1.smithi137.stdout:4/183: mkdir d4/d10/d3d 0 2024-03-14T13:02:10.490 INFO:tasks.workunit.client.1.smithi137.stdout:0/70: dread d1/fd [0,4194304] 0 2024-03-14T13:02:10.491 INFO:tasks.workunit.client.1.smithi137.stdout:0/71: stat d1/l3 0 2024-03-14T13:02:10.491 INFO:tasks.workunit.client.1.smithi137.stdout:4/184: write f2 [2105466,102014] 0 2024-03-14T13:02:10.495 INFO:tasks.workunit.client.1.smithi137.stdout:6/97: truncate f0 623086 0 2024-03-14T13:02:10.496 INFO:tasks.workunit.client.1.smithi137.stdout:6/98: chown d2/d6/de/lf 67936 1 2024-03-14T13:02:10.497 INFO:tasks.workunit.client.1.smithi137.stdout:4/185: creat d4/d10/d3d/f3e x:0 0 0 2024-03-14T13:02:10.497 INFO:tasks.workunit.client.1.smithi137.stdout:6/99: read - d2/f25 zero size 2024-03-14T13:02:10.498 INFO:tasks.workunit.client.1.smithi137.stdout:0/72: rmdir d1/d5/d9 0 2024-03-14T13:02:10.499 INFO:tasks.workunit.client.1.smithi137.stdout:4/186: creat d4/d10/f3f x:0 0 0 2024-03-14T13:02:10.502 INFO:tasks.workunit.client.1.smithi137.stdout:0/73: creat d1/f13 x:0 0 0 2024-03-14T13:02:10.503 INFO:tasks.workunit.client.1.smithi137.stdout:0/74: readlink d1/d5/l7 0 2024-03-14T13:02:10.504 INFO:tasks.workunit.client.1.smithi137.stdout:0/75: stat d1/f12 0 2024-03-14T13:02:10.505 INFO:tasks.workunit.client.1.smithi137.stdout:6/100: dwrite d2/f25 [0,4194304] 0 2024-03-14T13:02:10.508 INFO:tasks.workunit.client.1.smithi137.stdout:4/187: dread d4/f11 [0,4194304] 0 2024-03-14T13:02:10.508 INFO:tasks.workunit.client.1.smithi137.stdout:6/101: write d2/d6/de/f13 [3136,115211] 0 2024-03-14T13:02:10.509 INFO:tasks.workunit.client.1.smithi137.stdout:4/188: dread - d4/f3c zero size 2024-03-14T13:02:10.509 INFO:tasks.workunit.client.1.smithi137.stdout:4/189: read - d4/f31 zero size 2024-03-14T13:02:10.510 INFO:tasks.workunit.client.1.smithi137.stdout:0/76: creat d1/d11/f14 x:0 0 0 2024-03-14T13:02:10.512 INFO:tasks.workunit.client.1.smithi137.stdout:6/102: mkdir d2/d26 0 2024-03-14T13:02:10.513 INFO:tasks.workunit.client.1.smithi137.stdout:0/77: creat d1/d5/f15 x:0 0 0 2024-03-14T13:02:10.513 INFO:tasks.workunit.client.1.smithi137.stdout:6/103: readlink d2/d6/de/lf 0 2024-03-14T13:02:10.514 INFO:tasks.workunit.client.0.smithi111.stdout:5/307: write d2/d16/f42 [3898525,38230] 0 2024-03-14T13:02:10.514 INFO:tasks.workunit.client.1.smithi137.stdout:6/104: chown l1 12370 1 2024-03-14T13:02:10.514 INFO:tasks.workunit.client.0.smithi111.stdout:8/408: dread d3/f3e [0,4194304] 0 2024-03-14T13:02:10.515 INFO:tasks.workunit.client.0.smithi111.stdout:8/409: chown d3/dc/l1f 30 1 2024-03-14T13:02:10.516 INFO:tasks.workunit.client.1.smithi137.stdout:0/78: rename d1/cb to d1/d5/d10/c16 0 2024-03-14T13:02:10.516 INFO:tasks.workunit.client.0.smithi111.stdout:8/410: chown d3/d1b/d20/d35/l4f 33576439 1 2024-03-14T13:02:10.517 INFO:tasks.workunit.client.1.smithi137.stdout:6/105: symlink d2/d6/l27 0 2024-03-14T13:02:10.517 INFO:tasks.workunit.client.0.smithi111.stdout:5/308: rename d2/d13/f2b to d2/d13/f6c 0 2024-03-14T13:02:10.517 INFO:tasks.workunit.client.0.smithi111.stdout:5/309: stat d2/f35 0 2024-03-14T13:02:10.518 INFO:tasks.workunit.client.1.smithi137.stdout:0/79: rmdir d1/d5/d10 39 2024-03-14T13:02:10.520 INFO:tasks.workunit.client.0.smithi111.stdout:5/310: mknod d2/d16/d49/d33/d68/c6d 0 2024-03-14T13:02:10.521 INFO:tasks.workunit.client.1.smithi137.stdout:0/80: creat d1/f17 x:0 0 0 2024-03-14T13:02:10.523 INFO:tasks.workunit.client.0.smithi111.stdout:5/311: truncate d2/d16/d49/f5d 144664 0 2024-03-14T13:02:10.523 INFO:tasks.workunit.client.0.smithi111.stdout:8/411: dwrite d3/dc/d31/f4b [0,4194304] 0 2024-03-14T13:02:10.523 INFO:tasks.workunit.client.1.smithi137.stdout:6/106: dwrite d2/f25 [0,4194304] 0 2024-03-14T13:02:10.524 INFO:tasks.workunit.client.0.smithi111.stdout:8/412: read d3/d1b/f75 [44926,88801] 0 2024-03-14T13:02:10.527 INFO:tasks.workunit.client.1.smithi137.stdout:0/81: rename d1/l3 to d1/d5/d10/l18 0 2024-03-14T13:02:10.529 INFO:tasks.workunit.client.1.smithi137.stdout:6/107: mkdir d2/d6/d8/d28 0 2024-03-14T13:02:10.531 INFO:tasks.workunit.client.1.smithi137.stdout:0/82: fsync d1/d5/f15 0 2024-03-14T13:02:10.532 INFO:tasks.workunit.client.1.smithi137.stdout:0/83: stat d1/d11/f14 0 2024-03-14T13:02:10.532 INFO:tasks.workunit.client.1.smithi137.stdout:0/84: read - d1/f17 zero size 2024-03-14T13:02:10.536 INFO:tasks.workunit.client.1.smithi137.stdout:0/85: unlink d1/d5/l7 0 2024-03-14T13:02:10.537 INFO:tasks.workunit.client.1.smithi137.stdout:6/108: link d2/d6/cb d2/d6/c29 0 2024-03-14T13:02:10.539 INFO:tasks.workunit.client.1.smithi137.stdout:0/86: creat d1/d11/f19 x:0 0 0 2024-03-14T13:02:10.539 INFO:tasks.workunit.client.1.smithi137.stdout:6/109: mknod d2/d6/d1d/c2a 0 2024-03-14T13:02:10.540 INFO:tasks.workunit.client.0.smithi111.stdout:2/145: sync 2024-03-14T13:02:10.540 INFO:tasks.workunit.client.0.smithi111.stdout:6/201: sync 2024-03-14T13:02:10.540 INFO:tasks.workunit.client.0.smithi111.stdout:8/413: dwrite d3/dc/f4e [0,4194304] 0 2024-03-14T13:02:10.540 INFO:tasks.workunit.client.0.smithi111.stdout:7/214: sync 2024-03-14T13:02:10.542 INFO:tasks.workunit.client.0.smithi111.stdout:2/146: stat dd/cf 0 2024-03-14T13:02:10.542 INFO:tasks.workunit.client.0.smithi111.stdout:8/414: write d3/d1b/d20/f2b [3964250,905] 0 2024-03-14T13:02:10.543 INFO:tasks.workunit.client.0.smithi111.stdout:6/202: mkdir d0/d17/d20/d4a 0 2024-03-14T13:02:10.543 INFO:tasks.workunit.client.0.smithi111.stdout:7/215: truncate d0/d1/d21/f3f 802698 0 2024-03-14T13:02:10.544 INFO:tasks.workunit.client.0.smithi111.stdout:2/147: creat dd/d11/f2c x:0 0 0 2024-03-14T13:02:10.544 INFO:tasks.workunit.client.0.smithi111.stdout:7/216: fsync d0/f2e 0 2024-03-14T13:02:10.545 INFO:tasks.workunit.client.0.smithi111.stdout:6/203: creat d0/d17/d20/f4b x:0 0 0 2024-03-14T13:02:10.546 INFO:tasks.workunit.client.0.smithi111.stdout:2/148: symlink dd/d24/l2d 0 2024-03-14T13:02:10.547 INFO:tasks.workunit.client.0.smithi111.stdout:7/217: write d0/d1/d5/d38/d9/f3d [5131344,96606] 0 2024-03-14T13:02:10.550 INFO:tasks.workunit.client.0.smithi111.stdout:7/218: symlink d0/d1/d5/l41 0 2024-03-14T13:02:10.551 INFO:tasks.workunit.client.1.smithi137.stdout:0/87: dwrite d1/f17 [0,4194304] 0 2024-03-14T13:02:10.554 INFO:tasks.workunit.client.0.smithi111.stdout:8/415: dread d3/d22/f3f [0,4194304] 0 2024-03-14T13:02:10.555 INFO:tasks.workunit.client.1.smithi137.stdout:0/88: write d1/fd [2762524,43095] 0 2024-03-14T13:02:10.557 INFO:tasks.workunit.client.0.smithi111.stdout:8/416: mknod d3/d49/c86 0 2024-03-14T13:02:10.558 INFO:tasks.workunit.client.1.smithi137.stdout:6/110: dwrite f0 [0,4194304] 0 2024-03-14T13:02:10.559 INFO:tasks.workunit.client.1.smithi137.stdout:6/111: fsync d2/d6/fc 0 2024-03-14T13:02:10.562 INFO:tasks.workunit.client.0.smithi111.stdout:7/219: dwrite d0/d1/d21/f15 [0,4194304] 0 2024-03-14T13:02:10.562 INFO:tasks.workunit.client.0.smithi111.stdout:8/417: mknod d3/d49/c87 0 2024-03-14T13:02:10.562 INFO:tasks.workunit.client.0.smithi111.stdout:2/149: write f1 [1805265,9876] 0 2024-03-14T13:02:10.565 INFO:tasks.workunit.client.1.smithi137.stdout:6/112: symlink d2/d6/l2b 0 2024-03-14T13:02:10.566 INFO:tasks.workunit.client.0.smithi111.stdout:2/150: creat dd/d24/f2e x:0 0 0 2024-03-14T13:02:10.567 INFO:tasks.workunit.client.0.smithi111.stdout:2/151: dread - dd/d11/f25 zero size 2024-03-14T13:02:10.567 INFO:tasks.workunit.client.0.smithi111.stdout:7/220: unlink d0/d1/d5/d38/d9/c18 0 2024-03-14T13:02:10.568 INFO:tasks.workunit.client.1.smithi137.stdout:0/89: dwrite d1/d11/f14 [0,4194304] 0 2024-03-14T13:02:10.569 INFO:tasks.workunit.client.0.smithi111.stdout:2/152: mknod dd/d11/c2f 0 2024-03-14T13:02:10.570 INFO:tasks.workunit.client.0.smithi111.stdout:6/204: dread d0/ff [0,4194304] 0 2024-03-14T13:02:10.570 INFO:tasks.workunit.client.1.smithi137.stdout:6/113: fsync d2/d6/d8/f15 0 2024-03-14T13:02:10.571 INFO:tasks.workunit.client.0.smithi111.stdout:7/221: rename d0/d1/d21/f34 to d0/d1/d22/f42 0 2024-03-14T13:02:10.572 INFO:tasks.workunit.client.1.smithi137.stdout:6/114: chown d2/l5 2857 1 2024-03-14T13:02:10.573 INFO:tasks.workunit.client.0.smithi111.stdout:2/153: write dd/fe [3283706,115015] 0 2024-03-14T13:02:10.573 INFO:tasks.workunit.client.0.smithi111.stdout:7/222: chown d0/d1/d5/d38/d9/c32 1067 1 2024-03-14T13:02:10.574 INFO:tasks.workunit.client.1.smithi137.stdout:0/90: symlink d1/d5/l1a 0 2024-03-14T13:02:10.574 INFO:tasks.workunit.client.0.smithi111.stdout:2/154: dread - dd/d11/f25 zero size 2024-03-14T13:02:10.576 INFO:tasks.workunit.client.1.smithi137.stdout:0/91: fsync d1/d5/f15 0 2024-03-14T13:02:10.579 INFO:tasks.workunit.client.0.smithi111.stdout:6/205: dwrite d0/d17/f2b [0,4194304] 0 2024-03-14T13:02:10.579 INFO:tasks.workunit.client.1.smithi137.stdout:0/92: rmdir d1/d5/d10 39 2024-03-14T13:02:10.583 INFO:tasks.workunit.client.1.smithi137.stdout:6/115: dwrite d2/d6/de/f13 [0,4194304] 0 2024-03-14T13:02:10.584 INFO:tasks.workunit.client.1.smithi137.stdout:6/116: readlink d2/d6/l27 0 2024-03-14T13:02:10.585 INFO:tasks.workunit.client.0.smithi111.stdout:7/223: dread d0/f2e [0,4194304] 0 2024-03-14T13:02:10.588 INFO:tasks.workunit.client.0.smithi111.stdout:2/155: dwrite dd/d24/f2e [0,4194304] 0 2024-03-14T13:02:10.588 INFO:tasks.workunit.client.1.smithi137.stdout:6/117: rename d2/d6/d1d/c2a to d2/d6/de/c2c 0 2024-03-14T13:02:10.590 INFO:tasks.workunit.client.0.smithi111.stdout:2/156: chown dd/d11/d1a/f1c 3 1 2024-03-14T13:02:10.590 INFO:tasks.workunit.client.1.smithi137.stdout:6/118: truncate d2/f25 4552066 0 2024-03-14T13:02:10.590 INFO:tasks.workunit.client.1.smithi137.stdout:6/119: fdatasync d2/f7 0 2024-03-14T13:02:10.593 INFO:tasks.workunit.client.0.smithi111.stdout:2/157: rmdir dd 39 2024-03-14T13:02:10.596 INFO:tasks.workunit.client.0.smithi111.stdout:2/158: truncate dd/d11/f2c 991837 0 2024-03-14T13:02:10.598 INFO:tasks.workunit.client.0.smithi111.stdout:2/159: chown dd/d24/l23 2288 1 2024-03-14T13:02:10.598 INFO:tasks.workunit.client.1.smithi137.stdout:6/120: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:02:10.599 INFO:tasks.workunit.client.0.smithi111.stdout:2/160: chown dd/c15 12123331 1 2024-03-14T13:02:10.601 INFO:tasks.workunit.client.1.smithi137.stdout:6/121: creat d2/f2d x:0 0 0 2024-03-14T13:02:10.601 INFO:tasks.workunit.client.0.smithi111.stdout:7/224: truncate d0/d1/d5/d38/d9/f3d 2126020 0 2024-03-14T13:02:10.602 INFO:tasks.workunit.client.0.smithi111.stdout:2/161: mknod dd/d24/c30 0 2024-03-14T13:02:10.605 INFO:tasks.workunit.client.1.smithi137.stdout:6/122: link d2/l5 d2/d6/l2e 0 2024-03-14T13:02:10.606 INFO:tasks.workunit.client.0.smithi111.stdout:7/225: rename d0/d1/d5/d38/c33 to d0/d1/d21/d2a/c43 0 2024-03-14T13:02:10.608 INFO:tasks.workunit.client.1.smithi137.stdout:6/123: link d2/d6/de/lf d2/d6/l2f 0 2024-03-14T13:02:10.609 INFO:tasks.workunit.client.0.smithi111.stdout:2/162: creat dd/d17/f31 x:0 0 0 2024-03-14T13:02:10.609 INFO:tasks.workunit.client.1.smithi137.stdout:6/124: fsync d2/d6/d8/f1e 0 2024-03-14T13:02:10.610 INFO:tasks.workunit.client.1.smithi137.stdout:6/125: getdents d2/d26 0 2024-03-14T13:02:10.610 INFO:tasks.workunit.client.0.smithi111.stdout:7/226: creat d0/d1/d22/d3e/f44 x:0 0 0 2024-03-14T13:02:10.611 INFO:tasks.workunit.client.0.smithi111.stdout:2/163: symlink dd/d11/d1a/l32 0 2024-03-14T13:02:10.612 INFO:tasks.workunit.client.1.smithi137.stdout:6/126: write d2/d6/d8/f15 [1888654,2499] 0 2024-03-14T13:02:10.615 INFO:tasks.workunit.client.1.smithi137.stdout:0/93: truncate d1/f17 1963448 0 2024-03-14T13:02:10.616 INFO:tasks.workunit.client.1.smithi137.stdout:0/94: chown d1/f8 3 1 2024-03-14T13:02:10.617 INFO:tasks.workunit.client.1.smithi137.stdout:0/95: rmdir d1/d5 39 2024-03-14T13:02:10.620 INFO:tasks.workunit.client.1.smithi137.stdout:0/96: mkdir d1/d5/d1b 0 2024-03-14T13:02:10.621 INFO:tasks.workunit.client.1.smithi137.stdout:0/97: write d1/d11/f19 [353291,77789] 0 2024-03-14T13:02:10.623 INFO:tasks.workunit.client.1.smithi137.stdout:0/98: getdents d1 0 2024-03-14T13:02:10.625 INFO:tasks.workunit.client.1.smithi137.stdout:0/99: creat d1/f1c x:0 0 0 2024-03-14T13:02:10.633 INFO:tasks.workunit.client.1.smithi137.stdout:0/100: symlink d1/l1d 0 2024-03-14T13:02:10.635 INFO:tasks.workunit.client.1.smithi137.stdout:0/101: mkdir d1/d5/d1b/d1e 0 2024-03-14T13:02:10.638 INFO:tasks.workunit.client.1.smithi137.stdout:0/102: rename d1/d5/d10 to d1/d1f 0 2024-03-14T13:02:10.644 INFO:tasks.workunit.client.1.smithi137.stdout:0/103: dwrite d1/d11/f19 [0,4194304] 0 2024-03-14T13:02:10.645 INFO:tasks.workunit.client.1.smithi137.stdout:0/104: write d1/f12 [1059131,23356] 0 2024-03-14T13:02:10.654 INFO:tasks.workunit.client.1.smithi137.stdout:0/105: truncate d1/d11/f19 2856452 0 2024-03-14T13:02:10.656 INFO:tasks.workunit.client.1.smithi137.stdout:0/106: symlink d1/d5/d1b/l20 0 2024-03-14T13:02:10.657 INFO:tasks.workunit.client.1.smithi137.stdout:0/107: creat d1/d11/f21 x:0 0 0 2024-03-14T13:02:10.661 INFO:tasks.workunit.client.1.smithi137.stdout:0/108: read d1/d11/f19 [1913099,29875] 0 2024-03-14T13:02:10.662 INFO:tasks.workunit.client.1.smithi137.stdout:0/109: read d1/fd [928773,113780] 0 2024-03-14T13:02:10.663 INFO:tasks.workunit.client.0.smithi111.stdout:4/513: sync 2024-03-14T13:02:10.663 INFO:tasks.workunit.client.0.smithi111.stdout:1/222: sync 2024-03-14T13:02:10.663 INFO:tasks.workunit.client.0.smithi111.stdout:3/291: sync 2024-03-14T13:02:10.663 INFO:tasks.workunit.client.0.smithi111.stdout:0/201: sync 2024-03-14T13:02:10.664 INFO:tasks.workunit.client.0.smithi111.stdout:0/202: fsync d6/d9/de/f27 0 2024-03-14T13:02:10.666 INFO:tasks.workunit.client.1.smithi137.stdout:0/110: getdents d1/d5 0 2024-03-14T13:02:10.666 INFO:tasks.workunit.client.0.smithi111.stdout:4/514: write d1d/d22/d30/d5e/fb3 [3923928,65431] 0 2024-03-14T13:02:10.667 INFO:tasks.workunit.client.1.smithi137.stdout:0/111: symlink d1/d5/l22 0 2024-03-14T13:02:10.672 INFO:tasks.workunit.client.1.smithi137.stdout:0/112: write d1/f8 [1180819,86001] 0 2024-03-14T13:02:10.672 INFO:tasks.workunit.client.0.smithi111.stdout:0/203: fdatasync d6/f1a 0 2024-03-14T13:02:10.674 INFO:tasks.workunit.client.0.smithi111.stdout:1/223: dwrite d20/d21/d3f/f47 [0,4194304] 0 2024-03-14T13:02:10.675 INFO:tasks.workunit.client.0.smithi111.stdout:1/224: chown d20/d21/l4a 0 1 2024-03-14T13:02:10.675 INFO:tasks.workunit.client.0.smithi111.stdout:1/225: stat d20/c43 0 2024-03-14T13:02:10.676 INFO:tasks.workunit.client.1.smithi137.stdout:0/113: creat d1/d5/d1b/f23 x:0 0 0 2024-03-14T13:02:10.676 INFO:tasks.workunit.client.0.smithi111.stdout:3/292: symlink d0/d2d/d35/d50/d5e/d19/d65/l6d 0 2024-03-14T13:02:10.676 INFO:tasks.workunit.client.1.smithi137.stdout:0/114: stat d1/d5/d1b/d1e 0 2024-03-14T13:02:10.677 INFO:tasks.workunit.client.0.smithi111.stdout:1/226: write d20/d25/f38 [755528,32825] 0 2024-03-14T13:02:10.677 INFO:tasks.workunit.client.0.smithi111.stdout:0/204: creat d6/d21/f42 x:0 0 0 2024-03-14T13:02:10.678 INFO:tasks.workunit.client.0.smithi111.stdout:3/293: write d0/d30/f57 [4014963,48019] 0 2024-03-14T13:02:10.680 INFO:tasks.workunit.client.0.smithi111.stdout:3/294: readlink d0/d2d/d35/d50/d5e/d19/l3f 0 2024-03-14T13:02:10.681 INFO:tasks.workunit.client.0.smithi111.stdout:4/515: mkdir d1d/d22/d30/d5e/db8 0 2024-03-14T13:02:10.682 INFO:tasks.workunit.client.0.smithi111.stdout:1/227: chown d20/d21/d28/c2d 1 1 2024-03-14T13:02:10.682 INFO:tasks.workunit.client.1.smithi137.stdout:0/115: dread d1/f12 [0,4194304] 0 2024-03-14T13:02:10.682 INFO:tasks.workunit.client.0.smithi111.stdout:1/228: stat d20/d21/d28/l35 0 2024-03-14T13:02:10.683 INFO:tasks.workunit.client.0.smithi111.stdout:0/205: unlink d6/f10 0 2024-03-14T13:02:10.685 INFO:tasks.workunit.client.1.smithi137.stdout:0/116: rename d1/d5/d1b/f23 to d1/d1f/f24 0 2024-03-14T13:02:10.687 INFO:tasks.workunit.client.0.smithi111.stdout:4/516: mknod d1d/d22/d30/d5e/db8/cb9 0 2024-03-14T13:02:10.687 INFO:tasks.workunit.client.1.smithi137.stdout:0/117: mkdir d1/d5/d1b/d1e/d25 0 2024-03-14T13:02:10.687 INFO:tasks.workunit.client.1.smithi137.stdout:0/118: read - d1/f13 zero size 2024-03-14T13:02:10.688 INFO:tasks.workunit.client.0.smithi111.stdout:4/517: write d1d/f36 [13938,69624] 0 2024-03-14T13:02:10.688 INFO:tasks.workunit.client.1.smithi137.stdout:0/119: readlink d1/l1d 0 2024-03-14T13:02:10.688 INFO:tasks.workunit.client.0.smithi111.stdout:1/229: symlink d20/l56 0 2024-03-14T13:02:10.689 INFO:tasks.workunit.client.1.smithi137.stdout:0/120: write d1/d11/f14 [2554433,56488] 0 2024-03-14T13:02:10.689 INFO:tasks.workunit.client.0.smithi111.stdout:0/206: mkdir d6/d3d/d43 0 2024-03-14T13:02:10.690 INFO:tasks.workunit.client.0.smithi111.stdout:4/518: mkdir d1d/d22/d30/d5e/d6f/d7e/d66/dba 0 2024-03-14T13:02:10.691 INFO:tasks.workunit.client.0.smithi111.stdout:3/295: rmdir d0/d2d/d35/d50 39 2024-03-14T13:02:10.692 INFO:tasks.workunit.client.1.smithi137.stdout:0/121: mkdir d1/d5/d1b/d1e/d25/d26 0 2024-03-14T13:02:10.692 INFO:tasks.workunit.client.0.smithi111.stdout:0/207: mknod d6/d9/de/d32/c44 0 2024-03-14T13:02:10.692 INFO:tasks.workunit.client.0.smithi111.stdout:0/208: readlink d6/l8 0 2024-03-14T13:02:10.694 INFO:tasks.workunit.client.1.smithi137.stdout:0/122: mkdir d1/d5/d1b/d27 0 2024-03-14T13:02:10.695 INFO:tasks.workunit.client.0.smithi111.stdout:0/209: write d6/d9/d15/f16 [988587,45642] 0 2024-03-14T13:02:10.695 INFO:tasks.workunit.client.0.smithi111.stdout:4/519: rmdir d1d/d22/d40/d69/d86 39 2024-03-14T13:02:10.696 INFO:tasks.workunit.client.1.smithi137.stdout:0/123: link d1/f13 d1/d1f/f28 0 2024-03-14T13:02:10.696 INFO:tasks.workunit.client.0.smithi111.stdout:4/520: read d1d/f20 [1270170,74860] 0 2024-03-14T13:02:10.697 INFO:tasks.workunit.client.0.smithi111.stdout:0/210: write d6/d9/de/f3b [572212,19791] 0 2024-03-14T13:02:10.700 INFO:tasks.workunit.client.1.smithi137.stdout:0/124: rename d1/d5/f15 to d1/f29 0 2024-03-14T13:02:10.701 INFO:tasks.workunit.client.0.smithi111.stdout:4/521: unlink d1d/d22/d30/d7c/f5c 0 2024-03-14T13:02:10.706 INFO:tasks.workunit.client.1.smithi137.stdout:0/125: dwrite d1/f1c [0,4194304] 0 2024-03-14T13:02:10.708 INFO:tasks.workunit.client.0.smithi111.stdout:3/296: rmdir d0/d2d/d35/d50/d5e/d40/d67/d6b 0 2024-03-14T13:02:10.708 INFO:tasks.workunit.client.0.smithi111.stdout:4/522: dwrite d1d/d22/d23/fa2 [0,4194304] 0 2024-03-14T13:02:10.710 INFO:tasks.workunit.client.0.smithi111.stdout:0/211: mkdir d6/d45 0 2024-03-14T13:02:10.711 INFO:tasks.workunit.client.0.smithi111.stdout:0/212: fdatasync d6/d9/d15/f1b 0 2024-03-14T13:02:10.712 INFO:tasks.workunit.client.0.smithi111.stdout:0/213: dread - d6/d21/f42 zero size 2024-03-14T13:02:10.713 INFO:tasks.workunit.client.0.smithi111.stdout:3/297: mknod d0/d2d/d35/d4c/c6e 0 2024-03-14T13:02:10.714 INFO:tasks.workunit.client.1.smithi137.stdout:7/105: sync 2024-03-14T13:02:10.715 INFO:tasks.workunit.client.1.smithi137.stdout:7/106: readlink d3/d6/lc 0 2024-03-14T13:02:10.715 INFO:tasks.workunit.client.1.smithi137.stdout:0/126: truncate d1/d11/f14 3392310 0 2024-03-14T13:02:10.717 INFO:tasks.workunit.client.1.smithi137.stdout:7/107: fsync d3/f18 0 2024-03-14T13:02:10.719 INFO:tasks.workunit.client.1.smithi137.stdout:7/108: write d3/d6/f9 [2094300,5271] 0 2024-03-14T13:02:10.719 INFO:tasks.workunit.client.0.smithi111.stdout:3/298: mkdir d0/d2d/d35/d50/d5e/d40/d6f 0 2024-03-14T13:02:10.721 INFO:tasks.workunit.client.1.smithi137.stdout:0/127: rename d1/f12 to d1/d5/f2a 0 2024-03-14T13:02:10.721 INFO:tasks.workunit.client.0.smithi111.stdout:4/523: link d1d/d22/f54 d1d/fbb 0 2024-03-14T13:02:10.722 INFO:tasks.workunit.client.0.smithi111.stdout:4/524: dread - d1d/d22/d56/f7f zero size 2024-03-14T13:02:10.723 INFO:tasks.workunit.client.1.smithi137.stdout:7/109: dread d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:10.723 INFO:tasks.workunit.client.0.smithi111.stdout:4/525: write d1d/d22/f55 [993486,89414] 0 2024-03-14T13:02:10.725 INFO:tasks.workunit.client.0.smithi111.stdout:0/214: mkdir d6/d9/de/d3a/d46 0 2024-03-14T13:02:10.725 INFO:tasks.workunit.client.0.smithi111.stdout:3/299: link d0/d2d/d35/d50/d5e/d19/c32 d0/d2d/d35/d50/d6c/c70 0 2024-03-14T13:02:10.725 INFO:tasks.workunit.client.1.smithi137.stdout:0/128: dread d1/fd [0,4194304] 0 2024-03-14T13:02:10.726 INFO:tasks.workunit.client.0.smithi111.stdout:4/526: mknod d1d/d22/d30/d7c/cbc 0 2024-03-14T13:02:10.729 INFO:tasks.workunit.client.1.smithi137.stdout:0/129: mknod d1/d5/c2b 0 2024-03-14T13:02:10.730 INFO:tasks.workunit.client.0.smithi111.stdout:0/215: mknod d6/c47 0 2024-03-14T13:02:10.731 INFO:tasks.workunit.client.1.smithi137.stdout:7/110: dwrite d3/f18 [0,4194304] 0 2024-03-14T13:02:10.731 INFO:tasks.workunit.client.1.smithi137.stdout:0/130: symlink d1/d1f/l2c 0 2024-03-14T13:02:10.734 INFO:tasks.workunit.client.1.smithi137.stdout:7/111: symlink d3/d6/l20 0 2024-03-14T13:02:10.735 INFO:tasks.workunit.client.0.smithi111.stdout:3/300: dwrite d0/d2d/d35/f63 [0,4194304] 0 2024-03-14T13:02:10.736 INFO:tasks.workunit.client.1.smithi137.stdout:0/131: creat d1/d5/d1b/d1e/d25/d26/f2d x:0 0 0 2024-03-14T13:02:10.738 INFO:tasks.workunit.client.0.smithi111.stdout:3/301: readlink d0/d2d/d35/d50/d5e/d19/l29 0 2024-03-14T13:02:10.740 INFO:tasks.workunit.client.0.smithi111.stdout:4/527: dwrite d1d/d22/d28/fb6 [0,4194304] 0 2024-03-14T13:02:10.741 INFO:tasks.workunit.client.1.smithi137.stdout:7/112: dread f1 [0,4194304] 0 2024-03-14T13:02:10.746 INFO:tasks.workunit.client.0.smithi111.stdout:4/528: unlink d1d/d22/d23/fb2 0 2024-03-14T13:02:10.749 INFO:tasks.workunit.client.0.smithi111.stdout:0/216: dwrite d6/d9/de/f18 [0,4194304] 0 2024-03-14T13:02:10.750 INFO:tasks.workunit.client.0.smithi111.stdout:4/529: stat d1d/d22/d30/d5e/d6f/c8c 0 2024-03-14T13:02:10.752 INFO:tasks.workunit.client.1.smithi137.stdout:0/132: dwrite d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:10.755 INFO:tasks.workunit.client.0.smithi111.stdout:4/530: mkdir d1d/d22/d30/d5e/dbd 0 2024-03-14T13:02:10.757 INFO:tasks.workunit.client.1.smithi137.stdout:7/113: dwrite d3/d6/d14/f1b [0,4194304] 0 2024-03-14T13:02:10.758 INFO:tasks.workunit.client.0.smithi111.stdout:4/531: mkdir d1d/d48/dbe 0 2024-03-14T13:02:10.759 INFO:tasks.workunit.client.0.smithi111.stdout:4/532: write d1d/d22/f41 [1544185,54793] 0 2024-03-14T13:02:10.761 INFO:tasks.workunit.client.1.smithi137.stdout:0/133: rename d1/d5/ce to d1/d5/d1b/d1e/c2e 0 2024-03-14T13:02:10.763 INFO:tasks.workunit.client.1.smithi137.stdout:7/114: symlink d3/l21 0 2024-03-14T13:02:10.765 INFO:tasks.workunit.client.0.smithi111.stdout:4/533: creat d1d/d22/d40/d69/d86/db1/fbf x:0 0 0 2024-03-14T13:02:10.767 INFO:tasks.workunit.client.1.smithi137.stdout:0/134: dread d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:10.769 INFO:tasks.workunit.client.1.smithi137.stdout:0/135: read - d1/d11/f21 zero size 2024-03-14T13:02:10.769 INFO:tasks.workunit.client.0.smithi111.stdout:4/534: unlink d1d/d22/d30/d5e/d6f/d7e/c8b 0 2024-03-14T13:02:10.770 INFO:tasks.workunit.client.0.smithi111.stdout:3/302: truncate d0/f8 440519 0 2024-03-14T13:02:10.774 INFO:tasks.workunit.client.0.smithi111.stdout:3/303: getdents d0/d2d/d35/d50 0 2024-03-14T13:02:10.778 INFO:tasks.workunit.client.1.smithi137.stdout:0/136: dwrite d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:10.778 INFO:tasks.workunit.client.1.smithi137.stdout:0/137: read - d1/f29 zero size 2024-03-14T13:02:10.780 INFO:tasks.workunit.client.0.smithi111.stdout:3/304: dwrite d0/d2d/d35/d50/d5e/f12 [4194304,4194304] 0 2024-03-14T13:02:10.781 INFO:tasks.workunit.client.1.smithi137.stdout:0/138: unlink d1/fd 0 2024-03-14T13:02:10.784 INFO:tasks.workunit.client.0.smithi111.stdout:3/305: mkdir d0/d2d/d35/d50/d5e/d40/d71 0 2024-03-14T13:02:10.785 INFO:tasks.workunit.client.0.smithi111.stdout:0/217: write d6/d9/f12 [2463735,39476] 0 2024-03-14T13:02:10.786 INFO:tasks.workunit.client.1.smithi137.stdout:7/115: fsync d3/f18 0 2024-03-14T13:02:10.786 INFO:tasks.workunit.client.0.smithi111.stdout:3/306: symlink d0/d2d/l72 0 2024-03-14T13:02:10.788 INFO:tasks.workunit.client.1.smithi137.stdout:5/67: sync 2024-03-14T13:02:10.789 INFO:tasks.workunit.client.0.smithi111.stdout:3/307: rmdir d0/d2d/d35/d50/d5e/d19/d65 39 2024-03-14T13:02:10.792 INFO:tasks.workunit.client.0.smithi111.stdout:0/218: symlink d6/d9/de/d32/l48 0 2024-03-14T13:02:10.793 INFO:tasks.workunit.client.1.smithi137.stdout:2/92: sync 2024-03-14T13:02:10.793 INFO:tasks.workunit.client.0.smithi111.stdout:4/535: write d1d/d22/f54 [3676081,102936] 0 2024-03-14T13:02:10.793 INFO:tasks.workunit.client.0.smithi111.stdout:0/219: chown d6/f1a 2 1 2024-03-14T13:02:10.795 INFO:tasks.workunit.client.0.smithi111.stdout:4/536: dread - d1d/d22/d28/fb0 zero size 2024-03-14T13:02:10.796 INFO:tasks.workunit.client.1.smithi137.stdout:2/93: symlink d2/d7/l15 0 2024-03-14T13:02:10.796 INFO:tasks.workunit.client.0.smithi111.stdout:4/537: chown d1d/d22/f37 844 1 2024-03-14T13:02:10.796 INFO:tasks.workunit.client.0.smithi111.stdout:3/308: rename d0/d2d/d35/d50/d5e/d19/l45 to d0/d2d/d35/d50/d5e/d19/d65/l73 0 2024-03-14T13:02:10.797 INFO:tasks.workunit.client.1.smithi137.stdout:7/116: dwrite d3/f18 [0,4194304] 0 2024-03-14T13:02:10.798 INFO:tasks.workunit.client.0.smithi111.stdout:3/309: creat d0/d2d/d35/d50/f74 x:0 0 0 2024-03-14T13:02:10.799 INFO:tasks.workunit.client.1.smithi137.stdout:5/68: dwrite f4 [0,4194304] 0 2024-03-14T13:02:10.800 INFO:tasks.workunit.client.1.smithi137.stdout:7/117: mkdir d3/d6/d14/d22 0 2024-03-14T13:02:10.800 INFO:tasks.workunit.client.0.smithi111.stdout:0/220: getdents d6/d9/de 0 2024-03-14T13:02:10.800 INFO:tasks.workunit.client.0.smithi111.stdout:3/310: chown d0/d30/l4f 5433 1 2024-03-14T13:02:10.801 INFO:tasks.workunit.client.1.smithi137.stdout:7/118: dread - d3/d6/d14/f1a zero size 2024-03-14T13:02:10.803 INFO:tasks.workunit.client.1.smithi137.stdout:2/94: link d2/d7/f13 d2/d7/db/f16 0 2024-03-14T13:02:10.803 INFO:tasks.workunit.client.0.smithi111.stdout:0/221: unlink d6/d9/de/c14 0 2024-03-14T13:02:10.804 INFO:tasks.workunit.client.0.smithi111.stdout:0/222: read - d6/d9/d15/f2e zero size 2024-03-14T13:02:10.806 INFO:tasks.workunit.client.1.smithi137.stdout:7/119: write d3/d5/f11 [4302211,121007] 0 2024-03-14T13:02:10.807 INFO:tasks.workunit.client.0.smithi111.stdout:0/223: mkdir d6/d9/d15/d49 0 2024-03-14T13:02:10.809 INFO:tasks.workunit.client.0.smithi111.stdout:0/224: truncate d6/d9/d15/f2e 85337 0 2024-03-14T13:02:10.812 INFO:tasks.workunit.client.0.smithi111.stdout:3/311: rename d0/d2d/d35/d50/d5e/d19/d65 to d0/d2d/d35/d50/d5e/d19/d75 0 2024-03-14T13:02:10.814 INFO:tasks.workunit.client.1.smithi137.stdout:7/120: read d3/fe [726662,82501] 0 2024-03-14T13:02:10.815 INFO:tasks.workunit.client.0.smithi111.stdout:0/225: chown d6/d9/de/d3a/d46 1146149061 1 2024-03-14T13:02:10.815 INFO:tasks.workunit.client.0.smithi111.stdout:5/312: sync 2024-03-14T13:02:10.816 INFO:tasks.workunit.client.0.smithi111.stdout:9/235: sync 2024-03-14T13:02:10.817 INFO:tasks.workunit.client.0.smithi111.stdout:5/313: chown d2/d16/l41 2764325 1 2024-03-14T13:02:10.819 INFO:tasks.workunit.client.0.smithi111.stdout:9/236: creat d5/d6/d8/f49 x:0 0 0 2024-03-14T13:02:10.819 INFO:tasks.workunit.client.1.smithi137.stdout:5/69: dwrite d7/ff [0,4194304] 0 2024-03-14T13:02:10.820 INFO:tasks.workunit.client.0.smithi111.stdout:0/226: mkdir d6/d9/de/d3a/d46/d4a 0 2024-03-14T13:02:10.821 INFO:tasks.workunit.client.0.smithi111.stdout:3/312: dwrite d0/d2d/d35/d50/d5e/d40/f47 [0,4194304] 0 2024-03-14T13:02:10.821 INFO:tasks.workunit.client.0.smithi111.stdout:5/314: write d2/f35 [927697,8074] 0 2024-03-14T13:02:10.823 INFO:tasks.workunit.client.1.smithi137.stdout:0/139: dwrite d1/f13 [0,4194304] 0 2024-03-14T13:02:10.827 INFO:tasks.workunit.client.1.smithi137.stdout:5/70: fdatasync d7/f10 0 2024-03-14T13:02:10.827 INFO:tasks.workunit.client.1.smithi137.stdout:2/95: dwrite d2/f5 [4194304,4194304] 0 2024-03-14T13:02:10.830 INFO:tasks.workunit.client.1.smithi137.stdout:2/96: read d2/f5 [1971703,31268] 0 2024-03-14T13:02:10.830 INFO:tasks.workunit.client.1.smithi137.stdout:0/140: dread - d1/f29 zero size 2024-03-14T13:02:10.834 INFO:tasks.workunit.client.1.smithi137.stdout:5/71: mkdir d7/d16 0 2024-03-14T13:02:10.836 INFO:tasks.workunit.client.1.smithi137.stdout:0/141: fsync d1/f29 0 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.0.smithi111.stdout:0/227: dwrite d6/d9/de/f23 [0,4194304] 0 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:1/161: sync 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:6/127: sync 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:3/123: sync 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:9/177: sync 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:4/190: sync 2024-03-14T13:02:10.838 INFO:tasks.workunit.client.1.smithi137.stdout:8/92: sync 2024-03-14T13:02:10.839 INFO:tasks.workunit.client.1.smithi137.stdout:5/72: read - d7/f15 zero size 2024-03-14T13:02:10.839 INFO:tasks.workunit.client.0.smithi111.stdout:9/237: dread d5/d6/d2a/d2d/f32 [0,4194304] 0 2024-03-14T13:02:10.839 INFO:tasks.workunit.client.0.smithi111.stdout:5/315: dwrite d2/d16/d49/d44/d4a/f63 [0,4194304] 0 2024-03-14T13:02:10.842 INFO:tasks.workunit.client.1.smithi137.stdout:2/97: dwrite d2/fa [0,4194304] 0 2024-03-14T13:02:10.842 INFO:tasks.workunit.client.1.smithi137.stdout:3/124: chown d4/de/d16/f1a 294005 1 2024-03-14T13:02:10.843 INFO:tasks.workunit.client.0.smithi111.stdout:9/238: dread d5/d6/d2a/d2d/f32 [0,4194304] 0 2024-03-14T13:02:10.844 INFO:tasks.workunit.client.1.smithi137.stdout:9/178: write d6/df/d19/d1f/f2a [1138422,31966] 0 2024-03-14T13:02:10.847 INFO:tasks.workunit.client.0.smithi111.stdout:9/239: fsync f2 0 2024-03-14T13:02:10.848 INFO:tasks.workunit.client.0.smithi111.stdout:9/240: fdatasync d5/fa 0 2024-03-14T13:02:10.848 INFO:tasks.workunit.client.0.smithi111.stdout:5/316: truncate d2/d16/d49/d44/d4a/f62 296190 0 2024-03-14T13:02:10.848 INFO:tasks.workunit.client.1.smithi137.stdout:1/162: write d7/d9/d19/d1f/d22/f25 [1147146,130294] 0 2024-03-14T13:02:10.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/128: symlink d2/d26/l30 0 2024-03-14T13:02:10.849 INFO:tasks.workunit.client.1.smithi137.stdout:3/125: write d4/de/d16/f1a [4806382,105924] 0 2024-03-14T13:02:10.853 INFO:tasks.workunit.client.0.smithi111.stdout:0/228: dwrite d6/f1e [0,4194304] 0 2024-03-14T13:02:10.853 INFO:tasks.workunit.client.1.smithi137.stdout:5/73: rename f4 to d7/d16/f17 0 2024-03-14T13:02:10.853 INFO:tasks.workunit.client.1.smithi137.stdout:9/179: dread - d6/df/d19/d33/f3c zero size 2024-03-14T13:02:10.853 INFO:tasks.workunit.client.1.smithi137.stdout:2/98: mkdir d2/d7/db/de/d17 0 2024-03-14T13:02:10.854 INFO:tasks.workunit.client.1.smithi137.stdout:5/74: chown d7/ld 20095241 1 2024-03-14T13:02:10.856 INFO:tasks.workunit.client.0.smithi111.stdout:5/317: truncate d2/d16/d49/d44/f50 1038723 0 2024-03-14T13:02:10.858 INFO:tasks.workunit.client.1.smithi137.stdout:1/163: creat d7/d9/d19/f2f x:0 0 0 2024-03-14T13:02:10.859 INFO:tasks.workunit.client.0.smithi111.stdout:9/241: unlink d5/c17 0 2024-03-14T13:02:10.860 INFO:tasks.workunit.client.0.smithi111.stdout:0/229: creat d6/d45/f4b x:0 0 0 2024-03-14T13:02:10.861 INFO:tasks.workunit.client.0.smithi111.stdout:5/318: dread d2/d16/d49/d33/f40 [0,4194304] 0 2024-03-14T13:02:10.862 INFO:tasks.workunit.client.0.smithi111.stdout:9/242: chown d5/d6/d8/f3a 402 1 2024-03-14T13:02:10.862 INFO:tasks.workunit.client.1.smithi137.stdout:3/126: mkdir d4/d22 0 2024-03-14T13:02:10.862 INFO:tasks.workunit.client.0.smithi111.stdout:0/230: mknod d6/d3d/c4c 0 2024-03-14T13:02:10.863 INFO:tasks.workunit.client.1.smithi137.stdout:7/121: truncate d3/fe 3600338 0 2024-03-14T13:02:10.863 INFO:tasks.workunit.client.1.smithi137.stdout:4/191: creat d4/d10/d26/f40 x:0 0 0 2024-03-14T13:02:10.863 INFO:tasks.workunit.client.0.smithi111.stdout:0/231: chown d6/d9/f12 0 1 2024-03-14T13:02:10.864 INFO:tasks.workunit.client.0.smithi111.stdout:5/319: chown d2/d13/f4b 491686512 1 2024-03-14T13:02:10.864 INFO:tasks.workunit.client.0.smithi111.stdout:5/320: read d2/d16/d49/d33/f40 [53217,82258] 0 2024-03-14T13:02:10.866 INFO:tasks.workunit.client.0.smithi111.stdout:5/321: write d2/d13/f29 [2037139,77917] 0 2024-03-14T13:02:10.866 INFO:tasks.workunit.client.1.smithi137.stdout:9/180: rename d6/df/d19/d33/f3c to d6/df/d19/d3a/d40/f41 0 2024-03-14T13:02:10.866 INFO:tasks.workunit.client.1.smithi137.stdout:6/129: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:02:10.867 INFO:tasks.workunit.client.1.smithi137.stdout:3/127: dread - d4/de/d13/f21 zero size 2024-03-14T13:02:10.867 INFO:tasks.workunit.client.1.smithi137.stdout:2/99: mknod d2/d7/db/c18 0 2024-03-14T13:02:10.867 INFO:tasks.workunit.client.0.smithi111.stdout:0/232: symlink d6/d9/de/d32/l4d 0 2024-03-14T13:02:10.867 INFO:tasks.workunit.client.0.smithi111.stdout:5/322: chown d2/d45/f66 1265 1 2024-03-14T13:02:10.869 INFO:tasks.workunit.client.1.smithi137.stdout:1/164: symlink d7/d9/d19/d1f/l30 0 2024-03-14T13:02:10.870 INFO:tasks.workunit.client.1.smithi137.stdout:5/75: dread d7/fb [0,4194304] 0 2024-03-14T13:02:10.870 INFO:tasks.workunit.client.0.smithi111.stdout:0/233: mknod d6/d9/d15/d49/c4e 0 2024-03-14T13:02:10.870 INFO:tasks.workunit.client.1.smithi137.stdout:4/192: fdatasync d4/d10/f25 0 2024-03-14T13:02:10.871 INFO:tasks.workunit.client.0.smithi111.stdout:5/323: creat d2/d1f/d6a/f6e x:0 0 0 2024-03-14T13:02:10.871 INFO:tasks.workunit.client.1.smithi137.stdout:2/100: write d2/d7/db/ff [250866,19796] 0 2024-03-14T13:02:10.873 INFO:tasks.workunit.client.1.smithi137.stdout:1/165: stat d7/d9/d19/d1f/f23 0 2024-03-14T13:02:10.873 INFO:tasks.workunit.client.1.smithi137.stdout:3/128: creat d4/d19/f23 x:0 0 0 2024-03-14T13:02:10.874 INFO:tasks.workunit.client.1.smithi137.stdout:5/76: mkdir d7/d16/d18 0 2024-03-14T13:02:10.875 INFO:tasks.workunit.client.1.smithi137.stdout:2/101: fsync d2/d7/f13 0 2024-03-14T13:02:10.878 INFO:tasks.workunit.client.1.smithi137.stdout:1/166: mkdir d7/d31 0 2024-03-14T13:02:10.879 INFO:tasks.workunit.client.1.smithi137.stdout:5/77: creat d7/f19 x:0 0 0 2024-03-14T13:02:10.880 INFO:tasks.workunit.client.1.smithi137.stdout:3/129: symlink d4/d19/l24 0 2024-03-14T13:02:10.881 INFO:tasks.workunit.client.0.smithi111.stdout:5/324: dwrite d2/d16/d49/d33/f43 [0,4194304] 0 2024-03-14T13:02:10.882 INFO:tasks.workunit.client.1.smithi137.stdout:4/193: rename d4/d9/d1b to d4/d9/d3a/d41 0 2024-03-14T13:02:10.883 INFO:tasks.workunit.client.1.smithi137.stdout:9/181: dread d6/df/f20 [4194304,4194304] 0 2024-03-14T13:02:10.883 INFO:tasks.workunit.client.1.smithi137.stdout:1/167: fsync f5 0 2024-03-14T13:02:10.883 INFO:tasks.workunit.client.1.smithi137.stdout:6/130: dwrite d2/f2d [0,4194304] 0 2024-03-14T13:02:10.884 INFO:tasks.workunit.client.0.smithi111.stdout:3/313: rmdir d0/d2d 39 2024-03-14T13:02:10.884 INFO:tasks.workunit.client.0.smithi111.stdout:0/234: dwrite d6/d9/de/f26 [0,4194304] 0 2024-03-14T13:02:10.885 INFO:tasks.workunit.client.0.smithi111.stdout:5/325: symlink d2/d16/d49/l6f 0 2024-03-14T13:02:10.885 INFO:tasks.workunit.client.1.smithi137.stdout:2/102: symlink d2/l19 0 2024-03-14T13:02:10.887 INFO:tasks.workunit.client.0.smithi111.stdout:5/326: chown d2/d13/l30 71888296 1 2024-03-14T13:02:10.888 INFO:tasks.workunit.client.0.smithi111.stdout:3/314: mknod d0/d2d/d35/d50/c76 0 2024-03-14T13:02:10.891 INFO:tasks.workunit.client.0.smithi111.stdout:5/327: rename d2/d1f/f57 to d2/d16/f70 0 2024-03-14T13:02:10.892 INFO:tasks.workunit.client.0.smithi111.stdout:9/243: truncate f4 2915646 0 2024-03-14T13:02:10.893 INFO:tasks.workunit.client.0.smithi111.stdout:9/244: readlink d5/d6/d8/l44 0 2024-03-14T13:02:10.893 INFO:tasks.workunit.client.0.smithi111.stdout:5/328: write d2/d16/d49/d33/f43 [72727,75502] 0 2024-03-14T13:02:10.894 INFO:tasks.workunit.client.1.smithi137.stdout:5/78: unlink d7/fe 0 2024-03-14T13:02:10.894 INFO:tasks.workunit.client.1.smithi137.stdout:3/130: stat d4/de/d13/c1b 0 2024-03-14T13:02:10.894 INFO:tasks.workunit.client.0.smithi111.stdout:3/315: creat d0/d2d/d35/d50/d5e/d19/d75/f77 x:0 0 0 2024-03-14T13:02:10.895 INFO:tasks.workunit.client.1.smithi137.stdout:3/131: chown d4/de/d13/c15 3634737 1 2024-03-14T13:02:10.897 INFO:tasks.workunit.client.0.smithi111.stdout:9/245: creat d5/d6/d2a/d2d/f4a x:0 0 0 2024-03-14T13:02:10.898 INFO:tasks.workunit.client.1.smithi137.stdout:0/142: dwrite d1/f17 [0,4194304] 0 2024-03-14T13:02:10.902 INFO:tasks.workunit.client.0.smithi111.stdout:5/329: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:02:10.904 INFO:tasks.workunit.client.1.smithi137.stdout:4/194: mkdir d4/d10/d42 0 2024-03-14T13:02:10.904 INFO:tasks.workunit.client.0.smithi111.stdout:5/330: chown d2/d16/d2c/l4e 0 1 2024-03-14T13:02:10.905 INFO:tasks.workunit.client.0.smithi111.stdout:5/331: chown d2/d1f/d6a 11205681 1 2024-03-14T13:02:10.905 INFO:tasks.workunit.client.0.smithi111.stdout:3/316: dread d0/d2d/d35/d50/d5e/f12 [4194304,4194304] 0 2024-03-14T13:02:10.906 INFO:tasks.workunit.client.0.smithi111.stdout:9/246: getdents d5/d42 0 2024-03-14T13:02:10.907 INFO:tasks.workunit.client.0.smithi111.stdout:9/247: write d5/d6/f3d [3532112,12136] 0 2024-03-14T13:02:10.907 INFO:tasks.workunit.client.0.smithi111.stdout:5/332: creat d2/d1f/d6a/f71 x:0 0 0 2024-03-14T13:02:10.909 INFO:tasks.workunit.client.0.smithi111.stdout:5/333: truncate d2/d13/f4f 562583 0 2024-03-14T13:02:10.911 INFO:tasks.workunit.client.1.smithi137.stdout:6/131: dwrite d2/d6/d8/f1e [0,4194304] 0 2024-03-14T13:02:10.911 INFO:tasks.workunit.client.1.smithi137.stdout:1/168: symlink d7/l32 0 2024-03-14T13:02:10.911 INFO:tasks.workunit.client.1.smithi137.stdout:2/103: dread d2/fa [0,4194304] 0 2024-03-14T13:02:10.911 INFO:tasks.workunit.client.0.smithi111.stdout:3/317: mknod d0/d2d/d35/c78 0 2024-03-14T13:02:10.912 INFO:tasks.workunit.client.1.smithi137.stdout:4/195: dread - d4/d9/d3a/d41/d38/f3b zero size 2024-03-14T13:02:10.916 INFO:tasks.workunit.client.0.smithi111.stdout:3/318: dread d0/d2d/d35/d50/d5e/d19/f5a [0,4194304] 0 2024-03-14T13:02:10.918 INFO:tasks.workunit.client.0.smithi111.stdout:5/334: rename d2/d16/d2c to d2/d16/d49/d44/d72 0 2024-03-14T13:02:10.918 INFO:tasks.workunit.client.1.smithi137.stdout:2/104: dread - d2/d7/db/f16 zero size 2024-03-14T13:02:10.918 INFO:tasks.workunit.client.0.smithi111.stdout:5/335: dread - d2/d45/f5c zero size 2024-03-14T13:02:10.919 INFO:tasks.workunit.client.1.smithi137.stdout:5/79: dwrite d7/fa [0,4194304] 0 2024-03-14T13:02:10.919 INFO:tasks.workunit.client.1.smithi137.stdout:3/132: dread d4/f18 [0,4194304] 0 2024-03-14T13:02:10.920 INFO:tasks.workunit.client.0.smithi111.stdout:9/248: link d5/d6/d14/c48 d5/c4b 0 2024-03-14T13:02:10.921 INFO:tasks.workunit.client.1.smithi137.stdout:1/169: write d7/d29/f2b [90729,128005] 0 2024-03-14T13:02:10.921 INFO:tasks.workunit.client.1.smithi137.stdout:8/93: truncate d1/f8 114868 0 2024-03-14T13:02:10.922 INFO:tasks.workunit.client.0.smithi111.stdout:5/336: creat d2/d16/d49/d4c/f73 x:0 0 0 2024-03-14T13:02:10.924 INFO:tasks.workunit.client.1.smithi137.stdout:1/170: chown d7/d9/c17 10518 1 2024-03-14T13:02:10.924 INFO:tasks.workunit.client.0.smithi111.stdout:9/249: symlink d5/l4c 0 2024-03-14T13:02:10.926 INFO:tasks.workunit.client.0.smithi111.stdout:9/250: write d5/d6/d2a/f3e [921476,59450] 0 2024-03-14T13:02:10.928 INFO:tasks.workunit.client.1.smithi137.stdout:6/132: symlink d2/d6/d1d/l31 0 2024-03-14T13:02:10.929 INFO:tasks.workunit.client.1.smithi137.stdout:9/182: rename d6/d26/l2f to d6/df/l42 0 2024-03-14T13:02:10.931 INFO:tasks.workunit.client.1.smithi137.stdout:4/196: symlink d4/d9/d3a/d41/d38/l43 0 2024-03-14T13:02:10.932 INFO:tasks.workunit.client.1.smithi137.stdout:5/80: creat d7/d16/f1a x:0 0 0 2024-03-14T13:02:10.934 INFO:tasks.workunit.client.1.smithi137.stdout:2/105: rmdir d2/d7/db 39 2024-03-14T13:02:10.935 INFO:tasks.workunit.client.1.smithi137.stdout:3/133: creat d4/de/d13/f25 x:0 0 0 2024-03-14T13:02:10.935 INFO:tasks.workunit.client.0.smithi111.stdout:9/251: dread d5/fe [0,4194304] 0 2024-03-14T13:02:10.936 INFO:tasks.workunit.client.1.smithi137.stdout:4/197: write d4/f12 [3241570,96675] 0 2024-03-14T13:02:10.936 INFO:tasks.workunit.client.1.smithi137.stdout:3/134: dread - d4/de/d13/f21 zero size 2024-03-14T13:02:10.937 INFO:tasks.workunit.client.1.smithi137.stdout:5/81: write d7/fa [3882005,4116] 0 2024-03-14T13:02:10.938 INFO:tasks.workunit.client.0.smithi111.stdout:9/252: symlink d5/d6/l4d 0 2024-03-14T13:02:10.938 INFO:tasks.workunit.client.1.smithi137.stdout:3/135: fsync d4/de/d16/f1a 0 2024-03-14T13:02:10.939 INFO:tasks.workunit.client.0.smithi111.stdout:9/253: chown d5/d6/f2f 0 1 2024-03-14T13:02:10.939 INFO:tasks.workunit.client.1.smithi137.stdout:6/133: readlink d2/d6/l2e 0 2024-03-14T13:02:10.941 INFO:tasks.workunit.client.1.smithi137.stdout:1/171: rename d7/d9/d19/d1f/l30 to d7/d9/d19/d1f/d22/l33 0 2024-03-14T13:02:10.941 INFO:tasks.workunit.client.1.smithi137.stdout:3/136: fdatasync d4/d19/f23 0 2024-03-14T13:02:10.942 INFO:tasks.workunit.client.0.smithi111.stdout:3/319: truncate d0/f7 231097 0 2024-03-14T13:02:10.943 INFO:tasks.workunit.client.1.smithi137.stdout:1/172: fdatasync d7/d9/fd 0 2024-03-14T13:02:10.943 INFO:tasks.workunit.client.1.smithi137.stdout:6/134: dread d2/d6/d8/f1f [0,4194304] 0 2024-03-14T13:02:10.944 INFO:tasks.workunit.client.1.smithi137.stdout:9/183: creat d6/df/d19/d3a/f43 x:0 0 0 2024-03-14T13:02:10.946 INFO:tasks.workunit.client.1.smithi137.stdout:3/137: truncate f1 2834933 0 2024-03-14T13:02:10.946 INFO:tasks.workunit.client.0.smithi111.stdout:9/254: dwrite d5/d6/fc [0,4194304] 0 2024-03-14T13:02:10.947 INFO:tasks.workunit.client.0.smithi111.stdout:3/320: rename d0/d30/f4a to d0/d2d/d35/d50/d5e/d40/d6f/f79 0 2024-03-14T13:02:10.948 INFO:tasks.workunit.client.1.smithi137.stdout:6/135: fsync f0 0 2024-03-14T13:02:10.949 INFO:tasks.workunit.client.1.smithi137.stdout:8/94: fdatasync d1/f8 0 2024-03-14T13:02:10.949 INFO:tasks.workunit.client.1.smithi137.stdout:0/143: getdents d1/d1f 0 2024-03-14T13:02:10.949 INFO:tasks.workunit.client.0.smithi111.stdout:9/255: rename d5/d6/f3d to d5/d6/d2a/f4e 0 2024-03-14T13:02:10.950 INFO:tasks.workunit.client.0.smithi111.stdout:3/321: symlink d0/d2d/l7a 0 2024-03-14T13:02:10.951 INFO:tasks.workunit.client.0.smithi111.stdout:3/322: write d0/d2d/d35/f61 [1028415,107330] 0 2024-03-14T13:02:10.952 INFO:tasks.workunit.client.1.smithi137.stdout:4/198: dread d4/f1a [4194304,4194304] 0 2024-03-14T13:02:10.953 INFO:tasks.workunit.client.0.smithi111.stdout:9/256: creat d5/f4f x:0 0 0 2024-03-14T13:02:10.956 INFO:tasks.workunit.client.1.smithi137.stdout:9/184: write d6/df/d19/d1f/f34 [821338,98317] 0 2024-03-14T13:02:10.959 INFO:tasks.workunit.client.1.smithi137.stdout:5/82: dread d7/d16/f17 [0,4194304] 0 2024-03-14T13:02:10.960 INFO:tasks.workunit.client.0.smithi111.stdout:3/323: link d0/d2d/d35/d50/d5e/d40/d67/l68 d0/d2d/d35/d4c/l7b 0 2024-03-14T13:02:10.961 INFO:tasks.workunit.client.0.smithi111.stdout:9/257: link d5/d6/f2f d5/f50 0 2024-03-14T13:02:10.961 INFO:tasks.workunit.client.1.smithi137.stdout:0/144: creat d1/d1f/f2f x:0 0 0 2024-03-14T13:02:10.962 INFO:tasks.workunit.client.0.smithi111.stdout:3/324: truncate d0/d2d/d35/d50/f56 955114 0 2024-03-14T13:02:10.962 INFO:tasks.workunit.client.1.smithi137.stdout:4/199: chown d4/c39 65 1 2024-03-14T13:02:10.962 INFO:tasks.workunit.client.1.smithi137.stdout:1/173: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:10.964 INFO:tasks.workunit.client.0.smithi111.stdout:9/258: mknod d5/d6/c51 0 2024-03-14T13:02:10.964 INFO:tasks.workunit.client.0.smithi111.stdout:3/325: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:10.964 INFO:tasks.workunit.client.1.smithi137.stdout:0/145: chown d1/d5/l22 324 1 2024-03-14T13:02:10.969 INFO:tasks.workunit.client.0.smithi111.stdout:3/326: creat d0/d2d/d35/d50/d5e/d40/d71/f7c x:0 0 0 2024-03-14T13:02:10.969 INFO:tasks.workunit.client.1.smithi137.stdout:3/138: rename d4/de/d13/c1b to d4/de/d16/c26 0 2024-03-14T13:02:10.972 INFO:tasks.workunit.client.1.smithi137.stdout:6/136: creat d2/d6/de/f32 x:0 0 0 2024-03-14T13:02:10.973 INFO:tasks.workunit.client.0.smithi111.stdout:9/259: dwrite d5/d6/d8/d1a/f43 [0,4194304] 0 2024-03-14T13:02:10.975 INFO:tasks.workunit.client.0.smithi111.stdout:3/327: dread d0/d30/f38 [0,4194304] 0 2024-03-14T13:02:10.975 INFO:tasks.workunit.client.1.smithi137.stdout:2/106: mknod d2/d7/db/de/d17/c1a 0 2024-03-14T13:02:10.977 INFO:tasks.workunit.client.0.smithi111.stdout:9/260: write d5/d42/f46 [437587,2644] 0 2024-03-14T13:02:10.978 INFO:tasks.workunit.client.0.smithi111.stdout:5/337: dread d2/d16/f70 [4194304,4194304] 0 2024-03-14T13:02:10.983 INFO:tasks.workunit.client.1.smithi137.stdout:3/139: mkdir d4/de/d16/d27 0 2024-03-14T13:02:10.983 INFO:tasks.workunit.client.1.smithi137.stdout:8/95: mknod d1/d17/c1b 0 2024-03-14T13:02:10.984 INFO:tasks.workunit.client.0.smithi111.stdout:3/328: dread d0/d2d/d35/d50/f54 [0,4194304] 0 2024-03-14T13:02:10.984 INFO:tasks.workunit.client.1.smithi137.stdout:6/137: mkdir d2/d6/d8/d33 0 2024-03-14T13:02:10.986 INFO:tasks.workunit.client.0.smithi111.stdout:3/329: chown d0/d30/l4f 98842 1 2024-03-14T13:02:10.986 INFO:tasks.workunit.client.0.smithi111.stdout:9/261: chown d5/fe 2911360 1 2024-03-14T13:02:10.987 INFO:tasks.workunit.client.1.smithi137.stdout:6/138: write d2/f25 [540619,38160] 0 2024-03-14T13:02:10.988 INFO:tasks.workunit.client.0.smithi111.stdout:5/338: mkdir d2/d74 0 2024-03-14T13:02:10.988 INFO:tasks.workunit.client.1.smithi137.stdout:0/146: dwrite d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:10.990 INFO:tasks.workunit.client.1.smithi137.stdout:2/107: creat d2/d7/db/de/f1b x:0 0 0 2024-03-14T13:02:10.991 INFO:tasks.workunit.client.0.smithi111.stdout:9/262: rmdir d5/d6/d2a 39 2024-03-14T13:02:10.992 INFO:tasks.workunit.client.0.smithi111.stdout:3/330: mknod d0/d2d/d35/d50/d5e/c7d 0 2024-03-14T13:02:10.992 INFO:tasks.workunit.client.1.smithi137.stdout:4/200: mknod d4/c44 0 2024-03-14T13:02:10.993 INFO:tasks.workunit.client.1.smithi137.stdout:1/174: creat d7/d9/d19/d1f/d2e/f34 x:0 0 0 2024-03-14T13:02:10.994 INFO:tasks.workunit.client.1.smithi137.stdout:3/140: creat d4/de/f28 x:0 0 0 2024-03-14T13:02:10.995 INFO:tasks.workunit.client.1.smithi137.stdout:3/141: stat d4/de/d13/f21 0 2024-03-14T13:02:10.996 INFO:tasks.workunit.client.1.smithi137.stdout:8/96: creat d1/d17/f1c x:0 0 0 2024-03-14T13:02:10.996 INFO:tasks.workunit.client.0.smithi111.stdout:3/331: creat d0/d2d/d35/d50/d5e/d19/d75/f7e x:0 0 0 2024-03-14T13:02:10.997 INFO:tasks.workunit.client.1.smithi137.stdout:3/142: truncate d4/de/d16/f20 747188 0 2024-03-14T13:02:10.997 INFO:tasks.workunit.client.1.smithi137.stdout:8/97: read - d1/d17/f1c zero size 2024-03-14T13:02:10.998 INFO:tasks.workunit.client.1.smithi137.stdout:3/143: fsync d4/de/d13/f21 0 2024-03-14T13:02:10.999 INFO:tasks.workunit.client.1.smithi137.stdout:3/144: truncate d4/fd 978393 0 2024-03-14T13:02:11.000 INFO:tasks.workunit.client.1.smithi137.stdout:9/185: truncate d6/fe 3487475 0 2024-03-14T13:02:11.001 INFO:tasks.workunit.client.1.smithi137.stdout:3/145: write d4/de/f28 [1018341,43065] 0 2024-03-14T13:02:11.001 INFO:tasks.workunit.client.0.smithi111.stdout:5/339: link d2/d16/l41 d2/d1f/l75 0 2024-03-14T13:02:11.002 INFO:tasks.workunit.client.1.smithi137.stdout:3/146: stat d4/de/d16 0 2024-03-14T13:02:11.003 INFO:tasks.workunit.client.1.smithi137.stdout:3/147: readlink d4/l8 0 2024-03-14T13:02:11.003 INFO:tasks.workunit.client.0.smithi111.stdout:5/340: mkdir d2/d45/d76 0 2024-03-14T13:02:11.004 INFO:tasks.workunit.client.1.smithi137.stdout:1/175: dread d7/d9/d19/d1f/d22/f25 [0,4194304] 0 2024-03-14T13:02:11.004 INFO:tasks.workunit.client.1.smithi137.stdout:0/147: creat d1/d5/d1b/d1e/f30 x:0 0 0 2024-03-14T13:02:11.004 INFO:tasks.workunit.client.1.smithi137.stdout:2/108: creat d2/d7/db/f1c x:0 0 0 2024-03-14T13:02:11.006 INFO:tasks.workunit.client.1.smithi137.stdout:3/148: truncate d4/de/d13/f21 744482 0 2024-03-14T13:02:11.006 INFO:tasks.workunit.client.0.smithi111.stdout:9/263: dwrite d5/d6/d2a/f4e [4194304,4194304] 0 2024-03-14T13:02:11.006 INFO:tasks.workunit.client.1.smithi137.stdout:4/201: rename d4/d9/c30 to d4/d9/d3a/d41/c45 0 2024-03-14T13:02:11.007 INFO:tasks.workunit.client.1.smithi137.stdout:9/186: creat d6/df/d19/d3a/f44 x:0 0 0 2024-03-14T13:02:11.008 INFO:tasks.workunit.client.0.smithi111.stdout:5/341: link d2/f7 d2/d74/f77 0 2024-03-14T13:02:11.008 INFO:tasks.workunit.client.1.smithi137.stdout:3/149: fsync d4/de/d16/f1a 0 2024-03-14T13:02:11.009 INFO:tasks.workunit.client.1.smithi137.stdout:4/202: chown d4/d9/d3a 1140 1 2024-03-14T13:02:11.009 INFO:tasks.workunit.client.1.smithi137.stdout:6/139: unlink d2/d6/de/c2c 0 2024-03-14T13:02:11.009 INFO:tasks.workunit.client.0.smithi111.stdout:9/264: mkdir d5/d42/d52 0 2024-03-14T13:02:11.010 INFO:tasks.workunit.client.1.smithi137.stdout:0/148: mkdir d1/d5/d1b/d1e/d31 0 2024-03-14T13:02:11.011 INFO:tasks.workunit.client.0.smithi111.stdout:5/342: creat d2/d16/d49/f78 x:0 0 0 2024-03-14T13:02:11.014 INFO:tasks.workunit.client.1.smithi137.stdout:8/98: rename d1/dc/d10/l18 to d1/l1d 0 2024-03-14T13:02:11.014 INFO:tasks.workunit.client.1.smithi137.stdout:3/150: rmdir d4/de/d13 39 2024-03-14T13:02:11.015 INFO:tasks.workunit.client.0.smithi111.stdout:9/265: link d5/d6/d8/l44 d5/d6/d2a/l53 0 2024-03-14T13:02:11.015 INFO:tasks.workunit.client.1.smithi137.stdout:4/203: unlink d4/d9/lb 0 2024-03-14T13:02:11.016 INFO:tasks.workunit.client.1.smithi137.stdout:3/151: stat d4/de 0 2024-03-14T13:02:11.016 INFO:tasks.workunit.client.1.smithi137.stdout:1/176: dread f5 [0,4194304] 0 2024-03-14T13:02:11.017 INFO:tasks.workunit.client.1.smithi137.stdout:5/83: link d7/f19 d7/f1b 0 2024-03-14T13:02:11.017 INFO:tasks.workunit.client.1.smithi137.stdout:9/187: dread d6/df/d19/d1f/f3b [0,4194304] 0 2024-03-14T13:02:11.018 INFO:tasks.workunit.client.0.smithi111.stdout:9/266: rename d5/d6/d8/f41 to d5/d6/d14/f54 0 2024-03-14T13:02:11.019 INFO:tasks.workunit.client.0.smithi111.stdout:3/332: chown d0/f7 10 1 2024-03-14T13:02:11.019 INFO:tasks.workunit.client.1.smithi137.stdout:3/152: stat d4/fd 0 2024-03-14T13:02:11.020 INFO:tasks.workunit.client.1.smithi137.stdout:6/140: creat d2/d6/f34 x:0 0 0 2024-03-14T13:02:11.020 INFO:tasks.workunit.client.1.smithi137.stdout:4/204: write d4/d9/f18 [373439,115433] 0 2024-03-14T13:02:11.020 INFO:tasks.workunit.client.0.smithi111.stdout:9/267: write d5/d6/d2a/d2d/f32 [363973,58189] 0 2024-03-14T13:02:11.024 INFO:tasks.workunit.client.0.smithi111.stdout:9/268: unlink d5/d6/d2a/d2d/f32 0 2024-03-14T13:02:11.025 INFO:tasks.workunit.client.1.smithi137.stdout:1/177: dread d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:11.025 INFO:tasks.workunit.client.1.smithi137.stdout:2/109: dwrite d2/fa [0,4194304] 0 2024-03-14T13:02:11.027 INFO:tasks.workunit.client.1.smithi137.stdout:0/149: dread d1/d11/f19 [0,4194304] 0 2024-03-14T13:02:11.029 INFO:tasks.workunit.client.1.smithi137.stdout:6/141: creat d2/d6/f35 x:0 0 0 2024-03-14T13:02:11.033 INFO:tasks.workunit.client.0.smithi111.stdout:9/269: getdents d5/d6/d8 0 2024-03-14T13:02:11.035 INFO:tasks.workunit.client.1.smithi137.stdout:8/99: dwrite d1/d17/f1c [0,4194304] 0 2024-03-14T13:02:11.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/84: rename d7/d16/f17 to d7/d16/f1c 0 2024-03-14T13:02:11.041 INFO:tasks.workunit.client.1.smithi137.stdout:3/153: write d4/de/d13/f21 [1279265,73184] 0 2024-03-14T13:02:11.042 INFO:tasks.workunit.client.1.smithi137.stdout:0/150: dwrite d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:11.043 INFO:tasks.workunit.client.0.smithi111.stdout:9/270: dread d5/d6/ff [0,4194304] 0 2024-03-14T13:02:11.044 INFO:tasks.workunit.client.1.smithi137.stdout:9/188: dread d6/df/f10 [0,4194304] 0 2024-03-14T13:02:11.044 INFO:tasks.workunit.client.1.smithi137.stdout:0/151: readlink d1/d5/d1b/l20 0 2024-03-14T13:02:11.045 INFO:tasks.workunit.client.0.smithi111.stdout:9/271: truncate d5/d6/d2a/d2d/f4a 810006 0 2024-03-14T13:02:11.048 INFO:tasks.workunit.client.1.smithi137.stdout:0/152: dread d1/d11/f19 [0,4194304] 0 2024-03-14T13:02:11.050 INFO:tasks.workunit.client.1.smithi137.stdout:2/110: rmdir d2/d7/db/de 39 2024-03-14T13:02:11.051 INFO:tasks.workunit.client.0.smithi111.stdout:9/272: dwrite d5/f31 [0,4194304] 0 2024-03-14T13:02:11.052 INFO:tasks.workunit.client.1.smithi137.stdout:8/100: creat d1/d17/f1e x:0 0 0 2024-03-14T13:02:11.052 INFO:tasks.workunit.client.1.smithi137.stdout:4/205: symlink d4/d10/d42/l46 0 2024-03-14T13:02:11.053 INFO:tasks.workunit.client.1.smithi137.stdout:1/178: symlink d7/d31/l35 0 2024-03-14T13:02:11.054 INFO:tasks.workunit.client.1.smithi137.stdout:3/154: creat d4/de/d13/f29 x:0 0 0 2024-03-14T13:02:11.054 INFO:tasks.workunit.client.0.smithi111.stdout:9/273: mkdir d5/d6/d14/d55 0 2024-03-14T13:02:11.055 INFO:tasks.workunit.client.1.smithi137.stdout:5/85: chown d7/fb 402482670 1 2024-03-14T13:02:11.057 INFO:tasks.workunit.client.1.smithi137.stdout:6/142: dread d2/d6/d8/f15 [0,4194304] 0 2024-03-14T13:02:11.057 INFO:tasks.workunit.client.1.smithi137.stdout:5/86: stat d7/d16/f1c 0 2024-03-14T13:02:11.058 INFO:tasks.workunit.client.0.smithi111.stdout:7/227: sync 2024-03-14T13:02:11.058 INFO:tasks.workunit.client.0.smithi111.stdout:6/206: sync 2024-03-14T13:02:11.058 INFO:tasks.workunit.client.0.smithi111.stdout:2/164: sync 2024-03-14T13:02:11.058 INFO:tasks.workunit.client.0.smithi111.stdout:8/418: sync 2024-03-14T13:02:11.058 INFO:tasks.workunit.client.0.smithi111.stdout:9/274: creat d5/d6/f56 x:0 0 0 2024-03-14T13:02:11.059 INFO:tasks.workunit.client.1.smithi137.stdout:8/101: creat d1/dc/f1f x:0 0 0 2024-03-14T13:02:11.059 INFO:tasks.workunit.client.1.smithi137.stdout:4/206: readlink d4/d9/d3a/d41/l35 0 2024-03-14T13:02:11.059 INFO:tasks.workunit.client.1.smithi137.stdout:7/122: sync 2024-03-14T13:02:11.060 INFO:tasks.workunit.client.0.smithi111.stdout:6/207: mknod d0/c4c 0 2024-03-14T13:02:11.061 INFO:tasks.workunit.client.0.smithi111.stdout:2/165: creat dd/d11/f33 x:0 0 0 2024-03-14T13:02:11.061 INFO:tasks.workunit.client.1.smithi137.stdout:0/153: creat d1/d5/d1b/d1e/d31/f32 x:0 0 0 2024-03-14T13:02:11.062 INFO:tasks.workunit.client.0.smithi111.stdout:7/228: mknod d0/d1/c45 0 2024-03-14T13:02:11.063 INFO:tasks.workunit.client.1.smithi137.stdout:1/179: symlink d7/d9/d19/d1f/l36 0 2024-03-14T13:02:11.063 INFO:tasks.workunit.client.0.smithi111.stdout:7/229: chown d0/d1/d5/l41 855970716 1 2024-03-14T13:02:11.063 INFO:tasks.workunit.client.0.smithi111.stdout:8/419: rename d3/d1b/f3a to d3/d22/f88 0 2024-03-14T13:02:11.064 INFO:tasks.workunit.client.1.smithi137.stdout:2/111: mkdir d2/d7/db/de/d1d 0 2024-03-14T13:02:11.064 INFO:tasks.workunit.client.0.smithi111.stdout:2/166: rmdir dd/d11/d1a 39 2024-03-14T13:02:11.065 INFO:tasks.workunit.client.1.smithi137.stdout:6/143: symlink d2/d6/de/l36 0 2024-03-14T13:02:11.066 INFO:tasks.workunit.client.1.smithi137.stdout:5/87: unlink d7/d16/f1c 0 2024-03-14T13:02:11.066 INFO:tasks.workunit.client.0.smithi111.stdout:7/230: chown d0/d11/f29 33234 1 2024-03-14T13:02:11.067 INFO:tasks.workunit.client.1.smithi137.stdout:8/102: mknod d1/d17/c20 0 2024-03-14T13:02:11.067 INFO:tasks.workunit.client.0.smithi111.stdout:9/275: link d5/d6/d8/d1a/f43 d5/d6/d8/f57 0 2024-03-14T13:02:11.067 INFO:tasks.workunit.client.0.smithi111.stdout:7/231: fdatasync d0/f2e 0 2024-03-14T13:02:11.070 INFO:tasks.workunit.client.0.smithi111.stdout:6/208: dwrite d0/d17/d24/f38 [0,4194304] 0 2024-03-14T13:02:11.070 INFO:tasks.workunit.client.1.smithi137.stdout:0/154: symlink d1/d1f/l33 0 2024-03-14T13:02:11.071 INFO:tasks.workunit.client.0.smithi111.stdout:9/276: rmdir d5/d6/d2a/d2d 39 2024-03-14T13:02:11.071 INFO:tasks.workunit.client.1.smithi137.stdout:1/180: mknod d7/d9/d19/d1f/c37 0 2024-03-14T13:02:11.072 INFO:tasks.workunit.client.1.smithi137.stdout:1/181: dread - d7/d9/d19/d1f/d2e/f34 zero size 2024-03-14T13:02:11.073 INFO:tasks.workunit.client.0.smithi111.stdout:7/232: creat d0/d1/d22/f46 x:0 0 0 2024-03-14T13:02:11.075 INFO:tasks.workunit.client.0.smithi111.stdout:2/167: dwrite f1 [0,4194304] 0 2024-03-14T13:02:11.076 INFO:tasks.workunit.client.1.smithi137.stdout:4/207: symlink d4/d9/l47 0 2024-03-14T13:02:11.077 INFO:tasks.workunit.client.0.smithi111.stdout:6/209: rename d0/d17/f40 to d0/d17/d21/d42/d49/f4d 0 2024-03-14T13:02:11.078 INFO:tasks.workunit.client.1.smithi137.stdout:4/208: chown d4/d9/f1f 55003931 1 2024-03-14T13:02:11.079 INFO:tasks.workunit.client.0.smithi111.stdout:9/277: creat d5/d6/d2a/f58 x:0 0 0 2024-03-14T13:02:11.081 INFO:tasks.workunit.client.1.smithi137.stdout:8/103: creat d1/dc/d10/f21 x:0 0 0 2024-03-14T13:02:11.082 INFO:tasks.workunit.client.1.smithi137.stdout:0/155: symlink d1/d5/d1b/d1e/d31/l34 0 2024-03-14T13:02:11.082 INFO:tasks.workunit.client.0.smithi111.stdout:2/168: write dd/d11/d1a/f1c [2353397,124552] 0 2024-03-14T13:02:11.083 INFO:tasks.workunit.client.1.smithi137.stdout:3/155: creat d4/de/f2a x:0 0 0 2024-03-14T13:02:11.088 INFO:tasks.workunit.client.1.smithi137.stdout:2/112: creat d2/f1e x:0 0 0 2024-03-14T13:02:11.088 INFO:tasks.workunit.client.0.smithi111.stdout:2/169: rename dd/d24 to dd/d17/d34 0 2024-03-14T13:02:11.089 INFO:tasks.workunit.client.1.smithi137.stdout:6/144: write d2/d6/d8/f1f [264491,58592] 0 2024-03-14T13:02:11.089 INFO:tasks.workunit.client.0.smithi111.stdout:6/210: dwrite d0/d17/d21/d42/d49/f4d [0,4194304] 0 2024-03-14T13:02:11.089 INFO:tasks.workunit.client.1.smithi137.stdout:8/104: creat d1/dc/d11/d13/f22 x:0 0 0 2024-03-14T13:02:11.091 INFO:tasks.workunit.client.0.smithi111.stdout:6/211: write d0/d17/d20/f26 [4040131,54116] 0 2024-03-14T13:02:11.091 INFO:tasks.workunit.client.1.smithi137.stdout:0/156: dread d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:11.091 INFO:tasks.workunit.client.1.smithi137.stdout:9/189: write d6/df/f14 [2695689,113420] 0 2024-03-14T13:02:11.092 INFO:tasks.workunit.client.1.smithi137.stdout:3/156: creat d4/f2b x:0 0 0 2024-03-14T13:02:11.092 INFO:tasks.workunit.client.1.smithi137.stdout:6/145: dread - d2/d6/f35 zero size 2024-03-14T13:02:11.094 INFO:tasks.workunit.client.1.smithi137.stdout:8/105: fdatasync d1/dc/d10/f1a 0 2024-03-14T13:02:11.094 INFO:tasks.workunit.client.1.smithi137.stdout:1/182: symlink d7/l38 0 2024-03-14T13:02:11.096 INFO:tasks.workunit.client.1.smithi137.stdout:5/88: dwrite d7/f11 [0,4194304] 0 2024-03-14T13:02:11.096 INFO:tasks.workunit.client.1.smithi137.stdout:7/123: getdents d3 0 2024-03-14T13:02:11.098 INFO:tasks.workunit.client.0.smithi111.stdout:9/278: rmdir d5 39 2024-03-14T13:02:11.099 INFO:tasks.workunit.client.1.smithi137.stdout:1/183: stat d7/c16 0 2024-03-14T13:02:11.099 INFO:tasks.workunit.client.1.smithi137.stdout:2/113: chown d2/d7/db/de/f1b 6 1 2024-03-14T13:02:11.100 INFO:tasks.workunit.client.1.smithi137.stdout:0/157: readlink d1/d5/d1b/d1e/d31/l34 0 2024-03-14T13:02:11.101 INFO:tasks.workunit.client.1.smithi137.stdout:5/89: chown d7/fb 26313 1 2024-03-14T13:02:11.103 INFO:tasks.workunit.client.1.smithi137.stdout:2/114: read - d2/d7/db/f1c zero size 2024-03-14T13:02:11.105 INFO:tasks.workunit.client.0.smithi111.stdout:9/279: truncate d5/d6/f22 5509950 0 2024-03-14T13:02:11.105 INFO:tasks.workunit.client.1.smithi137.stdout:0/158: fdatasync d1/d5/f2a 0 2024-03-14T13:02:11.106 INFO:tasks.workunit.client.1.smithi137.stdout:7/124: mkdir d3/d5/d23 0 2024-03-14T13:02:11.109 INFO:tasks.workunit.client.0.smithi111.stdout:9/280: rename d5/d6/d14/c48 to d5/d6/d8/c59 0 2024-03-14T13:02:11.109 INFO:tasks.workunit.client.1.smithi137.stdout:9/190: creat d6/df/d19/f45 x:0 0 0 2024-03-14T13:02:11.109 INFO:tasks.workunit.client.0.smithi111.stdout:9/281: readlink d5/d6/l37 0 2024-03-14T13:02:11.110 INFO:tasks.workunit.client.1.smithi137.stdout:9/191: write d6/f2b [1842853,69480] 0 2024-03-14T13:02:11.112 INFO:tasks.workunit.client.1.smithi137.stdout:8/106: mknod d1/d17/c23 0 2024-03-14T13:02:11.112 INFO:tasks.workunit.client.1.smithi137.stdout:5/90: rmdir d7 39 2024-03-14T13:02:11.113 INFO:tasks.workunit.client.1.smithi137.stdout:9/192: dread - d6/df/d19/d33/f3e zero size 2024-03-14T13:02:11.113 INFO:tasks.workunit.client.1.smithi137.stdout:2/115: rmdir d2/d7/db 39 2024-03-14T13:02:11.114 INFO:tasks.workunit.client.1.smithi137.stdout:6/146: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:11.115 INFO:tasks.workunit.client.1.smithi137.stdout:0/159: chown d1/d5/d1b/d1e/d31/f32 45729869 1 2024-03-14T13:02:11.115 INFO:tasks.workunit.client.0.smithi111.stdout:9/282: dwrite d5/d42/f46 [0,4194304] 0 2024-03-14T13:02:11.116 INFO:tasks.workunit.client.1.smithi137.stdout:7/125: rmdir d3 39 2024-03-14T13:02:11.117 INFO:tasks.workunit.client.0.smithi111.stdout:9/283: fdatasync d5/d6/d2a/f58 0 2024-03-14T13:02:11.117 INFO:tasks.workunit.client.1.smithi137.stdout:8/107: creat d1/dc/d10/f24 x:0 0 0 2024-03-14T13:02:11.118 INFO:tasks.workunit.client.1.smithi137.stdout:8/108: dread - d1/d17/f1e zero size 2024-03-14T13:02:11.119 INFO:tasks.workunit.client.0.smithi111.stdout:9/284: mkdir d5/d6/d2a/d5a 0 2024-03-14T13:02:11.121 INFO:tasks.workunit.client.1.smithi137.stdout:9/193: rename d6/fa to d6/df/d19/d33/f46 0 2024-03-14T13:02:11.124 INFO:tasks.workunit.client.0.smithi111.stdout:6/212: dwrite d0/ff [0,4194304] 0 2024-03-14T13:02:11.127 INFO:tasks.workunit.client.1.smithi137.stdout:7/126: mknod d3/d5/c24 0 2024-03-14T13:02:11.127 INFO:tasks.workunit.client.1.smithi137.stdout:2/116: symlink d2/d7/l1f 0 2024-03-14T13:02:11.128 INFO:tasks.workunit.client.1.smithi137.stdout:5/91: mkdir d7/d1d 0 2024-03-14T13:02:11.129 INFO:tasks.workunit.client.1.smithi137.stdout:9/194: creat d6/df/d19/d3a/f47 x:0 0 0 2024-03-14T13:02:11.129 INFO:tasks.workunit.client.0.smithi111.stdout:2/170: dread dd/fe [0,4194304] 0 2024-03-14T13:02:11.130 INFO:tasks.workunit.client.1.smithi137.stdout:6/147: creat d2/d6/d8/d28/f37 x:0 0 0 2024-03-14T13:02:11.131 INFO:tasks.workunit.client.1.smithi137.stdout:9/195: chown d6/df/d19/d3a 1025212251 1 2024-03-14T13:02:11.132 INFO:tasks.workunit.client.1.smithi137.stdout:5/92: truncate d7/fb 1390889 0 2024-03-14T13:02:11.132 INFO:tasks.workunit.client.1.smithi137.stdout:0/160: rename d1/d5/d1b/d27 to d1/d5/d1b/d1e/d25/d35 0 2024-03-14T13:02:11.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/184: link d7/c28 d7/c39 0 2024-03-14T13:02:11.134 INFO:tasks.workunit.client.0.smithi111.stdout:9/285: dwrite d5/f26 [0,4194304] 0 2024-03-14T13:02:11.135 INFO:tasks.workunit.client.1.smithi137.stdout:0/161: dread - d1/d11/f21 zero size 2024-03-14T13:02:11.138 INFO:tasks.workunit.client.0.smithi111.stdout:9/286: mkdir d5/d6/d8/d5b 0 2024-03-14T13:02:11.138 INFO:tasks.workunit.client.0.smithi111.stdout:6/213: link d0/d17/d20/l3a d0/d17/l4e 0 2024-03-14T13:02:11.139 INFO:tasks.workunit.client.1.smithi137.stdout:9/196: dread - d6/df/d19/d3a/f43 zero size 2024-03-14T13:02:11.141 INFO:tasks.workunit.client.1.smithi137.stdout:6/148: write d2/d6/d8/f15 [179575,28478] 0 2024-03-14T13:02:11.142 INFO:tasks.workunit.client.0.smithi111.stdout:9/287: fsync d5/d6/d8/d1a/f43 0 2024-03-14T13:02:11.143 INFO:tasks.workunit.client.0.smithi111.stdout:6/214: symlink d0/d17/d21/d42/d49/l4f 0 2024-03-14T13:02:11.143 INFO:tasks.workunit.client.0.smithi111.stdout:2/171: dwrite dd/d11/d1a/f21 [0,4194304] 0 2024-03-14T13:02:11.143 INFO:tasks.workunit.client.1.smithi137.stdout:8/109: rename d1/dc/d11/d13/l14 to d1/dc/l25 0 2024-03-14T13:02:11.145 INFO:tasks.workunit.client.1.smithi137.stdout:7/127: dwrite d3/d6/d14/f1d [0,4194304] 0 2024-03-14T13:02:11.147 INFO:tasks.workunit.client.0.smithi111.stdout:9/288: fdatasync d5/d6/f22 0 2024-03-14T13:02:11.148 INFO:tasks.workunit.client.1.smithi137.stdout:0/162: chown d1/d5/d1b/d1e/d31/l34 0 1 2024-03-14T13:02:11.148 INFO:tasks.workunit.client.0.smithi111.stdout:6/215: chown d0/l13 414 1 2024-03-14T13:02:11.149 INFO:tasks.workunit.client.0.smithi111.stdout:6/216: fsync d0/f1c 0 2024-03-14T13:02:11.151 INFO:tasks.workunit.client.0.smithi111.stdout:2/172: symlink dd/d17/l35 0 2024-03-14T13:02:11.151 INFO:tasks.workunit.client.1.smithi137.stdout:8/110: dread - d1/dc/f1f zero size 2024-03-14T13:02:11.153 INFO:tasks.workunit.client.0.smithi111.stdout:2/173: write dd/d11/d1a/f1b [1274160,90756] 0 2024-03-14T13:02:11.153 INFO:tasks.workunit.client.0.smithi111.stdout:6/217: mkdir d0/d50 0 2024-03-14T13:02:11.155 INFO:tasks.workunit.client.0.smithi111.stdout:9/289: dwrite d5/f26 [0,4194304] 0 2024-03-14T13:02:11.157 INFO:tasks.workunit.client.0.smithi111.stdout:6/218: read d0/d17/d21/f44 [2311283,73327] 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:5/93: dread d7/fa [0,4194304] 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:7/128: mknod d3/c25 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:6/149: mkdir d2/d6/d8/d38 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:8/111: fdatasync d1/d17/f1c 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:9/197: unlink d6/df/d19/d3a/f44 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:6/150: write d2/d6/de/f32 [655543,48464] 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.1.smithi137.stdout:9/198: truncate d6/fe 3958072 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.0.smithi111.stdout:9/290: unlink d5/d6/d8/f57 0 2024-03-14T13:02:11.160 INFO:tasks.workunit.client.0.smithi111.stdout:6/219: write d0/d17/d24/f46 [243537,79814] 0 2024-03-14T13:02:11.161 INFO:tasks.workunit.client.1.smithi137.stdout:1/185: dread d7/d9/ff [0,4194304] 0 2024-03-14T13:02:11.161 INFO:tasks.workunit.client.1.smithi137.stdout:5/94: symlink d7/l1e 0 2024-03-14T13:02:11.163 INFO:tasks.workunit.client.0.smithi111.stdout:2/174: getdents dd/d11/d1a 0 2024-03-14T13:02:11.165 INFO:tasks.workunit.client.0.smithi111.stdout:2/175: chown dd/d17/d34/l1e 42848104 1 2024-03-14T13:02:11.166 INFO:tasks.workunit.client.1.smithi137.stdout:8/112: creat d1/dc/d11/d13/f26 x:0 0 0 2024-03-14T13:02:11.167 INFO:tasks.workunit.client.0.smithi111.stdout:6/220: creat d0/d17/d20/d4a/f51 x:0 0 0 2024-03-14T13:02:11.169 INFO:tasks.workunit.client.1.smithi137.stdout:0/163: getdents d1/d5/d1b/d1e/d25/d26 0 2024-03-14T13:02:11.170 INFO:tasks.workunit.client.0.smithi111.stdout:6/221: unlink d0/f1d 0 2024-03-14T13:02:11.170 INFO:tasks.workunit.client.0.smithi111.stdout:9/291: dwrite d5/d6/d8/f30 [0,4194304] 0 2024-03-14T13:02:11.173 INFO:tasks.workunit.client.0.smithi111.stdout:9/292: chown d5/d6/d8/f12 27698561 1 2024-03-14T13:02:11.173 INFO:tasks.workunit.client.1.smithi137.stdout:1/186: dwrite f3 [4194304,4194304] 0 2024-03-14T13:02:11.174 INFO:tasks.workunit.client.1.smithi137.stdout:1/187: fdatasync d7/f1d 0 2024-03-14T13:02:11.175 INFO:tasks.workunit.client.0.smithi111.stdout:6/222: creat d0/d17/d21/d42/d49/f52 x:0 0 0 2024-03-14T13:02:11.175 INFO:tasks.workunit.client.0.smithi111.stdout:2/176: dwrite dd/d17/f31 [0,4194304] 0 2024-03-14T13:02:11.176 INFO:tasks.workunit.client.1.smithi137.stdout:9/199: mknod d6/df/d19/d3a/d40/c48 0 2024-03-14T13:02:11.176 INFO:tasks.workunit.client.1.smithi137.stdout:6/151: mkdir d2/d6/d8/d28/d39 0 2024-03-14T13:02:11.177 INFO:tasks.workunit.client.0.smithi111.stdout:9/293: creat d5/d42/f5c x:0 0 0 2024-03-14T13:02:11.179 INFO:tasks.workunit.client.1.smithi137.stdout:1/188: write d7/d9/d19/d1f/d2e/f34 [177564,118432] 0 2024-03-14T13:02:11.181 INFO:tasks.workunit.client.0.smithi111.stdout:6/223: mkdir d0/d17/d21/d42/d49/d3d/d53 0 2024-03-14T13:02:11.183 INFO:tasks.workunit.client.0.smithi111.stdout:2/177: dread dd/d17/d34/f2e [0,4194304] 0 2024-03-14T13:02:11.184 INFO:tasks.workunit.client.0.smithi111.stdout:2/178: stat dd/d11/d1a/c27 0 2024-03-14T13:02:11.185 INFO:tasks.workunit.client.1.smithi137.stdout:0/164: fdatasync d1/d1f/f24 0 2024-03-14T13:02:11.185 INFO:tasks.workunit.client.1.smithi137.stdout:6/152: write d2/f7 [412979,5862] 0 2024-03-14T13:02:11.185 INFO:tasks.workunit.client.1.smithi137.stdout:0/165: fsync d1/d11/f21 0 2024-03-14T13:02:11.186 INFO:tasks.workunit.client.0.smithi111.stdout:9/294: creat d5/d6/d2a/d5a/f5d x:0 0 0 2024-03-14T13:02:11.186 INFO:tasks.workunit.client.0.smithi111.stdout:6/224: symlink d0/d17/d21/d42/d49/d3d/l54 0 2024-03-14T13:02:11.188 INFO:tasks.workunit.client.1.smithi137.stdout:0/166: write d1/d11/f21 [117446,87640] 0 2024-03-14T13:02:11.188 INFO:tasks.workunit.client.0.smithi111.stdout:2/179: creat dd/d17/d34/f36 x:0 0 0 2024-03-14T13:02:11.190 INFO:tasks.workunit.client.1.smithi137.stdout:2/117: write d2/d7/db/f16 [672118,32801] 0 2024-03-14T13:02:11.191 INFO:tasks.workunit.client.0.smithi111.stdout:9/295: symlink d5/d6/d2a/d2d/l5e 0 2024-03-14T13:02:11.192 INFO:tasks.workunit.client.1.smithi137.stdout:8/113: dwrite d1/dc/d11/d13/f16 [0,4194304] 0 2024-03-14T13:02:11.193 INFO:tasks.workunit.client.0.smithi111.stdout:6/225: creat d0/d17/f55 x:0 0 0 2024-03-14T13:02:11.194 INFO:tasks.workunit.client.0.smithi111.stdout:2/180: creat dd/d17/f37 x:0 0 0 2024-03-14T13:02:11.198 INFO:tasks.workunit.client.0.smithi111.stdout:9/296: link d5/d6/d2a/c2c d5/d42/c5f 0 2024-03-14T13:02:11.198 INFO:tasks.workunit.client.1.smithi137.stdout:6/153: symlink d2/d6/d8/d33/l3a 0 2024-03-14T13:02:11.202 INFO:tasks.workunit.client.1.smithi137.stdout:8/114: creat d1/dc/d10/f27 x:0 0 0 2024-03-14T13:02:11.202 INFO:tasks.workunit.client.1.smithi137.stdout:6/154: rmdir d2/d26 39 2024-03-14T13:02:11.202 INFO:tasks.workunit.client.1.smithi137.stdout:7/129: write d3/ff [558291,21617] 0 2024-03-14T13:02:11.204 INFO:tasks.workunit.client.1.smithi137.stdout:1/189: getdents d7/d9/d19/d1f/d26 0 2024-03-14T13:02:11.204 INFO:tasks.workunit.client.1.smithi137.stdout:2/118: dwrite d2/d7/db/f1c [0,4194304] 0 2024-03-14T13:02:11.205 INFO:tasks.workunit.client.1.smithi137.stdout:8/115: fdatasync d1/dc/d10/f27 0 2024-03-14T13:02:11.209 INFO:tasks.workunit.client.1.smithi137.stdout:8/116: symlink d1/dc/d11/l28 0 2024-03-14T13:02:11.209 INFO:tasks.workunit.client.0.smithi111.stdout:9/297: rmdir d5/d6/d2a/d5a 39 2024-03-14T13:02:11.210 INFO:tasks.workunit.client.1.smithi137.stdout:5/95: write d7/f15 [420086,69548] 0 2024-03-14T13:02:11.211 INFO:tasks.workunit.client.1.smithi137.stdout:6/155: mknod d2/c3b 0 2024-03-14T13:02:11.212 INFO:tasks.workunit.client.1.smithi137.stdout:2/119: dread d2/d7/db/f16 [0,4194304] 0 2024-03-14T13:02:11.213 INFO:tasks.workunit.client.0.smithi111.stdout:9/298: chown d5/d6/d2a/d5a 12484510 1 2024-03-14T13:02:11.213 INFO:tasks.workunit.client.1.smithi137.stdout:1/190: unlink d7/c28 0 2024-03-14T13:02:11.216 INFO:tasks.workunit.client.0.smithi111.stdout:2/181: getdents dd/d17/d34 0 2024-03-14T13:02:11.218 INFO:tasks.workunit.client.1.smithi137.stdout:7/130: dwrite d3/ff [0,4194304] 0 2024-03-14T13:02:11.219 INFO:tasks.workunit.client.0.smithi111.stdout:9/299: dwrite d5/d6/d2a/d5a/f5d [0,4194304] 0 2024-03-14T13:02:11.223 INFO:tasks.workunit.client.0.smithi111.stdout:2/182: dread dd/d17/d34/f2e [0,4194304] 0 2024-03-14T13:02:11.224 INFO:tasks.workunit.client.1.smithi137.stdout:5/96: write d7/ff [5077653,68217] 0 2024-03-14T13:02:11.225 INFO:tasks.workunit.client.1.smithi137.stdout:8/117: unlink d1/f2 0 2024-03-14T13:02:11.225 INFO:tasks.workunit.client.1.smithi137.stdout:2/120: rmdir d2/d7/db/de 39 2024-03-14T13:02:11.225 INFO:tasks.workunit.client.1.smithi137.stdout:1/191: creat d7/d31/f3a x:0 0 0 2024-03-14T13:02:11.225 INFO:tasks.workunit.client.1.smithi137.stdout:0/167: dwrite d1/d1f/f24 [4194304,4194304] 0 2024-03-14T13:02:11.225 INFO:tasks.workunit.client.0.smithi111.stdout:2/183: truncate dd/d11/f33 368325 0 2024-03-14T13:02:11.226 INFO:tasks.workunit.client.1.smithi137.stdout:8/118: fsync d1/dc/d10/f21 0 2024-03-14T13:02:11.227 INFO:tasks.workunit.client.0.smithi111.stdout:2/184: rmdir dd 39 2024-03-14T13:02:11.227 INFO:tasks.workunit.client.0.smithi111.stdout:9/300: dwrite d5/f31 [0,4194304] 0 2024-03-14T13:02:11.228 INFO:tasks.workunit.client.0.smithi111.stdout:2/185: stat dd/c2a 0 2024-03-14T13:02:11.231 INFO:tasks.workunit.client.1.smithi137.stdout:5/97: write d7/f11 [1404461,11606] 0 2024-03-14T13:02:11.231 INFO:tasks.workunit.client.1.smithi137.stdout:6/156: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:11.233 INFO:tasks.workunit.client.0.smithi111.stdout:2/186: creat dd/d11/d1a/f38 x:0 0 0 2024-03-14T13:02:11.233 INFO:tasks.workunit.client.1.smithi137.stdout:5/98: truncate d7/f11 4578463 0 2024-03-14T13:02:11.233 INFO:tasks.workunit.client.1.smithi137.stdout:1/192: mknod d7/d9/d19/c3b 0 2024-03-14T13:02:11.234 INFO:tasks.workunit.client.0.smithi111.stdout:2/187: chown dd/d11/d1a/l2b 1702822 1 2024-03-14T13:02:11.234 INFO:tasks.workunit.client.1.smithi137.stdout:6/157: chown d2/d6/l2b 11060949 1 2024-03-14T13:02:11.235 INFO:tasks.workunit.client.0.smithi111.stdout:9/301: rename d5/f26 to d5/d42/f60 0 2024-03-14T13:02:11.236 INFO:tasks.workunit.client.1.smithi137.stdout:6/158: read d2/f2d [1425873,84439] 0 2024-03-14T13:02:11.238 INFO:tasks.workunit.client.1.smithi137.stdout:1/193: write d7/d9/d19/d1f/f23 [32258,106572] 0 2024-03-14T13:02:11.239 INFO:tasks.workunit.client.1.smithi137.stdout:7/131: rename d3/d6/d14/f1b to d3/f26 0 2024-03-14T13:02:11.240 INFO:tasks.workunit.client.1.smithi137.stdout:6/159: readlink d2/d6/d1d/l31 0 2024-03-14T13:02:11.240 INFO:tasks.workunit.client.1.smithi137.stdout:2/121: dread d2/d7/f10 [0,4194304] 0 2024-03-14T13:02:11.240 INFO:tasks.workunit.client.1.smithi137.stdout:1/194: chown f3 32522386 1 2024-03-14T13:02:11.241 INFO:tasks.workunit.client.1.smithi137.stdout:5/99: symlink d7/d16/d18/l1f 0 2024-03-14T13:02:11.243 INFO:tasks.workunit.client.0.smithi111.stdout:2/188: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:11.244 INFO:tasks.workunit.client.1.smithi137.stdout:2/122: unlink l1 0 2024-03-14T13:02:11.246 INFO:tasks.workunit.client.1.smithi137.stdout:7/132: dread d3/f26 [0,4194304] 0 2024-03-14T13:02:11.246 INFO:tasks.workunit.client.0.smithi111.stdout:2/189: symlink dd/l39 0 2024-03-14T13:02:11.247 INFO:tasks.workunit.client.1.smithi137.stdout:6/160: dread d2/f25 [0,4194304] 0 2024-03-14T13:02:11.247 INFO:tasks.workunit.client.1.smithi137.stdout:8/119: link d1/fd d1/dc/d11/d13/f29 0 2024-03-14T13:02:11.247 INFO:tasks.workunit.client.0.smithi111.stdout:2/190: truncate dd/d11/d1a/f38 407311 0 2024-03-14T13:02:11.248 INFO:tasks.workunit.client.0.smithi111.stdout:2/191: chown dd/l1f 95014606 1 2024-03-14T13:02:11.248 INFO:tasks.workunit.client.1.smithi137.stdout:1/195: truncate d7/d9/d19/d1f/d22/f25 1377795 0 2024-03-14T13:02:11.251 INFO:tasks.workunit.client.0.smithi111.stdout:2/192: creat dd/d17/f3a x:0 0 0 2024-03-14T13:02:11.251 INFO:tasks.workunit.client.1.smithi137.stdout:6/161: fdatasync d2/f2d 0 2024-03-14T13:02:11.251 INFO:tasks.workunit.client.0.smithi111.stdout:2/193: chown dd/l29 238554156 1 2024-03-14T13:02:11.252 INFO:tasks.workunit.client.0.smithi111.stdout:2/194: fdatasync dd/d11/f25 0 2024-03-14T13:02:11.252 INFO:tasks.workunit.client.0.smithi111.stdout:2/195: read f1 [3051954,11529] 0 2024-03-14T13:02:11.253 INFO:tasks.workunit.client.1.smithi137.stdout:1/196: chown d7/l12 10894 1 2024-03-14T13:02:11.253 INFO:tasks.workunit.client.1.smithi137.stdout:1/197: readlink d7/l38 0 2024-03-14T13:02:11.253 INFO:tasks.workunit.client.1.smithi137.stdout:1/198: stat d7/d9/f14 0 2024-03-14T13:02:11.253 INFO:tasks.workunit.client.0.smithi111.stdout:2/196: stat dd/d17/d34/c19 0 2024-03-14T13:02:11.254 INFO:tasks.workunit.client.1.smithi137.stdout:8/120: creat d1/dc/d11/f2a x:0 0 0 2024-03-14T13:02:11.255 INFO:tasks.workunit.client.1.smithi137.stdout:2/123: creat d2/d7/db/de/f20 x:0 0 0 2024-03-14T13:02:11.256 INFO:tasks.workunit.client.0.smithi111.stdout:2/197: mkdir dd/d11/d3b 0 2024-03-14T13:02:11.256 INFO:tasks.workunit.client.0.smithi111.stdout:2/198: chown c8 484454186 1 2024-03-14T13:02:11.256 INFO:tasks.workunit.client.1.smithi137.stdout:7/133: mkdir d3/d6/d14/d27 0 2024-03-14T13:02:11.256 INFO:tasks.workunit.client.1.smithi137.stdout:7/134: readlink d3/d6/lc 0 2024-03-14T13:02:11.258 INFO:tasks.workunit.client.1.smithi137.stdout:1/199: mknod d7/d9/d19/d1f/d2e/c3c 0 2024-03-14T13:02:11.259 INFO:tasks.workunit.client.1.smithi137.stdout:8/121: mknod d1/dc/d11/c2b 0 2024-03-14T13:02:11.260 INFO:tasks.workunit.client.1.smithi137.stdout:6/162: mkdir d2/d6/d8/d28/d39/d3c 0 2024-03-14T13:02:11.261 INFO:tasks.workunit.client.1.smithi137.stdout:2/124: mknod d2/d7/db/de/c21 0 2024-03-14T13:02:11.263 INFO:tasks.workunit.client.1.smithi137.stdout:1/200: dread d7/fb [4194304,4194304] 0 2024-03-14T13:02:11.264 INFO:tasks.workunit.client.1.smithi137.stdout:5/100: link d7/f10 d7/d16/d18/f20 0 2024-03-14T13:02:11.265 INFO:tasks.workunit.client.0.smithi111.stdout:2/199: getdents dd 0 2024-03-14T13:02:11.267 INFO:tasks.workunit.client.1.smithi137.stdout:6/163: creat d2/d6/f3d x:0 0 0 2024-03-14T13:02:11.267 INFO:tasks.workunit.client.1.smithi137.stdout:7/135: dread d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:11.268 INFO:tasks.workunit.client.1.smithi137.stdout:6/164: dread - d2/d6/d8/d28/f37 zero size 2024-03-14T13:02:11.269 INFO:tasks.workunit.client.1.smithi137.stdout:2/125: truncate d2/d7/f10 193570 0 2024-03-14T13:02:11.270 INFO:tasks.workunit.client.0.smithi111.stdout:9/302: dread d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:11.270 INFO:tasks.workunit.client.1.smithi137.stdout:5/101: write d7/fa [3099212,94936] 0 2024-03-14T13:02:11.271 INFO:tasks.workunit.client.1.smithi137.stdout:2/126: dread d2/d7/db/ff [0,4194304] 0 2024-03-14T13:02:11.273 INFO:tasks.workunit.client.1.smithi137.stdout:0/168: truncate d1/f8 1663259 0 2024-03-14T13:02:11.274 INFO:tasks.workunit.client.1.smithi137.stdout:1/201: creat d7/d9/d19/d1f/d2e/f3d x:0 0 0 2024-03-14T13:02:11.280 INFO:tasks.workunit.client.1.smithi137.stdout:6/165: fsync d2/f7 0 2024-03-14T13:02:11.281 INFO:tasks.workunit.client.1.smithi137.stdout:6/166: write f0 [2671876,28353] 0 2024-03-14T13:02:11.282 INFO:tasks.workunit.client.1.smithi137.stdout:6/167: read - d2/d6/f34 zero size 2024-03-14T13:02:11.284 INFO:tasks.workunit.client.1.smithi137.stdout:2/127: rename d2/d7/db/de/d17 to d2/d7/db/de/d22 0 2024-03-14T13:02:11.286 INFO:tasks.workunit.client.1.smithi137.stdout:0/169: mknod d1/d5/d1b/d1e/d25/c36 0 2024-03-14T13:02:11.287 INFO:tasks.workunit.client.1.smithi137.stdout:0/170: fsync d1/d5/f2a 0 2024-03-14T13:02:11.288 INFO:tasks.workunit.client.1.smithi137.stdout:7/136: link f1 d3/d5/f28 0 2024-03-14T13:02:11.288 INFO:tasks.workunit.client.0.smithi111.stdout:4/538: sync 2024-03-14T13:02:11.288 INFO:tasks.workunit.client.0.smithi111.stdout:1/230: sync 2024-03-14T13:02:11.293 INFO:tasks.workunit.client.1.smithi137.stdout:5/102: fsync d7/f10 0 2024-03-14T13:02:11.294 INFO:tasks.workunit.client.1.smithi137.stdout:5/103: dread - d7/f19 zero size 2024-03-14T13:02:11.295 INFO:tasks.workunit.client.0.smithi111.stdout:2/200: creat dd/d11/d3b/f3c x:0 0 0 2024-03-14T13:02:11.295 INFO:tasks.workunit.client.1.smithi137.stdout:5/104: chown d7/d16/d18 996 1 2024-03-14T13:02:11.296 INFO:tasks.workunit.client.0.smithi111.stdout:4/539: fdatasync f11 0 2024-03-14T13:02:11.296 INFO:tasks.workunit.client.0.smithi111.stdout:2/201: write dd/d11/d1a/f38 [866530,54044] 0 2024-03-14T13:02:11.297 INFO:tasks.workunit.client.0.smithi111.stdout:9/303: mknod d5/d42/d52/c61 0 2024-03-14T13:02:11.297 INFO:tasks.workunit.client.0.smithi111.stdout:1/231: chown d20/f24 584 1 2024-03-14T13:02:11.298 INFO:tasks.workunit.client.1.smithi137.stdout:6/168: truncate d2/f7 814043 0 2024-03-14T13:02:11.298 INFO:tasks.workunit.client.0.smithi111.stdout:1/232: fdatasync fe 0 2024-03-14T13:02:11.299 INFO:tasks.workunit.client.0.smithi111.stdout:4/540: creat d1d/d22/d23/fc0 x:0 0 0 2024-03-14T13:02:11.299 INFO:tasks.workunit.client.0.smithi111.stdout:1/233: write d20/d21/f42 [633730,67027] 0 2024-03-14T13:02:11.300 INFO:tasks.workunit.client.0.smithi111.stdout:4/541: truncate fe 4702216 0 2024-03-14T13:02:11.300 INFO:tasks.workunit.client.0.smithi111.stdout:2/202: creat dd/d11/d1a/f3d x:0 0 0 2024-03-14T13:02:11.301 INFO:tasks.workunit.client.0.smithi111.stdout:4/542: stat d1d/d48/dbe 0 2024-03-14T13:02:11.303 INFO:tasks.workunit.client.0.smithi111.stdout:2/203: write dd/d11/d1a/f1c [1395886,13118] 0 2024-03-14T13:02:11.305 INFO:tasks.workunit.client.1.smithi137.stdout:0/171: creat d1/d5/f37 x:0 0 0 2024-03-14T13:02:11.305 INFO:tasks.workunit.client.0.smithi111.stdout:1/234: unlink d20/d25/d31/c52 0 2024-03-14T13:02:11.307 INFO:tasks.workunit.client.1.smithi137.stdout:6/169: dread d2/f11 [0,4194304] 0 2024-03-14T13:02:11.307 INFO:tasks.workunit.client.0.smithi111.stdout:2/204: truncate f1 3296368 0 2024-03-14T13:02:11.307 INFO:tasks.workunit.client.0.smithi111.stdout:2/205: readlink l6 0 2024-03-14T13:02:11.309 INFO:tasks.workunit.client.1.smithi137.stdout:2/128: rmdir d2/d7/db/de 39 2024-03-14T13:02:11.310 INFO:tasks.workunit.client.0.smithi111.stdout:4/543: rename d1d/d22/d30/f4d to d1d/d22/d30/d87/fc1 0 2024-03-14T13:02:11.310 INFO:tasks.workunit.client.1.smithi137.stdout:7/137: dwrite d3/f1e [0,4194304] 0 2024-03-14T13:02:11.311 INFO:tasks.workunit.client.1.smithi137.stdout:0/172: symlink d1/d11/l38 0 2024-03-14T13:02:11.312 INFO:tasks.workunit.client.0.smithi111.stdout:2/206: mknod dd/d11/c3e 0 2024-03-14T13:02:11.313 INFO:tasks.workunit.client.1.smithi137.stdout:1/202: write d7/f1d [35169,95154] 0 2024-03-14T13:02:11.314 INFO:tasks.workunit.client.1.smithi137.stdout:6/170: unlink d2/d6/de/l1c 0 2024-03-14T13:02:11.321 INFO:tasks.workunit.client.0.smithi111.stdout:4/544: write d1d/d22/d30/d7c/d59/fb7 [1217538,25315] 0 2024-03-14T13:02:11.322 INFO:tasks.workunit.client.1.smithi137.stdout:7/138: symlink d3/d5/d23/l29 0 2024-03-14T13:02:11.326 INFO:tasks.workunit.client.1.smithi137.stdout:2/129: mknod d2/d7/db/de/d1d/c23 0 2024-03-14T13:02:11.329 INFO:tasks.workunit.client.1.smithi137.stdout:1/203: creat d7/d9/f3e x:0 0 0 2024-03-14T13:02:11.329 INFO:tasks.workunit.client.1.smithi137.stdout:6/171: link d2/d6/l27 d2/d6/d1d/l3e 0 2024-03-14T13:02:11.330 INFO:tasks.workunit.client.1.smithi137.stdout:2/130: mknod d2/d7/db/de/d1d/c24 0 2024-03-14T13:02:11.331 INFO:tasks.workunit.client.0.smithi111.stdout:1/235: dread d20/d25/d31/f3a [0,4194304] 0 2024-03-14T13:02:11.331 INFO:tasks.workunit.client.1.smithi137.stdout:7/139: mknod d3/d6/d14/d27/c2a 0 2024-03-14T13:02:11.332 INFO:tasks.workunit.client.0.smithi111.stdout:1/236: write d20/d40/f4f [414562,53175] 0 2024-03-14T13:02:11.334 INFO:tasks.workunit.client.1.smithi137.stdout:6/172: mknod d2/d6/d8/d28/c3f 0 2024-03-14T13:02:11.335 INFO:tasks.workunit.client.1.smithi137.stdout:1/204: rmdir d7/d9/d19/d1f/d26 39 2024-03-14T13:02:11.335 INFO:tasks.workunit.client.1.smithi137.stdout:6/173: dread - d2/d6/f35 zero size 2024-03-14T13:02:11.335 INFO:tasks.workunit.client.1.smithi137.stdout:7/140: dread d3/f1e [0,4194304] 0 2024-03-14T13:02:11.336 INFO:tasks.workunit.client.0.smithi111.stdout:1/237: link d20/d40/f4c d20/f57 0 2024-03-14T13:02:11.339 INFO:tasks.workunit.client.1.smithi137.stdout:0/173: dread d1/f8 [0,4194304] 0 2024-03-14T13:02:11.339 INFO:tasks.workunit.client.0.smithi111.stdout:1/238: dread d20/d21/f42 [0,4194304] 0 2024-03-14T13:02:11.339 INFO:tasks.workunit.client.1.smithi137.stdout:1/205: symlink d7/d9/l3f 0 2024-03-14T13:02:11.341 INFO:tasks.workunit.client.1.smithi137.stdout:7/141: creat d3/d5/d23/f2b x:0 0 0 2024-03-14T13:02:11.342 INFO:tasks.workunit.client.1.smithi137.stdout:6/174: creat d2/d6/d8/d28/f40 x:0 0 0 2024-03-14T13:02:11.342 INFO:tasks.workunit.client.1.smithi137.stdout:6/175: fsync d2/d6/de/f32 0 2024-03-14T13:02:11.344 INFO:tasks.workunit.client.1.smithi137.stdout:2/131: rename d2/fa to d2/f25 0 2024-03-14T13:02:11.345 INFO:tasks.workunit.client.1.smithi137.stdout:0/174: fsync d1/f8 0 2024-03-14T13:02:11.345 INFO:tasks.workunit.client.0.smithi111.stdout:1/239: truncate d20/d25/f2e 1944792 0 2024-03-14T13:02:11.346 INFO:tasks.workunit.client.1.smithi137.stdout:1/206: mkdir d7/d9/d19/d40 0 2024-03-14T13:02:11.347 INFO:tasks.workunit.client.1.smithi137.stdout:7/142: mkdir d3/d5/d23/d2c 0 2024-03-14T13:02:11.350 INFO:tasks.workunit.client.0.smithi111.stdout:1/240: write d20/d25/d31/f3b [636958,11928] 0 2024-03-14T13:02:11.351 INFO:tasks.workunit.client.1.smithi137.stdout:2/132: mkdir d2/d7/db/de/d22/d26 0 2024-03-14T13:02:11.352 INFO:tasks.workunit.client.0.smithi111.stdout:1/241: readlink d20/l27 0 2024-03-14T13:02:11.352 INFO:tasks.workunit.client.1.smithi137.stdout:2/133: write d2/f1e [994947,109167] 0 2024-03-14T13:02:11.353 INFO:tasks.workunit.client.1.smithi137.stdout:0/175: unlink d1/f8 0 2024-03-14T13:02:11.355 INFO:tasks.workunit.client.1.smithi137.stdout:7/143: truncate d3/f26 236956 0 2024-03-14T13:02:11.357 INFO:tasks.workunit.client.0.smithi111.stdout:1/242: fsync d20/d25/f2e 0 2024-03-14T13:02:11.357 INFO:tasks.workunit.client.1.smithi137.stdout:6/176: mknod d2/d26/c41 0 2024-03-14T13:02:11.357 INFO:tasks.workunit.client.1.smithi137.stdout:0/176: truncate d1/d5/d1b/d1e/d25/d26/f2d 764362 0 2024-03-14T13:02:11.358 INFO:tasks.workunit.client.1.smithi137.stdout:6/177: write d2/f17 [4717329,75602] 0 2024-03-14T13:02:11.359 INFO:tasks.workunit.client.1.smithi137.stdout:7/144: symlink d3/d6/d14/l2d 0 2024-03-14T13:02:11.360 INFO:tasks.workunit.client.0.smithi111.stdout:1/243: rename ff to d20/f58 0 2024-03-14T13:02:11.360 INFO:tasks.workunit.client.0.smithi111.stdout:1/244: fdatasync d20/d40/f4c 0 2024-03-14T13:02:11.361 INFO:tasks.workunit.client.1.smithi137.stdout:0/177: write d1/d5/d1b/d1e/f30 [950341,68443] 0 2024-03-14T13:02:11.361 INFO:tasks.workunit.client.1.smithi137.stdout:6/178: chown d2/d6/f3d 23783 1 2024-03-14T13:02:11.361 INFO:tasks.workunit.client.1.smithi137.stdout:1/207: dread d7/f10 [0,4194304] 0 2024-03-14T13:02:11.362 INFO:tasks.workunit.client.1.smithi137.stdout:7/145: mkdir d3/d6/d2e 0 2024-03-14T13:02:11.362 INFO:tasks.workunit.client.0.smithi111.stdout:1/245: mkdir d20/d21/d3f/d59 0 2024-03-14T13:02:11.364 INFO:tasks.workunit.client.1.smithi137.stdout:1/208: readlink d7/d9/d19/d1f/d22/l33 0 2024-03-14T13:02:11.366 INFO:tasks.workunit.client.1.smithi137.stdout:1/209: write d7/d9/d19/f2f [788062,38585] 0 2024-03-14T13:02:11.366 INFO:tasks.workunit.client.1.smithi137.stdout:0/178: creat d1/d5/d1b/d1e/d25/d26/f39 x:0 0 0 2024-03-14T13:02:11.367 INFO:tasks.workunit.client.1.smithi137.stdout:7/146: creat d3/d6/d14/f2f x:0 0 0 2024-03-14T13:02:11.367 INFO:tasks.workunit.client.0.smithi111.stdout:1/246: dwrite d20/d40/f4f [0,4194304] 0 2024-03-14T13:02:11.368 INFO:tasks.workunit.client.0.smithi111.stdout:1/247: readlink l7 0 2024-03-14T13:02:11.368 INFO:tasks.workunit.client.1.smithi137.stdout:0/179: dread - d1/d5/f37 zero size 2024-03-14T13:02:11.368 INFO:tasks.workunit.client.1.smithi137.stdout:2/134: getdents d2/d7/db/de/d22 0 2024-03-14T13:02:11.369 INFO:tasks.workunit.client.0.smithi111.stdout:1/248: dread - d20/d21/f41 zero size 2024-03-14T13:02:11.371 INFO:tasks.workunit.client.1.smithi137.stdout:7/147: read d3/d6/f9 [2300223,121227] 0 2024-03-14T13:02:11.371 INFO:tasks.workunit.client.1.smithi137.stdout:1/210: write d7/f10 [473346,4114] 0 2024-03-14T13:02:11.371 INFO:tasks.workunit.client.0.smithi111.stdout:1/249: mkdir d20/d21/d3f/d5a 0 2024-03-14T13:02:11.372 INFO:tasks.workunit.client.1.smithi137.stdout:6/179: dwrite d2/d6/fc [0,4194304] 0 2024-03-14T13:02:11.375 INFO:tasks.workunit.client.1.smithi137.stdout:1/211: write f5 [4487732,4073] 0 2024-03-14T13:02:11.376 INFO:tasks.workunit.client.1.smithi137.stdout:2/135: creat d2/d7/db/de/d22/d26/f27 x:0 0 0 2024-03-14T13:02:11.376 INFO:tasks.workunit.client.1.smithi137.stdout:6/180: symlink d2/d6/l42 0 2024-03-14T13:02:11.377 INFO:tasks.workunit.client.0.smithi111.stdout:1/250: dread d20/d25/d31/f3a [0,4194304] 0 2024-03-14T13:02:11.379 INFO:tasks.workunit.client.1.smithi137.stdout:0/180: dread d1/d1f/f28 [0,4194304] 0 2024-03-14T13:02:11.379 INFO:tasks.workunit.client.1.smithi137.stdout:1/212: unlink d7/l32 0 2024-03-14T13:02:11.381 INFO:tasks.workunit.client.1.smithi137.stdout:2/136: creat d2/f28 x:0 0 0 2024-03-14T13:02:11.383 INFO:tasks.workunit.client.0.smithi111.stdout:1/251: dwrite d20/f57 [0,4194304] 0 2024-03-14T13:02:11.384 INFO:tasks.workunit.client.1.smithi137.stdout:1/213: dread d7/d9/d19/f2f [0,4194304] 0 2024-03-14T13:02:11.386 INFO:tasks.workunit.client.1.smithi137.stdout:6/181: mkdir d2/d6/d8/d28/d39/d3c/d43 0 2024-03-14T13:02:11.386 INFO:tasks.workunit.client.1.smithi137.stdout:0/181: mkdir d1/d5/d1b/d1e/d31/d3a 0 2024-03-14T13:02:11.387 INFO:tasks.workunit.client.1.smithi137.stdout:1/214: rmdir d7/d9/d19/d1f/d2e 39 2024-03-14T13:02:11.394 INFO:tasks.workunit.client.1.smithi137.stdout:0/182: creat d1/d1f/f3b x:0 0 0 2024-03-14T13:02:11.394 INFO:tasks.workunit.client.1.smithi137.stdout:1/215: dread f5 [0,4194304] 0 2024-03-14T13:02:11.395 INFO:tasks.workunit.client.1.smithi137.stdout:6/182: dread d2/d6/fc [0,4194304] 0 2024-03-14T13:02:11.395 INFO:tasks.workunit.client.1.smithi137.stdout:0/183: stat d1/d1f/l33 0 2024-03-14T13:02:11.396 INFO:tasks.workunit.client.1.smithi137.stdout:2/137: getdents d2/d7 0 2024-03-14T13:02:11.398 INFO:tasks.workunit.client.1.smithi137.stdout:1/216: unlink d7/d9/l3f 0 2024-03-14T13:02:11.400 INFO:tasks.workunit.client.1.smithi137.stdout:6/183: rename d2/f11 to d2/d6/d8/d33/f44 0 2024-03-14T13:02:11.401 INFO:tasks.workunit.client.1.smithi137.stdout:2/138: mknod d2/d7/db/de/d22/c29 0 2024-03-14T13:02:11.403 INFO:tasks.workunit.client.1.smithi137.stdout:1/217: unlink d7/c16 0 2024-03-14T13:02:11.403 INFO:tasks.workunit.client.1.smithi137.stdout:0/184: dwrite d1/f29 [0,4194304] 0 2024-03-14T13:02:11.404 INFO:tasks.workunit.client.0.smithi111.stdout:1/252: dread d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:11.406 INFO:tasks.workunit.client.0.smithi111.stdout:1/253: fsync d20/f24 0 2024-03-14T13:02:11.407 INFO:tasks.workunit.client.0.smithi111.stdout:1/254: fsync d20/f44 0 2024-03-14T13:02:11.409 INFO:tasks.workunit.client.1.smithi137.stdout:1/218: creat d7/d9/f41 x:0 0 0 2024-03-14T13:02:11.410 INFO:tasks.workunit.client.0.smithi111.stdout:1/255: dread d20/d25/f30 [0,4194304] 0 2024-03-14T13:02:11.410 INFO:tasks.workunit.client.0.smithi111.stdout:1/256: chown d20/d25 12322 1 2024-03-14T13:02:11.410 INFO:tasks.workunit.client.0.smithi111.stdout:1/257: stat l8 0 2024-03-14T13:02:11.411 INFO:tasks.workunit.client.1.smithi137.stdout:6/184: chown d2/d6/cb 0 1 2024-03-14T13:02:11.411 INFO:tasks.workunit.client.1.smithi137.stdout:2/139: mknod d2/c2a 0 2024-03-14T13:02:11.412 INFO:tasks.workunit.client.1.smithi137.stdout:0/185: fsync d1/d11/f14 0 2024-03-14T13:02:11.412 INFO:tasks.workunit.client.0.smithi111.stdout:1/258: mkdir d20/d21/d3f/d5b 0 2024-03-14T13:02:11.414 INFO:tasks.workunit.client.1.smithi137.stdout:6/185: mkdir d2/d6/de/d45 0 2024-03-14T13:02:11.415 INFO:tasks.workunit.client.0.smithi111.stdout:1/259: rename d20/d21/d4e to d20/d21/d28/d5c 0 2024-03-14T13:02:11.417 INFO:tasks.workunit.client.0.smithi111.stdout:1/260: readlink d20/d21/l32 0 2024-03-14T13:02:11.417 INFO:tasks.workunit.client.1.smithi137.stdout:0/186: dread d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:11.417 INFO:tasks.workunit.client.0.smithi111.stdout:1/261: stat d20/d25/f39 0 2024-03-14T13:02:11.418 INFO:tasks.workunit.client.1.smithi137.stdout:1/219: mkdir d7/d9/d19/d40/d42 0 2024-03-14T13:02:11.418 INFO:tasks.workunit.client.0.smithi111.stdout:1/262: write fa [5060194,7305] 0 2024-03-14T13:02:11.419 INFO:tasks.workunit.client.0.smithi111.stdout:1/263: symlink d20/d21/d28/d5c/l5d 0 2024-03-14T13:02:11.420 INFO:tasks.workunit.client.1.smithi137.stdout:0/187: symlink d1/d1f/l3c 0 2024-03-14T13:02:11.421 INFO:tasks.workunit.client.1.smithi137.stdout:9/200: sync 2024-03-14T13:02:11.421 INFO:tasks.workunit.client.1.smithi137.stdout:3/157: sync 2024-03-14T13:02:11.422 INFO:tasks.workunit.client.1.smithi137.stdout:4/209: sync 2024-03-14T13:02:11.423 INFO:tasks.workunit.client.1.smithi137.stdout:2/140: link d2/d7/db/de/c11 d2/d7/db/de/c2b 0 2024-03-14T13:02:11.425 INFO:tasks.workunit.client.1.smithi137.stdout:1/220: creat d7/d9/d19/d1f/f43 x:0 0 0 2024-03-14T13:02:11.426 INFO:tasks.workunit.client.1.smithi137.stdout:3/158: write d4/de/d13/f25 [546432,35230] 0 2024-03-14T13:02:11.431 INFO:tasks.workunit.client.1.smithi137.stdout:6/186: dwrite d2/f17 [0,4194304] 0 2024-03-14T13:02:11.433 INFO:tasks.workunit.client.1.smithi137.stdout:1/221: symlink d7/d29/l44 0 2024-03-14T13:02:11.434 INFO:tasks.workunit.client.1.smithi137.stdout:0/188: dread d1/f1c [0,4194304] 0 2024-03-14T13:02:11.436 INFO:tasks.workunit.client.1.smithi137.stdout:4/210: creat d4/f48 x:0 0 0 2024-03-14T13:02:11.436 INFO:tasks.workunit.client.1.smithi137.stdout:3/159: dread d4/de/f28 [0,4194304] 0 2024-03-14T13:02:11.437 INFO:tasks.workunit.client.1.smithi137.stdout:4/211: fdatasync d4/f8 0 2024-03-14T13:02:11.438 INFO:tasks.workunit.client.1.smithi137.stdout:4/212: chown d4/d10/d42 0 1 2024-03-14T13:02:11.439 INFO:tasks.workunit.client.1.smithi137.stdout:3/160: truncate d4/de/d13/f25 853249 0 2024-03-14T13:02:11.439 INFO:tasks.workunit.client.1.smithi137.stdout:4/213: read d4/d27/f2b [1671968,81176] 0 2024-03-14T13:02:11.440 INFO:tasks.workunit.client.1.smithi137.stdout:3/161: write d4/de/f1f [420998,64121] 0 2024-03-14T13:02:11.441 INFO:tasks.workunit.client.1.smithi137.stdout:1/222: creat d7/d9/d19/d1f/d22/f45 x:0 0 0 2024-03-14T13:02:11.443 INFO:tasks.workunit.client.1.smithi137.stdout:0/189: creat d1/d5/d1b/d1e/f3d x:0 0 0 2024-03-14T13:02:11.446 INFO:tasks.workunit.client.1.smithi137.stdout:2/141: creat d2/d7/db/f2c x:0 0 0 2024-03-14T13:02:11.448 INFO:tasks.workunit.client.1.smithi137.stdout:6/187: mknod d2/d6/de/d45/c46 0 2024-03-14T13:02:11.449 INFO:tasks.workunit.client.1.smithi137.stdout:3/162: dread d4/de/d13/f25 [0,4194304] 0 2024-03-14T13:02:11.452 INFO:tasks.workunit.client.1.smithi137.stdout:9/201: write d6/df/f10 [1136836,50433] 0 2024-03-14T13:02:11.452 INFO:tasks.workunit.client.1.smithi137.stdout:3/163: chown d4/d19/l24 147141417 1 2024-03-14T13:02:11.454 INFO:tasks.workunit.client.0.smithi111.stdout:7/233: sync 2024-03-14T13:02:11.454 INFO:tasks.workunit.client.0.smithi111.stdout:8/420: sync 2024-03-14T13:02:11.454 INFO:tasks.workunit.client.0.smithi111.stdout:0/235: sync 2024-03-14T13:02:11.455 INFO:tasks.workunit.client.0.smithi111.stdout:3/333: sync 2024-03-14T13:02:11.455 INFO:tasks.workunit.client.0.smithi111.stdout:5/343: sync 2024-03-14T13:02:11.455 INFO:tasks.workunit.client.1.smithi137.stdout:1/223: dwrite d7/f1d [0,4194304] 0 2024-03-14T13:02:11.458 INFO:tasks.workunit.client.1.smithi137.stdout:3/164: write d4/fd [820127,126776] 0 2024-03-14T13:02:11.461 INFO:tasks.workunit.client.1.smithi137.stdout:6/188: dread d2/d6/fc [0,4194304] 0 2024-03-14T13:02:11.462 INFO:tasks.workunit.client.1.smithi137.stdout:9/202: fdatasync d6/df/d19/d33/f3e 0 2024-03-14T13:02:11.463 INFO:tasks.workunit.client.1.smithi137.stdout:2/142: rmdir d2/d7/db 39 2024-03-14T13:02:11.464 INFO:tasks.workunit.client.1.smithi137.stdout:3/165: chown d4/l8 948436 1 2024-03-14T13:02:11.464 INFO:tasks.workunit.client.1.smithi137.stdout:4/214: dwrite f2 [0,4194304] 0 2024-03-14T13:02:11.466 INFO:tasks.workunit.client.1.smithi137.stdout:6/189: readlink d2/d26/l30 0 2024-03-14T13:02:11.470 INFO:tasks.workunit.client.0.smithi111.stdout:7/234: dread d0/f2e [0,4194304] 0 2024-03-14T13:02:11.471 INFO:tasks.workunit.client.0.smithi111.stdout:8/421: truncate d3/f65 285622 0 2024-03-14T13:02:11.471 INFO:tasks.workunit.client.0.smithi111.stdout:3/334: link d0/d2d/d35/d50/d5e/d40/l41 d0/d2d/d35/d50/d5e/d19/d75/l7f 0 2024-03-14T13:02:11.472 INFO:tasks.workunit.client.0.smithi111.stdout:5/344: dwrite d2/d45/f22 [0,4194304] 0 2024-03-14T13:02:11.473 INFO:tasks.workunit.client.0.smithi111.stdout:7/235: dread - d0/d1/d22/f46 zero size 2024-03-14T13:02:11.474 INFO:tasks.workunit.client.1.smithi137.stdout:0/190: truncate d1/d1f/f24 3519294 0 2024-03-14T13:02:11.474 INFO:tasks.workunit.client.1.smithi137.stdout:9/203: creat d6/f49 x:0 0 0 2024-03-14T13:02:11.474 INFO:tasks.workunit.client.1.smithi137.stdout:3/166: mknod d4/d22/c2c 0 2024-03-14T13:02:11.474 INFO:tasks.workunit.client.0.smithi111.stdout:5/345: dread - d2/d1f/d6a/f71 zero size 2024-03-14T13:02:11.474 INFO:tasks.workunit.client.0.smithi111.stdout:7/236: fsync d0/d1/d21/f15 0 2024-03-14T13:02:11.477 INFO:tasks.workunit.client.0.smithi111.stdout:7/237: fsync d0/d1/d21/f3f 0 2024-03-14T13:02:11.477 INFO:tasks.workunit.client.1.smithi137.stdout:3/167: chown d4/de/d13/f21 190644223 1 2024-03-14T13:02:11.477 INFO:tasks.workunit.client.0.smithi111.stdout:3/335: symlink d0/d2d/d35/d50/d5e/d40/d55/l80 0 2024-03-14T13:02:11.478 INFO:tasks.workunit.client.1.smithi137.stdout:9/204: creat d6/df/d19/d1f/f4a x:0 0 0 2024-03-14T13:02:11.478 INFO:tasks.workunit.client.1.smithi137.stdout:0/191: creat d1/f3e x:0 0 0 2024-03-14T13:02:11.478 INFO:tasks.workunit.client.0.smithi111.stdout:7/238: stat d0/d1/d22/f42 0 2024-03-14T13:02:11.479 INFO:tasks.workunit.client.0.smithi111.stdout:7/239: dread - d0/d1/d22/f42 zero size 2024-03-14T13:02:11.481 INFO:tasks.workunit.client.0.smithi111.stdout:7/240: read - d0/d1/d22/f42 zero size 2024-03-14T13:02:11.481 INFO:tasks.workunit.client.1.smithi137.stdout:9/205: chown d6/fe 1 1 2024-03-14T13:02:11.481 INFO:tasks.workunit.client.1.smithi137.stdout:2/143: mkdir d2/d7/db/de/d1d/d2d 0 2024-03-14T13:02:11.482 INFO:tasks.workunit.client.1.smithi137.stdout:4/215: read d4/d9/fc [3201703,126928] 0 2024-03-14T13:02:11.484 INFO:tasks.workunit.client.1.smithi137.stdout:3/168: creat d4/de/d16/d1c/f2d x:0 0 0 2024-03-14T13:02:11.485 INFO:tasks.workunit.client.0.smithi111.stdout:7/241: mknod d0/d1/d21/d3a/c47 0 2024-03-14T13:02:11.485 INFO:tasks.workunit.client.0.smithi111.stdout:0/236: dread d6/d9/d15/f37 [0,4194304] 0 2024-03-14T13:02:11.485 INFO:tasks.workunit.client.1.smithi137.stdout:0/192: truncate d1/d11/f21 510134 0 2024-03-14T13:02:11.488 INFO:tasks.workunit.client.1.smithi137.stdout:2/144: mkdir d2/d7/d2e 0 2024-03-14T13:02:11.488 INFO:tasks.workunit.client.0.smithi111.stdout:5/346: dwrite d2/d16/d49/d44/d4a/f63 [0,4194304] 0 2024-03-14T13:02:11.489 INFO:tasks.workunit.client.1.smithi137.stdout:3/169: creat d4/d22/f2e x:0 0 0 2024-03-14T13:02:11.489 INFO:tasks.workunit.client.0.smithi111.stdout:0/237: stat d6/d9/de/l17 0 2024-03-14T13:02:11.489 INFO:tasks.workunit.client.0.smithi111.stdout:3/336: dwrite d0/d2d/d35/d50/d5e/f12 [0,4194304] 0 2024-03-14T13:02:11.490 INFO:tasks.workunit.client.1.smithi137.stdout:1/224: dwrite d7/d9/d19/d1f/f43 [0,4194304] 0 2024-03-14T13:02:11.491 INFO:tasks.workunit.client.0.smithi111.stdout:5/347: write d2/d45/f5c [775535,16634] 0 2024-03-14T13:02:11.492 INFO:tasks.workunit.client.1.smithi137.stdout:6/190: dwrite d2/d6/d8/f1f [0,4194304] 0 2024-03-14T13:02:11.492 INFO:tasks.workunit.client.1.smithi137.stdout:4/216: rename d4/d10/f34 to d4/d9/d3a/d41/f49 0 2024-03-14T13:02:11.494 INFO:tasks.workunit.client.1.smithi137.stdout:4/217: write d4/d27/f28 [41388,68319] 0 2024-03-14T13:02:11.497 INFO:tasks.workunit.client.1.smithi137.stdout:0/193: symlink d1/d5/d1b/d1e/l3f 0 2024-03-14T13:02:11.498 INFO:tasks.workunit.client.0.smithi111.stdout:5/348: mkdir d2/d45/d79 0 2024-03-14T13:02:11.501 INFO:tasks.workunit.client.0.smithi111.stdout:5/349: chown d2/d1f/f24 55597 1 2024-03-14T13:02:11.501 INFO:tasks.workunit.client.0.smithi111.stdout:0/238: dwrite f5 [0,4194304] 0 2024-03-14T13:02:11.504 INFO:tasks.workunit.client.0.smithi111.stdout:7/242: link d0/d1/d5/l37 d0/d1/l48 0 2024-03-14T13:02:11.506 INFO:tasks.workunit.client.0.smithi111.stdout:0/239: symlink d6/d3d/l4f 0 2024-03-14T13:02:11.507 INFO:tasks.workunit.client.1.smithi137.stdout:0/194: dread - d1/d1f/f2f zero size 2024-03-14T13:02:11.508 INFO:tasks.workunit.client.0.smithi111.stdout:3/337: link d0/d2d/d35/d50/d5e/d19/l29 d0/d2d/d35/d50/l81 0 2024-03-14T13:02:11.511 INFO:tasks.workunit.client.0.smithi111.stdout:7/243: rmdir d0/d11 39 2024-03-14T13:02:11.511 INFO:tasks.workunit.client.1.smithi137.stdout:3/170: dwrite d4/f2b [0,4194304] 0 2024-03-14T13:02:11.511 INFO:tasks.workunit.client.1.smithi137.stdout:2/145: mkdir d2/d7/db/d2f 0 2024-03-14T13:02:11.511 INFO:tasks.workunit.client.1.smithi137.stdout:6/191: creat d2/d6/d8/d28/d39/d3c/f47 x:0 0 0 2024-03-14T13:02:11.512 INFO:tasks.workunit.client.1.smithi137.stdout:4/218: symlink d4/d9/l4a 0 2024-03-14T13:02:11.514 INFO:tasks.workunit.client.1.smithi137.stdout:0/195: write d1/d5/d1b/d1e/d25/d26/f39 [268355,109577] 0 2024-03-14T13:02:11.514 INFO:tasks.workunit.client.0.smithi111.stdout:3/338: rename d0/d2d/d35/d50/d5e/d19/d75 to d0/d2d/d35/d4c/d82 0 2024-03-14T13:02:11.515 INFO:tasks.workunit.client.0.smithi111.stdout:5/350: link d2/d16/d49/ce d2/d13/c7a 0 2024-03-14T13:02:11.515 INFO:tasks.workunit.client.1.smithi137.stdout:1/225: link d7/f10 d7/d9/d19/d40/f46 0 2024-03-14T13:02:11.516 INFO:tasks.workunit.client.0.smithi111.stdout:7/244: creat d0/d1/d5/d38/d9/f49 x:0 0 0 2024-03-14T13:02:11.519 INFO:tasks.workunit.client.1.smithi137.stdout:6/192: readlink d2/d6/de/l36 0 2024-03-14T13:02:11.519 INFO:tasks.workunit.client.1.smithi137.stdout:4/219: dread - d4/d9/d3a/d41/d38/f3b zero size 2024-03-14T13:02:11.521 INFO:tasks.workunit.client.0.smithi111.stdout:5/351: chown d2/d1f/c54 339 1 2024-03-14T13:02:11.523 INFO:tasks.workunit.client.1.smithi137.stdout:0/196: symlink d1/d11/l40 0 2024-03-14T13:02:11.523 INFO:tasks.workunit.client.1.smithi137.stdout:1/226: stat d7/d9/d19/d1f 0 2024-03-14T13:02:11.523 INFO:tasks.workunit.client.0.smithi111.stdout:8/422: read d3/f65 [33472,64119] 0 2024-03-14T13:02:11.526 INFO:tasks.workunit.client.1.smithi137.stdout:4/220: readlink d4/d10/l36 0 2024-03-14T13:02:11.530 INFO:tasks.workunit.client.1.smithi137.stdout:0/197: creat d1/d5/d1b/d1e/d25/d26/f41 x:0 0 0 2024-03-14T13:02:11.530 INFO:tasks.workunit.client.0.smithi111.stdout:8/423: dwrite f0 [4194304,4194304] 0 2024-03-14T13:02:11.531 INFO:tasks.workunit.client.1.smithi137.stdout:9/206: dwrite d6/df/f20 [4194304,4194304] 0 2024-03-14T13:02:11.532 INFO:tasks.workunit.client.1.smithi137.stdout:3/171: dwrite d4/d19/f23 [0,4194304] 0 2024-03-14T13:02:11.532 INFO:tasks.workunit.client.1.smithi137.stdout:1/227: creat d7/d9/d19/d40/f47 x:0 0 0 2024-03-14T13:02:11.533 INFO:tasks.workunit.client.1.smithi137.stdout:4/221: mknod d4/d9/c4b 0 2024-03-14T13:02:11.536 INFO:tasks.workunit.client.0.smithi111.stdout:8/424: link d3/dc/f6c d3/d1b/d20/d35/d40/d48/f89 0 2024-03-14T13:02:11.539 INFO:tasks.workunit.client.0.smithi111.stdout:8/425: write d3/d22/d30/f47 [895427,13065] 0 2024-03-14T13:02:11.539 INFO:tasks.workunit.client.1.smithi137.stdout:1/228: read d7/d9/d19/d1f/f43 [2809535,40514] 0 2024-03-14T13:02:11.539 INFO:tasks.workunit.client.1.smithi137.stdout:9/207: write d6/df/f20 [3780221,118101] 0 2024-03-14T13:02:11.541 INFO:tasks.workunit.client.0.smithi111.stdout:3/339: rmdir d0 39 2024-03-14T13:02:11.542 INFO:tasks.workunit.client.0.smithi111.stdout:8/426: mknod d3/d1b/c8a 0 2024-03-14T13:02:11.544 INFO:tasks.workunit.client.1.smithi137.stdout:9/208: truncate d6/f49 621347 0 2024-03-14T13:02:11.545 INFO:tasks.workunit.client.1.smithi137.stdout:2/146: rmdir d2 39 2024-03-14T13:02:11.545 INFO:tasks.workunit.client.0.smithi111.stdout:5/352: truncate d2/d16/d49/f11 1525512 0 2024-03-14T13:02:11.545 INFO:tasks.workunit.client.0.smithi111.stdout:8/427: write d3/d22/d30/f74 [590021,42270] 0 2024-03-14T13:02:11.546 INFO:tasks.workunit.client.1.smithi137.stdout:3/172: creat d4/d19/f2f x:0 0 0 2024-03-14T13:02:11.546 INFO:tasks.workunit.client.1.smithi137.stdout:4/222: creat d4/d9/d3a/d41/d38/f4c x:0 0 0 2024-03-14T13:02:11.549 INFO:tasks.workunit.client.1.smithi137.stdout:0/198: rename d1/d5/d1b/d1e/d31 to d1/d5/d1b/d42 0 2024-03-14T13:02:11.549 INFO:tasks.workunit.client.0.smithi111.stdout:3/340: mknod d0/d2d/d35/d69/c83 0 2024-03-14T13:02:11.549 INFO:tasks.workunit.client.1.smithi137.stdout:4/223: dread - d4/d9/d3a/d41/d38/f4c zero size 2024-03-14T13:02:11.550 INFO:tasks.workunit.client.0.smithi111.stdout:5/353: creat d2/d16/d49/d33/f7b x:0 0 0 2024-03-14T13:02:11.550 INFO:tasks.workunit.client.1.smithi137.stdout:1/229: readlink d7/d31/l35 0 2024-03-14T13:02:11.552 INFO:tasks.workunit.client.1.smithi137.stdout:6/193: truncate d2/d6/d8/f15 673270 0 2024-03-14T13:02:11.555 INFO:tasks.workunit.client.1.smithi137.stdout:3/173: creat d4/df/f30 x:0 0 0 2024-03-14T13:02:11.555 INFO:tasks.workunit.client.1.smithi137.stdout:0/199: fdatasync d1/d5/d1b/d1e/d25/d26/f39 0 2024-03-14T13:02:11.556 INFO:tasks.workunit.client.0.smithi111.stdout:8/428: dwrite d3/d1b/d66/f7b [0,4194304] 0 2024-03-14T13:02:11.557 INFO:tasks.workunit.client.1.smithi137.stdout:6/194: read d2/f25 [2421714,128395] 0 2024-03-14T13:02:11.558 INFO:tasks.workunit.client.1.smithi137.stdout:4/224: dread - d4/d9/d3a/d41/d38/f4c zero size 2024-03-14T13:02:11.560 INFO:tasks.workunit.client.1.smithi137.stdout:9/209: dwrite d6/f49 [0,4194304] 0 2024-03-14T13:02:11.561 INFO:tasks.workunit.client.1.smithi137.stdout:0/200: readlink d1/d5/d1b/d42/l34 0 2024-03-14T13:02:11.564 INFO:tasks.workunit.client.1.smithi137.stdout:3/174: creat d4/de/d16/f31 x:0 0 0 2024-03-14T13:02:11.564 INFO:tasks.workunit.client.0.smithi111.stdout:7/245: dread d0/d1/f28 [0,4194304] 0 2024-03-14T13:02:11.566 INFO:tasks.workunit.client.0.smithi111.stdout:7/246: mkdir d0/d1/d4a 0 2024-03-14T13:02:11.567 INFO:tasks.workunit.client.1.smithi137.stdout:8/122: sync 2024-03-14T13:02:11.570 INFO:tasks.workunit.client.1.smithi137.stdout:9/210: creat d6/df/d19/d3a/d40/f4b x:0 0 0 2024-03-14T13:02:11.571 INFO:tasks.workunit.client.1.smithi137.stdout:2/147: rename d2/d7/db/de/d22/c29 to d2/d7/c30 0 2024-03-14T13:02:11.572 INFO:tasks.workunit.client.1.smithi137.stdout:8/123: symlink d1/dc/l2c 0 2024-03-14T13:02:11.575 INFO:tasks.workunit.client.1.smithi137.stdout:6/195: symlink d2/d6/d8/d28/d39/d3c/d43/l48 0 2024-03-14T13:02:11.578 INFO:tasks.workunit.client.1.smithi137.stdout:9/211: chown d6/df/f10 50111 1 2024-03-14T13:02:11.578 INFO:tasks.workunit.client.1.smithi137.stdout:8/124: chown d1/d17/c1b 3 1 2024-03-14T13:02:11.579 INFO:tasks.workunit.client.1.smithi137.stdout:6/196: truncate f0 4311912 0 2024-03-14T13:02:11.579 INFO:tasks.workunit.client.1.smithi137.stdout:2/148: fsync d2/d7/db/de/f1b 0 2024-03-14T13:02:11.579 INFO:tasks.workunit.client.0.smithi111.stdout:3/341: dread d0/d2d/d35/d50/d5e/f5f [0,4194304] 0 2024-03-14T13:02:11.580 INFO:tasks.workunit.client.0.smithi111.stdout:8/429: dread d3/d49/f53 [0,4194304] 0 2024-03-14T13:02:11.580 INFO:tasks.workunit.client.0.smithi111.stdout:8/430: stat d3/d5/f1e 0 2024-03-14T13:02:11.582 INFO:tasks.workunit.client.0.smithi111.stdout:3/342: mkdir d0/d2d/d35/d69/d84 0 2024-03-14T13:02:11.583 INFO:tasks.workunit.client.1.smithi137.stdout:3/175: dwrite d4/de/f1f [0,4194304] 0 2024-03-14T13:02:11.584 INFO:tasks.workunit.client.1.smithi137.stdout:9/212: chown d6/df/l2e 599379 1 2024-03-14T13:02:11.584 INFO:tasks.workunit.client.1.smithi137.stdout:0/201: rename d1/d5/d1b/d1e/d25/d35 to d1/d5/d43 0 2024-03-14T13:02:11.585 INFO:tasks.workunit.client.1.smithi137.stdout:6/197: rmdir d2/d26 39 2024-03-14T13:02:11.587 INFO:tasks.workunit.client.1.smithi137.stdout:4/225: dwrite d4/d10/d3d/f3e [0,4194304] 0 2024-03-14T13:02:11.588 INFO:tasks.workunit.client.0.smithi111.stdout:8/431: rename d3/d1b/d20/d35/d40/l6e to d3/d22/d30/l8b 0 2024-03-14T13:02:11.588 INFO:tasks.workunit.client.1.smithi137.stdout:3/176: mknod d4/d19/c32 0 2024-03-14T13:02:11.591 INFO:tasks.workunit.client.1.smithi137.stdout:8/125: creat d1/dc/d11/f2d x:0 0 0 2024-03-14T13:02:11.591 INFO:tasks.workunit.client.0.smithi111.stdout:7/247: dread d0/d1/d5/d38/d9/f3d [0,4194304] 0 2024-03-14T13:02:11.591 INFO:tasks.workunit.client.1.smithi137.stdout:9/213: write d6/df/f10 [423961,94577] 0 2024-03-14T13:02:11.592 INFO:tasks.workunit.client.1.smithi137.stdout:0/202: mkdir d1/d44 0 2024-03-14T13:02:11.594 INFO:tasks.workunit.client.0.smithi111.stdout:3/343: rename d0/d2d/d35/d50/l81 to d0/d2d/d35/d50/d5e/l85 0 2024-03-14T13:02:11.595 INFO:tasks.workunit.client.0.smithi111.stdout:7/248: truncate d0/d1/d5/d38/d9/f3d 103140 0 2024-03-14T13:02:11.596 INFO:tasks.workunit.client.1.smithi137.stdout:4/226: dread - d4/d10/f3f zero size 2024-03-14T13:02:11.598 INFO:tasks.workunit.client.0.smithi111.stdout:7/249: truncate d0/d1/d22/f42 653215 0 2024-03-14T13:02:11.600 INFO:tasks.workunit.client.0.smithi111.stdout:7/250: write d0/d1/d22/f39 [739426,75312] 0 2024-03-14T13:02:11.600 INFO:tasks.workunit.client.1.smithi137.stdout:3/177: truncate d4/df/f30 929864 0 2024-03-14T13:02:11.600 INFO:tasks.workunit.client.1.smithi137.stdout:1/230: dwrite d7/d9/d19/d1f/f2a [0,4194304] 0 2024-03-14T13:02:11.603 INFO:tasks.workunit.client.1.smithi137.stdout:2/149: link d2/d7/db/ff d2/d7/db/de/d22/d26/f31 0 2024-03-14T13:02:11.603 INFO:tasks.workunit.client.1.smithi137.stdout:4/227: mkdir d4/d4d 0 2024-03-14T13:02:11.604 INFO:tasks.workunit.client.0.smithi111.stdout:7/251: fsync d0/d1/d5/f1a 0 2024-03-14T13:02:11.607 INFO:tasks.workunit.client.0.smithi111.stdout:8/432: dwrite d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:11.607 INFO:tasks.workunit.client.1.smithi137.stdout:2/150: read d2/f1e [998906,59649] 0 2024-03-14T13:02:11.607 INFO:tasks.workunit.client.1.smithi137.stdout:8/126: link d1/dc/d11/d13/f22 d1/f2e 0 2024-03-14T13:02:11.608 INFO:tasks.workunit.client.1.smithi137.stdout:6/198: getdents d2/d6/de/d45 0 2024-03-14T13:02:11.609 INFO:tasks.workunit.client.0.smithi111.stdout:7/252: dread d0/d1/d22/f42 [0,4194304] 0 2024-03-14T13:02:11.611 INFO:tasks.workunit.client.0.smithi111.stdout:8/433: symlink d3/d1b/d20/d35/d40/d48/d77/l8c 0 2024-03-14T13:02:11.612 INFO:tasks.workunit.client.1.smithi137.stdout:9/214: rename d6/df/d19/d1f/f34 to d6/df/d19/f4c 0 2024-03-14T13:02:11.613 INFO:tasks.workunit.client.1.smithi137.stdout:3/178: dwrite d4/fd [0,4194304] 0 2024-03-14T13:02:11.614 INFO:tasks.workunit.client.0.smithi111.stdout:8/434: write d3/d22/d4a/f5c [1863280,9553] 0 2024-03-14T13:02:11.617 INFO:tasks.workunit.client.0.smithi111.stdout:3/344: truncate d0/d30/f48 526039 0 2024-03-14T13:02:11.617 INFO:tasks.workunit.client.1.smithi137.stdout:3/179: write d4/fd [904954,1981] 0 2024-03-14T13:02:11.617 INFO:tasks.workunit.client.1.smithi137.stdout:8/127: symlink d1/l2f 0 2024-03-14T13:02:11.617 INFO:tasks.workunit.client.1.smithi137.stdout:3/180: chown d4/df/f30 531416 1 2024-03-14T13:02:11.617 INFO:tasks.workunit.client.1.smithi137.stdout:1/231: creat d7/d9/d19/d1f/d26/f48 x:0 0 0 2024-03-14T13:02:11.619 INFO:tasks.workunit.client.0.smithi111.stdout:3/345: creat d0/d2d/f86 x:0 0 0 2024-03-14T13:02:11.619 INFO:tasks.workunit.client.1.smithi137.stdout:6/199: symlink d2/d6/d8/d28/d39/l49 0 2024-03-14T13:02:11.620 INFO:tasks.workunit.client.1.smithi137.stdout:1/232: write d7/d9/f15 [2759725,130647] 0 2024-03-14T13:02:11.622 INFO:tasks.workunit.client.1.smithi137.stdout:9/215: symlink d6/d26/l4d 0 2024-03-14T13:02:11.622 INFO:tasks.workunit.client.1.smithi137.stdout:8/128: mknod d1/dc/c30 0 2024-03-14T13:02:11.623 INFO:tasks.workunit.client.0.smithi111.stdout:3/346: rename d0/d30/f38 to d0/d2d/d35/d69/d84/f87 0 2024-03-14T13:02:11.623 INFO:tasks.workunit.client.1.smithi137.stdout:6/200: readlink d2/d6/l2f 0 2024-03-14T13:02:11.624 INFO:tasks.workunit.client.1.smithi137.stdout:9/216: mknod d6/c4e 0 2024-03-14T13:02:11.626 INFO:tasks.workunit.client.1.smithi137.stdout:2/151: getdents d2 0 2024-03-14T13:02:11.627 INFO:tasks.workunit.client.1.smithi137.stdout:8/129: rename d1/f2e to d1/f31 0 2024-03-14T13:02:11.629 INFO:tasks.workunit.client.0.smithi111.stdout:3/347: dread d0/d2d/d35/d50/d5e/d40/f47 [0,4194304] 0 2024-03-14T13:02:11.630 INFO:tasks.workunit.client.1.smithi137.stdout:9/217: mknod d6/df/d19/d3a/c4f 0 2024-03-14T13:02:11.630 INFO:tasks.workunit.client.0.smithi111.stdout:3/348: rmdir d0/d2d/d35/d50/d5e/d40/d67 39 2024-03-14T13:02:11.631 INFO:tasks.workunit.client.1.smithi137.stdout:1/233: creat d7/d9/d19/d40/d42/f49 x:0 0 0 2024-03-14T13:02:11.632 INFO:tasks.workunit.client.0.smithi111.stdout:3/349: write d0/d2d/d35/d50/f54 [324611,64402] 0 2024-03-14T13:02:11.632 INFO:tasks.workunit.client.1.smithi137.stdout:2/152: rmdir d2/d7 39 2024-03-14T13:02:11.633 INFO:tasks.workunit.client.1.smithi137.stdout:1/234: fsync f5 0 2024-03-14T13:02:11.633 INFO:tasks.workunit.client.1.smithi137.stdout:9/218: truncate d6/df/d19/d3a/f47 869416 0 2024-03-14T13:02:11.634 INFO:tasks.workunit.client.0.smithi111.stdout:3/350: truncate d0/d2d/d35/d4c/d82/f7e 823313 0 2024-03-14T13:02:11.636 INFO:tasks.workunit.client.0.smithi111.stdout:3/351: creat d0/d2d/d35/d50/f88 x:0 0 0 2024-03-14T13:02:11.636 INFO:tasks.workunit.client.1.smithi137.stdout:4/228: dread d4/d9/d3a/d41/f21 [0,4194304] 0 2024-03-14T13:02:11.638 INFO:tasks.workunit.client.1.smithi137.stdout:2/153: write d2/d7/db/de/f20 [403186,112505] 0 2024-03-14T13:02:11.638 INFO:tasks.workunit.client.1.smithi137.stdout:6/201: rename d2/d6/de/c18 to d2/c4a 0 2024-03-14T13:02:11.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:11 smithi111 ceph-mon[39688]: pgmap v20: 65 pgs: 65 active+clean; 553 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.7 MiB/s rd, 36 MiB/s wr, 65 op/s 2024-03-14T13:02:11.640 INFO:tasks.workunit.client.0.smithi111.stdout:3/352: rename d0/d2d/d35/d50/d5e/c7d to d0/d2d/d35/d69/d84/c89 0 2024-03-14T13:02:11.640 INFO:tasks.workunit.client.1.smithi137.stdout:9/219: mknod d6/df/c50 0 2024-03-14T13:02:11.641 INFO:tasks.workunit.client.0.smithi111.stdout:8/435: truncate d3/d22/d4a/f5c 2069560 0 2024-03-14T13:02:11.641 INFO:tasks.workunit.client.0.smithi111.stdout:8/436: chown d3/d22 86508269 1 2024-03-14T13:02:11.642 INFO:tasks.workunit.client.1.smithi137.stdout:1/235: dread d7/d9/d19/d40/f46 [0,4194304] 0 2024-03-14T13:02:11.642 INFO:tasks.workunit.client.0.smithi111.stdout:3/353: symlink d0/d30/l8a 0 2024-03-14T13:02:11.643 INFO:tasks.workunit.client.0.smithi111.stdout:3/354: fdatasync d0/d2d/d35/d50/d5e/d19/f26 0 2024-03-14T13:02:11.643 INFO:tasks.workunit.client.1.smithi137.stdout:4/229: rename d4/d9/fc to d4/d10/d42/f4e 0 2024-03-14T13:02:11.644 INFO:tasks.workunit.client.0.smithi111.stdout:8/437: rmdir d3/d5 39 2024-03-14T13:02:11.648 INFO:tasks.workunit.client.0.smithi111.stdout:8/438: write d3/f32 [1749751,50725] 0 2024-03-14T13:02:11.650 INFO:tasks.workunit.client.1.smithi137.stdout:2/154: rename d2/d7/db/de/d1d/c23 to d2/d7/d2e/c32 0 2024-03-14T13:02:11.652 INFO:tasks.workunit.client.0.smithi111.stdout:8/439: getdents d3/d1b/d66 0 2024-03-14T13:02:11.653 INFO:tasks.workunit.client.0.smithi111.stdout:8/440: dread - d3/dc/f6c zero size 2024-03-14T13:02:11.655 INFO:tasks.workunit.client.1.smithi137.stdout:6/202: rename d2/d6/l2b to d2/d26/l4b 0 2024-03-14T13:02:11.655 INFO:tasks.workunit.client.1.smithi137.stdout:1/236: write f3 [6679436,121793] 0 2024-03-14T13:02:11.657 INFO:tasks.workunit.client.1.smithi137.stdout:4/230: link f2 d4/d9/d3a/f4f 0 2024-03-14T13:02:11.658 INFO:tasks.workunit.client.0.smithi111.stdout:8/441: mkdir d3/dc/d45/d8d 0 2024-03-14T13:02:11.658 INFO:tasks.workunit.client.1.smithi137.stdout:4/231: chown d4/d9/d3a/d41/f49 2278742 1 2024-03-14T13:02:11.659 INFO:tasks.workunit.client.1.smithi137.stdout:2/155: creat d2/d7/db/de/d22/d26/f33 x:0 0 0 2024-03-14T13:02:11.659 INFO:tasks.workunit.client.1.smithi137.stdout:9/220: creat d6/df/f51 x:0 0 0 2024-03-14T13:02:11.661 INFO:tasks.workunit.client.1.smithi137.stdout:6/203: mkdir d2/d4c 0 2024-03-14T13:02:11.661 INFO:tasks.workunit.client.0.smithi111.stdout:3/355: dwrite d0/f23 [0,4194304] 0 2024-03-14T13:02:11.663 INFO:tasks.workunit.client.1.smithi137.stdout:1/237: symlink d7/d9/d19/d1f/l4a 0 2024-03-14T13:02:11.664 INFO:tasks.workunit.client.1.smithi137.stdout:6/204: stat d2/d6/d8/f1f 0 2024-03-14T13:02:11.665 INFO:tasks.workunit.client.0.smithi111.stdout:3/356: mknod d0/d2d/d35/d50/d5e/d40/c8b 0 2024-03-14T13:02:11.670 INFO:tasks.workunit.client.1.smithi137.stdout:1/238: creat d7/d9/f4b x:0 0 0 2024-03-14T13:02:11.670 INFO:tasks.workunit.client.1.smithi137.stdout:9/221: link d6/df/c12 d6/c52 0 2024-03-14T13:02:11.673 INFO:tasks.workunit.client.0.smithi111.stdout:8/442: dwrite d3/d1b/f1d [0,4194304] 0 2024-03-14T13:02:11.674 INFO:tasks.workunit.client.1.smithi137.stdout:4/232: dwrite d4/f37 [0,4194304] 0 2024-03-14T13:02:11.675 INFO:tasks.workunit.client.1.smithi137.stdout:9/222: mkdir d6/d53 0 2024-03-14T13:02:11.676 INFO:tasks.workunit.client.1.smithi137.stdout:2/156: dwrite d2/d7/db/de/f20 [0,4194304] 0 2024-03-14T13:02:11.677 INFO:tasks.workunit.client.1.smithi137.stdout:4/233: fdatasync d4/d10/d26/f40 0 2024-03-14T13:02:11.678 INFO:tasks.workunit.client.1.smithi137.stdout:6/205: rename d2/c3 to d2/d6/d8/d28/d39/d3c/d43/c4d 0 2024-03-14T13:02:11.678 INFO:tasks.workunit.client.1.smithi137.stdout:1/239: write d7/d9/d19/d1f/d26/f48 [634211,12639] 0 2024-03-14T13:02:11.680 INFO:tasks.workunit.client.1.smithi137.stdout:6/206: read d2/d6/fc [2018923,26700] 0 2024-03-14T13:02:11.681 INFO:tasks.workunit.client.1.smithi137.stdout:9/223: write d6/fc [1323728,85520] 0 2024-03-14T13:02:11.682 INFO:tasks.workunit.client.1.smithi137.stdout:4/234: mknod d4/d9/d3a/d41/c50 0 2024-03-14T13:02:11.682 INFO:tasks.workunit.client.0.smithi111.stdout:3/357: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:11.683 INFO:tasks.workunit.client.1.smithi137.stdout:9/224: chown d6/df/d19/d3a/c4f 13942655 1 2024-03-14T13:02:11.686 INFO:tasks.workunit.client.1.smithi137.stdout:1/240: read d7/d9/d19/d1f/d2e/f34 [254166,97290] 0 2024-03-14T13:02:11.687 INFO:tasks.workunit.client.1.smithi137.stdout:2/157: mkdir d2/d34 0 2024-03-14T13:02:11.689 INFO:tasks.workunit.client.1.smithi137.stdout:7/148: sync 2024-03-14T13:02:11.690 INFO:tasks.workunit.client.1.smithi137.stdout:5/105: sync 2024-03-14T13:02:11.693 INFO:tasks.workunit.client.1.smithi137.stdout:2/158: write d2/d7/db/f1c [3331779,78583] 0 2024-03-14T13:02:11.694 INFO:tasks.workunit.client.0.smithi111.stdout:8/443: dwrite d3/d1b/d34/f3b [0,4194304] 0 2024-03-14T13:02:11.694 INFO:tasks.workunit.client.1.smithi137.stdout:7/149: mknod d3/d5/c30 0 2024-03-14T13:02:11.695 INFO:tasks.workunit.client.1.smithi137.stdout:1/241: chown d7/d9/d19/d1f/l24 199 1 2024-03-14T13:02:11.697 INFO:tasks.workunit.client.1.smithi137.stdout:4/235: dread d4/d9/f16 [0,4194304] 0 2024-03-14T13:02:11.697 INFO:tasks.workunit.client.1.smithi137.stdout:1/242: write d7/d31/f3a [212958,109112] 0 2024-03-14T13:02:11.699 INFO:tasks.workunit.client.0.smithi111.stdout:8/444: creat d3/d1b/d20/f8e x:0 0 0 2024-03-14T13:02:11.700 INFO:tasks.workunit.client.1.smithi137.stdout:9/225: dread d6/f49 [0,4194304] 0 2024-03-14T13:02:11.702 INFO:tasks.workunit.client.1.smithi137.stdout:2/159: readlink d2/l3 0 2024-03-14T13:02:11.703 INFO:tasks.workunit.client.1.smithi137.stdout:9/226: fsync d6/df/d19/d3a/d40/f4b 0 2024-03-14T13:02:11.706 INFO:tasks.workunit.client.1.smithi137.stdout:2/160: write d2/d7/db/de/d22/d26/f27 [408514,25703] 0 2024-03-14T13:02:11.707 INFO:tasks.workunit.client.1.smithi137.stdout:1/243: dread d7/d9/ff [0,4194304] 0 2024-03-14T13:02:11.708 INFO:tasks.workunit.client.1.smithi137.stdout:1/244: dread - d7/d9/d19/d40/d42/f49 zero size 2024-03-14T13:02:11.709 INFO:tasks.workunit.client.1.smithi137.stdout:9/227: symlink d6/df/d19/d1f/l54 0 2024-03-14T13:02:11.709 INFO:tasks.workunit.client.1.smithi137.stdout:4/236: rmdir d4/d9 39 2024-03-14T13:02:11.710 INFO:tasks.workunit.client.1.smithi137.stdout:1/245: chown d7/d9 67199 1 2024-03-14T13:02:11.710 INFO:tasks.workunit.client.1.smithi137.stdout:5/106: link d7/l1e d7/l21 0 2024-03-14T13:02:11.712 INFO:tasks.workunit.client.1.smithi137.stdout:7/150: link d3/d5/c30 d3/d6/d2e/c31 0 2024-03-14T13:02:11.713 INFO:tasks.workunit.client.1.smithi137.stdout:7/151: fsync d3/f1c 0 2024-03-14T13:02:11.713 INFO:tasks.workunit.client.1.smithi137.stdout:9/228: readlink d6/df/l1b 0 2024-03-14T13:02:11.714 INFO:tasks.workunit.client.1.smithi137.stdout:4/237: fsync d4/d9/f1f 0 2024-03-14T13:02:11.715 INFO:tasks.workunit.client.1.smithi137.stdout:2/161: symlink d2/d7/db/de/d1d/d2d/l35 0 2024-03-14T13:02:11.715 INFO:tasks.workunit.client.1.smithi137.stdout:9/229: write d6/df/d19/f45 [868628,96371] 0 2024-03-14T13:02:11.717 INFO:tasks.workunit.client.1.smithi137.stdout:5/107: rename d7/ld to d7/d16/d18/l22 0 2024-03-14T13:02:11.718 INFO:tasks.workunit.client.1.smithi137.stdout:9/230: chown d6/df/d19/d3a/f47 25210 1 2024-03-14T13:02:11.719 INFO:tasks.workunit.client.1.smithi137.stdout:9/231: fsync d6/df/f20 0 2024-03-14T13:02:11.720 INFO:tasks.workunit.client.1.smithi137.stdout:1/246: getdents d7/d29 0 2024-03-14T13:02:11.722 INFO:tasks.workunit.client.1.smithi137.stdout:6/207: truncate d2/d6/d8/f1e 2694372 0 2024-03-14T13:02:11.722 INFO:tasks.workunit.client.1.smithi137.stdout:1/247: write d7/d9/d19/d1f/d2e/f3d [912778,20376] 0 2024-03-14T13:02:11.725 INFO:tasks.workunit.client.1.smithi137.stdout:2/162: unlink d2/f25 0 2024-03-14T13:02:11.726 INFO:tasks.workunit.client.1.smithi137.stdout:5/108: mkdir d7/d23 0 2024-03-14T13:02:11.728 INFO:tasks.workunit.client.0.smithi111.stdout:8/445: dread d3/dc/f4e [0,4194304] 0 2024-03-14T13:02:11.728 INFO:tasks.workunit.client.1.smithi137.stdout:9/232: mknod d6/df/d19/c55 0 2024-03-14T13:02:11.730 INFO:tasks.workunit.client.0.smithi111.stdout:8/446: creat d3/d1b/d20/d83/f8f x:0 0 0 2024-03-14T13:02:11.730 INFO:tasks.workunit.client.1.smithi137.stdout:5/109: truncate d7/d16/f1a 878952 0 2024-03-14T13:02:11.730 INFO:tasks.workunit.client.1.smithi137.stdout:1/248: symlink d7/d9/d19/d40/l4c 0 2024-03-14T13:02:11.731 INFO:tasks.workunit.client.1.smithi137.stdout:6/208: dwrite d2/d6/de/f32 [0,4194304] 0 2024-03-14T13:02:11.734 INFO:tasks.workunit.client.0.smithi111.stdout:8/447: write d3/d1b/d20/d35/d40/d48/f89 [64073,38247] 0 2024-03-14T13:02:11.734 INFO:tasks.workunit.client.0.smithi111.stdout:8/448: dread - d3/d1b/d20/d35/d40/f76 zero size 2024-03-14T13:02:11.738 INFO:tasks.workunit.client.0.smithi111.stdout:8/449: getdents d3/d49 0 2024-03-14T13:02:11.738 INFO:tasks.workunit.client.1.smithi137.stdout:7/152: rename d3/d5/f28 to d3/f32 0 2024-03-14T13:02:11.739 INFO:tasks.workunit.client.1.smithi137.stdout:9/233: chown d6/df/d19/d1f/f24 13 1 2024-03-14T13:02:11.739 INFO:tasks.workunit.client.0.smithi111.stdout:8/450: creat d3/d1b/d20/d35/d40/f90 x:0 0 0 2024-03-14T13:02:11.740 INFO:tasks.workunit.client.0.smithi111.stdout:8/451: chown d3/d22/d30/f47 7 1 2024-03-14T13:02:11.746 INFO:tasks.workunit.client.1.smithi137.stdout:5/110: mknod d7/d23/c24 0 2024-03-14T13:02:11.746 INFO:tasks.workunit.client.0.smithi111.stdout:8/452: truncate d3/d5/f19 3687515 0 2024-03-14T13:02:11.748 INFO:tasks.workunit.client.0.smithi111.stdout:8/453: fdatasync d3/d22/d4a/f5c 0 2024-03-14T13:02:11.749 INFO:tasks.workunit.client.1.smithi137.stdout:1/249: mkdir d7/d4d 0 2024-03-14T13:02:11.754 INFO:tasks.workunit.client.1.smithi137.stdout:5/111: write d7/fb [2256527,99779] 0 2024-03-14T13:02:11.754 INFO:tasks.workunit.client.1.smithi137.stdout:2/163: dwrite d2/d7/db/de/d22/d26/f33 [0,4194304] 0 2024-03-14T13:02:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:11 smithi137 ceph-mon[50336]: pgmap v20: 65 pgs: 65 active+clean; 553 MiB data, 3.2 GiB used, 533 GiB / 536 GiB avail; 1.7 MiB/s rd, 36 MiB/s wr, 65 op/s 2024-03-14T13:02:11.755 INFO:tasks.workunit.client.0.smithi111.stdout:8/454: dwrite d3/f6b [0,4194304] 0 2024-03-14T13:02:11.758 INFO:tasks.workunit.client.1.smithi137.stdout:1/250: symlink d7/d9/d19/d1f/d26/l4e 0 2024-03-14T13:02:11.758 INFO:tasks.workunit.client.0.smithi111.stdout:8/455: creat d3/d1b/d20/d35/d40/f91 x:0 0 0 2024-03-14T13:02:11.760 INFO:tasks.workunit.client.1.smithi137.stdout:7/153: dwrite d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:11.765 INFO:tasks.workunit.client.1.smithi137.stdout:7/154: readlink d3/d6/lc 0 2024-03-14T13:02:11.766 INFO:tasks.workunit.client.1.smithi137.stdout:6/209: dwrite d2/f17 [4194304,4194304] 0 2024-03-14T13:02:11.771 INFO:tasks.workunit.client.0.smithi111.stdout:8/456: dwrite d3/d1b/d20/d35/d40/f76 [0,4194304] 0 2024-03-14T13:02:11.775 INFO:tasks.workunit.client.0.smithi111.stdout:8/457: rmdir d3/d49 39 2024-03-14T13:02:11.776 INFO:tasks.workunit.client.1.smithi137.stdout:1/251: fdatasync d7/fb 0 2024-03-14T13:02:11.776 INFO:tasks.workunit.client.0.smithi111.stdout:8/458: dread - d3/d1b/d20/f8e zero size 2024-03-14T13:02:11.777 INFO:tasks.workunit.client.0.smithi111.stdout:8/459: dread - d3/d1b/d20/d83/f8f zero size 2024-03-14T13:02:11.778 INFO:tasks.workunit.client.1.smithi137.stdout:5/112: dwrite d7/f15 [0,4194304] 0 2024-03-14T13:02:11.783 INFO:tasks.workunit.client.1.smithi137.stdout:1/252: fsync f6 0 2024-03-14T13:02:11.784 INFO:tasks.workunit.client.1.smithi137.stdout:6/210: dwrite f0 [0,4194304] 0 2024-03-14T13:02:11.785 INFO:tasks.workunit.client.1.smithi137.stdout:1/253: creat d7/d9/f4f x:0 0 0 2024-03-14T13:02:11.791 INFO:tasks.workunit.client.1.smithi137.stdout:7/155: getdents d3/d6/d2e 0 2024-03-14T13:02:11.792 INFO:tasks.workunit.client.1.smithi137.stdout:2/164: link d2/c2a d2/c36 0 2024-03-14T13:02:11.792 INFO:tasks.workunit.client.1.smithi137.stdout:7/156: readlink d3/d5/d23/l29 0 2024-03-14T13:02:11.800 INFO:tasks.workunit.client.1.smithi137.stdout:5/113: dwrite d7/d16/f1a [0,4194304] 0 2024-03-14T13:02:11.800 INFO:tasks.workunit.client.1.smithi137.stdout:1/254: mkdir d7/d9/d19/d40/d50 0 2024-03-14T13:02:11.801 INFO:tasks.workunit.client.1.smithi137.stdout:5/114: chown d7/d16 1 1 2024-03-14T13:02:11.803 INFO:tasks.workunit.client.1.smithi137.stdout:9/234: dread d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:11.807 INFO:tasks.workunit.client.1.smithi137.stdout:7/157: getdents d3/d6/d14/d27 0 2024-03-14T13:02:11.808 INFO:tasks.workunit.client.1.smithi137.stdout:1/255: write d7/d9/d19/d1f/f23 [1391133,348] 0 2024-03-14T13:02:11.818 INFO:tasks.workunit.client.1.smithi137.stdout:9/235: dwrite d6/df/d19/d3a/f43 [0,4194304] 0 2024-03-14T13:02:11.818 INFO:tasks.workunit.client.1.smithi137.stdout:5/115: dwrite d7/fa [0,4194304] 0 2024-03-14T13:02:11.820 INFO:tasks.workunit.client.1.smithi137.stdout:5/116: dread - d7/f10 zero size 2024-03-14T13:02:11.821 INFO:tasks.workunit.client.1.smithi137.stdout:9/236: mkdir d6/d56 0 2024-03-14T13:02:11.821 INFO:tasks.workunit.client.1.smithi137.stdout:7/158: link d3/c25 d3/d6/c33 0 2024-03-14T13:02:11.822 INFO:tasks.workunit.client.1.smithi137.stdout:9/237: readlink d6/d26/l4d 0 2024-03-14T13:02:11.826 INFO:tasks.workunit.client.1.smithi137.stdout:9/238: getdents d6/d26 0 2024-03-14T13:02:11.826 INFO:tasks.workunit.client.1.smithi137.stdout:7/159: link d3/d5/la d3/d5/d23/d2c/l34 0 2024-03-14T13:02:11.831 INFO:tasks.workunit.client.1.smithi137.stdout:9/239: rename d6/df/f14 to d6/f57 0 2024-03-14T13:02:11.832 INFO:tasks.workunit.client.1.smithi137.stdout:7/160: write d3/f18 [1059714,44263] 0 2024-03-14T13:02:11.833 INFO:tasks.workunit.client.1.smithi137.stdout:7/161: fsync d3/d6/f15 0 2024-03-14T13:02:11.833 INFO:tasks.workunit.client.1.smithi137.stdout:7/162: readlink d3/d6/l20 0 2024-03-14T13:02:11.834 INFO:tasks.workunit.client.1.smithi137.stdout:7/163: chown d3/d6/d2e 104972 1 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.0.smithi111.stdout:1/264: sync 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.0.smithi111.stdout:2/207: sync 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.0.smithi111.stdout:6/226: sync 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.0.smithi111.stdout:4/545: sync 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.0.smithi111.stdout:9/304: sync 2024-03-14T13:02:11.837 INFO:tasks.workunit.client.1.smithi137.stdout:7/164: rmdir d3/d6/d2e 39 2024-03-14T13:02:11.838 INFO:tasks.workunit.client.0.smithi111.stdout:9/305: read - d5/d6/d8/f49 zero size 2024-03-14T13:02:11.840 INFO:tasks.workunit.client.0.smithi111.stdout:9/306: readlink d5/d42/l47 0 2024-03-14T13:02:11.840 INFO:tasks.workunit.client.0.smithi111.stdout:1/265: creat d20/d21/d3f/d5a/f5e x:0 0 0 2024-03-14T13:02:11.841 INFO:tasks.workunit.client.1.smithi137.stdout:1/256: write d7/d9/ff [845226,10664] 0 2024-03-14T13:02:11.842 INFO:tasks.workunit.client.0.smithi111.stdout:9/307: chown d5/c33 1 1 2024-03-14T13:02:11.844 INFO:tasks.workunit.client.0.smithi111.stdout:9/308: fsync d5/d6/d8/f3a 0 2024-03-14T13:02:11.846 INFO:tasks.workunit.client.0.smithi111.stdout:1/266: dread d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:11.846 INFO:tasks.workunit.client.0.smithi111.stdout:6/227: dwrite d0/d17/d21/f2d [0,4194304] 0 2024-03-14T13:02:11.847 INFO:tasks.workunit.client.1.smithi137.stdout:7/165: rename d3/d6/d14/f1d to d3/d6/d2e/f35 0 2024-03-14T13:02:11.848 INFO:tasks.workunit.client.1.smithi137.stdout:1/257: chown d7/d9/d19/d1f/l4a 3 1 2024-03-14T13:02:11.848 INFO:tasks.workunit.client.0.smithi111.stdout:1/267: write d20/d25/f46 [325576,89386] 0 2024-03-14T13:02:11.848 INFO:tasks.workunit.client.0.smithi111.stdout:4/546: truncate d1d/d22/d30/d5e/fb3 3208986 0 2024-03-14T13:02:11.850 INFO:tasks.workunit.client.0.smithi111.stdout:4/547: dread - d1d/d22/d40/d69/d86/db1/fbf zero size 2024-03-14T13:02:11.851 INFO:tasks.workunit.client.1.smithi137.stdout:9/240: getdents d6 0 2024-03-14T13:02:11.852 INFO:tasks.workunit.client.0.smithi111.stdout:1/268: creat d20/d21/f5f x:0 0 0 2024-03-14T13:02:11.852 INFO:tasks.workunit.client.0.smithi111.stdout:9/309: dread d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:11.853 INFO:tasks.workunit.client.0.smithi111.stdout:6/228: creat d0/d50/f56 x:0 0 0 2024-03-14T13:02:11.854 INFO:tasks.workunit.client.0.smithi111.stdout:1/269: read f11 [1194281,70888] 0 2024-03-14T13:02:11.854 INFO:tasks.workunit.client.0.smithi111.stdout:9/310: stat d5/c4b 0 2024-03-14T13:02:11.856 INFO:tasks.workunit.client.1.smithi137.stdout:6/211: dwrite d2/d6/d8/f1e [0,4194304] 0 2024-03-14T13:02:11.858 INFO:tasks.workunit.client.1.smithi137.stdout:7/166: readlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:11.858 INFO:tasks.workunit.client.0.smithi111.stdout:6/229: getdents d0/d17/d21/d42 0 2024-03-14T13:02:11.860 INFO:tasks.workunit.client.1.smithi137.stdout:5/117: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:11.862 INFO:tasks.workunit.client.1.smithi137.stdout:1/258: truncate d7/fb 7055801 0 2024-03-14T13:02:11.862 INFO:tasks.workunit.client.1.smithi137.stdout:2/165: dwrite d2/d7/db/f16 [0,4194304] 0 2024-03-14T13:02:11.865 INFO:tasks.workunit.client.1.smithi137.stdout:5/118: write d7/d16/d18/f20 [3776319,101058] 0 2024-03-14T13:02:11.865 INFO:tasks.workunit.client.0.smithi111.stdout:9/311: link d5/d42/l47 d5/d6/l62 0 2024-03-14T13:02:11.865 INFO:tasks.workunit.client.0.smithi111.stdout:6/230: dread d0/d17/d24/f46 [0,4194304] 0 2024-03-14T13:02:11.865 INFO:tasks.workunit.client.0.smithi111.stdout:1/270: dwrite d20/d21/d3f/d5a/f5e [0,4194304] 0 2024-03-14T13:02:11.869 INFO:tasks.workunit.client.1.smithi137.stdout:6/212: creat d2/d6/d1d/f4e x:0 0 0 2024-03-14T13:02:11.869 INFO:tasks.workunit.client.1.smithi137.stdout:1/259: rmdir d7 39 2024-03-14T13:02:11.871 INFO:tasks.workunit.client.0.smithi111.stdout:6/231: link d0/f14 d0/f57 0 2024-03-14T13:02:11.871 INFO:tasks.workunit.client.0.smithi111.stdout:6/232: chown d0/d17/d24 60127250 1 2024-03-14T13:02:11.872 INFO:tasks.workunit.client.0.smithi111.stdout:9/312: symlink d5/d6/l63 0 2024-03-14T13:02:11.873 INFO:tasks.workunit.client.0.smithi111.stdout:1/271: mknod d20/d25/d31/c60 0 2024-03-14T13:02:11.873 INFO:tasks.workunit.client.0.smithi111.stdout:6/233: write d0/d17/d21/f28 [4044900,121767] 0 2024-03-14T13:02:11.876 INFO:tasks.workunit.client.1.smithi137.stdout:5/119: mkdir d7/d16/d18/d25 0 2024-03-14T13:02:11.877 INFO:tasks.workunit.client.0.smithi111.stdout:6/234: creat d0/d17/d21/d42/f58 x:0 0 0 2024-03-14T13:02:11.878 INFO:tasks.workunit.client.1.smithi137.stdout:2/166: dread d2/d7/f10 [0,4194304] 0 2024-03-14T13:02:11.879 INFO:tasks.workunit.client.0.smithi111.stdout:6/235: write d0/f1c [938188,19038] 0 2024-03-14T13:02:11.882 INFO:tasks.workunit.client.0.smithi111.stdout:6/236: mkdir d0/d17/d20/d4a/d59 0 2024-03-14T13:02:11.883 INFO:tasks.workunit.client.1.smithi137.stdout:6/213: rmdir d2/d6/d8/d28 39 2024-03-14T13:02:11.883 INFO:tasks.workunit.client.1.smithi137.stdout:5/120: write d7/ff [1942301,118490] 0 2024-03-14T13:02:11.886 INFO:tasks.workunit.client.1.smithi137.stdout:9/241: dwrite d6/df/d19/d1f/f3b [0,4194304] 0 2024-03-14T13:02:11.887 INFO:tasks.workunit.client.1.smithi137.stdout:1/260: readlink d7/d29/l2c 0 2024-03-14T13:02:11.889 INFO:tasks.workunit.client.1.smithi137.stdout:1/261: read f5 [3725112,79781] 0 2024-03-14T13:02:11.891 INFO:tasks.workunit.client.1.smithi137.stdout:6/214: rename d2/d6/d8/f15 to d2/d6/de/f4f 0 2024-03-14T13:02:11.892 INFO:tasks.workunit.client.1.smithi137.stdout:9/242: symlink d6/df/d19/d1f/l58 0 2024-03-14T13:02:11.893 INFO:tasks.workunit.client.1.smithi137.stdout:2/167: dread d2/d7/db/f16 [0,4194304] 0 2024-03-14T13:02:11.894 INFO:tasks.workunit.client.1.smithi137.stdout:5/121: rename d7/ff to d7/d16/f26 0 2024-03-14T13:02:11.896 INFO:tasks.workunit.client.1.smithi137.stdout:9/243: mknod d6/df/d19/d1f/c59 0 2024-03-14T13:02:11.898 INFO:tasks.workunit.client.1.smithi137.stdout:1/262: rename d7/d9/d19/d1f/f43 to d7/d9/d19/d1f/d2e/f51 0 2024-03-14T13:02:11.900 INFO:tasks.workunit.client.1.smithi137.stdout:2/168: creat d2/d7/db/d2f/f37 x:0 0 0 2024-03-14T13:02:11.901 INFO:tasks.workunit.client.1.smithi137.stdout:9/244: symlink d6/df/d19/d3a/d40/l5a 0 2024-03-14T13:02:11.902 INFO:tasks.workunit.client.1.smithi137.stdout:1/263: mknod d7/d9/d19/c52 0 2024-03-14T13:02:11.903 INFO:tasks.workunit.client.1.smithi137.stdout:6/215: link d2/c3b d2/d6/c50 0 2024-03-14T13:02:11.904 INFO:tasks.workunit.client.1.smithi137.stdout:9/245: creat d6/df/d19/d3a/f5b x:0 0 0 2024-03-14T13:02:11.904 INFO:tasks.workunit.client.0.smithi111.stdout:4/548: dread d1d/d22/d23/f34 [0,4194304] 0 2024-03-14T13:02:11.905 INFO:tasks.workunit.client.1.smithi137.stdout:5/122: rename d7/l1e to d7/l27 0 2024-03-14T13:02:11.908 INFO:tasks.workunit.client.1.smithi137.stdout:6/216: chown d2/d6/d8/d28/c3f 792782 1 2024-03-14T13:02:11.908 INFO:tasks.workunit.client.1.smithi137.stdout:9/246: creat d6/df/d19/d3a/d40/f5c x:0 0 0 2024-03-14T13:02:11.910 INFO:tasks.workunit.client.0.smithi111.stdout:1/272: dread f19 [0,4194304] 0 2024-03-14T13:02:11.911 INFO:tasks.workunit.client.1.smithi137.stdout:9/247: read d6/df/f20 [615142,56511] 0 2024-03-14T13:02:11.915 INFO:tasks.workunit.client.0.smithi111.stdout:9/313: write d5/f50 [1278395,5989] 0 2024-03-14T13:02:11.915 INFO:tasks.workunit.client.0.smithi111.stdout:9/314: chown d5/f4f 127202 1 2024-03-14T13:02:11.916 INFO:tasks.workunit.client.1.smithi137.stdout:7/167: truncate d3/d6/d2e/f35 1949224 0 2024-03-14T13:02:11.916 INFO:tasks.workunit.client.0.smithi111.stdout:6/237: write d0/f4 [3272256,51841] 0 2024-03-14T13:02:11.917 INFO:tasks.workunit.client.1.smithi137.stdout:2/169: dwrite d2/d7/db/de/d22/d26/f27 [0,4194304] 0 2024-03-14T13:02:11.917 INFO:tasks.workunit.client.0.smithi111.stdout:9/315: chown d5/d6/d8/d5b 160 1 2024-03-14T13:02:11.918 INFO:tasks.workunit.client.0.smithi111.stdout:1/273: truncate d20/d25/d31/f3a 1945860 0 2024-03-14T13:02:11.919 INFO:tasks.workunit.client.1.smithi137.stdout:7/168: stat d3/d6/d14/l2d 0 2024-03-14T13:02:11.919 INFO:tasks.workunit.client.1.smithi137.stdout:6/217: creat d2/d6/d8/d38/f51 x:0 0 0 2024-03-14T13:02:11.920 INFO:tasks.workunit.client.0.smithi111.stdout:6/238: rmdir d0/d17/d20/d4a 39 2024-03-14T13:02:11.920 INFO:tasks.workunit.client.1.smithi137.stdout:7/169: stat d3/d6/l10 0 2024-03-14T13:02:11.922 INFO:tasks.workunit.client.1.smithi137.stdout:6/218: chown d2/d6 11081 1 2024-03-14T13:02:11.923 INFO:tasks.workunit.client.1.smithi137.stdout:2/170: fdatasync d2/f5 0 2024-03-14T13:02:11.924 INFO:tasks.workunit.client.0.smithi111.stdout:9/316: rmdir d5/d6/d14/d55 0 2024-03-14T13:02:11.924 INFO:tasks.workunit.client.1.smithi137.stdout:9/248: dwrite d6/df/d19/d3a/f47 [0,4194304] 0 2024-03-14T13:02:11.925 INFO:tasks.workunit.client.0.smithi111.stdout:6/239: mkdir d0/d17/d20/d5a 0 2024-03-14T13:02:11.925 INFO:tasks.workunit.client.0.smithi111.stdout:9/317: write d5/d6/d8/f30 [2011469,14728] 0 2024-03-14T13:02:11.929 INFO:tasks.workunit.client.0.smithi111.stdout:9/318: creat d5/d6/d8/d1a/f64 x:0 0 0 2024-03-14T13:02:11.930 INFO:tasks.workunit.client.1.smithi137.stdout:9/249: write d6/df/d19/d1f/f4a [486593,41617] 0 2024-03-14T13:02:11.931 INFO:tasks.workunit.client.1.smithi137.stdout:6/219: dread d2/d6/d8/f1f [0,4194304] 0 2024-03-14T13:02:11.931 INFO:tasks.workunit.client.0.smithi111.stdout:6/240: rmdir d0/d17/d21/d42/d49 39 2024-03-14T13:02:11.931 INFO:tasks.workunit.client.0.smithi111.stdout:6/241: read - d0/f3e zero size 2024-03-14T13:02:11.932 INFO:tasks.workunit.client.1.smithi137.stdout:2/171: readlink d2/d7/l1f 0 2024-03-14T13:02:11.934 INFO:tasks.workunit.client.1.smithi137.stdout:7/170: truncate f1 845346 0 2024-03-14T13:02:11.935 INFO:tasks.workunit.client.0.smithi111.stdout:9/319: creat d5/f65 x:0 0 0 2024-03-14T13:02:11.938 INFO:tasks.workunit.client.0.smithi111.stdout:9/320: rename d5/d6/d2a/f58 to d5/d6/d2a/d2d/f66 0 2024-03-14T13:02:11.939 INFO:tasks.workunit.client.1.smithi137.stdout:1/264: write d7/d9/d19/d1f/d22/f25 [1686065,16435] 0 2024-03-14T13:02:11.939 INFO:tasks.workunit.client.1.smithi137.stdout:5/123: truncate d7/f10 749855 0 2024-03-14T13:02:11.940 INFO:tasks.workunit.client.0.smithi111.stdout:6/242: rmdir d0/d17/d21/d47 0 2024-03-14T13:02:11.942 INFO:tasks.workunit.client.1.smithi137.stdout:9/250: creat d6/df/f5d x:0 0 0 2024-03-14T13:02:11.943 INFO:tasks.workunit.client.1.smithi137.stdout:6/220: dwrite f0 [4194304,4194304] 0 2024-03-14T13:02:11.943 INFO:tasks.workunit.client.0.smithi111.stdout:1/274: dread d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:11.943 INFO:tasks.workunit.client.0.smithi111.stdout:9/321: dread d5/d6/f2f [0,4194304] 0 2024-03-14T13:02:11.944 INFO:tasks.workunit.client.0.smithi111.stdout:9/322: stat d5/l4c 0 2024-03-14T13:02:11.944 INFO:tasks.workunit.client.0.smithi111.stdout:6/243: symlink d0/d17/d20/d5a/l5b 0 2024-03-14T13:02:11.945 INFO:tasks.workunit.client.1.smithi137.stdout:2/172: creat d2/d7/db/f38 x:0 0 0 2024-03-14T13:02:11.946 INFO:tasks.workunit.client.0.smithi111.stdout:0/240: sync 2024-03-14T13:02:11.946 INFO:tasks.workunit.client.0.smithi111.stdout:5/354: sync 2024-03-14T13:02:11.946 INFO:tasks.workunit.client.0.smithi111.stdout:1/275: creat d20/f61 x:0 0 0 2024-03-14T13:02:11.947 INFO:tasks.workunit.client.0.smithi111.stdout:9/323: mknod d5/d6/d14/c67 0 2024-03-14T13:02:11.949 INFO:tasks.workunit.client.0.smithi111.stdout:6/244: readlink d0/d17/d21/d42/d49/d3d/l54 0 2024-03-14T13:02:11.949 INFO:tasks.workunit.client.0.smithi111.stdout:9/324: readlink d5/d6/l37 0 2024-03-14T13:02:11.950 INFO:tasks.workunit.client.0.smithi111.stdout:5/355: mkdir d2/d16/d49/d7c 0 2024-03-14T13:02:11.950 INFO:tasks.workunit.client.1.smithi137.stdout:5/124: fdatasync d7/f19 0 2024-03-14T13:02:11.950 INFO:tasks.workunit.client.1.smithi137.stdout:7/171: dwrite d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:11.953 INFO:tasks.workunit.client.0.smithi111.stdout:1/276: symlink d20/d25/l62 0 2024-03-14T13:02:11.954 INFO:tasks.workunit.client.1.smithi137.stdout:1/265: creat d7/f53 x:0 0 0 2024-03-14T13:02:11.956 INFO:tasks.workunit.client.1.smithi137.stdout:7/172: dread d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:11.958 INFO:tasks.workunit.client.1.smithi137.stdout:5/125: rename d7/d23 to d7/d16/d18/d25/d28 0 2024-03-14T13:02:11.958 INFO:tasks.workunit.client.1.smithi137.stdout:6/221: symlink d2/l52 0 2024-03-14T13:02:11.961 INFO:tasks.workunit.client.1.smithi137.stdout:1/266: mkdir d7/d31/d54 0 2024-03-14T13:02:11.962 INFO:tasks.workunit.client.1.smithi137.stdout:5/126: readlink d7/l13 0 2024-03-14T13:02:11.962 INFO:tasks.workunit.client.1.smithi137.stdout:5/127: chown d7/fa 28 1 2024-03-14T13:02:11.965 INFO:tasks.workunit.client.0.smithi111.stdout:6/245: dwrite d0/d17/f55 [0,4194304] 0 2024-03-14T13:02:11.965 INFO:tasks.workunit.client.1.smithi137.stdout:9/251: creat d6/df/f5e x:0 0 0 2024-03-14T13:02:11.966 INFO:tasks.workunit.client.0.smithi111.stdout:5/356: dwrite d2/d16/f70 [0,4194304] 0 2024-03-14T13:02:11.967 INFO:tasks.workunit.client.1.smithi137.stdout:9/252: truncate d6/df/d19/f45 1487426 0 2024-03-14T13:02:11.967 INFO:tasks.workunit.client.0.smithi111.stdout:6/246: chown d0/d17/d20/f32 1326 1 2024-03-14T13:02:11.968 INFO:tasks.workunit.client.1.smithi137.stdout:9/253: truncate d6/f2b 5128161 0 2024-03-14T13:02:11.968 INFO:tasks.workunit.client.0.smithi111.stdout:1/277: dwrite d20/d25/f46 [0,4194304] 0 2024-03-14T13:02:11.968 INFO:tasks.workunit.client.0.smithi111.stdout:5/357: chown d2/d16/c2d 5 1 2024-03-14T13:02:11.969 INFO:tasks.workunit.client.1.smithi137.stdout:9/254: dread - d6/df/d19/d3a/d40/f4b zero size 2024-03-14T13:02:11.970 INFO:tasks.workunit.client.0.smithi111.stdout:5/358: dread - d2/d1f/d6a/f71 zero size 2024-03-14T13:02:11.972 INFO:tasks.workunit.client.1.smithi137.stdout:9/255: stat d6/df/d19/c55 0 2024-03-14T13:02:11.972 INFO:tasks.workunit.client.1.smithi137.stdout:7/173: truncate d3/f26 215451 0 2024-03-14T13:02:11.972 INFO:tasks.workunit.client.1.smithi137.stdout:5/128: chown d7/l27 94509798 1 2024-03-14T13:02:11.972 INFO:tasks.workunit.client.0.smithi111.stdout:5/359: write d2/d16/d49/d4c/f73 [487905,43309] 0 2024-03-14T13:02:11.973 INFO:tasks.workunit.client.1.smithi137.stdout:2/173: link d2/f1e d2/d7/db/de/f39 0 2024-03-14T13:02:11.974 INFO:tasks.workunit.client.0.smithi111.stdout:9/325: write d5/d6/d8/d1a/f43 [943563,108962] 0 2024-03-14T13:02:11.974 INFO:tasks.workunit.client.1.smithi137.stdout:5/129: read d7/fb [1407911,121602] 0 2024-03-14T13:02:11.976 INFO:tasks.workunit.client.0.smithi111.stdout:0/241: dread d6/d9/d15/f1b [0,4194304] 0 2024-03-14T13:02:11.978 INFO:tasks.workunit.client.0.smithi111.stdout:9/326: dread - d5/d6/d8/f3a zero size 2024-03-14T13:02:11.981 INFO:tasks.workunit.client.1.smithi137.stdout:2/174: mkdir d2/d7/db/de/d1d/d3a 0 2024-03-14T13:02:11.981 INFO:tasks.workunit.client.1.smithi137.stdout:6/222: getdents d2/d6/d8/d28 0 2024-03-14T13:02:11.982 INFO:tasks.workunit.client.0.smithi111.stdout:9/327: dread d5/d6/f2f [0,4194304] 0 2024-03-14T13:02:11.983 INFO:tasks.workunit.client.0.smithi111.stdout:0/242: dwrite f3 [0,4194304] 0 2024-03-14T13:02:11.985 INFO:tasks.workunit.client.1.smithi137.stdout:6/223: mknod d2/d6/c53 0 2024-03-14T13:02:11.985 INFO:tasks.workunit.client.0.smithi111.stdout:9/328: chown d5/d6/d8/l44 298 1 2024-03-14T13:02:11.986 INFO:tasks.workunit.client.0.smithi111.stdout:0/243: dread - d6/d9/de/f41 zero size 2024-03-14T13:02:11.987 INFO:tasks.workunit.client.0.smithi111.stdout:9/329: symlink d5/d42/d52/l68 0 2024-03-14T13:02:11.988 INFO:tasks.workunit.client.0.smithi111.stdout:0/244: write d6/d9/de/f27 [2494345,65510] 0 2024-03-14T13:02:11.988 INFO:tasks.workunit.client.0.smithi111.stdout:9/330: write d5/f31 [4875320,113923] 0 2024-03-14T13:02:11.992 INFO:tasks.workunit.client.1.smithi137.stdout:0/203: sync 2024-03-14T13:02:11.994 INFO:tasks.workunit.client.1.smithi137.stdout:1/267: link d7/l20 d7/l55 0 2024-03-14T13:02:11.994 INFO:tasks.workunit.client.1.smithi137.stdout:0/204: fdatasync d1/d5/f37 0 2024-03-14T13:02:11.997 INFO:tasks.workunit.client.0.smithi111.stdout:0/245: dwrite f3 [4194304,4194304] 0 2024-03-14T13:02:11.997 INFO:tasks.workunit.client.0.smithi111.stdout:1/278: write d20/d25/f2e [2660041,64256] 0 2024-03-14T13:02:11.999 INFO:tasks.workunit.client.1.smithi137.stdout:0/205: read d1/f1c [3623458,101050] 0 2024-03-14T13:02:12.000 INFO:tasks.workunit.client.1.smithi137.stdout:9/256: dwrite d6/df/d19/f4c [0,4194304] 0 2024-03-14T13:02:12.001 INFO:tasks.workunit.client.1.smithi137.stdout:6/224: dwrite d2/d6/de/f13 [0,4194304] 0 2024-03-14T13:02:12.003 INFO:tasks.workunit.client.0.smithi111.stdout:0/246: mknod d6/d9/d15/c50 0 2024-03-14T13:02:12.004 INFO:tasks.workunit.client.1.smithi137.stdout:9/257: dread - d6/df/d19/d33/f36 zero size 2024-03-14T13:02:12.004 INFO:tasks.workunit.client.1.smithi137.stdout:0/206: mknod d1/c45 0 2024-03-14T13:02:12.007 INFO:tasks.workunit.client.0.smithi111.stdout:0/247: mknod d6/d9/d15/d49/c51 0 2024-03-14T13:02:12.008 INFO:tasks.workunit.client.0.smithi111.stdout:6/247: dread d0/d17/f1e [0,4194304] 0 2024-03-14T13:02:12.015 INFO:tasks.workunit.client.0.smithi111.stdout:1/279: dwrite fa [0,4194304] 0 2024-03-14T13:02:12.015 INFO:tasks.workunit.client.0.smithi111.stdout:6/248: creat d0/d17/d21/d42/d49/d3d/f5c x:0 0 0 2024-03-14T13:02:12.017 INFO:tasks.workunit.client.1.smithi137.stdout:9/258: getdents d6/d26 0 2024-03-14T13:02:12.018 INFO:tasks.workunit.client.1.smithi137.stdout:0/207: creat d1/f46 x:0 0 0 2024-03-14T13:02:12.018 INFO:tasks.workunit.client.1.smithi137.stdout:1/268: getdents d7/d9 0 2024-03-14T13:02:12.021 INFO:tasks.workunit.client.1.smithi137.stdout:9/259: write d6/df/f20 [4486895,9378] 0 2024-03-14T13:02:12.021 INFO:tasks.workunit.client.1.smithi137.stdout:0/208: chown d1/d5/d1b/d1e/d25 2103921 1 2024-03-14T13:02:12.021 INFO:tasks.workunit.client.0.smithi111.stdout:1/280: write d20/d25/f30 [240177,62058] 0 2024-03-14T13:02:12.022 INFO:tasks.workunit.client.1.smithi137.stdout:2/175: dwrite d2/d7/db/de/d22/d26/f27 [4194304,4194304] 0 2024-03-14T13:02:12.023 INFO:tasks.workunit.client.1.smithi137.stdout:1/269: unlink d7/d9/fd 0 2024-03-14T13:02:12.024 INFO:tasks.workunit.client.1.smithi137.stdout:9/260: dread - d6/df/d19/d3a/d40/f5c zero size 2024-03-14T13:02:12.025 INFO:tasks.workunit.client.1.smithi137.stdout:2/176: truncate d2/d7/db/f2c 92653 0 2024-03-14T13:02:12.025 INFO:tasks.workunit.client.0.smithi111.stdout:1/281: write f19 [1379355,108964] 0 2024-03-14T13:02:12.026 INFO:tasks.workunit.client.0.smithi111.stdout:6/249: getdents d0/d17 0 2024-03-14T13:02:12.027 INFO:tasks.workunit.client.0.smithi111.stdout:6/250: stat d0/d17/f1e 0 2024-03-14T13:02:12.028 INFO:tasks.workunit.client.1.smithi137.stdout:0/209: symlink d1/d5/l47 0 2024-03-14T13:02:12.029 INFO:tasks.workunit.client.0.smithi111.stdout:6/251: creat d0/d17/d21/f5d x:0 0 0 2024-03-14T13:02:12.029 INFO:tasks.workunit.client.0.smithi111.stdout:1/282: mknod d20/d21/d3f/d59/c63 0 2024-03-14T13:02:12.031 INFO:tasks.workunit.client.0.smithi111.stdout:5/360: dread d2/d16/d49/d44/d4a/f60 [0,4194304] 0 2024-03-14T13:02:12.033 INFO:tasks.workunit.client.0.smithi111.stdout:6/252: mkdir d0/d17/d20/d5a/d5e 0 2024-03-14T13:02:12.033 INFO:tasks.workunit.client.0.smithi111.stdout:5/361: truncate d2/d13/f6c 477365 0 2024-03-14T13:02:12.033 INFO:tasks.workunit.client.1.smithi137.stdout:9/261: write d6/df/d19/d1f/f24 [2785465,90294] 0 2024-03-14T13:02:12.033 INFO:tasks.workunit.client.1.smithi137.stdout:2/177: rename d2/d7/db/cd to d2/d7/d2e/c3b 0 2024-03-14T13:02:12.035 INFO:tasks.workunit.client.0.smithi111.stdout:1/283: rmdir d20/d21/d3f/d5b 0 2024-03-14T13:02:12.037 INFO:tasks.workunit.client.0.smithi111.stdout:6/253: creat d0/d50/f5f x:0 0 0 2024-03-14T13:02:12.038 INFO:tasks.workunit.client.1.smithi137.stdout:9/262: creat d6/d26/f5f x:0 0 0 2024-03-14T13:02:12.038 INFO:tasks.workunit.client.0.smithi111.stdout:5/362: read d2/f7 [1964213,83025] 0 2024-03-14T13:02:12.038 INFO:tasks.workunit.client.0.smithi111.stdout:6/254: dread - d0/d17/d20/f4b zero size 2024-03-14T13:02:12.039 INFO:tasks.workunit.client.1.smithi137.stdout:7/174: dwrite d3/f1e [0,4194304] 0 2024-03-14T13:02:12.040 INFO:tasks.workunit.client.0.smithi111.stdout:1/284: getdents d20/d21/d3f/d5a 0 2024-03-14T13:02:12.040 INFO:tasks.workunit.client.1.smithi137.stdout:0/210: link d1/f1c d1/d5/d1b/d42/d3a/f48 0 2024-03-14T13:02:12.041 INFO:tasks.workunit.client.0.smithi111.stdout:6/255: dread - d0/d17/d21/d42/d49/f45 zero size 2024-03-14T13:02:12.041 INFO:tasks.workunit.client.1.smithi137.stdout:1/270: dread d7/d9/d19/d1f/d22/f25 [0,4194304] 0 2024-03-14T13:02:12.041 INFO:tasks.workunit.client.0.smithi111.stdout:5/363: read d2/d16/d49/d33/f43 [899249,59662] 0 2024-03-14T13:02:12.042 INFO:tasks.workunit.client.1.smithi137.stdout:7/175: readlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:12.046 INFO:tasks.workunit.client.0.smithi111.stdout:5/364: creat d2/d16/d49/d4c/f7d x:0 0 0 2024-03-14T13:02:12.047 INFO:tasks.workunit.client.1.smithi137.stdout:1/271: read d7/d9/ff [449028,102800] 0 2024-03-14T13:02:12.048 INFO:tasks.workunit.client.0.smithi111.stdout:1/285: getdents d20 0 2024-03-14T13:02:12.048 INFO:tasks.workunit.client.0.smithi111.stdout:6/256: mknod d0/d17/d20/d4a/d59/c60 0 2024-03-14T13:02:12.049 INFO:tasks.workunit.client.1.smithi137.stdout:2/178: dread d2/d7/db/de/f20 [0,4194304] 0 2024-03-14T13:02:12.049 INFO:tasks.workunit.client.0.smithi111.stdout:0/248: write f1 [2667278,71487] 0 2024-03-14T13:02:12.053 INFO:tasks.workunit.client.0.smithi111.stdout:1/286: creat d20/d25/f64 x:0 0 0 2024-03-14T13:02:12.054 INFO:tasks.workunit.client.0.smithi111.stdout:0/249: unlink d6/l8 0 2024-03-14T13:02:12.056 INFO:tasks.workunit.client.0.smithi111.stdout:6/257: creat d0/d17/d21/d42/d49/f61 x:0 0 0 2024-03-14T13:02:12.056 INFO:tasks.workunit.client.1.smithi137.stdout:2/179: mkdir d2/d34/d3c 0 2024-03-14T13:02:12.056 INFO:tasks.workunit.client.0.smithi111.stdout:5/365: dwrite d2/d16/d49/f78 [0,4194304] 0 2024-03-14T13:02:12.057 INFO:tasks.workunit.client.1.smithi137.stdout:6/225: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:02:12.058 INFO:tasks.workunit.client.0.smithi111.stdout:5/366: truncate d2/d1f/f5e 520827 0 2024-03-14T13:02:12.058 INFO:tasks.workunit.client.0.smithi111.stdout:1/287: creat d20/d21/d3f/d5a/f65 x:0 0 0 2024-03-14T13:02:12.059 INFO:tasks.workunit.client.1.smithi137.stdout:9/263: dwrite d6/fe [0,4194304] 0 2024-03-14T13:02:12.060 INFO:tasks.workunit.client.1.smithi137.stdout:7/176: dwrite d3/f18 [4194304,4194304] 0 2024-03-14T13:02:12.062 INFO:tasks.workunit.client.0.smithi111.stdout:6/258: mknod d0/d17/d21/d42/c62 0 2024-03-14T13:02:12.062 INFO:tasks.workunit.client.1.smithi137.stdout:6/226: read - d2/d6/d8/d38/f51 zero size 2024-03-14T13:02:12.063 INFO:tasks.workunit.client.1.smithi137.stdout:1/272: link d7/d9/d19/f2f d7/d4d/f56 0 2024-03-14T13:02:12.063 INFO:tasks.workunit.client.1.smithi137.stdout:2/180: read d2/d7/db/de/f39 [1007161,23094] 0 2024-03-14T13:02:12.064 INFO:tasks.workunit.client.1.smithi137.stdout:7/177: mknod d3/d6/d14/d27/c36 0 2024-03-14T13:02:12.065 INFO:tasks.workunit.client.1.smithi137.stdout:9/264: creat d6/df/d19/d3a/d40/f60 x:0 0 0 2024-03-14T13:02:12.066 INFO:tasks.workunit.client.0.smithi111.stdout:1/288: truncate d20/f44 16464 0 2024-03-14T13:02:12.066 INFO:tasks.workunit.client.1.smithi137.stdout:1/273: stat d7/l1e 0 2024-03-14T13:02:12.067 INFO:tasks.workunit.client.0.smithi111.stdout:1/289: chown c1c 53145176 1 2024-03-14T13:02:12.069 INFO:tasks.workunit.client.0.smithi111.stdout:6/259: rename d0/d17/d21/d42/d49/d3d/d53 to d0/d17/d21/d42/d49/d3d/d53/d63 22 2024-03-14T13:02:12.069 INFO:tasks.workunit.client.0.smithi111.stdout:1/290: creat d20/d21/d3f/f66 x:0 0 0 2024-03-14T13:02:12.071 INFO:tasks.workunit.client.0.smithi111.stdout:5/367: dwrite d2/d1f/d6a/f6e [0,4194304] 0 2024-03-14T13:02:12.071 INFO:tasks.workunit.client.0.smithi111.stdout:0/250: write d6/d9/d15/f37 [2477384,7259] 0 2024-03-14T13:02:12.072 INFO:tasks.workunit.client.1.smithi137.stdout:9/265: fsync d6/f49 0 2024-03-14T13:02:12.075 INFO:tasks.workunit.client.0.smithi111.stdout:1/291: rename d20/d21/l4a to d20/d21/d28/d5c/l67 0 2024-03-14T13:02:12.079 INFO:tasks.workunit.client.0.smithi111.stdout:1/292: write d20/d21/f42 [950536,17194] 0 2024-03-14T13:02:12.080 INFO:tasks.workunit.client.0.smithi111.stdout:6/260: link d0/d17/d20/d5a/l5b d0/d17/d24/l64 0 2024-03-14T13:02:12.082 INFO:tasks.workunit.client.0.smithi111.stdout:1/293: write d20/d21/d3f/f66 [727929,30605] 0 2024-03-14T13:02:12.082 INFO:tasks.workunit.client.0.smithi111.stdout:0/251: dwrite d6/d21/f42 [0,4194304] 0 2024-03-14T13:02:12.084 INFO:tasks.workunit.client.0.smithi111.stdout:6/261: mkdir d0/d17/d20/d65 0 2024-03-14T13:02:12.085 INFO:tasks.workunit.client.0.smithi111.stdout:1/294: mknod d20/d25/c68 0 2024-03-14T13:02:12.087 INFO:tasks.workunit.client.0.smithi111.stdout:0/252: creat d6/d3d/f52 x:0 0 0 2024-03-14T13:02:12.088 INFO:tasks.workunit.client.0.smithi111.stdout:6/262: creat d0/d50/f66 x:0 0 0 2024-03-14T13:02:12.089 INFO:tasks.workunit.client.1.smithi137.stdout:6/227: dwrite d2/d6/d8/d38/f51 [0,4194304] 0 2024-03-14T13:02:12.089 INFO:tasks.workunit.client.1.smithi137.stdout:2/181: dwrite d2/d7/db/f1c [0,4194304] 0 2024-03-14T13:02:12.090 INFO:tasks.workunit.client.1.smithi137.stdout:1/274: getdents d7/d9/d19/d1f/d2e 0 2024-03-14T13:02:12.092 INFO:tasks.workunit.client.1.smithi137.stdout:7/178: dwrite d3/f1c [0,4194304] 0 2024-03-14T13:02:12.092 INFO:tasks.workunit.client.1.smithi137.stdout:0/211: dread d1/d11/f19 [0,4194304] 0 2024-03-14T13:02:12.092 INFO:tasks.workunit.client.1.smithi137.stdout:9/266: getdents d6/d26 0 2024-03-14T13:02:12.093 INFO:tasks.workunit.client.0.smithi111.stdout:0/253: truncate d6/d9/de/f3c 662561 0 2024-03-14T13:02:12.093 INFO:tasks.workunit.client.0.smithi111.stdout:6/263: write d0/d17/d20/f32 [404971,102742] 0 2024-03-14T13:02:12.095 INFO:tasks.workunit.client.0.smithi111.stdout:1/295: creat d20/d25/d31/f69 x:0 0 0 2024-03-14T13:02:12.095 INFO:tasks.workunit.client.0.smithi111.stdout:6/264: dread - d0/d50/f66 zero size 2024-03-14T13:02:12.096 INFO:tasks.workunit.client.0.smithi111.stdout:1/296: creat d20/d21/d3f/d5a/f6a x:0 0 0 2024-03-14T13:02:12.097 INFO:tasks.workunit.client.0.smithi111.stdout:6/265: fdatasync d0/d17/d21/d42/d49/d3d/f5c 0 2024-03-14T13:02:12.099 INFO:tasks.workunit.client.0.smithi111.stdout:6/266: write d0/d17/d24/f38 [947231,34071] 0 2024-03-14T13:02:12.100 INFO:tasks.workunit.client.1.smithi137.stdout:0/212: mknod d1/d5/c49 0 2024-03-14T13:02:12.104 INFO:tasks.workunit.client.0.smithi111.stdout:5/368: write d2/d13/f15 [4872671,54551] 0 2024-03-14T13:02:12.104 INFO:tasks.workunit.client.0.smithi111.stdout:1/297: mknod d20/d21/c6b 0 2024-03-14T13:02:12.105 INFO:tasks.workunit.client.1.smithi137.stdout:0/213: stat d1/d1f/l18 0 2024-03-14T13:02:12.107 INFO:tasks.workunit.client.0.smithi111.stdout:1/298: fsync d20/d21/d3f/d5a/f65 0 2024-03-14T13:02:12.111 INFO:tasks.workunit.client.0.smithi111.stdout:6/267: getdents d0 0 2024-03-14T13:02:12.111 INFO:tasks.workunit.client.0.smithi111.stdout:1/299: truncate d20/d21/d3f/d5a/f65 859090 0 2024-03-14T13:02:12.113 INFO:tasks.workunit.client.0.smithi111.stdout:1/300: stat d20/d25/d31/f3b 0 2024-03-14T13:02:12.113 INFO:tasks.workunit.client.1.smithi137.stdout:1/275: dwrite d7/d9/f15 [0,4194304] 0 2024-03-14T13:02:12.114 INFO:tasks.workunit.client.0.smithi111.stdout:6/268: write d0/d17/d20/d4a/f51 [606967,37804] 0 2024-03-14T13:02:12.114 INFO:tasks.workunit.client.0.smithi111.stdout:1/301: fsync d20/d40/f4f 0 2024-03-14T13:02:12.116 INFO:tasks.workunit.client.0.smithi111.stdout:6/269: write d0/f3e [203754,54932] 0 2024-03-14T13:02:12.116 INFO:tasks.workunit.client.1.smithi137.stdout:2/182: dwrite d2/d7/db/de/d22/d26/f33 [0,4194304] 0 2024-03-14T13:02:12.116 INFO:tasks.workunit.client.1.smithi137.stdout:0/214: rmdir d1/d5/d43 0 2024-03-14T13:02:12.117 INFO:tasks.workunit.client.0.smithi111.stdout:0/254: getdents d6/d9 0 2024-03-14T13:02:12.117 INFO:tasks.workunit.client.0.smithi111.stdout:1/302: chown d20/d25/d31/c51 231522004 1 2024-03-14T13:02:12.118 INFO:tasks.workunit.client.0.smithi111.stdout:6/270: chown d0/f14 63150656 1 2024-03-14T13:02:12.118 INFO:tasks.workunit.client.0.smithi111.stdout:1/303: chown l1e 141358 1 2024-03-14T13:02:12.118 INFO:tasks.workunit.client.1.smithi137.stdout:9/267: dwrite d6/df/f51 [0,4194304] 0 2024-03-14T13:02:12.119 INFO:tasks.workunit.client.1.smithi137.stdout:7/179: dread d3/f18 [4194304,4194304] 0 2024-03-14T13:02:12.120 INFO:tasks.workunit.client.1.smithi137.stdout:6/228: dwrite d2/d6/d1d/f4e [0,4194304] 0 2024-03-14T13:02:12.121 INFO:tasks.workunit.client.1.smithi137.stdout:1/276: write d7/d9/d19/d1f/f2a [3601829,65840] 0 2024-03-14T13:02:12.125 INFO:tasks.workunit.client.1.smithi137.stdout:7/180: write d3/f1c [1761491,106840] 0 2024-03-14T13:02:12.125 INFO:tasks.workunit.client.1.smithi137.stdout:2/183: creat d2/d7/d2e/f3d x:0 0 0 2024-03-14T13:02:12.125 INFO:tasks.workunit.client.1.smithi137.stdout:6/229: rmdir d2/d6/de 39 2024-03-14T13:02:12.127 INFO:tasks.workunit.client.0.smithi111.stdout:0/255: creat d6/d9/de/f53 x:0 0 0 2024-03-14T13:02:12.128 INFO:tasks.workunit.client.1.smithi137.stdout:1/277: rmdir d7/d31 39 2024-03-14T13:02:12.132 INFO:tasks.workunit.client.0.smithi111.stdout:1/304: dwrite d20/d21/f42 [0,4194304] 0 2024-03-14T13:02:12.132 INFO:tasks.workunit.client.1.smithi137.stdout:1/278: write d7/d9/d19/d40/d42/f49 [408529,69012] 0 2024-03-14T13:02:12.132 INFO:tasks.workunit.client.1.smithi137.stdout:9/268: dwrite d6/f49 [0,4194304] 0 2024-03-14T13:02:12.134 INFO:tasks.workunit.client.0.smithi111.stdout:5/369: dread d2/d16/d49/f12 [0,4194304] 0 2024-03-14T13:02:12.136 INFO:tasks.workunit.client.1.smithi137.stdout:1/279: dread d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:12.137 INFO:tasks.workunit.client.1.smithi137.stdout:7/181: dread d3/d6/f15 [0,4194304] 0 2024-03-14T13:02:12.140 INFO:tasks.workunit.client.0.smithi111.stdout:6/271: dwrite d0/d17/d21/d42/d49/f52 [0,4194304] 0 2024-03-14T13:02:12.140 INFO:tasks.workunit.client.0.smithi111.stdout:5/370: read d2/d13/f2f [55350,32480] 0 2024-03-14T13:02:12.144 INFO:tasks.workunit.client.1.smithi137.stdout:9/269: dread d6/f57 [0,4194304] 0 2024-03-14T13:02:12.145 INFO:tasks.workunit.client.0.smithi111.stdout:0/256: rename d6/d9 to d6/d9/de/d3a/d46/d4a/d54 22 2024-03-14T13:02:12.145 INFO:tasks.workunit.client.1.smithi137.stdout:1/280: dread d7/f1d [0,4194304] 0 2024-03-14T13:02:12.145 INFO:tasks.workunit.client.0.smithi111.stdout:6/272: creat d0/d50/f67 x:0 0 0 2024-03-14T13:02:12.146 INFO:tasks.workunit.client.0.smithi111.stdout:5/371: chown d2/d16/c2d 1251561 1 2024-03-14T13:02:12.147 INFO:tasks.workunit.client.0.smithi111.stdout:7/253: sync 2024-03-14T13:02:12.150 INFO:tasks.workunit.client.0.smithi111.stdout:0/257: chown d6/d9/de/f1d 75391 1 2024-03-14T13:02:12.153 INFO:tasks.workunit.client.0.smithi111.stdout:0/258: write f1 [2530882,110934] 0 2024-03-14T13:02:12.153 INFO:tasks.workunit.client.0.smithi111.stdout:6/273: mkdir d0/d17/d20/d4a/d68 0 2024-03-14T13:02:12.155 INFO:tasks.workunit.client.0.smithi111.stdout:5/372: write d2/d16/d49/f27 [4594067,39805] 0 2024-03-14T13:02:12.160 INFO:tasks.workunit.client.0.smithi111.stdout:0/259: mkdir d6/d9/d15/d55 0 2024-03-14T13:02:12.164 INFO:tasks.workunit.client.0.smithi111.stdout:7/254: dwrite d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:12.166 INFO:tasks.workunit.client.0.smithi111.stdout:3/358: sync 2024-03-14T13:02:12.169 INFO:tasks.workunit.client.0.smithi111.stdout:6/274: rename d0/d17/d21/d42/d49 to d0/d17/d21/d69 0 2024-03-14T13:02:12.171 INFO:tasks.workunit.client.0.smithi111.stdout:0/260: mkdir d6/d9/d15/d49/d56 0 2024-03-14T13:02:12.172 INFO:tasks.workunit.client.0.smithi111.stdout:7/255: creat d0/d1/d4a/f4b x:0 0 0 2024-03-14T13:02:12.173 INFO:tasks.workunit.client.0.smithi111.stdout:7/256: read - d0/d1/d4a/f4b zero size 2024-03-14T13:02:12.177 INFO:tasks.workunit.client.0.smithi111.stdout:7/257: creat d0/d1/d5/f4c x:0 0 0 2024-03-14T13:02:12.177 INFO:tasks.workunit.client.0.smithi111.stdout:3/359: symlink d0/d2d/d35/d50/d5e/d40/d67/l8c 0 2024-03-14T13:02:12.177 INFO:tasks.workunit.client.0.smithi111.stdout:7/258: rename d0/d1/d4a to d0/d1/d4a/d4d 22 2024-03-14T13:02:12.177 INFO:tasks.workunit.client.0.smithi111.stdout:7/259: chown d0/d1/d22 213 1 2024-03-14T13:02:12.179 INFO:tasks.workunit.client.1.smithi137.stdout:4/238: sync 2024-03-14T13:02:12.179 INFO:tasks.workunit.client.1.smithi137.stdout:3/181: sync 2024-03-14T13:02:12.179 INFO:tasks.workunit.client.1.smithi137.stdout:8/130: sync 2024-03-14T13:02:12.182 INFO:tasks.workunit.client.0.smithi111.stdout:7/260: symlink d0/d1/d22/d3e/l4e 0 2024-03-14T13:02:12.183 INFO:tasks.workunit.client.1.smithi137.stdout:4/239: write d4/d9/f18 [923854,5996] 0 2024-03-14T13:02:12.184 INFO:tasks.workunit.client.0.smithi111.stdout:1/305: dread f11 [0,4194304] 0 2024-03-14T13:02:12.184 INFO:tasks.workunit.client.0.smithi111.stdout:7/261: dread - d0/d1/d5/f4c zero size 2024-03-14T13:02:12.186 INFO:tasks.workunit.client.0.smithi111.stdout:3/360: dwrite d0/d2d/d35/d4c/d82/f7e [0,4194304] 0 2024-03-14T13:02:12.187 INFO:tasks.workunit.client.0.smithi111.stdout:1/306: readlink d20/d21/l55 0 2024-03-14T13:02:12.189 INFO:tasks.workunit.client.1.smithi137.stdout:7/182: rename d3/ff to d3/d5/f37 0 2024-03-14T13:02:12.189 INFO:tasks.workunit.client.1.smithi137.stdout:7/183: chown d3/d6 1 1 2024-03-14T13:02:12.193 INFO:tasks.workunit.client.0.smithi111.stdout:3/361: dread d0/d2d/d35/d50/d5e/d19/f1a [0,4194304] 0 2024-03-14T13:02:12.193 INFO:tasks.workunit.client.1.smithi137.stdout:9/270: creat d6/df/d19/d33/f61 x:0 0 0 2024-03-14T13:02:12.196 INFO:tasks.workunit.client.0.smithi111.stdout:1/307: dwrite d20/d21/d3f/d5a/f65 [0,4194304] 0 2024-03-14T13:02:12.199 INFO:tasks.workunit.client.0.smithi111.stdout:1/308: read f19 [2570797,126004] 0 2024-03-14T13:02:12.202 INFO:tasks.workunit.client.1.smithi137.stdout:8/131: creat d1/dc/d10/f32 x:0 0 0 2024-03-14T13:02:12.207 INFO:tasks.workunit.client.1.smithi137.stdout:3/182: symlink d4/de/d16/l33 0 2024-03-14T13:02:12.207 INFO:tasks.workunit.client.1.smithi137.stdout:9/271: write d6/df/d19/d33/f46 [152281,3829] 0 2024-03-14T13:02:12.209 INFO:tasks.workunit.client.1.smithi137.stdout:1/281: symlink d7/d31/l57 0 2024-03-14T13:02:12.215 INFO:tasks.workunit.client.1.smithi137.stdout:4/240: dwrite d4/d9/d3a/d41/f49 [0,4194304] 0 2024-03-14T13:02:12.216 INFO:tasks.workunit.client.1.smithi137.stdout:3/183: mkdir d4/df/d34 0 2024-03-14T13:02:12.218 INFO:tasks.workunit.client.1.smithi137.stdout:1/282: truncate d7/d9/f4f 504486 0 2024-03-14T13:02:12.218 INFO:tasks.workunit.client.1.smithi137.stdout:7/184: mknod d3/d6/c38 0 2024-03-14T13:02:12.219 INFO:tasks.workunit.client.1.smithi137.stdout:1/283: dread - d7/d9/d19/d40/f47 zero size 2024-03-14T13:02:12.225 INFO:tasks.workunit.client.1.smithi137.stdout:9/272: creat d6/df/d19/f62 x:0 0 0 2024-03-14T13:02:12.225 INFO:tasks.workunit.client.1.smithi137.stdout:1/284: unlink d7/d9/d19/d1f/d2e/f51 0 2024-03-14T13:02:12.225 INFO:tasks.workunit.client.1.smithi137.stdout:3/184: chown d4/de/d16/c26 122 1 2024-03-14T13:02:12.227 INFO:tasks.workunit.client.1.smithi137.stdout:9/273: chown d6/d26 1 1 2024-03-14T13:02:12.229 INFO:tasks.workunit.client.1.smithi137.stdout:4/241: getdents d4/d9/d3a/d41 0 2024-03-14T13:02:12.229 INFO:tasks.workunit.client.1.smithi137.stdout:7/185: getdents d3/d5/d23 0 2024-03-14T13:02:12.229 INFO:tasks.workunit.client.1.smithi137.stdout:9/274: dread - d6/df/d19/d33/f36 zero size 2024-03-14T13:02:12.230 INFO:tasks.workunit.client.1.smithi137.stdout:3/185: truncate d4/de/d16/d1c/f2d 475062 0 2024-03-14T13:02:12.230 INFO:tasks.workunit.client.1.smithi137.stdout:1/285: mkdir d7/d9/d19/d40/d58 0 2024-03-14T13:02:12.232 INFO:tasks.workunit.client.1.smithi137.stdout:7/186: mknod d3/d6/d14/d27/c39 0 2024-03-14T13:02:12.233 INFO:tasks.workunit.client.1.smithi137.stdout:4/242: symlink d4/d9/l51 0 2024-03-14T13:02:12.234 INFO:tasks.workunit.client.1.smithi137.stdout:3/186: mkdir d4/df/d35 0 2024-03-14T13:02:12.234 INFO:tasks.workunit.client.1.smithi137.stdout:1/286: rename d7 to d7/d9/d19/d1f/d59 22 2024-03-14T13:02:12.237 INFO:tasks.workunit.client.1.smithi137.stdout:4/243: rename d4/d10/d3d/f3e to d4/d10/d42/f52 0 2024-03-14T13:02:12.237 INFO:tasks.workunit.client.1.smithi137.stdout:1/287: mknod d7/d9/c5a 0 2024-03-14T13:02:12.241 INFO:tasks.workunit.client.1.smithi137.stdout:1/288: rename d7/d9/f41 to d7/d9/d19/d40/f5b 0 2024-03-14T13:02:12.243 INFO:tasks.workunit.client.1.smithi137.stdout:3/187: link d4/d19/f23 d4/de/d13/f36 0 2024-03-14T13:02:12.246 INFO:tasks.workunit.client.1.smithi137.stdout:1/289: mknod d7/d9/c5c 0 2024-03-14T13:02:12.247 INFO:tasks.workunit.client.1.smithi137.stdout:3/188: creat d4/f37 x:0 0 0 2024-03-14T13:02:12.248 INFO:tasks.workunit.client.1.smithi137.stdout:1/290: mknod d7/d4d/c5d 0 2024-03-14T13:02:12.249 INFO:tasks.workunit.client.1.smithi137.stdout:3/189: symlink d4/df/l38 0 2024-03-14T13:02:12.250 INFO:tasks.workunit.client.1.smithi137.stdout:3/190: chown d4/f2b 0 1 2024-03-14T13:02:12.250 INFO:tasks.workunit.client.1.smithi137.stdout:7/187: dread d3/f18 [0,4194304] 0 2024-03-14T13:02:12.252 INFO:tasks.workunit.client.1.smithi137.stdout:7/188: truncate d3/d5/f19 4876707 0 2024-03-14T13:02:12.252 INFO:tasks.workunit.client.1.smithi137.stdout:1/291: unlink d7/d9/d19/d40/f47 0 2024-03-14T13:02:12.253 INFO:tasks.workunit.client.1.smithi137.stdout:3/191: creat d4/d22/f39 x:0 0 0 2024-03-14T13:02:12.253 INFO:tasks.workunit.client.1.smithi137.stdout:7/189: mknod d3/d6/d14/c3a 0 2024-03-14T13:02:12.255 INFO:tasks.workunit.client.1.smithi137.stdout:3/192: write d4/d22/f2e [732577,127725] 0 2024-03-14T13:02:12.256 INFO:tasks.workunit.client.1.smithi137.stdout:7/190: chown d3/l21 405 1 2024-03-14T13:02:12.256 INFO:tasks.workunit.client.1.smithi137.stdout:1/292: write d7/d9/d19/d1f/d2e/f3d [1764480,88526] 0 2024-03-14T13:02:12.257 INFO:tasks.workunit.client.0.smithi111.stdout:3/362: dread d0/d2d/d35/d50/d5e/d19/f26 [0,4194304] 0 2024-03-14T13:02:12.258 INFO:tasks.workunit.client.1.smithi137.stdout:3/193: mknod d4/de/d13/c3a 0 2024-03-14T13:02:12.259 INFO:tasks.workunit.client.1.smithi137.stdout:1/293: readlink d7/d9/l13 0 2024-03-14T13:02:12.259 INFO:tasks.workunit.client.0.smithi111.stdout:3/363: rmdir d0/d2d/d35/d50/d5e 39 2024-03-14T13:02:12.260 INFO:tasks.workunit.client.0.smithi111.stdout:6/275: dread d0/d17/d24/f38 [0,4194304] 0 2024-03-14T13:02:12.260 INFO:tasks.workunit.client.1.smithi137.stdout:6/230: fsync d2/d6/d8/d38/f51 0 2024-03-14T13:02:12.261 INFO:tasks.workunit.client.1.smithi137.stdout:7/191: creat d3/d5/f3b x:0 0 0 2024-03-14T13:02:12.264 INFO:tasks.workunit.client.0.smithi111.stdout:3/364: write d0/d2d/d35/d50/d5e/d40/f47 [1384124,113635] 0 2024-03-14T13:02:12.264 INFO:tasks.workunit.client.1.smithi137.stdout:1/294: write d7/f1d [4033671,105884] 0 2024-03-14T13:02:12.265 INFO:tasks.workunit.client.0.smithi111.stdout:6/276: getdents d0/d17/d20/d4a/d68 0 2024-03-14T13:02:12.267 INFO:tasks.workunit.client.1.smithi137.stdout:1/295: chown d7/d9/d19/d1f/l36 126796 1 2024-03-14T13:02:12.268 INFO:tasks.workunit.client.0.smithi111.stdout:6/277: symlink d0/d17/l6a 0 2024-03-14T13:02:12.268 INFO:tasks.workunit.client.0.smithi111.stdout:3/365: read d0/d2d/d35/d50/d5e/d19/f1a [429980,82131] 0 2024-03-14T13:02:12.269 INFO:tasks.workunit.client.0.smithi111.stdout:6/278: stat d0/d17/d21/d69/d3d/d53 0 2024-03-14T13:02:12.270 INFO:tasks.workunit.client.0.smithi111.stdout:1/309: write d20/d21/d3f/d5a/f65 [4664406,12596] 0 2024-03-14T13:02:12.271 INFO:tasks.workunit.client.1.smithi137.stdout:8/132: truncate d1/dc/d11/d13/f16 3977318 0 2024-03-14T13:02:12.273 INFO:tasks.workunit.client.1.smithi137.stdout:1/296: mknod d7/d9/d19/d1f/d2e/c5e 0 2024-03-14T13:02:12.273 INFO:tasks.workunit.client.0.smithi111.stdout:1/310: symlink d20/d21/d3f/d5a/l6c 0 2024-03-14T13:02:12.273 INFO:tasks.workunit.client.0.smithi111.stdout:3/366: link d0/d30/c49 d0/d2d/d35/d50/d5e/d40/d6f/c8d 0 2024-03-14T13:02:12.274 INFO:tasks.workunit.client.1.smithi137.stdout:3/194: dwrite d4/d19/f23 [4194304,4194304] 0 2024-03-14T13:02:12.275 INFO:tasks.workunit.client.1.smithi137.stdout:7/192: dwrite d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:12.275 INFO:tasks.workunit.client.1.smithi137.stdout:8/133: symlink d1/dc/d11/d13/l33 0 2024-03-14T13:02:12.277 INFO:tasks.workunit.client.0.smithi111.stdout:3/367: rename d0/la to d0/d2d/d35/d69/l8e 0 2024-03-14T13:02:12.277 INFO:tasks.workunit.client.1.smithi137.stdout:7/193: write d3/d6/f15 [1407238,53946] 0 2024-03-14T13:02:12.277 INFO:tasks.workunit.client.1.smithi137.stdout:1/297: mkdir d7/d9/d19/d5f 0 2024-03-14T13:02:12.282 INFO:tasks.workunit.client.1.smithi137.stdout:8/134: mkdir d1/d17/d34 0 2024-03-14T13:02:12.283 INFO:tasks.workunit.client.1.smithi137.stdout:7/194: unlink d3/f26 0 2024-03-14T13:02:12.283 INFO:tasks.workunit.client.0.smithi111.stdout:1/311: rename d20/d25/f3e to d20/d21/d3f/d59/f6d 0 2024-03-14T13:02:12.285 INFO:tasks.workunit.client.0.smithi111.stdout:1/312: fdatasync d20/d21/d3f/d5a/f65 0 2024-03-14T13:02:12.285 INFO:tasks.workunit.client.0.smithi111.stdout:3/368: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:12.285 INFO:tasks.workunit.client.1.smithi137.stdout:3/195: symlink d4/df/d34/l3b 0 2024-03-14T13:02:12.286 INFO:tasks.workunit.client.1.smithi137.stdout:4/244: write d4/d10/f25 [4560201,42825] 0 2024-03-14T13:02:12.287 INFO:tasks.workunit.client.1.smithi137.stdout:3/196: chown d4/de/f1f 37 1 2024-03-14T13:02:12.288 INFO:tasks.workunit.client.0.smithi111.stdout:1/313: dread d20/d25/f30 [0,4194304] 0 2024-03-14T13:02:12.289 INFO:tasks.workunit.client.1.smithi137.stdout:5/130: sync 2024-03-14T13:02:12.289 INFO:tasks.workunit.client.0.smithi111.stdout:1/314: readlink d20/d25/l48 0 2024-03-14T13:02:12.290 INFO:tasks.workunit.client.0.smithi111.stdout:1/315: chown f11 170859414 1 2024-03-14T13:02:12.290 INFO:tasks.workunit.client.1.smithi137.stdout:1/298: link d7/d9/f15 d7/d31/d54/f60 0 2024-03-14T13:02:12.291 INFO:tasks.workunit.client.0.smithi111.stdout:1/316: mknod d20/d40/c6e 0 2024-03-14T13:02:12.293 INFO:tasks.workunit.client.1.smithi137.stdout:4/245: unlink d4/d9/c2c 0 2024-03-14T13:02:12.293 INFO:tasks.workunit.client.0.smithi111.stdout:3/369: rename d0/d2d/d35/d50/d5e/c2c to d0/d2d/d35/d50/d5e/d40/d6f/c8f 0 2024-03-14T13:02:12.294 INFO:tasks.workunit.client.1.smithi137.stdout:1/299: dread d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:12.296 INFO:tasks.workunit.client.1.smithi137.stdout:8/135: link d1/f6 d1/dc/d10/f35 0 2024-03-14T13:02:12.296 INFO:tasks.workunit.client.0.smithi111.stdout:3/370: mknod d0/d2d/d35/d69/d84/c90 0 2024-03-14T13:02:12.297 INFO:tasks.workunit.client.1.smithi137.stdout:3/197: truncate d4/de/f28 985960 0 2024-03-14T13:02:12.297 INFO:tasks.workunit.client.1.smithi137.stdout:3/198: fsync d4/de/d13/f21 0 2024-03-14T13:02:12.298 INFO:tasks.workunit.client.0.smithi111.stdout:3/371: fsync d0/d2d/d35/d50/d5e/f12 0 2024-03-14T13:02:12.298 INFO:tasks.workunit.client.1.smithi137.stdout:5/131: creat d7/d16/d18/d25/f29 x:0 0 0 2024-03-14T13:02:12.299 INFO:tasks.workunit.client.0.smithi111.stdout:1/317: dwrite d20/d21/f2a [0,4194304] 0 2024-03-14T13:02:12.299 INFO:tasks.workunit.client.1.smithi137.stdout:7/195: symlink d3/d5/d23/d2c/l3c 0 2024-03-14T13:02:12.300 INFO:tasks.workunit.client.1.smithi137.stdout:1/300: dread d7/f1d [0,4194304] 0 2024-03-14T13:02:12.300 INFO:tasks.workunit.client.0.smithi111.stdout:3/372: mkdir d0/d91 0 2024-03-14T13:02:12.301 INFO:tasks.workunit.client.1.smithi137.stdout:1/301: dread - d7/d9/f3e zero size 2024-03-14T13:02:12.302 INFO:tasks.workunit.client.1.smithi137.stdout:4/246: rename d4/d9/f16 to d4/d9/d3a/d41/d38/f53 0 2024-03-14T13:02:12.303 INFO:tasks.workunit.client.0.smithi111.stdout:3/373: write d0/f1c [1953713,53995] 0 2024-03-14T13:02:12.307 INFO:tasks.workunit.client.0.smithi111.stdout:1/318: dwrite d20/d25/d31/f3b [0,4194304] 0 2024-03-14T13:02:12.309 INFO:tasks.workunit.client.0.smithi111.stdout:1/319: write d20/d25/d31/f45 [564487,22656] 0 2024-03-14T13:02:12.310 INFO:tasks.workunit.client.0.smithi111.stdout:1/320: dread - d20/d25/f64 zero size 2024-03-14T13:02:12.322 INFO:tasks.workunit.client.0.smithi111.stdout:3/374: dread d0/d2d/d35/d50/d5e/f27 [0,4194304] 0 2024-03-14T13:02:12.323 INFO:tasks.workunit.client.0.smithi111.stdout:3/375: write d0/d30/f3d [3246437,19900] 0 2024-03-14T13:02:12.326 INFO:tasks.workunit.client.0.smithi111.stdout:3/376: getdents d0/d2d/d35/d4c/d82 0 2024-03-14T13:02:12.331 INFO:tasks.workunit.client.0.smithi111.stdout:3/377: link d0/d2d/d35/d50/d5e/f37 d0/d2d/d35/f92 0 2024-03-14T13:02:12.332 INFO:tasks.workunit.client.0.smithi111.stdout:1/321: dwrite d20/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:12.337 INFO:tasks.workunit.client.0.smithi111.stdout:3/378: link d0/d2d/d35/d69/d84/c90 d0/d91/c93 0 2024-03-14T13:02:12.339 INFO:tasks.workunit.client.0.smithi111.stdout:3/379: creat d0/d2d/d35/d69/f94 x:0 0 0 2024-03-14T13:02:12.340 INFO:tasks.workunit.client.0.smithi111.stdout:1/322: getdents d20/d21/d3f/d5a 0 2024-03-14T13:02:12.342 INFO:tasks.workunit.client.0.smithi111.stdout:1/323: stat d20/d21/d28/d5c/l67 0 2024-03-14T13:02:12.343 INFO:tasks.workunit.client.0.smithi111.stdout:3/380: write d0/d2d/d35/d50/d5e/d40/d6f/f79 [776017,2603] 0 2024-03-14T13:02:12.343 INFO:tasks.workunit.client.0.smithi111.stdout:1/324: creat d20/d21/d3f/f6f x:0 0 0 2024-03-14T13:02:12.345 INFO:tasks.workunit.client.0.smithi111.stdout:1/325: write d20/d25/d31/f3b [449150,64241] 0 2024-03-14T13:02:12.347 INFO:tasks.workunit.client.0.smithi111.stdout:3/381: rename d0/d30/f31 to d0/d30/f95 0 2024-03-14T13:02:12.350 INFO:tasks.workunit.client.0.smithi111.stdout:3/382: symlink d0/d91/l96 0 2024-03-14T13:02:12.350 INFO:tasks.workunit.client.0.smithi111.stdout:1/326: link l5 d20/d21/d3f/d5a/l70 0 2024-03-14T13:02:12.351 INFO:tasks.workunit.client.0.smithi111.stdout:3/383: rmdir d0/d2d/d35/d4c 39 2024-03-14T13:02:12.353 INFO:tasks.workunit.client.0.smithi111.stdout:1/327: truncate d20/d21/d3f/d5a/f65 3903354 0 2024-03-14T13:02:12.355 INFO:tasks.workunit.client.0.smithi111.stdout:1/328: chown d20/d25/d31/l36 272999 1 2024-03-14T13:02:12.356 INFO:tasks.workunit.client.0.smithi111.stdout:1/329: mknod d20/d25/c71 0 2024-03-14T13:02:12.357 INFO:tasks.workunit.client.0.smithi111.stdout:3/384: dwrite d0/d2d/d35/d4c/d82/f7e [0,4194304] 0 2024-03-14T13:02:12.358 INFO:tasks.workunit.client.0.smithi111.stdout:4/549: sync 2024-03-14T13:02:12.358 INFO:tasks.workunit.client.0.smithi111.stdout:2/208: sync 2024-03-14T13:02:12.358 INFO:tasks.workunit.client.0.smithi111.stdout:8/460: sync 2024-03-14T13:02:12.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/209: dread - dd/d11/d1a/f3d zero size 2024-03-14T13:02:12.359 INFO:tasks.workunit.client.0.smithi111.stdout:4/550: dread - d1d/d22/d30/f81 zero size 2024-03-14T13:02:12.361 INFO:tasks.workunit.client.0.smithi111.stdout:8/461: creat d3/d1b/f92 x:0 0 0 2024-03-14T13:02:12.363 INFO:tasks.workunit.client.0.smithi111.stdout:2/210: creat dd/f3f x:0 0 0 2024-03-14T13:02:12.365 INFO:tasks.workunit.client.1.smithi137.stdout:8/136: chown d1/dc/l25 3058140 1 2024-03-14T13:02:12.365 INFO:tasks.workunit.client.0.smithi111.stdout:1/330: dread d20/f24 [4194304,4194304] 0 2024-03-14T13:02:12.367 INFO:tasks.workunit.client.0.smithi111.stdout:2/211: unlink dd/d11/f25 0 2024-03-14T13:02:12.367 INFO:tasks.workunit.client.1.smithi137.stdout:5/132: creat d7/d16/d18/f2a x:0 0 0 2024-03-14T13:02:12.367 INFO:tasks.workunit.client.0.smithi111.stdout:1/331: write d20/d25/d31/f45 [104970,60903] 0 2024-03-14T13:02:12.368 INFO:tasks.workunit.client.0.smithi111.stdout:8/462: fsync d3/d49/f53 0 2024-03-14T13:02:12.368 INFO:tasks.workunit.client.0.smithi111.stdout:1/332: stat d20/d40/f54 0 2024-03-14T13:02:12.368 INFO:tasks.workunit.client.1.smithi137.stdout:7/196: symlink d3/d5/d23/d2c/l3d 0 2024-03-14T13:02:12.369 INFO:tasks.workunit.client.0.smithi111.stdout:2/212: write dd/d11/d3b/f3c [607334,72702] 0 2024-03-14T13:02:12.369 INFO:tasks.workunit.client.0.smithi111.stdout:8/463: chown d3/dc/l42 2092432 1 2024-03-14T13:02:12.370 INFO:tasks.workunit.client.1.smithi137.stdout:1/302: creat d7/d9/d19/d40/f61 x:0 0 0 2024-03-14T13:02:12.371 INFO:tasks.workunit.client.0.smithi111.stdout:2/213: chown c0 1962412 1 2024-03-14T13:02:12.371 INFO:tasks.workunit.client.0.smithi111.stdout:8/464: chown d3/dc/d45/f56 3157591 1 2024-03-14T13:02:12.371 INFO:tasks.workunit.client.0.smithi111.stdout:4/551: rmdir d1d/d22/d30/d5e/dbd 0 2024-03-14T13:02:12.371 INFO:tasks.workunit.client.1.smithi137.stdout:4/247: mkdir d4/d10/d3d/d54 0 2024-03-14T13:02:12.372 INFO:tasks.workunit.client.0.smithi111.stdout:1/333: symlink d20/d21/d3f/d59/l72 0 2024-03-14T13:02:12.372 INFO:tasks.workunit.client.0.smithi111.stdout:4/552: fsync d1d/d22/d30/d5e/f7a 0 2024-03-14T13:02:12.372 INFO:tasks.workunit.client.1.smithi137.stdout:8/137: symlink d1/dc/d10/l36 0 2024-03-14T13:02:12.372 INFO:tasks.workunit.client.0.smithi111.stdout:1/334: read - d20/d21/f41 zero size 2024-03-14T13:02:12.373 INFO:tasks.workunit.client.1.smithi137.stdout:5/133: dread d7/fa [0,4194304] 0 2024-03-14T13:02:12.373 INFO:tasks.workunit.client.0.smithi111.stdout:4/553: dread - d1d/d22/d23/fc0 zero size 2024-03-14T13:02:12.374 INFO:tasks.workunit.client.1.smithi137.stdout:7/197: creat d3/d6/d2e/f3e x:0 0 0 2024-03-14T13:02:12.375 INFO:tasks.workunit.client.1.smithi137.stdout:7/198: readlink d3/d5/d23/l29 0 2024-03-14T13:02:12.375 INFO:tasks.workunit.client.1.smithi137.stdout:3/199: creat d4/de/f3c x:0 0 0 2024-03-14T13:02:12.376 INFO:tasks.workunit.client.1.smithi137.stdout:7/199: chown d3/d6/d14/d27/c39 205 1 2024-03-14T13:02:12.376 INFO:tasks.workunit.client.1.smithi137.stdout:4/248: mkdir d4/d9/d3a/d41/d38/d55 0 2024-03-14T13:02:12.377 INFO:tasks.workunit.client.1.smithi137.stdout:7/200: chown d3/d6/d14 21 1 2024-03-14T13:02:12.377 INFO:tasks.workunit.client.0.smithi111.stdout:1/335: symlink d20/d25/d31/l73 0 2024-03-14T13:02:12.377 INFO:tasks.workunit.client.1.smithi137.stdout:8/138: mknod d1/d17/c37 0 2024-03-14T13:02:12.379 INFO:tasks.workunit.client.1.smithi137.stdout:7/201: write d3/d6/d14/f1a [625610,102749] 0 2024-03-14T13:02:12.379 INFO:tasks.workunit.client.1.smithi137.stdout:5/134: fsync d7/fb 0 2024-03-14T13:02:12.379 INFO:tasks.workunit.client.0.smithi111.stdout:2/214: rmdir dd/d11 39 2024-03-14T13:02:12.381 INFO:tasks.workunit.client.1.smithi137.stdout:1/303: creat d7/d31/d54/f62 x:0 0 0 2024-03-14T13:02:12.382 INFO:tasks.workunit.client.0.smithi111.stdout:1/336: creat d20/d21/d3f/d59/f74 x:0 0 0 2024-03-14T13:02:12.382 INFO:tasks.workunit.client.1.smithi137.stdout:3/200: rename d4/df/l38 to d4/de/d16/d1c/l3d 0 2024-03-14T13:02:12.383 INFO:tasks.workunit.client.1.smithi137.stdout:3/201: chown d4/de/d16/l33 2137407 1 2024-03-14T13:02:12.383 INFO:tasks.workunit.client.1.smithi137.stdout:4/249: creat d4/d9/d3a/f56 x:0 0 0 2024-03-14T13:02:12.385 INFO:tasks.workunit.client.1.smithi137.stdout:3/202: write d4/d22/f2e [1441023,121153] 0 2024-03-14T13:02:12.385 INFO:tasks.workunit.client.1.smithi137.stdout:4/250: chown d4/d10/d42/f4e 498 1 2024-03-14T13:02:12.385 INFO:tasks.workunit.client.0.smithi111.stdout:4/554: getdents d1d/d22/d28 0 2024-03-14T13:02:12.386 INFO:tasks.workunit.client.1.smithi137.stdout:3/203: truncate f1 3582280 0 2024-03-14T13:02:12.387 INFO:tasks.workunit.client.1.smithi137.stdout:7/202: mkdir d3/d5/d23/d3f 0 2024-03-14T13:02:12.387 INFO:tasks.workunit.client.0.smithi111.stdout:2/215: symlink dd/d11/d3b/l40 0 2024-03-14T13:02:12.387 INFO:tasks.workunit.client.0.smithi111.stdout:8/465: write d3/d5/fb [4807453,81360] 0 2024-03-14T13:02:12.388 INFO:tasks.workunit.client.1.smithi137.stdout:7/203: chown d3/d6/d14/d27/c36 359 1 2024-03-14T13:02:12.388 INFO:tasks.workunit.client.1.smithi137.stdout:5/135: symlink d7/d16/d18/d25/l2b 0 2024-03-14T13:02:12.389 INFO:tasks.workunit.client.0.smithi111.stdout:2/216: chown dd/d17/f3a 12728651 1 2024-03-14T13:02:12.390 INFO:tasks.workunit.client.1.smithi137.stdout:1/304: readlink d7/l38 0 2024-03-14T13:02:12.390 INFO:tasks.workunit.client.0.smithi111.stdout:4/555: symlink d1d/d22/d30/d5e/lc2 0 2024-03-14T13:02:12.391 INFO:tasks.workunit.client.1.smithi137.stdout:1/305: chown d7/d9/d19/d1f/d22/f45 26923781 1 2024-03-14T13:02:12.392 INFO:tasks.workunit.client.0.smithi111.stdout:1/337: dwrite d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:12.392 INFO:tasks.workunit.client.0.smithi111.stdout:2/217: unlink dd/d11/l26 0 2024-03-14T13:02:12.394 INFO:tasks.workunit.client.1.smithi137.stdout:3/204: mknod d4/de/d16/d1c/c3e 0 2024-03-14T13:02:12.395 INFO:tasks.workunit.client.0.smithi111.stdout:2/218: fsync dd/d11/d1a/f1c 0 2024-03-14T13:02:12.398 INFO:tasks.workunit.client.1.smithi137.stdout:8/139: rename d1/f8 to d1/d17/f38 0 2024-03-14T13:02:12.399 INFO:tasks.workunit.client.1.smithi137.stdout:4/251: dwrite f3 [0,4194304] 0 2024-03-14T13:02:12.400 INFO:tasks.workunit.client.1.smithi137.stdout:1/306: mkdir d7/d9/d19/d1f/d2e/d63 0 2024-03-14T13:02:12.400 INFO:tasks.workunit.client.1.smithi137.stdout:8/140: chown d1/dc/d11/d13/l33 215353 1 2024-03-14T13:02:12.401 INFO:tasks.workunit.client.0.smithi111.stdout:4/556: dwrite d1d/d22/d56/f5f [0,4194304] 0 2024-03-14T13:02:12.402 INFO:tasks.workunit.client.0.smithi111.stdout:8/466: dwrite d3/d1b/f6f [0,4194304] 0 2024-03-14T13:02:12.407 INFO:tasks.workunit.client.1.smithi137.stdout:1/307: creat d7/d4d/f64 x:0 0 0 2024-03-14T13:02:12.408 INFO:tasks.workunit.client.0.smithi111.stdout:1/338: dread d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:12.409 INFO:tasks.workunit.client.1.smithi137.stdout:5/136: creat d7/f2c x:0 0 0 2024-03-14T13:02:12.410 INFO:tasks.workunit.client.1.smithi137.stdout:5/137: write d7/f11 [5569876,107261] 0 2024-03-14T13:02:12.412 INFO:tasks.workunit.client.0.smithi111.stdout:4/557: creat d1d/d22/d30/d5e/d6f/fc3 x:0 0 0 2024-03-14T13:02:12.413 INFO:tasks.workunit.client.0.smithi111.stdout:2/219: dwrite dd/d17/d34/f2e [0,4194304] 0 2024-03-14T13:02:12.414 INFO:tasks.workunit.client.0.smithi111.stdout:1/339: rename d20/d21/d3f/d5a to d20/d25/d75 0 2024-03-14T13:02:12.416 INFO:tasks.workunit.client.1.smithi137.stdout:5/138: truncate d7/d16/d18/f2a 317318 0 2024-03-14T13:02:12.416 INFO:tasks.workunit.client.0.smithi111.stdout:1/340: stat d20/d25/d31/l36 0 2024-03-14T13:02:12.418 INFO:tasks.workunit.client.1.smithi137.stdout:4/252: rename d4/d10/d42/l46 to d4/l57 0 2024-03-14T13:02:12.419 INFO:tasks.workunit.client.0.smithi111.stdout:4/558: getdents d1d/d22/d56 0 2024-03-14T13:02:12.420 INFO:tasks.workunit.client.1.smithi137.stdout:4/253: chown d4/d10/f25 2 1 2024-03-14T13:02:12.420 INFO:tasks.workunit.client.0.smithi111.stdout:1/341: creat d20/d21/d3f/d59/f76 x:0 0 0 2024-03-14T13:02:12.423 INFO:tasks.workunit.client.1.smithi137.stdout:8/141: rename d1/dc/d10/f24 to d1/dc/f39 0 2024-03-14T13:02:12.423 INFO:tasks.workunit.client.0.smithi111.stdout:1/342: creat d20/d25/d75/f77 x:0 0 0 2024-03-14T13:02:12.424 INFO:tasks.workunit.client.1.smithi137.stdout:8/142: dread - d1/dc/d10/f27 zero size 2024-03-14T13:02:12.424 INFO:tasks.workunit.client.1.smithi137.stdout:5/139: fsync d7/f11 0 2024-03-14T13:02:12.426 INFO:tasks.workunit.client.1.smithi137.stdout:4/254: mkdir d4/d10/d3d/d54/d58 0 2024-03-14T13:02:12.426 INFO:tasks.workunit.client.1.smithi137.stdout:5/140: write d7/d16/f1a [3002063,39089] 0 2024-03-14T13:02:12.427 INFO:tasks.workunit.client.0.smithi111.stdout:1/343: rename d20/d21/d28/d5c/l5d to d20/d25/d75/l78 0 2024-03-14T13:02:12.428 INFO:tasks.workunit.client.0.smithi111.stdout:1/344: chown f1a 24693914 1 2024-03-14T13:02:12.428 INFO:tasks.workunit.client.1.smithi137.stdout:8/143: symlink d1/l3a 0 2024-03-14T13:02:12.428 INFO:tasks.workunit.client.0.smithi111.stdout:4/559: dread d1d/d22/d30/f38 [0,4194304] 0 2024-03-14T13:02:12.430 INFO:tasks.workunit.client.1.smithi137.stdout:5/141: truncate d7/f1b 164561 0 2024-03-14T13:02:12.431 INFO:tasks.workunit.client.1.smithi137.stdout:8/144: fdatasync d1/dc/d11/d13/f22 0 2024-03-14T13:02:12.432 INFO:tasks.workunit.client.0.smithi111.stdout:4/560: mkdir d1d/d22/d30/d7c/d59/dc4 0 2024-03-14T13:02:12.432 INFO:tasks.workunit.client.1.smithi137.stdout:0/215: sync 2024-03-14T13:02:12.434 INFO:tasks.workunit.client.1.smithi137.stdout:4/255: creat d4/d9/d3a/d41/d38/d55/f59 x:0 0 0 2024-03-14T13:02:12.435 INFO:tasks.workunit.client.0.smithi111.stdout:8/467: write d3/d1b/d20/d35/d40/d48/f50 [1769058,5104] 0 2024-03-14T13:02:12.435 INFO:tasks.workunit.client.1.smithi137.stdout:5/142: unlink d7/f11 0 2024-03-14T13:02:12.437 INFO:tasks.workunit.client.0.smithi111.stdout:1/345: getdents d20/d21/d3f/d59 0 2024-03-14T13:02:12.437 INFO:tasks.workunit.client.0.smithi111.stdout:8/468: chown d3/d5/c2f 41396 1 2024-03-14T13:02:12.437 INFO:tasks.workunit.client.1.smithi137.stdout:8/145: mknod d1/c3b 0 2024-03-14T13:02:12.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/220: truncate dd/d17/f31 4022107 0 2024-03-14T13:02:12.438 INFO:tasks.workunit.client.0.smithi111.stdout:1/346: stat d20/d25/c49 0 2024-03-14T13:02:12.440 INFO:tasks.workunit.client.0.smithi111.stdout:1/347: rmdir d20/d21/d28 39 2024-03-14T13:02:12.441 INFO:tasks.workunit.client.1.smithi137.stdout:0/216: rmdir d1 39 2024-03-14T13:02:12.441 INFO:tasks.workunit.client.0.smithi111.stdout:1/348: dread - d20/d21/d3f/f6f zero size 2024-03-14T13:02:12.441 INFO:tasks.workunit.client.0.smithi111.stdout:2/221: symlink dd/d17/l41 0 2024-03-14T13:02:12.443 INFO:tasks.workunit.client.1.smithi137.stdout:3/205: unlink d4/de/d16/d1c/l3d 0 2024-03-14T13:02:12.443 INFO:tasks.workunit.client.1.smithi137.stdout:3/206: chown d4/d19 3 1 2024-03-14T13:02:12.444 INFO:tasks.workunit.client.0.smithi111.stdout:1/349: unlink d20/d21/f2a 0 2024-03-14T13:02:12.444 INFO:tasks.workunit.client.1.smithi137.stdout:3/207: chown d4/de/d16/d27 14142 1 2024-03-14T13:02:12.445 INFO:tasks.workunit.client.0.smithi111.stdout:8/469: link d3/d1b/d34/l44 d3/d5/l93 0 2024-03-14T13:02:12.445 INFO:tasks.workunit.client.0.smithi111.stdout:8/470: stat d3/d1b/d34/c57 0 2024-03-14T13:02:12.445 INFO:tasks.workunit.client.1.smithi137.stdout:4/256: mkdir d4/d10/d5a 0 2024-03-14T13:02:12.446 INFO:tasks.workunit.client.0.smithi111.stdout:2/222: fdatasync dd/d17/f31 0 2024-03-14T13:02:12.447 INFO:tasks.workunit.client.0.smithi111.stdout:8/471: creat d3/dc/f94 x:0 0 0 2024-03-14T13:02:12.447 INFO:tasks.workunit.client.0.smithi111.stdout:2/223: write dd/f3f [1040701,122910] 0 2024-03-14T13:02:12.448 INFO:tasks.workunit.client.0.smithi111.stdout:2/224: stat dd/d17/l35 0 2024-03-14T13:02:12.449 INFO:tasks.workunit.client.1.smithi137.stdout:8/146: mkdir d1/dc/d3c 0 2024-03-14T13:02:12.449 INFO:tasks.workunit.client.1.smithi137.stdout:7/204: truncate d3/f18 4413558 0 2024-03-14T13:02:12.450 INFO:tasks.workunit.client.0.smithi111.stdout:8/472: creat d3/d22/d30/f95 x:0 0 0 2024-03-14T13:02:12.450 INFO:tasks.workunit.client.1.smithi137.stdout:7/205: dread - d3/d6/d2e/f3e zero size 2024-03-14T13:02:12.451 INFO:tasks.workunit.client.0.smithi111.stdout:2/225: creat dd/d17/d34/f42 x:0 0 0 2024-03-14T13:02:12.454 INFO:tasks.workunit.client.0.smithi111.stdout:2/226: creat dd/d11/d1a/f43 x:0 0 0 2024-03-14T13:02:12.455 INFO:tasks.workunit.client.1.smithi137.stdout:8/147: symlink d1/dc/d10/l3d 0 2024-03-14T13:02:12.457 INFO:tasks.workunit.client.1.smithi137.stdout:0/217: mkdir d1/d11/d4a 0 2024-03-14T13:02:12.457 INFO:tasks.workunit.client.1.smithi137.stdout:1/308: dwrite d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:12.457 INFO:tasks.workunit.client.0.smithi111.stdout:2/227: mknod dd/c44 0 2024-03-14T13:02:12.458 INFO:tasks.workunit.client.1.smithi137.stdout:0/218: dread - d1/d1f/f3b zero size 2024-03-14T13:02:12.458 INFO:tasks.workunit.client.0.smithi111.stdout:8/473: link d3/d1b/d20/d35/d40/d48/d7e/c84 d3/d1b/d20/d35/d40/d48/d77/c96 0 2024-03-14T13:02:12.459 INFO:tasks.workunit.client.1.smithi137.stdout:7/206: mknod d3/d5/c40 0 2024-03-14T13:02:12.459 INFO:tasks.workunit.client.1.smithi137.stdout:0/219: chown d1/d5/d1b/d42/l34 465841 1 2024-03-14T13:02:12.460 INFO:tasks.workunit.client.0.smithi111.stdout:2/228: mknod dd/c45 0 2024-03-14T13:02:12.460 INFO:tasks.workunit.client.1.smithi137.stdout:0/220: dread - d1/f3e zero size 2024-03-14T13:02:12.463 INFO:tasks.workunit.client.0.smithi111.stdout:8/474: getdents d3/d1b/d20/d35/d40/d48/d7e 0 2024-03-14T13:02:12.466 INFO:tasks.workunit.client.1.smithi137.stdout:3/208: rename d4/de/d13/f29 to d4/de/f3f 0 2024-03-14T13:02:12.466 INFO:tasks.workunit.client.0.smithi111.stdout:2/229: rename l6 to dd/l46 0 2024-03-14T13:02:12.468 INFO:tasks.workunit.client.1.smithi137.stdout:1/309: mknod d7/d9/d19/d40/c65 0 2024-03-14T13:02:12.468 INFO:tasks.workunit.client.0.smithi111.stdout:2/230: mknod dd/c47 0 2024-03-14T13:02:12.468 INFO:tasks.workunit.client.1.smithi137.stdout:4/257: creat d4/d9/f5b x:0 0 0 2024-03-14T13:02:12.470 INFO:tasks.workunit.client.0.smithi111.stdout:8/475: dwrite f0 [4194304,4194304] 0 2024-03-14T13:02:12.471 INFO:tasks.workunit.client.1.smithi137.stdout:0/221: creat d1/d5/d1b/d42/f4b x:0 0 0 2024-03-14T13:02:12.474 INFO:tasks.workunit.client.1.smithi137.stdout:5/143: truncate d7/f1b 1153061 0 2024-03-14T13:02:12.476 INFO:tasks.workunit.client.1.smithi137.stdout:3/209: creat d4/de/d13/f40 x:0 0 0 2024-03-14T13:02:12.477 INFO:tasks.workunit.client.0.smithi111.stdout:2/231: dwrite dd/d17/f31 [0,4194304] 0 2024-03-14T13:02:12.480 INFO:tasks.workunit.client.0.smithi111.stdout:2/232: write dd/d11/d1a/f1b [2129142,51472] 0 2024-03-14T13:02:12.481 INFO:tasks.workunit.client.0.smithi111.stdout:2/233: fsync dd/d11/d1a/f21 0 2024-03-14T13:02:12.482 INFO:tasks.workunit.client.0.smithi111.stdout:2/234: readlink dd/l39 0 2024-03-14T13:02:12.482 INFO:tasks.workunit.client.0.smithi111.stdout:8/476: dwrite d3/d1b/d20/d83/f8f [0,4194304] 0 2024-03-14T13:02:12.486 INFO:tasks.workunit.client.0.smithi111.stdout:8/477: write d3/d1b/d20/d35/d40/f91 [726144,94601] 0 2024-03-14T13:02:12.487 INFO:tasks.workunit.client.1.smithi137.stdout:5/144: mkdir d7/d16/d18/d2d 0 2024-03-14T13:02:12.491 INFO:tasks.workunit.client.1.smithi137.stdout:3/210: rename d4/de/d13/f25 to d4/de/d16/d1c/f41 0 2024-03-14T13:02:12.494 INFO:tasks.workunit.client.0.smithi111.stdout:8/478: dread d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:12.494 INFO:tasks.workunit.client.0.smithi111.stdout:2/235: dwrite dd/d11/d1a/f43 [0,4194304] 0 2024-03-14T13:02:12.494 INFO:tasks.workunit.client.1.smithi137.stdout:8/148: getdents d1 0 2024-03-14T13:02:12.494 INFO:tasks.workunit.client.1.smithi137.stdout:3/211: truncate d4/d19/f2f 956546 0 2024-03-14T13:02:12.494 INFO:tasks.workunit.client.1.smithi137.stdout:5/145: mknod d7/d16/d18/d25/d28/c2e 0 2024-03-14T13:02:12.496 INFO:tasks.workunit.client.1.smithi137.stdout:3/212: readlink d4/l8 0 2024-03-14T13:02:12.497 INFO:tasks.workunit.client.0.smithi111.stdout:2/236: unlink dd/c44 0 2024-03-14T13:02:12.500 INFO:tasks.workunit.client.1.smithi137.stdout:1/310: dwrite d7/d4d/f56 [4194304,4194304] 0 2024-03-14T13:02:12.501 INFO:tasks.workunit.client.0.smithi111.stdout:2/237: rename dd/d11/d3b to dd/d11/d48 0 2024-03-14T13:02:12.504 INFO:tasks.workunit.client.1.smithi137.stdout:1/311: write d7/d9/d19/d40/f5b [7875,68190] 0 2024-03-14T13:02:12.505 INFO:tasks.workunit.client.0.smithi111.stdout:2/238: dread dd/d11/f2c [0,4194304] 0 2024-03-14T13:02:12.509 INFO:tasks.workunit.client.0.smithi111.stdout:2/239: mkdir dd/d11/d48/d49 0 2024-03-14T13:02:12.511 INFO:tasks.workunit.client.0.smithi111.stdout:8/479: dread d3/d22/d30/f47 [0,4194304] 0 2024-03-14T13:02:12.513 INFO:tasks.workunit.client.0.smithi111.stdout:8/480: write d3/d1b/f75 [2691243,68339] 0 2024-03-14T13:02:12.515 INFO:tasks.workunit.client.0.smithi111.stdout:8/481: creat d3/d22/d30/f97 x:0 0 0 2024-03-14T13:02:12.515 INFO:tasks.workunit.client.1.smithi137.stdout:7/207: dwrite d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:12.516 INFO:tasks.workunit.client.0.smithi111.stdout:2/240: dwrite dd/d17/d34/f2e [0,4194304] 0 2024-03-14T13:02:12.517 INFO:tasks.workunit.client.1.smithi137.stdout:1/312: link d7/d31/l35 d7/d9/d19/d40/d50/l66 0 2024-03-14T13:02:12.518 INFO:tasks.workunit.client.1.smithi137.stdout:8/149: write d1/f31 [985163,44855] 0 2024-03-14T13:02:12.519 INFO:tasks.workunit.client.1.smithi137.stdout:7/208: stat d3/d6/c38 0 2024-03-14T13:02:12.524 INFO:tasks.workunit.client.1.smithi137.stdout:5/146: write d7/fa [304795,118090] 0 2024-03-14T13:02:12.525 INFO:tasks.workunit.client.0.smithi111.stdout:2/241: dwrite dd/f10 [4194304,4194304] 0 2024-03-14T13:02:12.527 INFO:tasks.workunit.client.1.smithi137.stdout:3/213: dread d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:12.528 INFO:tasks.workunit.client.1.smithi137.stdout:5/147: creat d7/d16/d18/d2d/f2f x:0 0 0 2024-03-14T13:02:12.529 INFO:tasks.workunit.client.1.smithi137.stdout:8/150: truncate d1/fd 1255310 0 2024-03-14T13:02:12.532 INFO:tasks.workunit.client.0.smithi111.stdout:2/242: dwrite dd/d11/d48/f3c [0,4194304] 0 2024-03-14T13:02:12.534 INFO:tasks.workunit.client.1.smithi137.stdout:5/148: creat d7/f30 x:0 0 0 2024-03-14T13:02:12.535 INFO:tasks.workunit.client.1.smithi137.stdout:7/209: dwrite d3/d5/f37 [0,4194304] 0 2024-03-14T13:02:12.535 INFO:tasks.workunit.client.0.smithi111.stdout:2/243: symlink dd/l4a 0 2024-03-14T13:02:12.539 INFO:tasks.workunit.client.0.smithi111.stdout:2/244: link dd/d11/f33 dd/f4b 0 2024-03-14T13:02:12.543 INFO:tasks.workunit.client.0.smithi111.stdout:2/245: creat dd/f4c x:0 0 0 2024-03-14T13:02:12.544 INFO:tasks.workunit.client.1.smithi137.stdout:1/313: dread d7/d9/d19/d1f/d2e/f3d [0,4194304] 0 2024-03-14T13:02:12.544 INFO:tasks.workunit.client.0.smithi111.stdout:2/246: fdatasync dd/d17/d34/f36 0 2024-03-14T13:02:12.545 INFO:tasks.workunit.client.0.smithi111.stdout:2/247: fdatasync dd/d11/d1a/f43 0 2024-03-14T13:02:12.547 INFO:tasks.workunit.client.0.smithi111.stdout:2/248: write dd/d11/d1a/f38 [860832,23752] 0 2024-03-14T13:02:12.547 INFO:tasks.workunit.client.0.smithi111.stdout:8/482: truncate d3/d1b/d34/f3c 3253260 0 2024-03-14T13:02:12.548 INFO:tasks.workunit.client.1.smithi137.stdout:1/314: truncate f6 4968465 0 2024-03-14T13:02:12.549 INFO:tasks.workunit.client.1.smithi137.stdout:3/214: dwrite f1 [0,4194304] 0 2024-03-14T13:02:12.549 INFO:tasks.workunit.client.0.smithi111.stdout:2/249: write dd/d17/f3a [603336,110016] 0 2024-03-14T13:02:12.550 INFO:tasks.workunit.client.1.smithi137.stdout:1/315: read - d7/d9/d19/d40/f61 zero size 2024-03-14T13:02:12.550 INFO:tasks.workunit.client.1.smithi137.stdout:3/215: chown d4/f37 28906018 1 2024-03-14T13:02:12.552 INFO:tasks.workunit.client.0.smithi111.stdout:2/250: getdents dd/d11/d48/d49 0 2024-03-14T13:02:12.552 INFO:tasks.workunit.client.0.smithi111.stdout:8/483: dread f0 [4194304,4194304] 0 2024-03-14T13:02:12.552 INFO:tasks.workunit.client.1.smithi137.stdout:3/216: write d4/d19/f2f [923086,21943] 0 2024-03-14T13:02:12.557 INFO:tasks.workunit.client.0.smithi111.stdout:2/251: dread dd/d17/d34/f2e [0,4194304] 0 2024-03-14T13:02:12.562 INFO:tasks.workunit.client.0.smithi111.stdout:2/252: rename dd/cf to dd/d11/d48/d49/c4d 0 2024-03-14T13:02:12.562 INFO:tasks.workunit.client.1.smithi137.stdout:1/316: symlink d7/d9/d19/d40/d42/l67 0 2024-03-14T13:02:12.563 INFO:tasks.workunit.client.1.smithi137.stdout:7/210: link d3/d5/d23/d2c/l34 d3/d6/d14/d22/l41 0 2024-03-14T13:02:12.563 INFO:tasks.workunit.client.0.smithi111.stdout:8/484: dwrite d3/d22/d30/f97 [0,4194304] 0 2024-03-14T13:02:12.563 INFO:tasks.workunit.client.1.smithi137.stdout:1/317: read - d7/d9/d19/d40/f61 zero size 2024-03-14T13:02:12.564 INFO:tasks.workunit.client.1.smithi137.stdout:7/211: fsync f1 0 2024-03-14T13:02:12.565 INFO:tasks.workunit.client.1.smithi137.stdout:3/217: dwrite f1 [0,4194304] 0 2024-03-14T13:02:12.565 INFO:tasks.workunit.client.0.smithi111.stdout:2/253: symlink dd/d11/d48/d49/l4e 0 2024-03-14T13:02:12.567 INFO:tasks.workunit.client.1.smithi137.stdout:2/184: sync 2024-03-14T13:02:12.568 INFO:tasks.workunit.client.0.smithi111.stdout:2/254: mkdir dd/d17/d34/d4f 0 2024-03-14T13:02:12.569 INFO:tasks.workunit.client.1.smithi137.stdout:7/212: read d3/d6/f15 [842514,8565] 0 2024-03-14T13:02:12.570 INFO:tasks.workunit.client.1.smithi137.stdout:3/218: link d4/d19/f23 d4/df/d34/f42 0 2024-03-14T13:02:12.570 INFO:tasks.workunit.client.1.smithi137.stdout:2/185: write d2/d7/db/de/d22/d26/f27 [5918692,17729] 0 2024-03-14T13:02:12.573 INFO:tasks.workunit.client.1.smithi137.stdout:1/318: read d7/fb [2772133,72163] 0 2024-03-14T13:02:12.574 INFO:tasks.workunit.client.0.smithi111.stdout:8/485: dwrite d3/dc/f6c [0,4194304] 0 2024-03-14T13:02:12.577 INFO:tasks.workunit.client.1.smithi137.stdout:1/319: truncate d7/d31/d54/f62 120566 0 2024-03-14T13:02:12.578 INFO:tasks.workunit.client.0.smithi111.stdout:8/486: truncate d3/dc/d45/f56 233879 0 2024-03-14T13:02:12.581 INFO:tasks.workunit.client.1.smithi137.stdout:2/186: mkdir d2/d7/d2e/d3e 0 2024-03-14T13:02:12.581 INFO:tasks.workunit.client.1.smithi137.stdout:8/151: dread d1/dc/d10/f35 [0,4194304] 0 2024-03-14T13:02:12.581 INFO:tasks.workunit.client.0.smithi111.stdout:8/487: getdents d3/dc/d31 0 2024-03-14T13:02:12.582 INFO:tasks.workunit.client.1.smithi137.stdout:1/320: rmdir d7/d9/d19/d1f 39 2024-03-14T13:02:12.583 INFO:tasks.workunit.client.0.smithi111.stdout:8/488: symlink d3/d22/d4a/l98 0 2024-03-14T13:02:12.583 INFO:tasks.workunit.client.0.smithi111.stdout:8/489: fsync d3/d22/d30/f74 0 2024-03-14T13:02:12.585 INFO:tasks.workunit.client.0.smithi111.stdout:8/490: creat d3/dc/f99 x:0 0 0 2024-03-14T13:02:12.586 INFO:tasks.workunit.client.1.smithi137.stdout:3/219: dwrite d4/de/f2a [0,4194304] 0 2024-03-14T13:02:12.589 INFO:tasks.workunit.client.1.smithi137.stdout:1/321: mknod d7/d31/d54/c68 0 2024-03-14T13:02:12.593 INFO:tasks.workunit.client.1.smithi137.stdout:8/152: read d1/dc/d11/d13/f16 [2737456,48254] 0 2024-03-14T13:02:12.594 INFO:tasks.workunit.client.0.smithi111.stdout:9/331: sync 2024-03-14T13:02:12.597 INFO:tasks.workunit.client.1.smithi137.stdout:5/149: dwrite d7/f1b [0,4194304] 0 2024-03-14T13:02:12.597 INFO:tasks.workunit.client.1.smithi137.stdout:8/153: chown d1/dc/f39 0 1 2024-03-14T13:02:12.598 INFO:tasks.workunit.client.0.smithi111.stdout:8/491: dwrite d3/d1b/d20/d83/f8f [0,4194304] 0 2024-03-14T13:02:12.601 INFO:tasks.workunit.client.0.smithi111.stdout:8/492: symlink d3/d1b/d20/d35/d40/l9a 0 2024-03-14T13:02:12.601 INFO:tasks.workunit.client.1.smithi137.stdout:8/154: creat d1/dc/d11/d13/f3e x:0 0 0 2024-03-14T13:02:12.603 INFO:tasks.workunit.client.1.smithi137.stdout:2/187: dwrite d2/d7/db/de/f1b [0,4194304] 0 2024-03-14T13:02:12.603 INFO:tasks.workunit.client.0.smithi111.stdout:9/332: link d5/d6/d2a/l53 d5/d6/d8/d5b/l69 0 2024-03-14T13:02:12.606 INFO:tasks.workunit.client.1.smithi137.stdout:5/150: symlink d7/l31 0 2024-03-14T13:02:12.609 INFO:tasks.workunit.client.1.smithi137.stdout:7/213: dwrite d3/fe [0,4194304] 0 2024-03-14T13:02:12.614 INFO:tasks.workunit.client.1.smithi137.stdout:2/188: read d2/f5 [4768232,103140] 0 2024-03-14T13:02:12.618 INFO:tasks.workunit.client.1.smithi137.stdout:2/189: rmdir d2/d7/d2e 39 2024-03-14T13:02:12.619 INFO:tasks.workunit.client.1.smithi137.stdout:5/151: dwrite d7/d16/d18/f2a [0,4194304] 0 2024-03-14T13:02:12.624 INFO:tasks.workunit.client.1.smithi137.stdout:2/190: rename d2/d7/db/de/d22/d26/f31 to d2/d7/db/de/f3f 0 2024-03-14T13:02:12.624 INFO:tasks.workunit.client.1.smithi137.stdout:5/152: symlink d7/d1d/l32 0 2024-03-14T13:02:12.626 INFO:tasks.workunit.client.1.smithi137.stdout:5/153: readlink d7/l12 0 2024-03-14T13:02:12.626 INFO:tasks.workunit.client.1.smithi137.stdout:5/154: readlink d7/l31 0 2024-03-14T13:02:12.626 INFO:tasks.workunit.client.1.smithi137.stdout:2/191: write d2/f12 [3462498,82652] 0 2024-03-14T13:02:12.629 INFO:tasks.workunit.client.1.smithi137.stdout:5/155: mkdir d7/d33 0 2024-03-14T13:02:12.629 INFO:tasks.workunit.client.1.smithi137.stdout:2/192: creat d2/d7/db/d2f/f40 x:0 0 0 2024-03-14T13:02:12.631 INFO:tasks.workunit.client.1.smithi137.stdout:5/156: rename d7/d16/d18/d25/f29 to d7/d16/d18/f34 0 2024-03-14T13:02:12.634 INFO:tasks.workunit.client.1.smithi137.stdout:5/157: mknod d7/d33/c35 0 2024-03-14T13:02:12.636 INFO:tasks.workunit.client.1.smithi137.stdout:2/193: dwrite d2/d7/db/d2f/f37 [0,4194304] 0 2024-03-14T13:02:12.638 INFO:tasks.workunit.client.1.smithi137.stdout:5/158: mkdir d7/d36 0 2024-03-14T13:02:12.641 INFO:tasks.workunit.client.1.smithi137.stdout:5/159: symlink d7/d16/d18/l37 0 2024-03-14T13:02:12.642 INFO:tasks.workunit.client.1.smithi137.stdout:5/160: chown d7/d16/d18/d2d/f2f 0 1 2024-03-14T13:02:12.642 INFO:tasks.workunit.client.1.smithi137.stdout:2/194: dread d2/d7/db/de/f1b [0,4194304] 0 2024-03-14T13:02:12.643 INFO:tasks.workunit.client.1.smithi137.stdout:5/161: creat d7/d16/f38 x:0 0 0 2024-03-14T13:02:12.644 INFO:tasks.workunit.client.1.smithi137.stdout:2/195: dread - d2/d7/d2e/f3d zero size 2024-03-14T13:02:12.645 INFO:tasks.workunit.client.1.smithi137.stdout:2/196: stat d2/d7/db/f38 0 2024-03-14T13:02:12.646 INFO:tasks.workunit.client.1.smithi137.stdout:2/197: dread - d2/d7/db/f38 zero size 2024-03-14T13:02:12.648 INFO:tasks.workunit.client.1.smithi137.stdout:5/162: getdents d7/d16/d18/d25/d28 0 2024-03-14T13:02:12.648 INFO:tasks.workunit.client.1.smithi137.stdout:2/198: truncate d2/d7/db/d2f/f40 790145 0 2024-03-14T13:02:12.651 INFO:tasks.workunit.client.0.smithi111.stdout:8/493: symlink d3/d1b/d20/d35/d40/d48/d7e/l9b 0 2024-03-14T13:02:12.653 INFO:tasks.workunit.client.0.smithi111.stdout:8/494: chown d3/d22/d30/f74 0 1 2024-03-14T13:02:12.654 INFO:tasks.workunit.client.1.smithi137.stdout:2/199: getdents d2/d7/db/de/d1d/d2d 0 2024-03-14T13:02:12.655 INFO:tasks.workunit.client.0.smithi111.stdout:9/333: creat d5/f6a x:0 0 0 2024-03-14T13:02:12.656 INFO:tasks.workunit.client.1.smithi137.stdout:2/200: fdatasync d2/d7/f10 0 2024-03-14T13:02:12.658 INFO:tasks.workunit.client.1.smithi137.stdout:7/214: truncate d3/d5/f19 442640 0 2024-03-14T13:02:12.659 INFO:tasks.workunit.client.0.smithi111.stdout:9/334: mknod d5/d6/d2a/c6b 0 2024-03-14T13:02:12.660 INFO:tasks.workunit.client.1.smithi137.stdout:3/220: dwrite d4/de/f28 [0,4194304] 0 2024-03-14T13:02:12.662 INFO:tasks.workunit.client.0.smithi111.stdout:8/495: dwrite d3/f14 [0,4194304] 0 2024-03-14T13:02:12.663 INFO:tasks.workunit.client.0.smithi111.stdout:9/335: rename d5/d42/f60 to d5/d42/d52/f6c 0 2024-03-14T13:02:12.665 INFO:tasks.workunit.client.0.smithi111.stdout:8/496: creat d3/d5/f9c x:0 0 0 2024-03-14T13:02:12.667 INFO:tasks.workunit.client.0.smithi111.stdout:9/336: unlink d5/d6/c2b 0 2024-03-14T13:02:12.667 INFO:tasks.workunit.client.1.smithi137.stdout:2/201: dread d2/f1e [0,4194304] 0 2024-03-14T13:02:12.669 INFO:tasks.workunit.client.1.smithi137.stdout:2/202: stat d2/f4 0 2024-03-14T13:02:12.669 INFO:tasks.workunit.client.0.smithi111.stdout:9/337: write d5/f65 [524633,57963] 0 2024-03-14T13:02:12.670 INFO:tasks.workunit.client.1.smithi137.stdout:2/203: rmdir d2/d7/db/de/d22 39 2024-03-14T13:02:12.671 INFO:tasks.workunit.client.0.smithi111.stdout:9/338: creat d5/d42/d52/f6d x:0 0 0 2024-03-14T13:02:12.673 INFO:tasks.workunit.client.0.smithi111.stdout:8/497: dwrite d3/f14 [0,4194304] 0 2024-03-14T13:02:12.676 INFO:tasks.workunit.client.0.smithi111.stdout:9/339: creat d5/d6/d8/d5b/f6e x:0 0 0 2024-03-14T13:02:12.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/340: write d5/f4f [147650,89757] 0 2024-03-14T13:02:12.679 INFO:tasks.workunit.client.1.smithi137.stdout:7/215: dread d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:12.680 INFO:tasks.workunit.client.0.smithi111.stdout:8/498: dwrite d3/d1b/f92 [0,4194304] 0 2024-03-14T13:02:12.682 INFO:tasks.workunit.client.0.smithi111.stdout:8/499: mkdir d3/dc/d31/d9d 0 2024-03-14T13:02:12.683 INFO:tasks.workunit.client.1.smithi137.stdout:7/216: mknod d3/d5/d23/d3f/c42 0 2024-03-14T13:02:12.684 INFO:tasks.workunit.client.1.smithi137.stdout:7/217: dread - d3/d6/d2e/f3e zero size 2024-03-14T13:02:12.684 INFO:tasks.workunit.client.0.smithi111.stdout:8/500: write d3/d22/f88 [879140,69845] 0 2024-03-14T13:02:12.685 INFO:tasks.workunit.client.1.smithi137.stdout:7/218: dread - d3/d6/d2e/f3e zero size 2024-03-14T13:02:12.685 INFO:tasks.workunit.client.1.smithi137.stdout:5/163: rmdir d7/d16/d18/d25 39 2024-03-14T13:02:12.687 INFO:tasks.workunit.client.0.smithi111.stdout:8/501: rename d3/d22/f3f to d3/d1b/d20/f9e 0 2024-03-14T13:02:12.688 INFO:tasks.workunit.client.1.smithi137.stdout:7/219: stat d3/d5/la 0 2024-03-14T13:02:12.692 INFO:tasks.workunit.client.1.smithi137.stdout:7/220: truncate d3/f1e 3086712 0 2024-03-14T13:02:12.695 INFO:tasks.workunit.client.1.smithi137.stdout:7/221: rename d3/d5/f3b to d3/d6/d14/d22/f43 0 2024-03-14T13:02:12.697 INFO:tasks.workunit.client.1.smithi137.stdout:2/204: dread d2/f4 [0,4194304] 0 2024-03-14T13:02:12.699 INFO:tasks.workunit.client.1.smithi137.stdout:7/222: dread d3/f1c [0,4194304] 0 2024-03-14T13:02:12.701 INFO:tasks.workunit.client.1.smithi137.stdout:2/205: unlink d2/d7/db/de/d22/d26/f33 0 2024-03-14T13:02:12.702 INFO:tasks.workunit.client.1.smithi137.stdout:7/223: symlink d3/d5/d23/l44 0 2024-03-14T13:02:12.703 INFO:tasks.workunit.client.1.smithi137.stdout:2/206: mkdir d2/d7/db/de/d1d/d41 0 2024-03-14T13:02:12.707 INFO:tasks.workunit.client.1.smithi137.stdout:2/207: dread d2/d7/db/f16 [0,4194304] 0 2024-03-14T13:02:12.708 INFO:tasks.workunit.client.1.smithi137.stdout:2/208: write d2/f12 [3425768,130634] 0 2024-03-14T13:02:12.709 INFO:tasks.workunit.client.0.smithi111.stdout:9/341: dread d5/d42/d52/f6c [0,4194304] 0 2024-03-14T13:02:12.710 INFO:tasks.workunit.client.1.smithi137.stdout:7/224: rename d3/f1e to d3/d6/f45 0 2024-03-14T13:02:12.711 INFO:tasks.workunit.client.0.smithi111.stdout:9/342: creat d5/d6/d2a/d5a/f6f x:0 0 0 2024-03-14T13:02:12.712 INFO:tasks.workunit.client.1.smithi137.stdout:7/225: truncate d3/d6/f9 4578636 0 2024-03-14T13:02:12.712 INFO:tasks.workunit.client.0.smithi111.stdout:9/343: fdatasync d5/f50 0 2024-03-14T13:02:12.714 INFO:tasks.workunit.client.0.smithi111.stdout:9/344: mknod d5/c70 0 2024-03-14T13:02:12.714 INFO:tasks.workunit.client.1.smithi137.stdout:2/209: rename d2/l19 to d2/d34/d3c/l42 0 2024-03-14T13:02:12.715 INFO:tasks.workunit.client.1.smithi137.stdout:7/226: unlink d3/d5/c24 0 2024-03-14T13:02:12.719 INFO:tasks.workunit.client.1.smithi137.stdout:2/210: mknod d2/d7/db/de/d1d/d3a/c43 0 2024-03-14T13:02:12.723 INFO:tasks.workunit.client.1.smithi137.stdout:2/211: dread d2/d7/db/d2f/f37 [0,4194304] 0 2024-03-14T13:02:12.730 INFO:tasks.workunit.client.1.smithi137.stdout:7/227: dwrite d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:12.730 INFO:tasks.workunit.client.1.smithi137.stdout:2/212: dwrite d2/f5 [0,4194304] 0 2024-03-14T13:02:12.732 INFO:tasks.workunit.client.1.smithi137.stdout:7/228: creat d3/d5/f46 x:0 0 0 2024-03-14T13:02:12.742 INFO:tasks.workunit.client.1.smithi137.stdout:7/229: dwrite d3/f32 [0,4194304] 0 2024-03-14T13:02:12.749 INFO:tasks.workunit.client.1.smithi137.stdout:2/213: truncate d2/d7/f13 3195447 0 2024-03-14T13:02:12.752 INFO:tasks.workunit.client.1.smithi137.stdout:2/214: mkdir d2/d7/d2e/d3e/d44 0 2024-03-14T13:02:12.754 INFO:tasks.workunit.client.1.smithi137.stdout:2/215: creat d2/d7/db/d2f/f45 x:0 0 0 2024-03-14T13:02:12.756 INFO:tasks.workunit.client.1.smithi137.stdout:2/216: mknod d2/d34/c46 0 2024-03-14T13:02:12.758 INFO:tasks.workunit.client.1.smithi137.stdout:2/217: creat d2/d7/d2e/f47 x:0 0 0 2024-03-14T13:02:12.762 INFO:tasks.workunit.client.1.smithi137.stdout:2/218: fdatasync d2/d7/db/ff 0 2024-03-14T13:02:12.767 INFO:tasks.workunit.client.1.smithi137.stdout:2/219: rename d2/d7/db/d2f/f37 to d2/d7/db/de/d1d/f48 0 2024-03-14T13:02:12.769 INFO:tasks.workunit.client.1.smithi137.stdout:2/220: write d2/d7/db/de/f1b [3758605,8778] 0 2024-03-14T13:02:12.773 INFO:tasks.workunit.client.0.smithi111.stdout:9/345: rmdir d5/d6 39 2024-03-14T13:02:12.775 INFO:tasks.workunit.client.0.smithi111.stdout:9/346: mkdir d5/d42/d71 0 2024-03-14T13:02:12.780 INFO:tasks.workunit.client.0.smithi111.stdout:8/502: dwrite d3/dc/f6c [4194304,4194304] 0 2024-03-14T13:02:12.781 INFO:tasks.workunit.client.0.smithi111.stdout:8/503: truncate d3/d5/f1e 4724741 0 2024-03-14T13:02:12.784 INFO:tasks.workunit.client.0.smithi111.stdout:3/385: sync 2024-03-14T13:02:12.784 INFO:tasks.workunit.client.0.smithi111.stdout:6/279: sync 2024-03-14T13:02:12.784 INFO:tasks.workunit.client.0.smithi111.stdout:5/373: sync 2024-03-14T13:02:12.784 INFO:tasks.workunit.client.0.smithi111.stdout:0/261: sync 2024-03-14T13:02:12.784 INFO:tasks.workunit.client.0.smithi111.stdout:7/262: sync 2024-03-14T13:02:12.786 INFO:tasks.workunit.client.0.smithi111.stdout:5/374: chown d2/d1f/d36/l4d 1466289 1 2024-03-14T13:02:12.787 INFO:tasks.workunit.client.0.smithi111.stdout:8/504: creat d3/d5/f9f x:0 0 0 2024-03-14T13:02:12.789 INFO:tasks.workunit.client.0.smithi111.stdout:5/375: chown d2/d16/d49/cf 3125 1 2024-03-14T13:02:12.789 INFO:tasks.workunit.client.0.smithi111.stdout:7/263: chown d0/d1/d21/d3a/c47 2434909 1 2024-03-14T13:02:12.790 INFO:tasks.workunit.client.0.smithi111.stdout:5/376: chown d2/d1f/f24 336269 1 2024-03-14T13:02:12.791 INFO:tasks.workunit.client.0.smithi111.stdout:7/264: dread - d0/d1/d4a/f4b zero size 2024-03-14T13:02:12.792 INFO:tasks.workunit.client.0.smithi111.stdout:6/280: rename d0/f6 to d0/d17/d20/f6b 0 2024-03-14T13:02:12.793 INFO:tasks.workunit.client.0.smithi111.stdout:6/281: readlink d0/l27 0 2024-03-14T13:02:12.793 INFO:tasks.workunit.client.0.smithi111.stdout:7/265: write d0/f7 [3082109,23042] 0 2024-03-14T13:02:12.795 INFO:tasks.workunit.client.0.smithi111.stdout:5/377: write d2/d16/d49/d33/f7b [661696,81843] 0 2024-03-14T13:02:12.796 INFO:tasks.workunit.client.0.smithi111.stdout:9/347: truncate d5/d6/d8/f30 2427000 0 2024-03-14T13:02:12.798 INFO:tasks.workunit.client.0.smithi111.stdout:9/348: write d5/f50 [3749611,99098] 0 2024-03-14T13:02:12.798 INFO:tasks.workunit.client.0.smithi111.stdout:6/282: truncate d0/d17/d21/d69/f39 989474 0 2024-03-14T13:02:12.798 INFO:tasks.workunit.client.1.smithi137.stdout:9/275: sync 2024-03-14T13:02:12.798 INFO:tasks.workunit.client.1.smithi137.stdout:6/231: sync 2024-03-14T13:02:12.802 INFO:tasks.workunit.client.0.smithi111.stdout:7/266: symlink d0/d11/l4f 0 2024-03-14T13:02:12.803 INFO:tasks.workunit.client.1.smithi137.stdout:9/276: mknod d6/df/d19/c63 0 2024-03-14T13:02:12.803 INFO:tasks.workunit.client.1.smithi137.stdout:9/277: dread - d6/df/f5e zero size 2024-03-14T13:02:12.805 INFO:tasks.workunit.client.1.smithi137.stdout:6/232: rename d2/d26 to d2/d54 0 2024-03-14T13:02:12.806 INFO:tasks.workunit.client.0.smithi111.stdout:6/283: dread d0/d17/f55 [0,4194304] 0 2024-03-14T13:02:12.806 INFO:tasks.workunit.client.1.smithi137.stdout:9/278: rmdir d6/df/d19/d1f 39 2024-03-14T13:02:12.806 INFO:tasks.workunit.client.0.smithi111.stdout:5/378: rename d2/d16/d49/c6b to d2/d45/c7e 0 2024-03-14T13:02:12.807 INFO:tasks.workunit.client.1.smithi137.stdout:2/221: getdents d2/d7/db/d2f 0 2024-03-14T13:02:12.808 INFO:tasks.workunit.client.0.smithi111.stdout:5/379: stat d2/d16/d49/d44/d72/f3e 0 2024-03-14T13:02:12.809 INFO:tasks.workunit.client.0.smithi111.stdout:8/505: truncate d3/d22/d30/f97 2745365 0 2024-03-14T13:02:12.809 INFO:tasks.workunit.client.0.smithi111.stdout:3/386: link d0/fd d0/d2d/d35/d50/d5e/f97 0 2024-03-14T13:02:12.810 INFO:tasks.workunit.client.1.smithi137.stdout:6/233: rename d2/d6/d8 to d2/d54/d55 0 2024-03-14T13:02:12.812 INFO:tasks.workunit.client.0.smithi111.stdout:9/349: rename d5/d6/l37 to d5/d6/d14/l72 0 2024-03-14T13:02:12.812 INFO:tasks.workunit.client.1.smithi137.stdout:2/222: write d2/d7/db/de/d1d/f48 [878644,25172] 0 2024-03-14T13:02:12.813 INFO:tasks.workunit.client.1.smithi137.stdout:6/234: mknod d2/d54/d55/d28/c56 0 2024-03-14T13:02:12.814 INFO:tasks.workunit.client.0.smithi111.stdout:0/262: getdents d6/d9/de/d32 0 2024-03-14T13:02:12.815 INFO:tasks.workunit.client.1.smithi137.stdout:2/223: rmdir d2/d7/db/de 39 2024-03-14T13:02:12.816 INFO:tasks.workunit.client.0.smithi111.stdout:5/380: symlink d2/d16/d49/d44/d72/l7f 0 2024-03-14T13:02:12.816 INFO:tasks.workunit.client.1.smithi137.stdout:6/235: mkdir d2/d54/d55/d28/d39/d3c/d43/d57 0 2024-03-14T13:02:12.817 INFO:tasks.workunit.client.0.smithi111.stdout:6/284: mknod d0/d17/d20/d4a/d68/c6c 0 2024-03-14T13:02:12.817 INFO:tasks.workunit.client.1.smithi137.stdout:9/279: creat d6/df/f64 x:0 0 0 2024-03-14T13:02:12.817 INFO:tasks.workunit.client.1.smithi137.stdout:9/280: stat d6/c4e 0 2024-03-14T13:02:12.818 INFO:tasks.workunit.client.0.smithi111.stdout:8/506: creat d3/d1b/d20/d35/fa0 x:0 0 0 2024-03-14T13:02:12.819 INFO:tasks.workunit.client.1.smithi137.stdout:9/281: write d6/df/d19/f37 [61918,101885] 0 2024-03-14T13:02:12.821 INFO:tasks.workunit.client.1.smithi137.stdout:9/282: dread - d6/df/f64 zero size 2024-03-14T13:02:12.821 INFO:tasks.workunit.client.0.smithi111.stdout:9/350: dwrite d5/f4f [0,4194304] 0 2024-03-14T13:02:12.821 INFO:tasks.workunit.client.0.smithi111.stdout:3/387: rename d0/d2d/d35/d50/d5e/d40/d55 to d0/d2d/d35/d50/d6c/d98 0 2024-03-14T13:02:12.822 INFO:tasks.workunit.client.0.smithi111.stdout:0/263: mknod d6/d45/c57 0 2024-03-14T13:02:12.823 INFO:tasks.workunit.client.1.smithi137.stdout:2/224: dread d2/d7/db/de/d1d/f48 [0,4194304] 0 2024-03-14T13:02:12.824 INFO:tasks.workunit.client.1.smithi137.stdout:9/283: creat d6/df/d19/d33/f65 x:0 0 0 2024-03-14T13:02:12.824 INFO:tasks.workunit.client.1.smithi137.stdout:6/236: dread d2/d54/d55/d33/f44 [0,4194304] 0 2024-03-14T13:02:12.824 INFO:tasks.workunit.client.0.smithi111.stdout:0/264: write d6/d9/d15/f1b [1796474,63125] 0 2024-03-14T13:02:12.825 INFO:tasks.workunit.client.1.smithi137.stdout:2/225: chown d2/d7/db/de/d1d/d3a 1496 1 2024-03-14T13:02:12.827 INFO:tasks.workunit.client.0.smithi111.stdout:3/388: creat d0/d2d/d35/d69/d84/f99 x:0 0 0 2024-03-14T13:02:12.829 INFO:tasks.workunit.client.1.smithi137.stdout:9/284: truncate d6/df/d19/d3a/d40/f60 404042 0 2024-03-14T13:02:12.829 INFO:tasks.workunit.client.1.smithi137.stdout:2/226: symlink d2/d34/l49 0 2024-03-14T13:02:12.829 INFO:tasks.workunit.client.0.smithi111.stdout:8/507: dread d3/d1b/f6f [0,4194304] 0 2024-03-14T13:02:12.829 INFO:tasks.workunit.client.0.smithi111.stdout:9/351: rename d5/c3c to d5/d42/d71/c73 0 2024-03-14T13:02:12.830 INFO:tasks.workunit.client.1.smithi137.stdout:6/237: mkdir d2/d54/d55/d28/d39/d3c/d43/d57/d58 0 2024-03-14T13:02:12.831 INFO:tasks.workunit.client.1.smithi137.stdout:6/238: write d2/f17 [1335131,44949] 0 2024-03-14T13:02:12.833 INFO:tasks.workunit.client.1.smithi137.stdout:9/285: creat d6/df/d19/d1f/f66 x:0 0 0 2024-03-14T13:02:12.834 INFO:tasks.workunit.client.0.smithi111.stdout:3/389: unlink d0/d2d/d35/d50/d5e/d40/c8b 0 2024-03-14T13:02:12.835 INFO:tasks.workunit.client.1.smithi137.stdout:2/227: creat d2/d7/db/de/f4a x:0 0 0 2024-03-14T13:02:12.835 INFO:tasks.workunit.client.1.smithi137.stdout:9/286: chown d6/df/f10 298194488 1 2024-03-14T13:02:12.835 INFO:tasks.workunit.client.0.smithi111.stdout:8/508: unlink d3/d5/f9f 0 2024-03-14T13:02:12.837 INFO:tasks.workunit.client.0.smithi111.stdout:9/352: mkdir d5/d6/d8/d74 0 2024-03-14T13:02:12.838 INFO:tasks.workunit.client.1.smithi137.stdout:9/287: creat d6/df/d19/d3a/f67 x:0 0 0 2024-03-14T13:02:12.838 INFO:tasks.workunit.client.1.smithi137.stdout:9/288: truncate d6/d26/f5f 496654 0 2024-03-14T13:02:12.839 INFO:tasks.workunit.client.1.smithi137.stdout:2/228: creat d2/f4b x:0 0 0 2024-03-14T13:02:12.840 INFO:tasks.workunit.client.1.smithi137.stdout:2/229: chown d2/d7/db/de/d1d/d3a 22 1 2024-03-14T13:02:12.840 INFO:tasks.workunit.client.0.smithi111.stdout:8/509: read d3/d1b/d20/d35/d40/d48/f68 [102103,129596] 0 2024-03-14T13:02:12.840 INFO:tasks.workunit.client.0.smithi111.stdout:3/390: symlink d0/d30/l9a 0 2024-03-14T13:02:12.841 INFO:tasks.workunit.client.1.smithi137.stdout:6/239: rename d2/d54/d55/d28 to d2/d6/de/d45/d59 0 2024-03-14T13:02:12.841 INFO:tasks.workunit.client.1.smithi137.stdout:2/230: write d2/d7/db/d2f/f40 [616474,101292] 0 2024-03-14T13:02:12.842 INFO:tasks.workunit.client.1.smithi137.stdout:9/289: write d6/f57 [3317576,35339] 0 2024-03-14T13:02:12.845 INFO:tasks.workunit.client.1.smithi137.stdout:9/290: rmdir d6/d26 39 2024-03-14T13:02:12.846 INFO:tasks.workunit.client.0.smithi111.stdout:3/391: read d0/fc [94771,83053] 0 2024-03-14T13:02:12.846 INFO:tasks.workunit.client.0.smithi111.stdout:0/265: dread d6/f1a [0,4194304] 0 2024-03-14T13:02:12.847 INFO:tasks.workunit.client.0.smithi111.stdout:5/381: truncate d2/f6 1029302 0 2024-03-14T13:02:12.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/240: creat d2/f5a x:0 0 0 2024-03-14T13:02:12.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/241: chown d2/l5 928 1 2024-03-14T13:02:12.848 INFO:tasks.workunit.client.1.smithi137.stdout:9/291: rmdir d6/d26 39 2024-03-14T13:02:12.849 INFO:tasks.workunit.client.0.smithi111.stdout:9/353: dwrite d5/d6/f2f [0,4194304] 0 2024-03-14T13:02:12.850 INFO:tasks.workunit.client.0.smithi111.stdout:8/510: dread d3/dc/d45/f56 [0,4194304] 0 2024-03-14T13:02:12.851 INFO:tasks.workunit.client.1.smithi137.stdout:6/242: write d2/d6/de/d45/d59/f37 [555503,55713] 0 2024-03-14T13:02:12.851 INFO:tasks.workunit.client.0.smithi111.stdout:0/266: creat d6/d9/f58 x:0 0 0 2024-03-14T13:02:12.852 INFO:tasks.workunit.client.0.smithi111.stdout:5/382: symlink d2/d13/l80 0 2024-03-14T13:02:12.856 INFO:tasks.workunit.client.0.smithi111.stdout:8/511: mkdir d3/d22/da1 0 2024-03-14T13:02:12.856 INFO:tasks.workunit.client.1.smithi137.stdout:6/243: write d2/d6/d1d/f4e [850353,63234] 0 2024-03-14T13:02:12.857 INFO:tasks.workunit.client.0.smithi111.stdout:8/512: chown d3/d1b/d34/c57 1922950316 1 2024-03-14T13:02:12.857 INFO:tasks.workunit.client.1.smithi137.stdout:6/244: fsync d2/d6/fc 0 2024-03-14T13:02:12.860 INFO:tasks.workunit.client.0.smithi111.stdout:5/383: rename d2/d16/d49/d44/d72/l4e to d2/d16/d49/d44/d72/l81 0 2024-03-14T13:02:12.861 INFO:tasks.workunit.client.1.smithi137.stdout:9/292: dread d6/df/d19/f45 [0,4194304] 0 2024-03-14T13:02:12.862 INFO:tasks.workunit.client.1.smithi137.stdout:2/231: dwrite d2/f1e [0,4194304] 0 2024-03-14T13:02:12.863 INFO:tasks.workunit.client.0.smithi111.stdout:9/354: creat d5/d6/d2a/f75 x:0 0 0 2024-03-14T13:02:12.865 INFO:tasks.workunit.client.0.smithi111.stdout:5/384: rmdir d2/d45 39 2024-03-14T13:02:12.868 INFO:tasks.workunit.client.1.smithi137.stdout:6/245: dwrite d2/f5a [0,4194304] 0 2024-03-14T13:02:12.870 INFO:tasks.workunit.client.1.smithi137.stdout:9/293: dread d6/df/d19/d1f/f3b [0,4194304] 0 2024-03-14T13:02:12.870 INFO:tasks.workunit.client.0.smithi111.stdout:9/355: creat d5/d6/d2a/d5a/f76 x:0 0 0 2024-03-14T13:02:12.872 INFO:tasks.workunit.client.1.smithi137.stdout:6/246: creat d2/d6/de/d45/f5b x:0 0 0 2024-03-14T13:02:12.873 INFO:tasks.workunit.client.0.smithi111.stdout:5/385: read - d2/d45/f66 zero size 2024-03-14T13:02:12.873 INFO:tasks.workunit.client.0.smithi111.stdout:9/356: readlink d5/d6/d8/l44 0 2024-03-14T13:02:12.875 INFO:tasks.workunit.client.1.smithi137.stdout:9/294: rmdir d6/df/d19/d33 39 2024-03-14T13:02:12.876 INFO:tasks.workunit.client.0.smithi111.stdout:5/386: stat d2/d16/d49/d44/d72/f3e 0 2024-03-14T13:02:12.876 INFO:tasks.workunit.client.1.smithi137.stdout:9/295: mknod d6/d56/c68 0 2024-03-14T13:02:12.876 INFO:tasks.workunit.client.1.smithi137.stdout:6/247: creat d2/d6/f5c x:0 0 0 2024-03-14T13:02:12.878 INFO:tasks.workunit.client.0.smithi111.stdout:8/513: dwrite d3/d1b/d20/f9e [0,4194304] 0 2024-03-14T13:02:12.878 INFO:tasks.workunit.client.0.smithi111.stdout:3/392: dread d0/f23 [0,4194304] 0 2024-03-14T13:02:12.879 INFO:tasks.workunit.client.0.smithi111.stdout:0/267: fdatasync d6/d9/de/f23 0 2024-03-14T13:02:12.879 INFO:tasks.workunit.client.1.smithi137.stdout:9/296: dread d6/df/d19/d1f/f3b [0,4194304] 0 2024-03-14T13:02:12.883 INFO:tasks.workunit.client.0.smithi111.stdout:9/357: creat d5/d6/d8/d74/f77 x:0 0 0 2024-03-14T13:02:12.883 INFO:tasks.workunit.client.0.smithi111.stdout:8/514: rmdir d3/d22/d4a 39 2024-03-14T13:02:12.885 INFO:tasks.workunit.client.1.smithi137.stdout:9/297: dwrite d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:12.886 INFO:tasks.workunit.client.1.smithi137.stdout:9/298: chown d6/d53 165493 1 2024-03-14T13:02:12.887 INFO:tasks.workunit.client.0.smithi111.stdout:3/393: symlink d0/d2d/d35/d50/d5e/l9b 0 2024-03-14T13:02:12.888 INFO:tasks.workunit.client.0.smithi111.stdout:5/387: write d2/d45/f34 [3061429,115713] 0 2024-03-14T13:02:12.889 INFO:tasks.workunit.client.0.smithi111.stdout:3/394: chown d0/d2d/d35/d4c/d82/f7e 6 1 2024-03-14T13:02:12.889 INFO:tasks.workunit.client.0.smithi111.stdout:5/388: dread - d2/d16/d49/d4c/f7d zero size 2024-03-14T13:02:12.890 INFO:tasks.workunit.client.0.smithi111.stdout:8/515: unlink d3/d5/f19 0 2024-03-14T13:02:12.894 INFO:tasks.workunit.client.0.smithi111.stdout:0/268: getdents d6/d21 0 2024-03-14T13:02:12.896 INFO:tasks.workunit.client.1.smithi137.stdout:9/299: truncate d6/df/f10 566342 0 2024-03-14T13:02:12.898 INFO:tasks.workunit.client.0.smithi111.stdout:9/358: truncate d5/d6/f2f 523721 0 2024-03-14T13:02:12.900 INFO:tasks.workunit.client.1.smithi137.stdout:9/300: rename d6/df/f5d to d6/f69 0 2024-03-14T13:02:12.901 INFO:tasks.workunit.client.0.smithi111.stdout:8/516: creat d3/d1b/d66/fa2 x:0 0 0 2024-03-14T13:02:12.901 INFO:tasks.workunit.client.0.smithi111.stdout:5/389: dwrite d2/d1f/f28 [0,4194304] 0 2024-03-14T13:02:12.901 INFO:tasks.workunit.client.0.smithi111.stdout:0/269: write d6/d9/f12 [5490671,111828] 0 2024-03-14T13:02:12.903 INFO:tasks.workunit.client.0.smithi111.stdout:9/359: symlink d5/d42/l78 0 2024-03-14T13:02:12.904 INFO:tasks.workunit.client.0.smithi111.stdout:9/360: chown c1 2206 1 2024-03-14T13:02:12.905 INFO:tasks.workunit.client.1.smithi137.stdout:9/301: creat d6/df/f6a x:0 0 0 2024-03-14T13:02:12.906 INFO:tasks.workunit.client.0.smithi111.stdout:5/390: rename d2/d13/f4b to d2/d45/f82 0 2024-03-14T13:02:12.908 INFO:tasks.workunit.client.0.smithi111.stdout:5/391: dread d2/d16/d49/f5d [0,4194304] 0 2024-03-14T13:02:12.908 INFO:tasks.workunit.client.0.smithi111.stdout:9/361: creat d5/d42/f79 x:0 0 0 2024-03-14T13:02:12.909 INFO:tasks.workunit.client.1.smithi137.stdout:9/302: rename d6/df/d19/l22 to d6/df/d19/l6b 0 2024-03-14T13:02:12.910 INFO:tasks.workunit.client.0.smithi111.stdout:0/270: symlink d6/d9/de/d3a/d46/d4a/l59 0 2024-03-14T13:02:12.911 INFO:tasks.workunit.client.1.smithi137.stdout:9/303: creat d6/df/d19/d1f/f6c x:0 0 0 2024-03-14T13:02:12.913 INFO:tasks.workunit.client.1.smithi137.stdout:9/304: rmdir d6/df/d19/d3a/d40 39 2024-03-14T13:02:12.915 INFO:tasks.workunit.client.0.smithi111.stdout:5/392: rename d2/d16/d49/f1e to d2/d16/d49/f83 0 2024-03-14T13:02:12.916 INFO:tasks.workunit.client.1.smithi137.stdout:9/305: creat d6/f6d x:0 0 0 2024-03-14T13:02:12.920 INFO:tasks.workunit.client.0.smithi111.stdout:8/517: truncate d3/d1b/d20/f9e 601250 0 2024-03-14T13:02:12.922 INFO:tasks.workunit.client.0.smithi111.stdout:9/362: dwrite d5/d6/fc [4194304,4194304] 0 2024-03-14T13:02:12.922 INFO:tasks.workunit.client.1.smithi137.stdout:9/306: dwrite d6/df/d19/d1f/f35 [0,4194304] 0 2024-03-14T13:02:12.924 INFO:tasks.workunit.client.0.smithi111.stdout:3/395: dwrite d0/d2d/d35/d69/d84/f87 [0,4194304] 0 2024-03-14T13:02:12.926 INFO:tasks.workunit.client.1.smithi137.stdout:9/307: chown d6/df/d19/d33/f46 50506895 1 2024-03-14T13:02:12.926 INFO:tasks.workunit.client.0.smithi111.stdout:0/271: dwrite d6/d9/de/d32/f33 [0,4194304] 0 2024-03-14T13:02:12.928 INFO:tasks.workunit.client.1.smithi137.stdout:9/308: creat d6/df/d19/d33/f6e x:0 0 0 2024-03-14T13:02:12.931 INFO:tasks.workunit.client.0.smithi111.stdout:5/393: write d2/d16/d49/f11 [2440365,47836] 0 2024-03-14T13:02:12.931 INFO:tasks.workunit.client.1.smithi137.stdout:9/309: mknod d6/d26/c6f 0 2024-03-14T13:02:12.931 INFO:tasks.workunit.client.0.smithi111.stdout:9/363: mkdir d5/d6/d14/d7a 0 2024-03-14T13:02:12.933 INFO:tasks.workunit.client.0.smithi111.stdout:0/272: mkdir d6/d5a 0 2024-03-14T13:02:12.937 INFO:tasks.workunit.client.0.smithi111.stdout:5/394: symlink d2/d74/l84 0 2024-03-14T13:02:12.939 INFO:tasks.workunit.client.0.smithi111.stdout:9/364: truncate d5/d6/d2a/d2d/f66 16640 0 2024-03-14T13:02:12.941 INFO:tasks.workunit.client.0.smithi111.stdout:8/518: dwrite d3/f3e [4194304,4194304] 0 2024-03-14T13:02:12.944 INFO:tasks.workunit.client.0.smithi111.stdout:8/519: write d3/dc/f99 [983486,67959] 0 2024-03-14T13:02:12.946 INFO:tasks.workunit.client.0.smithi111.stdout:0/273: dwrite d6/fd [0,4194304] 0 2024-03-14T13:02:12.947 INFO:tasks.workunit.client.0.smithi111.stdout:9/365: unlink d5/f6a 0 2024-03-14T13:02:12.949 INFO:tasks.workunit.client.0.smithi111.stdout:5/395: rename d2/d16/d49/d44/d4a/l59 to d2/l85 0 2024-03-14T13:02:12.951 INFO:tasks.workunit.client.0.smithi111.stdout:9/366: write d5/d6/d2a/d5a/f76 [69230,27962] 0 2024-03-14T13:02:12.956 INFO:tasks.workunit.client.0.smithi111.stdout:9/367: dread d5/d6/fc [4194304,4194304] 0 2024-03-14T13:02:12.957 INFO:tasks.workunit.client.0.smithi111.stdout:0/274: fsync d6/d9/de/d32/f33 0 2024-03-14T13:02:12.958 INFO:tasks.workunit.client.0.smithi111.stdout:0/275: write d6/d9/f58 [693922,60362] 0 2024-03-14T13:02:12.961 INFO:tasks.workunit.client.0.smithi111.stdout:9/368: rename d5/d6/d8/c11 to d5/d42/c7b 0 2024-03-14T13:02:12.961 INFO:tasks.workunit.client.1.smithi137.stdout:0/222: sync 2024-03-14T13:02:12.961 INFO:tasks.workunit.client.1.smithi137.stdout:4/258: sync 2024-03-14T13:02:12.963 INFO:tasks.workunit.client.0.smithi111.stdout:0/276: write d6/d9/de/f30 [479166,66994] 0 2024-03-14T13:02:12.968 INFO:tasks.workunit.client.0.smithi111.stdout:9/369: read d5/fa [3578769,27936] 0 2024-03-14T13:02:12.968 INFO:tasks.workunit.client.0.smithi111.stdout:5/396: dwrite d2/d16/d49/f5d [0,4194304] 0 2024-03-14T13:02:12.968 INFO:tasks.workunit.client.0.smithi111.stdout:0/277: rename d6/d3d/d43 to d6/d9/d5b 0 2024-03-14T13:02:12.973 INFO:tasks.workunit.client.0.smithi111.stdout:5/397: mknod d2/d74/c86 0 2024-03-14T13:02:12.978 INFO:tasks.workunit.client.0.smithi111.stdout:5/398: chown d2/d16/d49/d33/l5f 0 1 2024-03-14T13:02:12.980 INFO:tasks.workunit.client.0.smithi111.stdout:9/370: dwrite d5/d42/f79 [0,4194304] 0 2024-03-14T13:02:12.986 INFO:tasks.workunit.client.0.smithi111.stdout:0/278: dwrite d6/d9/de/f3b [0,4194304] 0 2024-03-14T13:02:12.986 INFO:tasks.workunit.client.0.smithi111.stdout:9/371: mkdir d5/d6/d8/d5b/d7c 0 2024-03-14T13:02:12.989 INFO:tasks.workunit.client.0.smithi111.stdout:9/372: rename c1 to d5/d6/d2a/d2d/c7d 0 2024-03-14T13:02:12.992 INFO:tasks.workunit.client.0.smithi111.stdout:5/399: write d2/f7 [4608555,55277] 0 2024-03-14T13:02:12.996 INFO:tasks.workunit.client.0.smithi111.stdout:5/400: fsync d2/d13/f2f 0 2024-03-14T13:02:12.998 INFO:tasks.workunit.client.0.smithi111.stdout:0/279: dwrite d6/d9/de/f36 [0,4194304] 0 2024-03-14T13:02:12.998 INFO:tasks.workunit.client.1.smithi137.stdout:9/310: creat d6/df/f70 x:0 0 0 2024-03-14T13:02:12.999 INFO:tasks.workunit.client.1.smithi137.stdout:0/223: creat d1/d44/f4c x:0 0 0 2024-03-14T13:02:13.000 INFO:tasks.workunit.client.0.smithi111.stdout:9/373: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:13.000 INFO:tasks.workunit.client.0.smithi111.stdout:5/401: unlink d2/d16/d49/l6f 0 2024-03-14T13:02:13.000 INFO:tasks.workunit.client.1.smithi137.stdout:0/224: read - d1/d1f/f2f zero size 2024-03-14T13:02:13.001 INFO:tasks.workunit.client.1.smithi137.stdout:9/311: chown d6/df/d19/d33/l39 2047888 1 2024-03-14T13:02:13.003 INFO:tasks.workunit.client.1.smithi137.stdout:4/259: dwrite d4/f3c [0,4194304] 0 2024-03-14T13:02:13.004 INFO:tasks.workunit.client.0.smithi111.stdout:9/374: readlink d5/d6/l62 0 2024-03-14T13:02:13.004 INFO:tasks.workunit.client.0.smithi111.stdout:9/375: chown d5/d6/d8/f12 0 1 2024-03-14T13:02:13.007 INFO:tasks.workunit.client.0.smithi111.stdout:9/376: dread d5/d6/d2a/d5a/f76 [0,4194304] 0 2024-03-14T13:02:13.007 INFO:tasks.workunit.client.1.smithi137.stdout:9/312: write d6/df/d19/d1f/f24 [1802126,98908] 0 2024-03-14T13:02:13.008 INFO:tasks.workunit.client.1.smithi137.stdout:0/225: symlink d1/d5/d1b/d1e/d25/l4d 0 2024-03-14T13:02:13.009 INFO:tasks.workunit.client.0.smithi111.stdout:5/402: getdents d2/d16/d49/d33 0 2024-03-14T13:02:13.010 INFO:tasks.workunit.client.1.smithi137.stdout:9/313: chown d6/df/f20 67034356 1 2024-03-14T13:02:13.011 INFO:tasks.workunit.client.0.smithi111.stdout:5/403: truncate d2/d16/d49/d4c/f55 780659 0 2024-03-14T13:02:13.012 INFO:tasks.workunit.client.1.smithi137.stdout:9/314: write d6/df/d19/d33/f46 [3645988,100809] 0 2024-03-14T13:02:13.012 INFO:tasks.workunit.client.0.smithi111.stdout:5/404: truncate d2/d16/d49/d44/d4a/f62 1260394 0 2024-03-14T13:02:13.017 INFO:tasks.workunit.client.0.smithi111.stdout:5/405: fdatasync d2/d16/d49/d44/d4a/f63 0 2024-03-14T13:02:13.017 INFO:tasks.workunit.client.0.smithi111.stdout:9/377: creat d5/d6/f7e x:0 0 0 2024-03-14T13:02:13.017 INFO:tasks.workunit.client.1.smithi137.stdout:9/315: mkdir d6/df/d71 0 2024-03-14T13:02:13.018 INFO:tasks.workunit.client.0.smithi111.stdout:5/406: symlink d2/d16/d49/d44/l87 0 2024-03-14T13:02:13.020 INFO:tasks.workunit.client.0.smithi111.stdout:9/378: mknod d5/d6/d2a/d2d/c7f 0 2024-03-14T13:02:13.020 INFO:tasks.workunit.client.0.smithi111.stdout:5/407: write d2/d1f/f5e [672199,117548] 0 2024-03-14T13:02:13.020 INFO:tasks.workunit.client.1.smithi137.stdout:0/226: dwrite d1/d11/f19 [0,4194304] 0 2024-03-14T13:02:13.020 INFO:tasks.workunit.client.1.smithi137.stdout:9/316: rename d6/d26/f5f to d6/df/d71/f72 0 2024-03-14T13:02:13.021 INFO:tasks.workunit.client.0.smithi111.stdout:9/379: chown d5/fe 28365 1 2024-03-14T13:02:13.024 INFO:tasks.workunit.client.0.smithi111.stdout:5/408: mkdir d2/d16/d49/d4c/d88 0 2024-03-14T13:02:13.024 INFO:tasks.workunit.client.1.smithi137.stdout:0/227: truncate d1/d11/f19 5151737 0 2024-03-14T13:02:13.025 INFO:tasks.workunit.client.1.smithi137.stdout:0/228: mknod d1/d11/c4e 0 2024-03-14T13:02:13.027 INFO:tasks.workunit.client.0.smithi111.stdout:5/409: mknod d2/d45/d79/c89 0 2024-03-14T13:02:13.028 INFO:tasks.workunit.client.1.smithi137.stdout:9/317: link d6/df/d19/d3a/d40/f4b d6/d56/f73 0 2024-03-14T13:02:13.028 INFO:tasks.workunit.client.1.smithi137.stdout:9/318: stat d6/c9 0 2024-03-14T13:02:13.028 INFO:tasks.workunit.client.1.smithi137.stdout:0/229: creat d1/d5/d1b/d42/f4f x:0 0 0 2024-03-14T13:02:13.029 INFO:tasks.workunit.client.0.smithi111.stdout:9/380: dread d5/d42/f79 [0,4194304] 0 2024-03-14T13:02:13.029 INFO:tasks.workunit.client.1.smithi137.stdout:9/319: truncate d6/f1d 1317683 0 2024-03-14T13:02:13.030 INFO:tasks.workunit.client.0.smithi111.stdout:0/280: dread d6/fd [4194304,4194304] 0 2024-03-14T13:02:13.031 INFO:tasks.workunit.client.1.smithi137.stdout:9/320: creat d6/d53/f74 x:0 0 0 2024-03-14T13:02:13.031 INFO:tasks.workunit.client.1.smithi137.stdout:0/230: read d1/d5/f2a [4098083,98387] 0 2024-03-14T13:02:13.034 INFO:tasks.workunit.client.1.smithi137.stdout:0/231: creat d1/d11/f50 x:0 0 0 2024-03-14T13:02:13.034 INFO:tasks.workunit.client.0.smithi111.stdout:0/281: unlink d6/fb 0 2024-03-14T13:02:13.035 INFO:tasks.workunit.client.1.smithi137.stdout:9/321: creat d6/df/d19/d33/f75 x:0 0 0 2024-03-14T13:02:13.036 INFO:tasks.workunit.client.0.smithi111.stdout:5/410: getdents d2 0 2024-03-14T13:02:13.037 INFO:tasks.workunit.client.1.smithi137.stdout:9/322: rmdir d6/d26 39 2024-03-14T13:02:13.039 INFO:tasks.workunit.client.0.smithi111.stdout:0/282: write d6/d9/de/f18 [3574929,66941] 0 2024-03-14T13:02:13.039 INFO:tasks.workunit.client.0.smithi111.stdout:9/381: dwrite d5/d6/d8/f3a [0,4194304] 0 2024-03-14T13:02:13.041 INFO:tasks.workunit.client.1.smithi137.stdout:0/232: link d1/d5/f37 d1/f51 0 2024-03-14T13:02:13.042 INFO:tasks.workunit.client.0.smithi111.stdout:5/411: creat d2/d16/d49/d33/d68/f8a x:0 0 0 2024-03-14T13:02:13.042 INFO:tasks.workunit.client.0.smithi111.stdout:0/283: rmdir d6/d9/de/d32 39 2024-03-14T13:02:13.043 INFO:tasks.workunit.client.0.smithi111.stdout:5/412: write d2/d45/f5c [1503189,87844] 0 2024-03-14T13:02:13.044 INFO:tasks.workunit.client.1.smithi137.stdout:4/260: dwrite f2 [0,4194304] 0 2024-03-14T13:02:13.044 INFO:tasks.workunit.client.1.smithi137.stdout:9/323: creat d6/df/d19/d3a/f76 x:0 0 0 2024-03-14T13:02:13.045 INFO:tasks.workunit.client.0.smithi111.stdout:9/382: rename d5/d42/d52/f6d to d5/d6/d8/d1a/f80 0 2024-03-14T13:02:13.046 INFO:tasks.workunit.client.0.smithi111.stdout:0/284: write d6/d9/de/f26 [4194802,80252] 0 2024-03-14T13:02:13.047 INFO:tasks.workunit.client.1.smithi137.stdout:0/233: truncate d1/d5/d1b/d42/f4b 364886 0 2024-03-14T13:02:13.048 INFO:tasks.workunit.client.0.smithi111.stdout:5/413: mkdir d2/d16/d49/d7c/d8b 0 2024-03-14T13:02:13.048 INFO:tasks.workunit.client.0.smithi111.stdout:0/285: chown d6/d9/de/f18 11641688 1 2024-03-14T13:02:13.049 INFO:tasks.workunit.client.0.smithi111.stdout:9/383: creat d5/d6/d14/f81 x:0 0 0 2024-03-14T13:02:13.050 INFO:tasks.workunit.client.0.smithi111.stdout:0/286: chown d6/d9/de/c34 87 1 2024-03-14T13:02:13.052 INFO:tasks.workunit.client.1.smithi137.stdout:4/261: mkdir d4/d27/d5c 0 2024-03-14T13:02:13.052 INFO:tasks.workunit.client.0.smithi111.stdout:0/287: write d6/d21/f42 [1163986,111750] 0 2024-03-14T13:02:13.052 INFO:tasks.workunit.client.0.smithi111.stdout:9/384: symlink d5/d6/d8/d5b/d7c/l82 0 2024-03-14T13:02:13.052 INFO:tasks.workunit.client.0.smithi111.stdout:5/414: read d2/d16/d49/f83 [3463165,109598] 0 2024-03-14T13:02:13.053 INFO:tasks.workunit.client.0.smithi111.stdout:5/415: chown d2 1084685 1 2024-03-14T13:02:13.054 INFO:tasks.workunit.client.1.smithi137.stdout:9/324: symlink d6/df/d19/d3a/l77 0 2024-03-14T13:02:13.055 INFO:tasks.workunit.client.0.smithi111.stdout:9/385: write d5/d6/d14/f54 [983911,72906] 0 2024-03-14T13:02:13.057 INFO:tasks.workunit.client.0.smithi111.stdout:0/288: getdents d6/d9/de/d3a 0 2024-03-14T13:02:13.057 INFO:tasks.workunit.client.1.smithi137.stdout:9/325: dread - d6/df/d19/d33/f61 zero size 2024-03-14T13:02:13.059 INFO:tasks.workunit.client.0.smithi111.stdout:5/416: rename d2/d16/d49/d44/d4a/f62 to d2/d16/d49/d44/d4a/f8c 0 2024-03-14T13:02:13.060 INFO:tasks.workunit.client.1.smithi137.stdout:9/326: chown d6/d56/f73 42577556 1 2024-03-14T13:02:13.060 INFO:tasks.workunit.client.1.smithi137.stdout:0/234: dwrite d1/d5/d1b/d42/f4f [0,4194304] 0 2024-03-14T13:02:13.060 INFO:tasks.workunit.client.0.smithi111.stdout:5/417: stat d2/d16/d49/d44/l67 0 2024-03-14T13:02:13.062 INFO:tasks.workunit.client.1.smithi137.stdout:9/327: write d6/f2b [5643154,64620] 0 2024-03-14T13:02:13.063 INFO:tasks.workunit.client.0.smithi111.stdout:0/289: dread d6/d21/f42 [0,4194304] 0 2024-03-14T13:02:13.064 INFO:tasks.workunit.client.1.smithi137.stdout:4/262: dwrite d4/d10/d42/f52 [0,4194304] 0 2024-03-14T13:02:13.065 INFO:tasks.workunit.client.0.smithi111.stdout:5/418: rename d2/d74 to d2/d45/d76/d8d 0 2024-03-14T13:02:13.066 INFO:tasks.workunit.client.1.smithi137.stdout:1/322: sync 2024-03-14T13:02:13.066 INFO:tasks.workunit.client.1.smithi137.stdout:3/221: sync 2024-03-14T13:02:13.066 INFO:tasks.workunit.client.1.smithi137.stdout:8/155: sync 2024-03-14T13:02:13.067 INFO:tasks.workunit.client.1.smithi137.stdout:9/328: fsync d6/d53/f74 0 2024-03-14T13:02:13.067 INFO:tasks.workunit.client.1.smithi137.stdout:3/222: readlink d4/l8 0 2024-03-14T13:02:13.069 INFO:tasks.workunit.client.1.smithi137.stdout:8/156: read - d1/dc/d11/d13/f26 zero size 2024-03-14T13:02:13.069 INFO:tasks.workunit.client.1.smithi137.stdout:3/223: chown d4/de/d13/f21 96647 1 2024-03-14T13:02:13.070 INFO:tasks.workunit.client.1.smithi137.stdout:8/157: chown d1/dc 21 1 2024-03-14T13:02:13.072 INFO:tasks.workunit.client.0.smithi111.stdout:0/290: creat d6/d9/de/d3a/d46/d4a/f5c x:0 0 0 2024-03-14T13:02:13.073 INFO:tasks.workunit.client.1.smithi137.stdout:8/158: chown d1/d17/c1b 28 1 2024-03-14T13:02:13.073 INFO:tasks.workunit.client.0.smithi111.stdout:0/291: write d6/d9/de/f53 [261555,116045] 0 2024-03-14T13:02:13.074 INFO:tasks.workunit.client.1.smithi137.stdout:8/159: fsync d1/dc/d10/f1a 0 2024-03-14T13:02:13.074 INFO:tasks.workunit.client.1.smithi137.stdout:9/329: mkdir d6/df/d19/d78 0 2024-03-14T13:02:13.076 INFO:tasks.workunit.client.0.smithi111.stdout:0/292: creat d6/d9/de/d3a/f5d x:0 0 0 2024-03-14T13:02:13.077 INFO:tasks.workunit.client.1.smithi137.stdout:9/330: write d6/df/d19/d33/f61 [562712,42655] 0 2024-03-14T13:02:13.077 INFO:tasks.workunit.client.1.smithi137.stdout:3/224: creat d4/df/d35/f43 x:0 0 0 2024-03-14T13:02:13.078 INFO:tasks.workunit.client.0.smithi111.stdout:0/293: write d6/d9/de/f23 [1087185,4392] 0 2024-03-14T13:02:13.079 INFO:tasks.workunit.client.1.smithi137.stdout:8/160: symlink d1/d17/l3f 0 2024-03-14T13:02:13.079 INFO:tasks.workunit.client.1.smithi137.stdout:1/323: creat d7/d9/d19/d1f/f69 x:0 0 0 2024-03-14T13:02:13.080 INFO:tasks.workunit.client.0.smithi111.stdout:0/294: write d6/d9/de/f27 [1150205,87866] 0 2024-03-14T13:02:13.081 INFO:tasks.workunit.client.1.smithi137.stdout:0/235: dwrite d1/d11/f14 [0,4194304] 0 2024-03-14T13:02:13.084 INFO:tasks.workunit.client.1.smithi137.stdout:8/161: creat d1/d17/d34/f40 x:0 0 0 2024-03-14T13:02:13.084 INFO:tasks.workunit.client.1.smithi137.stdout:1/324: mknod d7/d9/d19/d1f/d26/c6a 0 2024-03-14T13:02:13.088 INFO:tasks.workunit.client.1.smithi137.stdout:0/236: mknod d1/d11/c52 0 2024-03-14T13:02:13.088 INFO:tasks.workunit.client.1.smithi137.stdout:4/263: truncate d4/d10/d42/f52 283485 0 2024-03-14T13:02:13.088 INFO:tasks.workunit.client.1.smithi137.stdout:1/325: write d7/d29/f2b [736435,53216] 0 2024-03-14T13:02:13.089 INFO:tasks.workunit.client.1.smithi137.stdout:8/162: symlink d1/dc/d10/l41 0 2024-03-14T13:02:13.090 INFO:tasks.workunit.client.0.smithi111.stdout:0/295: dwrite d6/d9/de/f53 [0,4194304] 0 2024-03-14T13:02:13.091 INFO:tasks.workunit.client.1.smithi137.stdout:4/264: write d4/f3c [4350334,69813] 0 2024-03-14T13:02:13.091 INFO:tasks.workunit.client.1.smithi137.stdout:0/237: truncate d1/f46 1028491 0 2024-03-14T13:02:13.092 INFO:tasks.workunit.client.0.smithi111.stdout:5/419: dwrite d2/d45/f82 [0,4194304] 0 2024-03-14T13:02:13.092 INFO:tasks.workunit.client.1.smithi137.stdout:1/326: mkdir d7/d4d/d6b 0 2024-03-14T13:02:13.092 INFO:tasks.workunit.client.0.smithi111.stdout:0/296: readlink d6/d9/de/d32/l2b 0 2024-03-14T13:02:13.096 INFO:tasks.workunit.client.0.smithi111.stdout:0/297: creat d6/d9/de/d32/f5e x:0 0 0 2024-03-14T13:02:13.097 INFO:tasks.workunit.client.0.smithi111.stdout:5/420: fsync d2/d16/d49/f5d 0 2024-03-14T13:02:13.100 INFO:tasks.workunit.client.1.smithi137.stdout:8/163: fdatasync d1/d17/f38 0 2024-03-14T13:02:13.100 INFO:tasks.workunit.client.1.smithi137.stdout:1/327: mkdir d7/d9/d19/d1f/d26/d6c 0 2024-03-14T13:02:13.101 INFO:tasks.workunit.client.0.smithi111.stdout:5/421: link d2/d16/d49/d33/f43 d2/d16/d49/d44/f8e 0 2024-03-14T13:02:13.102 INFO:tasks.workunit.client.0.smithi111.stdout:5/422: read - d2/d45/f66 zero size 2024-03-14T13:02:13.103 INFO:tasks.workunit.client.1.smithi137.stdout:4/265: rename d4/f1a to d4/d9/f5d 0 2024-03-14T13:02:13.104 INFO:tasks.workunit.client.1.smithi137.stdout:8/164: write d1/d17/f1e [827148,87178] 0 2024-03-14T13:02:13.106 INFO:tasks.workunit.client.1.smithi137.stdout:8/165: symlink d1/l42 0 2024-03-14T13:02:13.107 INFO:tasks.workunit.client.0.smithi111.stdout:5/423: dwrite d2/d16/d49/d4c/f73 [0,4194304] 0 2024-03-14T13:02:13.111 INFO:tasks.workunit.client.1.smithi137.stdout:4/266: getdents d4/d10 0 2024-03-14T13:02:13.112 INFO:tasks.workunit.client.1.smithi137.stdout:4/267: rename d4/d10/d26 to d4/d10/d26/d5e 22 2024-03-14T13:02:13.125 INFO:tasks.workunit.client.1.smithi137.stdout:0/238: dwrite d1/f51 [0,4194304] 0 2024-03-14T13:02:13.129 INFO:tasks.workunit.client.1.smithi137.stdout:0/239: mkdir d1/d5/d1b/d1e/d53 0 2024-03-14T13:02:13.132 INFO:tasks.workunit.client.1.smithi137.stdout:4/268: truncate d4/d9/f5d 1748082 0 2024-03-14T13:02:13.133 INFO:tasks.workunit.client.1.smithi137.stdout:8/166: dwrite d1/d17/f38 [0,4194304] 0 2024-03-14T13:02:13.136 INFO:tasks.workunit.client.1.smithi137.stdout:0/240: mkdir d1/d11/d4a/d54 0 2024-03-14T13:02:13.137 INFO:tasks.workunit.client.1.smithi137.stdout:4/269: truncate d4/d10/d26/f40 540322 0 2024-03-14T13:02:13.137 INFO:tasks.workunit.client.1.smithi137.stdout:8/167: symlink d1/dc/d10/l43 0 2024-03-14T13:02:13.138 INFO:tasks.workunit.client.1.smithi137.stdout:4/270: write d4/f48 [624619,119205] 0 2024-03-14T13:02:13.139 INFO:tasks.workunit.client.1.smithi137.stdout:1/328: dread d7/d9/d19/d40/f46 [0,4194304] 0 2024-03-14T13:02:13.147 INFO:tasks.workunit.client.1.smithi137.stdout:0/241: getdents d1/d5/d1b/d42 0 2024-03-14T13:02:13.150 INFO:tasks.workunit.client.1.smithi137.stdout:0/242: mkdir d1/d5/d1b/d1e/d25/d55 0 2024-03-14T13:02:13.150 INFO:tasks.workunit.client.1.smithi137.stdout:1/329: dwrite d7/d9/d19/d1f/f2a [0,4194304] 0 2024-03-14T13:02:13.152 INFO:tasks.workunit.client.1.smithi137.stdout:0/243: mkdir d1/d44/d56 0 2024-03-14T13:02:13.155 INFO:tasks.workunit.client.1.smithi137.stdout:0/244: chown d1/d5/d1b/d1e/d25 3802825 1 2024-03-14T13:02:13.159 INFO:tasks.workunit.client.1.smithi137.stdout:0/245: symlink d1/d1f/l57 0 2024-03-14T13:02:13.160 INFO:tasks.workunit.client.1.smithi137.stdout:4/271: dread f2 [0,4194304] 0 2024-03-14T13:02:13.161 INFO:tasks.workunit.client.0.smithi111.stdout:1/350: sync 2024-03-14T13:02:13.161 INFO:tasks.workunit.client.0.smithi111.stdout:4/561: sync 2024-03-14T13:02:13.161 INFO:tasks.workunit.client.0.smithi111.stdout:7/267: sync 2024-03-14T13:02:13.161 INFO:tasks.workunit.client.0.smithi111.stdout:2/255: sync 2024-03-14T13:02:13.161 INFO:tasks.workunit.client.1.smithi137.stdout:4/272: chown d4/d27/f28 1868 1 2024-03-14T13:02:13.162 INFO:tasks.workunit.client.1.smithi137.stdout:0/246: creat d1/d5/d1b/d42/f58 x:0 0 0 2024-03-14T13:02:13.163 INFO:tasks.workunit.client.1.smithi137.stdout:8/168: dread d1/dc/d11/d13/f29 [0,4194304] 0 2024-03-14T13:02:13.163 INFO:tasks.workunit.client.1.smithi137.stdout:4/273: mknod d4/d9/d3a/c5f 0 2024-03-14T13:02:13.163 INFO:tasks.workunit.client.1.smithi137.stdout:8/169: chown d1/c3b 1341110 1 2024-03-14T13:02:13.165 INFO:tasks.workunit.client.0.smithi111.stdout:7/268: read d0/d1/d5/d38/d9/f3d [88609,34892] 0 2024-03-14T13:02:13.165 INFO:tasks.workunit.client.0.smithi111.stdout:4/562: unlink d1d/d22/d30/d7c/f6a 0 2024-03-14T13:02:13.166 INFO:tasks.workunit.client.1.smithi137.stdout:0/247: rename d1/d1f/c16 to d1/d5/d1b/c59 0 2024-03-14T13:02:13.168 INFO:tasks.workunit.client.1.smithi137.stdout:1/330: truncate f6 4149262 0 2024-03-14T13:02:13.168 INFO:tasks.workunit.client.0.smithi111.stdout:7/269: write d0/d1/d22/f42 [1608552,54504] 0 2024-03-14T13:02:13.169 INFO:tasks.workunit.client.1.smithi137.stdout:0/248: dread d1/f46 [0,4194304] 0 2024-03-14T13:02:13.169 INFO:tasks.workunit.client.0.smithi111.stdout:2/256: truncate dd/d11/d1a/f43 525271 0 2024-03-14T13:02:13.169 INFO:tasks.workunit.client.0.smithi111.stdout:1/351: link d20/d21/c6b d20/d25/d75/c79 0 2024-03-14T13:02:13.170 INFO:tasks.workunit.client.1.smithi137.stdout:8/170: rename d1/dc/d11/d13/f29 to d1/f44 0 2024-03-14T13:02:13.170 INFO:tasks.workunit.client.0.smithi111.stdout:1/352: truncate fe 4322782 0 2024-03-14T13:02:13.171 INFO:tasks.workunit.client.0.smithi111.stdout:4/563: symlink d1d/d22/d40/d69/lc5 0 2024-03-14T13:02:13.172 INFO:tasks.workunit.client.1.smithi137.stdout:8/171: truncate d1/dc/d11/d13/f3e 591052 0 2024-03-14T13:02:13.172 INFO:tasks.workunit.client.0.smithi111.stdout:2/257: symlink dd/d11/d1a/l50 0 2024-03-14T13:02:13.172 INFO:tasks.workunit.client.1.smithi137.stdout:4/274: link d4/d10/l36 d4/d4d/l60 0 2024-03-14T13:02:13.173 INFO:tasks.workunit.client.0.smithi111.stdout:2/258: readlink dd/d17/d34/l1e 0 2024-03-14T13:02:13.175 INFO:tasks.workunit.client.1.smithi137.stdout:1/331: getdents d7/d4d 0 2024-03-14T13:02:13.176 INFO:tasks.workunit.client.0.smithi111.stdout:1/353: creat d20/d21/d28/f7a x:0 0 0 2024-03-14T13:02:13.177 INFO:tasks.workunit.client.0.smithi111.stdout:4/564: mkdir d1d/d22/dc6 0 2024-03-14T13:02:13.177 INFO:tasks.workunit.client.0.smithi111.stdout:7/270: dwrite d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:13.178 INFO:tasks.workunit.client.1.smithi137.stdout:1/332: rmdir d7/d9/d19/d40 39 2024-03-14T13:02:13.180 INFO:tasks.workunit.client.0.smithi111.stdout:2/259: creat dd/d11/d48/d49/f51 x:0 0 0 2024-03-14T13:02:13.181 INFO:tasks.workunit.client.1.smithi137.stdout:1/333: mknod d7/d9/d19/d1f/d22/c6d 0 2024-03-14T13:02:13.182 INFO:tasks.workunit.client.0.smithi111.stdout:2/260: write dd/d11/d48/d49/f51 [428591,104814] 0 2024-03-14T13:02:13.186 INFO:tasks.workunit.client.0.smithi111.stdout:4/565: creat d1d/d22/d23/fc7 x:0 0 0 2024-03-14T13:02:13.186 INFO:tasks.workunit.client.0.smithi111.stdout:1/354: dread d20/d25/f38 [0,4194304] 0 2024-03-14T13:02:13.186 INFO:tasks.workunit.client.0.smithi111.stdout:7/271: symlink d0/d1/d21/d3a/l50 0 2024-03-14T13:02:13.186 INFO:tasks.workunit.client.0.smithi111.stdout:2/261: chown dd/d17 520762 1 2024-03-14T13:02:13.188 INFO:tasks.workunit.client.0.smithi111.stdout:4/566: chown d1d/d22/d56 387971 1 2024-03-14T13:02:13.189 INFO:tasks.workunit.client.0.smithi111.stdout:2/262: stat dd/d11/d48/d49/f51 0 2024-03-14T13:02:13.189 INFO:tasks.workunit.client.0.smithi111.stdout:1/355: readlink l1 0 2024-03-14T13:02:13.190 INFO:tasks.workunit.client.1.smithi137.stdout:5/164: sync 2024-03-14T13:02:13.190 INFO:tasks.workunit.client.1.smithi137.stdout:7/230: sync 2024-03-14T13:02:13.190 INFO:tasks.workunit.client.1.smithi137.stdout:6/248: sync 2024-03-14T13:02:13.190 INFO:tasks.workunit.client.1.smithi137.stdout:2/232: sync 2024-03-14T13:02:13.191 INFO:tasks.workunit.client.0.smithi111.stdout:4/567: write d1d/d22/d30/f3d [885611,95404] 0 2024-03-14T13:02:13.191 INFO:tasks.workunit.client.0.smithi111.stdout:4/568: fdatasync d1d/d22/d23/fc7 0 2024-03-14T13:02:13.192 INFO:tasks.workunit.client.1.smithi137.stdout:8/172: read d1/f44 [1137631,101946] 0 2024-03-14T13:02:13.193 INFO:tasks.workunit.client.1.smithi137.stdout:4/275: truncate d4/f32 1725241 0 2024-03-14T13:02:13.194 INFO:tasks.workunit.client.0.smithi111.stdout:7/272: fsync d0/d1/d5/d38/d9/f12 0 2024-03-14T13:02:13.195 INFO:tasks.workunit.client.1.smithi137.stdout:7/231: symlink d3/d5/l47 0 2024-03-14T13:02:13.196 INFO:tasks.workunit.client.1.smithi137.stdout:5/165: unlink d7/f19 0 2024-03-14T13:02:13.196 INFO:tasks.workunit.client.1.smithi137.stdout:0/249: dread d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:13.196 INFO:tasks.workunit.client.0.smithi111.stdout:7/273: fdatasync d0/d1/d21/f3f 0 2024-03-14T13:02:13.196 INFO:tasks.workunit.client.0.smithi111.stdout:2/263: rmdir dd 39 2024-03-14T13:02:13.197 INFO:tasks.workunit.client.1.smithi137.stdout:1/334: write d7/fb [1637113,104565] 0 2024-03-14T13:02:13.198 INFO:tasks.workunit.client.0.smithi111.stdout:7/274: chown d0/d1/d21/d3a 608180 1 2024-03-14T13:02:13.200 INFO:tasks.workunit.client.0.smithi111.stdout:7/275: fsync d0/d11/f29 0 2024-03-14T13:02:13.200 INFO:tasks.workunit.client.1.smithi137.stdout:4/276: symlink d4/d10/d3d/l61 0 2024-03-14T13:02:13.201 INFO:tasks.workunit.client.0.smithi111.stdout:4/569: dwrite d1d/d22/d30/f81 [0,4194304] 0 2024-03-14T13:02:13.202 INFO:tasks.workunit.client.0.smithi111.stdout:7/276: stat d0/l23 0 2024-03-14T13:02:13.202 INFO:tasks.workunit.client.1.smithi137.stdout:2/233: dwrite d2/f4b [0,4194304] 0 2024-03-14T13:02:13.205 INFO:tasks.workunit.client.0.smithi111.stdout:2/264: rename dd/f3f to dd/d11/d1a/f52 0 2024-03-14T13:02:13.206 INFO:tasks.workunit.client.1.smithi137.stdout:7/232: symlink d3/d6/d14/d27/l48 0 2024-03-14T13:02:13.207 INFO:tasks.workunit.client.0.smithi111.stdout:7/277: creat d0/d1/f51 x:0 0 0 2024-03-14T13:02:13.207 INFO:tasks.workunit.client.1.smithi137.stdout:2/234: write d2/d7/db/de/f39 [532004,44196] 0 2024-03-14T13:02:13.207 INFO:tasks.workunit.client.0.smithi111.stdout:1/356: dread f3 [0,4194304] 0 2024-03-14T13:02:13.207 INFO:tasks.workunit.client.0.smithi111.stdout:7/278: fsync d0/d1/d22/f42 0 2024-03-14T13:02:13.209 INFO:tasks.workunit.client.0.smithi111.stdout:4/570: creat d1d/d22/d30/d5e/d6f/d7e/fc8 x:0 0 0 2024-03-14T13:02:13.210 INFO:tasks.workunit.client.0.smithi111.stdout:7/279: fdatasync d0/d1/d22/d3e/f44 0 2024-03-14T13:02:13.211 INFO:tasks.workunit.client.0.smithi111.stdout:1/357: creat d20/d25/f7b x:0 0 0 2024-03-14T13:02:13.212 INFO:tasks.workunit.client.0.smithi111.stdout:4/571: chown d1d/d22/d30/d7c/d59/fb7 81174904 1 2024-03-14T13:02:13.213 INFO:tasks.workunit.client.0.smithi111.stdout:1/358: write d20/d40/f4c [2639294,54386] 0 2024-03-14T13:02:13.216 INFO:tasks.workunit.client.1.smithi137.stdout:8/173: link d1/dc/f1f d1/dc/d11/d13/f45 0 2024-03-14T13:02:13.218 INFO:tasks.workunit.client.0.smithi111.stdout:7/280: dwrite d0/d1/d21/f15 [0,4194304] 0 2024-03-14T13:02:13.219 INFO:tasks.workunit.client.1.smithi137.stdout:8/174: read d1/fd [585498,71796] 0 2024-03-14T13:02:13.220 INFO:tasks.workunit.client.1.smithi137.stdout:8/175: chown d1/dc/d10/f35 0 1 2024-03-14T13:02:13.220 INFO:tasks.workunit.client.0.smithi111.stdout:7/281: fdatasync d0/d1/d5/d38/d9/fb 0 2024-03-14T13:02:13.220 INFO:tasks.workunit.client.1.smithi137.stdout:6/249: truncate d2/d54/d55/f1f 2405930 0 2024-03-14T13:02:13.220 INFO:tasks.workunit.client.1.smithi137.stdout:2/235: creat d2/d7/db/de/d22/f4c x:0 0 0 2024-03-14T13:02:13.221 INFO:tasks.workunit.client.1.smithi137.stdout:4/277: symlink d4/d4d/l62 0 2024-03-14T13:02:13.222 INFO:tasks.workunit.client.1.smithi137.stdout:5/166: creat d7/d16/d18/f39 x:0 0 0 2024-03-14T13:02:13.222 INFO:tasks.workunit.client.0.smithi111.stdout:7/282: mkdir d0/d1/d4a/d52 0 2024-03-14T13:02:13.225 INFO:tasks.workunit.client.0.smithi111.stdout:7/283: creat d0/d11/f53 x:0 0 0 2024-03-14T13:02:13.228 INFO:tasks.workunit.client.0.smithi111.stdout:7/284: mknod d0/d1/d5/d38/d1b/c54 0 2024-03-14T13:02:13.228 INFO:tasks.workunit.client.1.smithi137.stdout:8/176: mkdir d1/dc/d10/d46 0 2024-03-14T13:02:13.229 INFO:tasks.workunit.client.1.smithi137.stdout:6/250: dread d2/d54/d55/d33/f44 [0,4194304] 0 2024-03-14T13:02:13.229 INFO:tasks.workunit.client.0.smithi111.stdout:7/285: fsync d0/d11/f29 0 2024-03-14T13:02:13.229 INFO:tasks.workunit.client.0.smithi111.stdout:4/572: getdents d1d/d22/d30/d5e/d6f/d7e 0 2024-03-14T13:02:13.230 INFO:tasks.workunit.client.0.smithi111.stdout:2/265: write dd/d11/d1a/f43 [145753,4052] 0 2024-03-14T13:02:13.230 INFO:tasks.workunit.client.0.smithi111.stdout:1/359: write d20/f23 [106347,127227] 0 2024-03-14T13:02:13.231 INFO:tasks.workunit.client.1.smithi137.stdout:1/335: symlink d7/d9/d19/d40/d58/l6e 0 2024-03-14T13:02:13.231 INFO:tasks.workunit.client.0.smithi111.stdout:4/573: read d1d/d22/d30/f38 [60929,93623] 0 2024-03-14T13:02:13.232 INFO:tasks.workunit.client.1.smithi137.stdout:6/251: dread - d2/d6/f3d zero size 2024-03-14T13:02:13.232 INFO:tasks.workunit.client.0.smithi111.stdout:2/266: write dd/d11/d1a/f21 [5091390,37837] 0 2024-03-14T13:02:13.233 INFO:tasks.workunit.client.1.smithi137.stdout:1/336: stat d7/d4d/f64 0 2024-03-14T13:02:13.234 INFO:tasks.workunit.client.0.smithi111.stdout:4/574: chown d1d/d22/d30/d5e/d6f/d7e/d66/l77 901818 1 2024-03-14T13:02:13.235 INFO:tasks.workunit.client.0.smithi111.stdout:4/575: truncate d1d/d22/d40/d69/d86/f9f 705779 0 2024-03-14T13:02:13.236 INFO:tasks.workunit.client.0.smithi111.stdout:2/267: mkdir dd/d17/d53 0 2024-03-14T13:02:13.239 INFO:tasks.workunit.client.0.smithi111.stdout:2/268: unlink dd/d11/d48/d49/l4e 0 2024-03-14T13:02:13.239 INFO:tasks.workunit.client.0.smithi111.stdout:2/269: readlink dd/l4a 0 2024-03-14T13:02:13.240 INFO:tasks.workunit.client.0.smithi111.stdout:4/576: mknod d1d/d22/d30/d7c/d59/dc4/cc9 0 2024-03-14T13:02:13.240 INFO:tasks.workunit.client.1.smithi137.stdout:0/250: dread d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:13.240 INFO:tasks.workunit.client.1.smithi137.stdout:4/278: dwrite d4/d10/f3f [0,4194304] 0 2024-03-14T13:02:13.241 INFO:tasks.workunit.client.0.smithi111.stdout:2/270: write dd/d11/d1a/f38 [1520592,22194] 0 2024-03-14T13:02:13.242 INFO:tasks.workunit.client.0.smithi111.stdout:4/577: creat d1d/d22/d56/fca x:0 0 0 2024-03-14T13:02:13.242 INFO:tasks.workunit.client.1.smithi137.stdout:2/236: dread d2/d7/db/de/f1b [0,4194304] 0 2024-03-14T13:02:13.243 INFO:tasks.workunit.client.1.smithi137.stdout:7/233: getdents d3/d6/d2e 0 2024-03-14T13:02:13.244 INFO:tasks.workunit.client.0.smithi111.stdout:4/578: creat d1d/d22/d30/d87/fcb x:0 0 0 2024-03-14T13:02:13.244 INFO:tasks.workunit.client.1.smithi137.stdout:6/252: creat d2/d6/de/f5d x:0 0 0 2024-03-14T13:02:13.246 INFO:tasks.workunit.client.1.smithi137.stdout:6/253: chown d2/d6/l2e 54841 1 2024-03-14T13:02:13.246 INFO:tasks.workunit.client.1.smithi137.stdout:6/254: chown d2/d54 6 1 2024-03-14T13:02:13.248 INFO:tasks.workunit.client.0.smithi111.stdout:4/579: rename d1d/d22/d56/l64 to d1d/lcc 0 2024-03-14T13:02:13.249 INFO:tasks.workunit.client.1.smithi137.stdout:2/237: mkdir d2/d7/db/de/d22/d4d 0 2024-03-14T13:02:13.249 INFO:tasks.workunit.client.1.smithi137.stdout:9/331: sync 2024-03-14T13:02:13.251 INFO:tasks.workunit.client.1.smithi137.stdout:7/234: creat d3/d6/d2e/f49 x:0 0 0 2024-03-14T13:02:13.251 INFO:tasks.workunit.client.1.smithi137.stdout:2/238: write d2/d7/db/d2f/f40 [1069257,88998] 0 2024-03-14T13:02:13.251 INFO:tasks.workunit.client.1.smithi137.stdout:9/332: chown d6/df/d19/d3a/c4f 376 1 2024-03-14T13:02:13.252 INFO:tasks.workunit.client.1.smithi137.stdout:8/177: link d1/d17/f1c d1/d17/d34/f47 0 2024-03-14T13:02:13.253 INFO:tasks.workunit.client.0.smithi111.stdout:1/360: write d20/d25/d31/f3a [380014,18041] 0 2024-03-14T13:02:13.255 INFO:tasks.workunit.client.0.smithi111.stdout:7/286: dwrite d0/d1/d5/d38/d9/f12 [0,4194304] 0 2024-03-14T13:02:13.255 INFO:tasks.workunit.client.0.smithi111.stdout:1/361: fdatasync d20/d25/f34 0 2024-03-14T13:02:13.255 INFO:tasks.workunit.client.1.smithi137.stdout:2/239: fdatasync d2/d7/db/de/f39 0 2024-03-14T13:02:13.255 INFO:tasks.workunit.client.1.smithi137.stdout:4/279: rename d4/f37 to d4/d10/d5a/f63 0 2024-03-14T13:02:13.256 INFO:tasks.workunit.client.0.smithi111.stdout:6/285: sync 2024-03-14T13:02:13.256 INFO:tasks.workunit.client.0.smithi111.stdout:3/396: sync 2024-03-14T13:02:13.256 INFO:tasks.workunit.client.0.smithi111.stdout:8/520: sync 2024-03-14T13:02:13.256 INFO:tasks.workunit.client.0.smithi111.stdout:2/271: write dd/f4b [684100,123228] 0 2024-03-14T13:02:13.259 INFO:tasks.workunit.client.1.smithi137.stdout:4/280: stat d4/d10/d26/f40 0 2024-03-14T13:02:13.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/521: write d3/f14 [2242800,27970] 0 2024-03-14T13:02:13.260 INFO:tasks.workunit.client.1.smithi137.stdout:1/337: creat d7/d9/d19/d1f/d26/d6c/f6f x:0 0 0 2024-03-14T13:02:13.261 INFO:tasks.workunit.client.1.smithi137.stdout:7/235: dwrite d3/fe [0,4194304] 0 2024-03-14T13:02:13.261 INFO:tasks.workunit.client.1.smithi137.stdout:6/255: creat d2/d6/de/d45/d59/f5e x:0 0 0 2024-03-14T13:02:13.263 INFO:tasks.workunit.client.1.smithi137.stdout:7/236: write d3/d5/d23/f2b [861304,101995] 0 2024-03-14T13:02:13.265 INFO:tasks.workunit.client.1.smithi137.stdout:8/178: creat d1/d17/f48 x:0 0 0 2024-03-14T13:02:13.268 INFO:tasks.workunit.client.0.smithi111.stdout:2/272: link c8 dd/d11/d48/c54 0 2024-03-14T13:02:13.268 INFO:tasks.workunit.client.1.smithi137.stdout:8/179: dread - d1/d17/d34/f40 zero size 2024-03-14T13:02:13.269 INFO:tasks.workunit.client.0.smithi111.stdout:7/287: getdents d0/d1/d22 0 2024-03-14T13:02:13.270 INFO:tasks.workunit.client.0.smithi111.stdout:8/522: rmdir d3/d1b/d20/d83 39 2024-03-14T13:02:13.271 INFO:tasks.workunit.client.0.smithi111.stdout:8/523: read d3/f3e [6545322,94864] 0 2024-03-14T13:02:13.276 INFO:tasks.workunit.client.0.smithi111.stdout:2/273: mkdir dd/d17/d34/d4f/d55 0 2024-03-14T13:02:13.276 INFO:tasks.workunit.client.0.smithi111.stdout:4/580: dread d1d/f20 [0,4194304] 0 2024-03-14T13:02:13.277 INFO:tasks.workunit.client.1.smithi137.stdout:0/251: link d1/d5/d1b/d1e/d25/d26/f2d d1/d44/d56/f5a 0 2024-03-14T13:02:13.277 INFO:tasks.workunit.client.1.smithi137.stdout:9/333: rename d6/df/d19/d3a/d40/l5a to d6/df/d19/d3a/d40/l79 0 2024-03-14T13:02:13.277 INFO:tasks.workunit.client.1.smithi137.stdout:2/240: mkdir d2/d7/d2e/d3e/d4e 0 2024-03-14T13:02:13.278 INFO:tasks.workunit.client.0.smithi111.stdout:7/288: dwrite d0/d1/f2 [0,4194304] 0 2024-03-14T13:02:13.279 INFO:tasks.workunit.client.1.smithi137.stdout:1/338: readlink d7/l55 0 2024-03-14T13:02:13.280 INFO:tasks.workunit.client.0.smithi111.stdout:8/524: dread d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:13.280 INFO:tasks.workunit.client.0.smithi111.stdout:4/581: chown d1d/d22/d30/d5e/d6f/d7e/d66/c9d 1565 1 2024-03-14T13:02:13.280 INFO:tasks.workunit.client.1.smithi137.stdout:5/167: rmdir d7/d16/d18/d25 39 2024-03-14T13:02:13.281 INFO:tasks.workunit.client.1.smithi137.stdout:4/281: link d4/f2e d4/d9/d3a/d41/d38/d55/f64 0 2024-03-14T13:02:13.282 INFO:tasks.workunit.client.0.smithi111.stdout:7/289: symlink d0/d1/d22/l55 0 2024-03-14T13:02:13.283 INFO:tasks.workunit.client.1.smithi137.stdout:7/237: mkdir d3/d4a 0 2024-03-14T13:02:13.283 INFO:tasks.workunit.client.1.smithi137.stdout:4/282: chown d4/d10/d26 16373079 1 2024-03-14T13:02:13.284 INFO:tasks.workunit.client.1.smithi137.stdout:6/256: mknod d2/d4c/c5f 0 2024-03-14T13:02:13.285 INFO:tasks.workunit.client.1.smithi137.stdout:4/283: readlink d4/d10/d3d/l61 0 2024-03-14T13:02:13.285 INFO:tasks.workunit.client.1.smithi137.stdout:0/252: truncate d1/f46 1870657 0 2024-03-14T13:02:13.287 INFO:tasks.workunit.client.1.smithi137.stdout:6/257: fsync d2/d6/de/f5d 0 2024-03-14T13:02:13.287 INFO:tasks.workunit.client.0.smithi111.stdout:6/286: truncate d0/d17/d20/f26 481538 0 2024-03-14T13:02:13.288 INFO:tasks.workunit.client.1.smithi137.stdout:0/253: fsync d1/d5/d1b/d42/f32 0 2024-03-14T13:02:13.289 INFO:tasks.workunit.client.1.smithi137.stdout:4/284: write d4/d9/f5b [569500,49514] 0 2024-03-14T13:02:13.289 INFO:tasks.workunit.client.0.smithi111.stdout:2/274: dwrite dd/d11/f33 [0,4194304] 0 2024-03-14T13:02:13.289 INFO:tasks.workunit.client.0.smithi111.stdout:7/290: unlink d0/d1/d22/d3e/f44 0 2024-03-14T13:02:13.290 INFO:tasks.workunit.client.0.smithi111.stdout:3/397: truncate d0/d2d/d35/d50/d5e/d19/f5a 2508970 0 2024-03-14T13:02:13.290 INFO:tasks.workunit.client.1.smithi137.stdout:7/238: write d3/d6/d2e/f49 [485782,47838] 0 2024-03-14T13:02:13.290 INFO:tasks.workunit.client.1.smithi137.stdout:5/168: dread - d7/d16/d18/d2d/f2f zero size 2024-03-14T13:02:13.291 INFO:tasks.workunit.client.0.smithi111.stdout:1/362: dread d20/d25/d75/f65 [0,4194304] 0 2024-03-14T13:02:13.293 INFO:tasks.workunit.client.1.smithi137.stdout:4/285: read d4/d10/f3f [975418,1413] 0 2024-03-14T13:02:13.295 INFO:tasks.workunit.client.0.smithi111.stdout:2/275: unlink dd/c47 0 2024-03-14T13:02:13.296 INFO:tasks.workunit.client.0.smithi111.stdout:7/291: fdatasync d0/d1/f28 0 2024-03-14T13:02:13.298 INFO:tasks.workunit.client.1.smithi137.stdout:1/339: dwrite d7/d9/d19/d40/f61 [0,4194304] 0 2024-03-14T13:02:13.302 INFO:tasks.workunit.client.0.smithi111.stdout:3/398: rename d0/d2d/d35/d50/d5e/l3c to d0/d2d/d35/d69/l9c 0 2024-03-14T13:02:13.302 INFO:tasks.workunit.client.0.smithi111.stdout:1/363: creat d20/d25/f7c x:0 0 0 2024-03-14T13:02:13.302 INFO:tasks.workunit.client.0.smithi111.stdout:4/582: link d1d/d22/d40/c57 d1d/d22/d40/d69/d86/ccd 0 2024-03-14T13:02:13.303 INFO:tasks.workunit.client.0.smithi111.stdout:2/276: write dd/fe [3486573,8982] 0 2024-03-14T13:02:13.303 INFO:tasks.workunit.client.1.smithi137.stdout:7/239: unlink d3/l21 0 2024-03-14T13:02:13.303 INFO:tasks.workunit.client.1.smithi137.stdout:9/334: rmdir d6/df/d19/d78 0 2024-03-14T13:02:13.304 INFO:tasks.workunit.client.0.smithi111.stdout:7/292: mkdir d0/d1/d5/d56 0 2024-03-14T13:02:13.304 INFO:tasks.workunit.client.0.smithi111.stdout:8/525: link d3/dc/d31/c5f d3/d22/d4a/ca3 0 2024-03-14T13:02:13.304 INFO:tasks.workunit.client.1.smithi137.stdout:9/335: dread - d6/df/f64 zero size 2024-03-14T13:02:13.305 INFO:tasks.workunit.client.1.smithi137.stdout:2/241: link d2/d7/db/de/c2b d2/d7/db/de/d22/c4f 0 2024-03-14T13:02:13.305 INFO:tasks.workunit.client.0.smithi111.stdout:7/293: chown d0/d1/d4a/d52 35912 1 2024-03-14T13:02:13.306 INFO:tasks.workunit.client.0.smithi111.stdout:6/287: link d0/d17/d20/f6b d0/d17/d20/d4a/d59/f6d 0 2024-03-14T13:02:13.306 INFO:tasks.workunit.client.1.smithi137.stdout:0/254: creat d1/d11/d4a/d54/f5b x:0 0 0 2024-03-14T13:02:13.306 INFO:tasks.workunit.client.1.smithi137.stdout:4/286: chown d4/l57 208534 1 2024-03-14T13:02:13.307 INFO:tasks.workunit.client.0.smithi111.stdout:2/277: mkdir dd/d11/d1a/d56 0 2024-03-14T13:02:13.308 INFO:tasks.workunit.client.1.smithi137.stdout:6/258: dwrite d2/d6/de/d45/d59/d39/d3c/f47 [0,4194304] 0 2024-03-14T13:02:13.309 INFO:tasks.workunit.client.0.smithi111.stdout:9/386: sync 2024-03-14T13:02:13.309 INFO:tasks.workunit.client.0.smithi111.stdout:0/298: sync 2024-03-14T13:02:13.309 INFO:tasks.workunit.client.0.smithi111.stdout:5/424: sync 2024-03-14T13:02:13.309 INFO:tasks.workunit.client.0.smithi111.stdout:2/278: stat dd/d17/d34/d4f/d55 0 2024-03-14T13:02:13.309 INFO:tasks.workunit.client.0.smithi111.stdout:1/364: creat d20/d25/d75/f7d x:0 0 0 2024-03-14T13:02:13.310 INFO:tasks.workunit.client.1.smithi137.stdout:5/169: symlink d7/d36/l3a 0 2024-03-14T13:02:13.311 INFO:tasks.workunit.client.0.smithi111.stdout:5/425: stat d2/d45/l2a 0 2024-03-14T13:02:13.312 INFO:tasks.workunit.client.0.smithi111.stdout:6/288: creat d0/f6e x:0 0 0 2024-03-14T13:02:13.313 INFO:tasks.workunit.client.0.smithi111.stdout:7/294: symlink d0/d1/d21/d3a/l57 0 2024-03-14T13:02:13.314 INFO:tasks.workunit.client.1.smithi137.stdout:9/336: dread - d6/df/d19/d3a/f67 zero size 2024-03-14T13:02:13.314 INFO:tasks.workunit.client.1.smithi137.stdout:1/340: dread d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:13.314 INFO:tasks.workunit.client.0.smithi111.stdout:4/583: dwrite d1d/d22/d56/f5f [0,4194304] 0 2024-03-14T13:02:13.315 INFO:tasks.workunit.client.1.smithi137.stdout:0/255: truncate d1/d1f/f2f 581358 0 2024-03-14T13:02:13.317 INFO:tasks.workunit.client.1.smithi137.stdout:2/242: symlink d2/d7/db/de/d22/l50 0 2024-03-14T13:02:13.318 INFO:tasks.workunit.client.0.smithi111.stdout:0/299: stat d6/d9/de/l20 0 2024-03-14T13:02:13.318 INFO:tasks.workunit.client.0.smithi111.stdout:4/584: chown d1d/d22/d30/d5e/d6f 98 1 2024-03-14T13:02:13.318 INFO:tasks.workunit.client.1.smithi137.stdout:7/240: dread d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:13.319 INFO:tasks.workunit.client.1.smithi137.stdout:6/259: rename d2/d6/f35 to d2/d6/de/d45/f60 0 2024-03-14T13:02:13.320 INFO:tasks.workunit.client.0.smithi111.stdout:1/365: mknod d20/d21/d3f/d59/c7e 0 2024-03-14T13:02:13.320 INFO:tasks.workunit.client.1.smithi137.stdout:2/243: rename d2 to d2/d7/d2e/d3e/d44/d51 22 2024-03-14T13:02:13.320 INFO:tasks.workunit.client.0.smithi111.stdout:5/426: mknod d2/d1f/d6a/c8f 0 2024-03-14T13:02:13.321 INFO:tasks.workunit.client.0.smithi111.stdout:1/366: dread - d20/f61 zero size 2024-03-14T13:02:13.321 INFO:tasks.workunit.client.1.smithi137.stdout:9/337: mkdir d6/d56/d7a 0 2024-03-14T13:02:13.322 INFO:tasks.workunit.client.0.smithi111.stdout:4/585: unlink d1d/f36 0 2024-03-14T13:02:13.322 INFO:tasks.workunit.client.0.smithi111.stdout:0/300: readlink d6/d9/de/l1c 0 2024-03-14T13:02:13.323 INFO:tasks.workunit.client.0.smithi111.stdout:4/586: write d1d/d22/f55 [1983482,91692] 0 2024-03-14T13:02:13.324 INFO:tasks.workunit.client.1.smithi137.stdout:4/287: mkdir d4/d10/d3d/d54/d58/d65 0 2024-03-14T13:02:13.324 INFO:tasks.workunit.client.1.smithi137.stdout:4/288: rename d4 to d4/d27/d66 22 2024-03-14T13:02:13.325 INFO:tasks.workunit.client.1.smithi137.stdout:4/289: chown d4/f11 6184 1 2024-03-14T13:02:13.325 INFO:tasks.workunit.client.0.smithi111.stdout:7/295: rename d0/d1/d22/c24 to d0/d1/d4a/c58 0 2024-03-14T13:02:13.329 INFO:tasks.workunit.client.0.smithi111.stdout:5/427: creat d2/d16/d49/d7c/f90 x:0 0 0 2024-03-14T13:02:13.329 INFO:tasks.workunit.client.1.smithi137.stdout:4/290: rename d4/d10/d3d/d54/d58 to d4/d10/d3d/d54/d58/d65/d67 22 2024-03-14T13:02:13.331 INFO:tasks.workunit.client.1.smithi137.stdout:0/256: dread d1/d5/d1b/d1e/f30 [0,4194304] 0 2024-03-14T13:02:13.331 INFO:tasks.workunit.client.1.smithi137.stdout:1/341: symlink d7/d9/d19/d1f/l70 0 2024-03-14T13:02:13.332 INFO:tasks.workunit.client.0.smithi111.stdout:1/367: dwrite d20/d25/f64 [0,4194304] 0 2024-03-14T13:02:13.335 INFO:tasks.workunit.client.0.smithi111.stdout:7/296: rmdir d0/d1/d22 39 2024-03-14T13:02:13.336 INFO:tasks.workunit.client.1.smithi137.stdout:6/260: unlink d2/d6/de/d45/d59/d39/d3c/d43/l48 0 2024-03-14T13:02:13.336 INFO:tasks.workunit.client.0.smithi111.stdout:7/297: chown d0/d1/d5/d38/d9/f19 281618 1 2024-03-14T13:02:13.337 INFO:tasks.workunit.client.1.smithi137.stdout:5/170: creat d7/f3b x:0 0 0 2024-03-14T13:02:13.339 INFO:tasks.workunit.client.1.smithi137.stdout:6/261: stat d2/d6/de/d45/d59/d39/d3c/d43/d57/d58 0 2024-03-14T13:02:13.341 INFO:tasks.workunit.client.1.smithi137.stdout:2/244: mknod d2/c52 0 2024-03-14T13:02:13.345 INFO:tasks.workunit.client.0.smithi111.stdout:5/428: creat d2/d16/d49/d7c/d8b/f91 x:0 0 0 2024-03-14T13:02:13.345 INFO:tasks.workunit.client.1.smithi137.stdout:9/338: mknod d6/c7b 0 2024-03-14T13:02:13.346 INFO:tasks.workunit.client.1.smithi137.stdout:4/291: rename d4/d9/ca to d4/d9/d3a/c68 0 2024-03-14T13:02:13.347 INFO:tasks.workunit.client.0.smithi111.stdout:0/301: dread d6/f1e [0,4194304] 0 2024-03-14T13:02:13.349 INFO:tasks.workunit.client.1.smithi137.stdout:1/342: dread d7/d9/d19/f2f [4194304,4194304] 0 2024-03-14T13:02:13.349 INFO:tasks.workunit.client.1.smithi137.stdout:6/262: symlink d2/d6/de/d45/d59/d39/d3c/l61 0 2024-03-14T13:02:13.350 INFO:tasks.workunit.client.0.smithi111.stdout:5/429: dread d2/d16/d49/d33/f40 [0,4194304] 0 2024-03-14T13:02:13.350 INFO:tasks.workunit.client.0.smithi111.stdout:6/289: getdents d0/d17/d20/d4a/d59 0 2024-03-14T13:02:13.350 INFO:tasks.workunit.client.0.smithi111.stdout:1/368: rename d20/f61 to d20/d25/d31/f7f 0 2024-03-14T13:02:13.351 INFO:tasks.workunit.client.1.smithi137.stdout:9/339: rmdir d6/df/d19/d1f 39 2024-03-14T13:02:13.351 INFO:tasks.workunit.client.1.smithi137.stdout:1/343: write d7/d9/d19/d40/f61 [322944,66750] 0 2024-03-14T13:02:13.354 INFO:tasks.workunit.client.1.smithi137.stdout:0/257: symlink d1/l5c 0 2024-03-14T13:02:13.354 INFO:tasks.workunit.client.0.smithi111.stdout:3/399: write d0/f8 [770390,45611] 0 2024-03-14T13:02:13.355 INFO:tasks.workunit.client.1.smithi137.stdout:0/258: write d1/d1f/f3b [220566,119058] 0 2024-03-14T13:02:13.358 INFO:tasks.workunit.client.1.smithi137.stdout:0/259: chown d1/f51 611421880 1 2024-03-14T13:02:13.361 INFO:tasks.workunit.client.0.smithi111.stdout:3/400: readlink d0/d91/l96 0 2024-03-14T13:02:13.361 INFO:tasks.workunit.client.0.smithi111.stdout:9/387: dread d5/d6/d8/f30 [0,4194304] 0 2024-03-14T13:02:13.363 INFO:tasks.workunit.client.1.smithi137.stdout:1/344: symlink d7/l71 0 2024-03-14T13:02:13.363 INFO:tasks.workunit.client.1.smithi137.stdout:7/241: write d3/d5/f19 [344075,47733] 0 2024-03-14T13:02:13.366 INFO:tasks.workunit.client.1.smithi137.stdout:0/260: rename d1/d5/f37 to d1/d5/d1b/d1e/d25/d26/f5d 0 2024-03-14T13:02:13.367 INFO:tasks.workunit.client.1.smithi137.stdout:5/171: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:13.372 INFO:tasks.workunit.client.0.smithi111.stdout:6/290: dwrite d0/f6e [0,4194304] 0 2024-03-14T13:02:13.372 INFO:tasks.workunit.client.0.smithi111.stdout:5/430: dwrite d2/d45/f66 [0,4194304] 0 2024-03-14T13:02:13.376 INFO:tasks.workunit.client.1.smithi137.stdout:5/172: read d7/d16/f1a [716102,78489] 0 2024-03-14T13:02:13.376 INFO:tasks.workunit.client.0.smithi111.stdout:2/279: dwrite dd/d11/f2c [0,4194304] 0 2024-03-14T13:02:13.380 INFO:tasks.workunit.client.0.smithi111.stdout:7/298: truncate d0/d1/d22/f46 1652212 0 2024-03-14T13:02:13.381 INFO:tasks.workunit.client.1.smithi137.stdout:9/340: dwrite d6/df/d19/d1f/f66 [0,4194304] 0 2024-03-14T13:02:13.381 INFO:tasks.workunit.client.1.smithi137.stdout:2/245: truncate d2/d7/db/f1c 227259 0 2024-03-14T13:02:13.385 INFO:tasks.workunit.client.0.smithi111.stdout:8/526: dwrite d3/d1b/d34/f3c [0,4194304] 0 2024-03-14T13:02:13.387 INFO:tasks.workunit.client.0.smithi111.stdout:5/431: stat d2/d16/d49/d33/d68/f8a 0 2024-03-14T13:02:13.387 INFO:tasks.workunit.client.0.smithi111.stdout:4/587: dread d1d/fbb [0,4194304] 0 2024-03-14T13:02:13.388 INFO:tasks.workunit.client.1.smithi137.stdout:4/292: dread d4/f11 [0,4194304] 0 2024-03-14T13:02:13.388 INFO:tasks.workunit.client.1.smithi137.stdout:1/345: mknod d7/d9/d19/c72 0 2024-03-14T13:02:13.389 INFO:tasks.workunit.client.1.smithi137.stdout:3/225: sync 2024-03-14T13:02:13.390 INFO:tasks.workunit.client.0.smithi111.stdout:8/527: readlink d3/d22/l55 0 2024-03-14T13:02:13.393 INFO:tasks.workunit.client.1.smithi137.stdout:0/261: symlink d1/d5/d1b/d1e/d53/l5e 0 2024-03-14T13:02:13.394 INFO:tasks.workunit.client.1.smithi137.stdout:2/246: mkdir d2/d34/d3c/d53 0 2024-03-14T13:02:13.394 INFO:tasks.workunit.client.1.smithi137.stdout:3/226: chown d4/de/l12 0 1 2024-03-14T13:02:13.397 INFO:tasks.workunit.client.1.smithi137.stdout:7/242: dwrite d3/d6/f15 [0,4194304] 0 2024-03-14T13:02:13.398 INFO:tasks.workunit.client.1.smithi137.stdout:5/173: mkdir d7/d16/d18/d25/d28/d3c 0 2024-03-14T13:02:13.398 INFO:tasks.workunit.client.0.smithi111.stdout:6/291: creat d0/d17/d20/d4a/d59/f6f x:0 0 0 2024-03-14T13:02:13.400 INFO:tasks.workunit.client.0.smithi111.stdout:9/388: dread d5/d42/d52/f6c [0,4194304] 0 2024-03-14T13:02:13.402 INFO:tasks.workunit.client.0.smithi111.stdout:1/369: mkdir d20/d25/d31/d80 0 2024-03-14T13:02:13.403 INFO:tasks.workunit.client.0.smithi111.stdout:0/302: truncate d6/d9/d15/f1b 863817 0 2024-03-14T13:02:13.403 INFO:tasks.workunit.client.0.smithi111.stdout:4/588: rmdir d1d/d22/d40 39 2024-03-14T13:02:13.404 INFO:tasks.workunit.client.0.smithi111.stdout:2/280: dwrite dd/d17/f31 [0,4194304] 0 2024-03-14T13:02:13.404 INFO:tasks.workunit.client.1.smithi137.stdout:6/263: truncate d2/d6/de/d45/d59/d39/d3c/f47 473500 0 2024-03-14T13:02:13.405 INFO:tasks.workunit.client.0.smithi111.stdout:6/292: symlink d0/d50/l70 0 2024-03-14T13:02:13.406 INFO:tasks.workunit.client.0.smithi111.stdout:5/432: mknod d2/d16/d49/d33/d68/c92 0 2024-03-14T13:02:13.406 INFO:tasks.workunit.client.0.smithi111.stdout:2/281: readlink dd/l4a 0 2024-03-14T13:02:13.407 INFO:tasks.workunit.client.1.smithi137.stdout:4/293: symlink d4/d10/d3d/d54/d58/l69 0 2024-03-14T13:02:13.408 INFO:tasks.workunit.client.1.smithi137.stdout:2/247: mkdir d2/d34/d3c/d53/d54 0 2024-03-14T13:02:13.408 INFO:tasks.workunit.client.0.smithi111.stdout:9/389: mkdir d5/d42/d52/d83 0 2024-03-14T13:02:13.409 INFO:tasks.workunit.client.1.smithi137.stdout:0/262: dread d1/f51 [0,4194304] 0 2024-03-14T13:02:13.410 INFO:tasks.workunit.client.1.smithi137.stdout:9/341: dwrite d6/df/f20 [4194304,4194304] 0 2024-03-14T13:02:13.410 INFO:tasks.workunit.client.0.smithi111.stdout:7/299: dwrite d0/d1/d21/f3f [0,4194304] 0 2024-03-14T13:02:13.411 INFO:tasks.workunit.client.1.smithi137.stdout:5/174: rename d7/f3b to d7/d16/d18/d25/d28/f3d 0 2024-03-14T13:02:13.414 INFO:tasks.workunit.client.0.smithi111.stdout:0/303: creat d6/d3d/f5f x:0 0 0 2024-03-14T13:02:13.415 INFO:tasks.workunit.client.1.smithi137.stdout:9/342: readlink d6/df/l2e 0 2024-03-14T13:02:13.415 INFO:tasks.workunit.client.0.smithi111.stdout:8/528: dread d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:13.415 INFO:tasks.workunit.client.0.smithi111.stdout:6/293: rmdir d0/d17/d21/d69 39 2024-03-14T13:02:13.415 INFO:tasks.workunit.client.0.smithi111.stdout:3/401: truncate d0/d2d/d35/d4c/d82/f7e 1932587 0 2024-03-14T13:02:13.416 INFO:tasks.workunit.client.0.smithi111.stdout:9/390: mkdir d5/d6/d8/d5b/d84 0 2024-03-14T13:02:13.417 INFO:tasks.workunit.client.0.smithi111.stdout:5/433: read d2/d16/d49/f83 [1447255,82984] 0 2024-03-14T13:02:13.418 INFO:tasks.workunit.client.0.smithi111.stdout:2/282: mknod dd/d17/d53/c57 0 2024-03-14T13:02:13.419 INFO:tasks.workunit.client.1.smithi137.stdout:6/264: write d2/d54/d55/d33/f44 [1404035,107702] 0 2024-03-14T13:02:13.419 INFO:tasks.workunit.client.0.smithi111.stdout:1/370: mknod d20/d25/d31/d80/c81 0 2024-03-14T13:02:13.422 INFO:tasks.workunit.client.0.smithi111.stdout:6/294: unlink d0/d17/d24/c29 0 2024-03-14T13:02:13.423 INFO:tasks.workunit.client.1.smithi137.stdout:4/294: truncate d4/d27/f28 131643 0 2024-03-14T13:02:13.423 INFO:tasks.workunit.client.0.smithi111.stdout:8/529: creat d3/d22/d30/fa4 x:0 0 0 2024-03-14T13:02:13.425 INFO:tasks.workunit.client.1.smithi137.stdout:8/180: sync 2024-03-14T13:02:13.425 INFO:tasks.workunit.client.1.smithi137.stdout:2/248: mknod d2/d7/db/de/d1d/c55 0 2024-03-14T13:02:13.425 INFO:tasks.workunit.client.0.smithi111.stdout:0/304: fdatasync d6/d9/de/d32/f5e 0 2024-03-14T13:02:13.426 INFO:tasks.workunit.client.0.smithi111.stdout:4/589: dwrite d1d/d22/fad [0,4194304] 0 2024-03-14T13:02:13.426 INFO:tasks.workunit.client.0.smithi111.stdout:9/391: rmdir d5/d6/d8/d5b 39 2024-03-14T13:02:13.427 INFO:tasks.workunit.client.1.smithi137.stdout:5/175: creat d7/d16/f3e x:0 0 0 2024-03-14T13:02:13.427 INFO:tasks.workunit.client.1.smithi137.stdout:4/295: chown d4/f12 5 1 2024-03-14T13:02:13.427 INFO:tasks.workunit.client.0.smithi111.stdout:2/283: fdatasync dd/d11/d48/d49/f51 0 2024-03-14T13:02:13.429 INFO:tasks.workunit.client.1.smithi137.stdout:0/263: dwrite d1/d11/f50 [0,4194304] 0 2024-03-14T13:02:13.429 INFO:tasks.workunit.client.1.smithi137.stdout:9/343: mknod d6/df/d71/c7c 0 2024-03-14T13:02:13.433 INFO:tasks.workunit.client.0.smithi111.stdout:1/371: write d20/d25/f7c [734428,108361] 0 2024-03-14T13:02:13.433 INFO:tasks.workunit.client.1.smithi137.stdout:0/264: stat d1/f17 0 2024-03-14T13:02:13.433 INFO:tasks.workunit.client.1.smithi137.stdout:9/344: fsync d6/f57 0 2024-03-14T13:02:13.435 INFO:tasks.workunit.client.0.smithi111.stdout:2/284: chown dd/d11/d1a/f21 1919289616 1 2024-03-14T13:02:13.437 INFO:tasks.workunit.client.1.smithi137.stdout:1/346: dwrite d7/d9/d19/d1f/d26/d6c/f6f [0,4194304] 0 2024-03-14T13:02:13.437 INFO:tasks.workunit.client.0.smithi111.stdout:5/434: write d2/d16/d49/d44/d4a/f8c [1345580,68782] 0 2024-03-14T13:02:13.439 INFO:tasks.workunit.client.1.smithi137.stdout:9/345: read - d6/df/d19/d3a/f76 zero size 2024-03-14T13:02:13.439 INFO:tasks.workunit.client.1.smithi137.stdout:7/243: link d3/d5/d23/l29 d3/d6/d14/d22/l4b 0 2024-03-14T13:02:13.440 INFO:tasks.workunit.client.0.smithi111.stdout:2/285: write dd/d11/d1a/f1c [918005,82188] 0 2024-03-14T13:02:13.440 INFO:tasks.workunit.client.0.smithi111.stdout:8/530: symlink d3/d22/la5 0 2024-03-14T13:02:13.441 INFO:tasks.workunit.client.0.smithi111.stdout:4/590: creat d1d/d22/d30/d7c/d59/dc4/fce x:0 0 0 2024-03-14T13:02:13.444 INFO:tasks.workunit.client.0.smithi111.stdout:0/305: dwrite d6/d9/de/f53 [0,4194304] 0 2024-03-14T13:02:13.446 INFO:tasks.workunit.client.1.smithi137.stdout:7/244: write d3/d5/f11 [4501192,54566] 0 2024-03-14T13:02:13.447 INFO:tasks.workunit.client.0.smithi111.stdout:1/372: mknod d20/d25/d75/c82 0 2024-03-14T13:02:13.447 INFO:tasks.workunit.client.1.smithi137.stdout:8/181: rename d1/dc/d10/f35 to d1/f49 0 2024-03-14T13:02:13.447 INFO:tasks.workunit.client.1.smithi137.stdout:1/347: truncate d7/d9/d19/d40/f46 5038461 0 2024-03-14T13:02:13.448 INFO:tasks.workunit.client.0.smithi111.stdout:2/286: readlink dd/l14 0 2024-03-14T13:02:13.449 INFO:tasks.workunit.client.1.smithi137.stdout:2/249: creat d2/d7/d2e/d3e/d44/f56 x:0 0 0 2024-03-14T13:02:13.451 INFO:tasks.workunit.client.0.smithi111.stdout:6/295: rename d0/d17/d20/d4a/d59/f6d to d0/d17/d21/d69/d3d/f71 0 2024-03-14T13:02:13.452 INFO:tasks.workunit.client.1.smithi137.stdout:8/182: chown d1/dc/d11 3100137 1 2024-03-14T13:02:13.452 INFO:tasks.workunit.client.1.smithi137.stdout:9/346: write d6/f69 [556421,21390] 0 2024-03-14T13:02:13.453 INFO:tasks.workunit.client.1.smithi137.stdout:4/296: truncate d4/d9/f5d 686648 0 2024-03-14T13:02:13.457 INFO:tasks.workunit.client.0.smithi111.stdout:7/300: dread d0/d1/f28 [0,4194304] 0 2024-03-14T13:02:13.457 INFO:tasks.workunit.client.1.smithi137.stdout:1/348: symlink d7/d4d/l73 0 2024-03-14T13:02:13.458 INFO:tasks.workunit.client.0.smithi111.stdout:2/287: creat dd/d17/d34/f58 x:0 0 0 2024-03-14T13:02:13.458 INFO:tasks.workunit.client.0.smithi111.stdout:6/296: dread - d0/d17/d20/d4a/d59/f6f zero size 2024-03-14T13:02:13.458 INFO:tasks.workunit.client.1.smithi137.stdout:2/250: fdatasync d2/f28 0 2024-03-14T13:02:13.458 INFO:tasks.workunit.client.1.smithi137.stdout:9/347: chown d6/d53 3336 1 2024-03-14T13:02:13.459 INFO:tasks.workunit.client.0.smithi111.stdout:8/531: fdatasync d3/d5/f12 0 2024-03-14T13:02:13.459 INFO:tasks.workunit.client.0.smithi111.stdout:0/306: mknod d6/d9/d15/d55/c60 0 2024-03-14T13:02:13.461 INFO:tasks.workunit.client.0.smithi111.stdout:1/373: write d20/d40/f54 [195892,63949] 0 2024-03-14T13:02:13.463 INFO:tasks.workunit.client.1.smithi137.stdout:3/227: truncate d4/de/f2a 2388595 0 2024-03-14T13:02:13.464 INFO:tasks.workunit.client.0.smithi111.stdout:3/402: truncate d0/d30/f3d 2385975 0 2024-03-14T13:02:13.464 INFO:tasks.workunit.client.0.smithi111.stdout:1/374: readlink l5 0 2024-03-14T13:02:13.465 INFO:tasks.workunit.client.1.smithi137.stdout:8/183: mkdir d1/d4a 0 2024-03-14T13:02:13.465 INFO:tasks.workunit.client.1.smithi137.stdout:7/245: rename d3/d5/c30 to d3/d6/d14/d27/c4c 0 2024-03-14T13:02:13.466 INFO:tasks.workunit.client.0.smithi111.stdout:2/288: mkdir dd/d17/d34/d4f/d55/d59 0 2024-03-14T13:02:13.466 INFO:tasks.workunit.client.0.smithi111.stdout:8/532: truncate d3/dc/d45/f7c 438278 0 2024-03-14T13:02:13.466 INFO:tasks.workunit.client.1.smithi137.stdout:4/297: write d4/d10/d42/f4e [5971013,74568] 0 2024-03-14T13:02:13.466 INFO:tasks.workunit.client.1.smithi137.stdout:0/265: creat d1/d44/d56/f5f x:0 0 0 2024-03-14T13:02:13.467 INFO:tasks.workunit.client.1.smithi137.stdout:8/184: chown d1/d17/l3f 58 1 2024-03-14T13:02:13.468 INFO:tasks.workunit.client.0.smithi111.stdout:0/307: creat d6/d9/de/d3a/f61 x:0 0 0 2024-03-14T13:02:13.468 INFO:tasks.workunit.client.0.smithi111.stdout:3/403: chown d0/d2d/d35/d50 611097234 1 2024-03-14T13:02:13.469 INFO:tasks.workunit.client.0.smithi111.stdout:4/591: dwrite d1d/d22/d30/f3d [0,4194304] 0 2024-03-14T13:02:13.469 INFO:tasks.workunit.client.1.smithi137.stdout:5/176: dread d7/fb [0,4194304] 0 2024-03-14T13:02:13.469 INFO:tasks.workunit.client.0.smithi111.stdout:7/301: mknod d0/d1/d5/d56/c59 0 2024-03-14T13:02:13.470 INFO:tasks.workunit.client.0.smithi111.stdout:5/435: dread d2/f6 [0,4194304] 0 2024-03-14T13:02:13.470 INFO:tasks.workunit.client.1.smithi137.stdout:2/251: mkdir d2/d7/db/de/d22/d4d/d57 0 2024-03-14T13:02:13.471 INFO:tasks.workunit.client.1.smithi137.stdout:8/185: read d1/f31 [169779,112017] 0 2024-03-14T13:02:13.471 INFO:tasks.workunit.client.0.smithi111.stdout:4/592: stat d1d/d22/lb4 0 2024-03-14T13:02:13.472 INFO:tasks.workunit.client.0.smithi111.stdout:6/297: getdents d0/d17/d20/d4a/d68 0 2024-03-14T13:02:13.473 INFO:tasks.workunit.client.1.smithi137.stdout:0/266: read d1/f46 [1464476,126493] 0 2024-03-14T13:02:13.473 INFO:tasks.workunit.client.1.smithi137.stdout:4/298: mkdir d4/d9/d3a/d41/d6a 0 2024-03-14T13:02:13.474 INFO:tasks.workunit.client.0.smithi111.stdout:8/533: write d3/d1b/d20/f37 [405387,115484] 0 2024-03-14T13:02:13.474 INFO:tasks.workunit.client.0.smithi111.stdout:1/375: creat d20/d21/d3f/f83 x:0 0 0 2024-03-14T13:02:13.475 INFO:tasks.workunit.client.0.smithi111.stdout:3/404: write d0/d2d/d35/d50/d5e/d19/f26 [2178086,43893] 0 2024-03-14T13:02:13.476 INFO:tasks.workunit.client.1.smithi137.stdout:5/177: chown d7/d16/d18/d25/d28/f3d 377780 1 2024-03-14T13:02:13.477 INFO:tasks.workunit.client.0.smithi111.stdout:7/302: creat d0/d1/d4a/f5a x:0 0 0 2024-03-14T13:02:13.478 INFO:tasks.workunit.client.0.smithi111.stdout:6/298: readlink d0/d17/l4e 0 2024-03-14T13:02:13.479 INFO:tasks.workunit.client.1.smithi137.stdout:6/265: dread d2/f25 [0,4194304] 0 2024-03-14T13:02:13.479 INFO:tasks.workunit.client.1.smithi137.stdout:9/348: getdents d6/d56 0 2024-03-14T13:02:13.479 INFO:tasks.workunit.client.0.smithi111.stdout:7/303: readlink d0/d11/l4f 0 2024-03-14T13:02:13.480 INFO:tasks.workunit.client.1.smithi137.stdout:1/349: rename d7/d9/ff to d7/d9/d19/f74 0 2024-03-14T13:02:13.480 INFO:tasks.workunit.client.0.smithi111.stdout:8/534: symlink d3/dc/d31/la6 0 2024-03-14T13:02:13.480 INFO:tasks.workunit.client.0.smithi111.stdout:1/376: truncate d20/d21/d3f/d59/f6d 2600306 0 2024-03-14T13:02:13.481 INFO:tasks.workunit.client.0.smithi111.stdout:9/392: truncate f2 520871 0 2024-03-14T13:02:13.482 INFO:tasks.workunit.client.1.smithi137.stdout:8/186: creat d1/dc/f4b x:0 0 0 2024-03-14T13:02:13.483 INFO:tasks.workunit.client.0.smithi111.stdout:8/535: chown d3/d22/l55 13937 1 2024-03-14T13:02:13.484 INFO:tasks.workunit.client.0.smithi111.stdout:6/299: chown d0/c5 79 1 2024-03-14T13:02:13.484 INFO:tasks.workunit.client.1.smithi137.stdout:2/252: dread d2/d7/db/de/f39 [0,4194304] 0 2024-03-14T13:02:13.485 INFO:tasks.workunit.client.0.smithi111.stdout:3/405: dread d0/fd [0,4194304] 0 2024-03-14T13:02:13.487 INFO:tasks.workunit.client.0.smithi111.stdout:0/308: link d6/d9/de/f23 d6/d9/d15/d49/f62 0 2024-03-14T13:02:13.488 INFO:tasks.workunit.client.1.smithi137.stdout:1/350: chown d7/d9/d19/d40 1771 1 2024-03-14T13:02:13.490 INFO:tasks.workunit.client.0.smithi111.stdout:2/289: rename dd/d11/d1a/f43 to dd/d11/f5a 0 2024-03-14T13:02:13.490 INFO:tasks.workunit.client.1.smithi137.stdout:2/253: dread d2/f4b [0,4194304] 0 2024-03-14T13:02:13.490 INFO:tasks.workunit.client.1.smithi137.stdout:9/349: write d6/df/d19/f45 [1987373,18880] 0 2024-03-14T13:02:13.491 INFO:tasks.workunit.client.0.smithi111.stdout:3/406: truncate d0/d2d/d35/d4c/d82/f77 435112 0 2024-03-14T13:02:13.492 INFO:tasks.workunit.client.0.smithi111.stdout:1/377: unlink d20/d25/f30 0 2024-03-14T13:02:13.494 INFO:tasks.workunit.client.0.smithi111.stdout:8/536: creat d3/dc/fa7 x:0 0 0 2024-03-14T13:02:13.495 INFO:tasks.workunit.client.0.smithi111.stdout:9/393: dwrite d5/d6/d2a/f75 [0,4194304] 0 2024-03-14T13:02:13.496 INFO:tasks.workunit.client.1.smithi137.stdout:5/178: creat d7/d16/d18/d25/f3f x:0 0 0 2024-03-14T13:02:13.497 INFO:tasks.workunit.client.0.smithi111.stdout:9/394: chown d5/f65 1 1 2024-03-14T13:02:13.499 INFO:tasks.workunit.client.1.smithi137.stdout:2/254: dwrite d2/f5 [4194304,4194304] 0 2024-03-14T13:02:13.499 INFO:tasks.workunit.client.0.smithi111.stdout:0/309: dread d6/d9/de/f3b [0,4194304] 0 2024-03-14T13:02:13.500 INFO:tasks.workunit.client.1.smithi137.stdout:2/255: chown d2/d34/c46 115 1 2024-03-14T13:02:13.501 INFO:tasks.workunit.client.1.smithi137.stdout:7/246: rename d3/d6/c33 to d3/d6/d14/d27/c4d 0 2024-03-14T13:02:13.502 INFO:tasks.workunit.client.1.smithi137.stdout:3/228: truncate d4/fd 4153495 0 2024-03-14T13:02:13.504 INFO:tasks.workunit.client.0.smithi111.stdout:9/395: dread d5/d6/d8/f3a [0,4194304] 0 2024-03-14T13:02:13.504 INFO:tasks.workunit.client.0.smithi111.stdout:0/310: dread d6/d9/d15/f37 [0,4194304] 0 2024-03-14T13:02:13.506 INFO:tasks.workunit.client.0.smithi111.stdout:1/378: creat d20/d40/f84 x:0 0 0 2024-03-14T13:02:13.509 INFO:tasks.workunit.client.0.smithi111.stdout:8/537: chown d3/d1b/d20/d83/f63 1562 1 2024-03-14T13:02:13.511 INFO:tasks.workunit.client.1.smithi137.stdout:6/266: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:13.511 INFO:tasks.workunit.client.1.smithi137.stdout:1/351: symlink d7/d9/d19/d1f/d22/l75 0 2024-03-14T13:02:13.512 INFO:tasks.workunit.client.0.smithi111.stdout:2/290: creat dd/d11/d1a/d56/f5b x:0 0 0 2024-03-14T13:02:13.514 INFO:tasks.workunit.client.1.smithi137.stdout:0/267: creat d1/d5/d1b/d1e/d25/d55/f60 x:0 0 0 2024-03-14T13:02:13.516 INFO:tasks.workunit.client.0.smithi111.stdout:5/436: dwrite d2/f6 [0,4194304] 0 2024-03-14T13:02:13.516 INFO:tasks.workunit.client.1.smithi137.stdout:7/247: dwrite d3/f32 [4194304,4194304] 0 2024-03-14T13:02:13.517 INFO:tasks.workunit.client.0.smithi111.stdout:2/291: write dd/d17/d34/f58 [353498,14322] 0 2024-03-14T13:02:13.517 INFO:tasks.workunit.client.0.smithi111.stdout:0/311: chown d6/d9/de/f2c 85 1 2024-03-14T13:02:13.519 INFO:tasks.workunit.client.1.smithi137.stdout:4/299: write d4/d9/d3a/d41/d38/f53 [1031746,102298] 0 2024-03-14T13:02:13.519 INFO:tasks.workunit.client.0.smithi111.stdout:2/292: chown dd/l39 23 1 2024-03-14T13:02:13.520 INFO:tasks.workunit.client.1.smithi137.stdout:2/256: symlink d2/d34/d3c/d53/l58 0 2024-03-14T13:02:13.521 INFO:tasks.workunit.client.0.smithi111.stdout:7/304: dread d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:13.521 INFO:tasks.workunit.client.1.smithi137.stdout:3/229: creat d4/df/d34/f44 x:0 0 0 2024-03-14T13:02:13.522 INFO:tasks.workunit.client.1.smithi137.stdout:2/257: write d2/f5 [6773301,32965] 0 2024-03-14T13:02:13.523 INFO:tasks.workunit.client.1.smithi137.stdout:1/352: dread d7/d9/d19/f2f [0,4194304] 0 2024-03-14T13:02:13.525 INFO:tasks.workunit.client.1.smithi137.stdout:8/187: rename d1/d17/c37 to d1/dc/d11/d13/c4c 0 2024-03-14T13:02:13.527 INFO:tasks.workunit.client.1.smithi137.stdout:7/248: mknod d3/d5/d23/d3f/c4e 0 2024-03-14T13:02:13.527 INFO:tasks.workunit.client.1.smithi137.stdout:0/268: dread d1/d11/f50 [0,4194304] 0 2024-03-14T13:02:13.528 INFO:tasks.workunit.client.1.smithi137.stdout:6/267: rename d2/d6 to d2/d6/de/d45/d59/d39/d3c/d43/d57/d58/d62 22 2024-03-14T13:02:13.528 INFO:tasks.workunit.client.0.smithi111.stdout:4/593: rename d1d/d22/d30/d87/fc1 to d1d/d22/d40/fcf 0 2024-03-14T13:02:13.528 INFO:tasks.workunit.client.0.smithi111.stdout:6/300: write d0/f57 [4863909,128221] 0 2024-03-14T13:02:13.529 INFO:tasks.workunit.client.0.smithi111.stdout:2/293: dwrite dd/d17/f31 [0,4194304] 0 2024-03-14T13:02:13.529 INFO:tasks.workunit.client.1.smithi137.stdout:8/188: write d1/dc/d11/d13/f3e [1214791,38073] 0 2024-03-14T13:02:13.529 INFO:tasks.workunit.client.1.smithi137.stdout:0/269: readlink d1/d1f/l2c 0 2024-03-14T13:02:13.530 INFO:tasks.workunit.client.1.smithi137.stdout:3/230: creat d4/d22/f45 x:0 0 0 2024-03-14T13:02:13.530 INFO:tasks.workunit.client.1.smithi137.stdout:5/179: rmdir d7 39 2024-03-14T13:02:13.532 INFO:tasks.workunit.client.1.smithi137.stdout:9/350: creat d6/df/f7d x:0 0 0 2024-03-14T13:02:13.533 INFO:tasks.workunit.client.1.smithi137.stdout:0/270: read d1/d1f/f24 [803682,13794] 0 2024-03-14T13:02:13.533 INFO:tasks.workunit.client.1.smithi137.stdout:8/189: truncate d1/dc/f1f 624859 0 2024-03-14T13:02:13.535 INFO:tasks.workunit.client.0.smithi111.stdout:4/594: dread - d1d/d22/d30/d7c/fa1 zero size 2024-03-14T13:02:13.538 INFO:tasks.workunit.client.0.smithi111.stdout:0/312: chown d6/d9/f12 223 1 2024-03-14T13:02:13.538 INFO:tasks.workunit.client.0.smithi111.stdout:1/379: truncate d20/f24 1706773 0 2024-03-14T13:02:13.538 INFO:tasks.workunit.client.1.smithi137.stdout:9/351: dread d6/df/d19/d3a/d40/f60 [0,4194304] 0 2024-03-14T13:02:13.541 INFO:tasks.workunit.client.1.smithi137.stdout:1/353: rename d7/d9/d19/d40/f61 to d7/d31/f76 0 2024-03-14T13:02:13.541 INFO:tasks.workunit.client.1.smithi137.stdout:9/352: write d6/df/d19/d1f/f35 [1174416,54394] 0 2024-03-14T13:02:13.541 INFO:tasks.workunit.client.0.smithi111.stdout:7/305: rename d0/d1/f2 to d0/d1/d21/d3a/f5b 0 2024-03-14T13:02:13.542 INFO:tasks.workunit.client.0.smithi111.stdout:3/407: link d0/f1 d0/d2d/d35/d69/f9d 0 2024-03-14T13:02:13.544 INFO:tasks.workunit.client.0.smithi111.stdout:5/437: creat d2/d13/f93 x:0 0 0 2024-03-14T13:02:13.545 INFO:tasks.workunit.client.1.smithi137.stdout:3/231: symlink d4/de/d16/d1c/l46 0 2024-03-14T13:02:13.545 INFO:tasks.workunit.client.1.smithi137.stdout:0/271: dwrite d1/d11/d4a/d54/f5b [0,4194304] 0 2024-03-14T13:02:13.546 INFO:tasks.workunit.client.0.smithi111.stdout:8/538: link d3/d1b/c8a d3/d1b/d20/d35/d40/d48/d77/ca8 0 2024-03-14T13:02:13.547 INFO:tasks.workunit.client.0.smithi111.stdout:0/313: write d6/fd [9399917,97181] 0 2024-03-14T13:02:13.548 INFO:tasks.workunit.client.0.smithi111.stdout:1/380: dread fa [4194304,4194304] 0 2024-03-14T13:02:13.552 INFO:tasks.workunit.client.0.smithi111.stdout:7/306: chown d0/d1/d5/d38/c27 1488 1 2024-03-14T13:02:13.553 INFO:tasks.workunit.client.0.smithi111.stdout:3/408: truncate d0/d2d/d35/d69/d84/f99 43477 0 2024-03-14T13:02:13.553 INFO:tasks.workunit.client.0.smithi111.stdout:1/381: dread - d20/d25/f3d zero size 2024-03-14T13:02:13.554 INFO:tasks.workunit.client.0.smithi111.stdout:4/595: rename d1d/d22/d30/d5e/ca3 to d1d/d22/dc6/cd0 0 2024-03-14T13:02:13.556 INFO:tasks.workunit.client.0.smithi111.stdout:2/294: getdents dd/d11 0 2024-03-14T13:02:13.556 INFO:tasks.workunit.client.1.smithi137.stdout:9/353: creat d6/df/d19/d1f/f7e x:0 0 0 2024-03-14T13:02:13.557 INFO:tasks.workunit.client.1.smithi137.stdout:1/354: write d7/d4d/f56 [6701799,27389] 0 2024-03-14T13:02:13.557 INFO:tasks.workunit.client.0.smithi111.stdout:4/596: fdatasync d1d/d22/f41 0 2024-03-14T13:02:13.558 INFO:tasks.workunit.client.1.smithi137.stdout:7/249: link d3/d6/d2e/f49 d3/d6/d2e/f4f 0 2024-03-14T13:02:13.558 INFO:tasks.workunit.client.0.smithi111.stdout:8/539: stat d3/d1b/d20/d35/d40/d48/d77/c96 0 2024-03-14T13:02:13.558 INFO:tasks.workunit.client.0.smithi111.stdout:6/301: dwrite d0/d17/d21/f44 [0,4194304] 0 2024-03-14T13:02:13.561 INFO:tasks.workunit.client.0.smithi111.stdout:8/540: symlink d3/dc/la9 0 2024-03-14T13:02:13.561 INFO:tasks.workunit.client.1.smithi137.stdout:3/232: mknod d4/d22/c47 0 2024-03-14T13:02:13.562 INFO:tasks.workunit.client.1.smithi137.stdout:0/272: mkdir d1/d5/d1b/d42/d61 0 2024-03-14T13:02:13.564 INFO:tasks.workunit.client.1.smithi137.stdout:3/233: readlink d4/d19/l24 0 2024-03-14T13:02:13.566 INFO:tasks.workunit.client.1.smithi137.stdout:0/273: dread - d1/d44/f4c zero size 2024-03-14T13:02:13.570 INFO:tasks.workunit.client.1.smithi137.stdout:5/180: dwrite d7/f15 [0,4194304] 0 2024-03-14T13:02:13.570 INFO:tasks.workunit.client.1.smithi137.stdout:6/268: getdents d2 0 2024-03-14T13:02:13.570 INFO:tasks.workunit.client.1.smithi137.stdout:0/274: stat d1/d11/d4a/d54/f5b 0 2024-03-14T13:02:13.571 INFO:tasks.workunit.client.1.smithi137.stdout:7/250: mknod d3/d5/d23/d3f/c50 0 2024-03-14T13:02:13.573 INFO:tasks.workunit.client.1.smithi137.stdout:1/355: mknod d7/d9/d19/d1f/d26/c77 0 2024-03-14T13:02:13.573 INFO:tasks.workunit.client.1.smithi137.stdout:5/181: chown d7/d16/d18/f20 21217835 1 2024-03-14T13:02:13.574 INFO:tasks.workunit.client.1.smithi137.stdout:3/234: mknod d4/d22/c48 0 2024-03-14T13:02:13.575 INFO:tasks.workunit.client.1.smithi137.stdout:4/300: truncate d4/d9/f5b 526099 0 2024-03-14T13:02:13.578 INFO:tasks.workunit.client.1.smithi137.stdout:4/301: readlink d4/d10/l23 0 2024-03-14T13:02:13.578 INFO:tasks.workunit.client.1.smithi137.stdout:3/235: chown d4/de/f3f 79496 1 2024-03-14T13:02:13.580 INFO:tasks.workunit.client.1.smithi137.stdout:3/236: dread - d4/df/d35/f43 zero size 2024-03-14T13:02:13.581 INFO:tasks.workunit.client.1.smithi137.stdout:3/237: read - d4/f37 zero size 2024-03-14T13:02:13.584 INFO:tasks.workunit.client.1.smithi137.stdout:6/269: dwrite d2/d6/f3d [0,4194304] 0 2024-03-14T13:02:13.585 INFO:tasks.workunit.client.1.smithi137.stdout:4/302: creat d4/d4d/f6b x:0 0 0 2024-03-14T13:02:13.587 INFO:tasks.workunit.client.1.smithi137.stdout:7/251: symlink d3/d6/l51 0 2024-03-14T13:02:13.587 INFO:tasks.workunit.client.1.smithi137.stdout:6/270: chown d2/d6/c19 0 1 2024-03-14T13:02:13.588 INFO:tasks.workunit.client.1.smithi137.stdout:6/271: chown d2/d6/de/d45/d59/f40 262661286 1 2024-03-14T13:02:13.588 INFO:tasks.workunit.client.1.smithi137.stdout:6/272: chown l1 722 1 2024-03-14T13:02:13.592 INFO:tasks.workunit.client.1.smithi137.stdout:5/182: symlink d7/d16/d18/l40 0 2024-03-14T13:02:13.593 INFO:tasks.workunit.client.1.smithi137.stdout:0/275: link d1/d5/l47 d1/d5/d1b/d1e/d25/d55/l62 0 2024-03-14T13:02:13.594 INFO:tasks.workunit.client.1.smithi137.stdout:3/238: rename d4/f2b to d4/df/d34/f49 0 2024-03-14T13:02:13.600 INFO:tasks.workunit.client.1.smithi137.stdout:6/273: unlink d2/d54/l30 0 2024-03-14T13:02:13.600 INFO:tasks.workunit.client.1.smithi137.stdout:1/356: creat d7/d9/d19/d1f/f78 x:0 0 0 2024-03-14T13:02:13.601 INFO:tasks.workunit.client.1.smithi137.stdout:4/303: rename d4/d10/f25 to d4/d10/d42/f6c 0 2024-03-14T13:02:13.603 INFO:tasks.workunit.client.1.smithi137.stdout:7/252: mkdir d3/d52 0 2024-03-14T13:02:13.607 INFO:tasks.workunit.client.1.smithi137.stdout:9/354: truncate d6/df/d71/f72 217499 0 2024-03-14T13:02:13.610 INFO:tasks.workunit.client.1.smithi137.stdout:1/357: creat d7/d9/d19/d1f/d2e/f79 x:0 0 0 2024-03-14T13:02:13.610 INFO:tasks.workunit.client.1.smithi137.stdout:7/253: dread d3/d6/d2e/f49 [0,4194304] 0 2024-03-14T13:02:13.611 INFO:tasks.workunit.client.1.smithi137.stdout:0/276: rename d1/d5/d1b/d1e/d53/l5e to d1/d5/l63 0 2024-03-14T13:02:13.613 INFO:tasks.workunit.client.1.smithi137.stdout:9/355: creat d6/df/d19/d3a/d40/f7f x:0 0 0 2024-03-14T13:02:13.614 INFO:tasks.workunit.client.1.smithi137.stdout:6/274: truncate d2/d6/de/d45/d59/d39/d3c/f47 976562 0 2024-03-14T13:02:13.615 INFO:tasks.workunit.client.1.smithi137.stdout:7/254: creat d3/d5/d23/d2c/f53 x:0 0 0 2024-03-14T13:02:13.615 INFO:tasks.workunit.client.1.smithi137.stdout:9/356: stat d6/df/d19/l23 0 2024-03-14T13:02:13.615 INFO:tasks.workunit.client.0.smithi111.stdout:7/307: mkdir d0/d1/d5/d38/d5c 0 2024-03-14T13:02:13.617 INFO:tasks.workunit.client.1.smithi137.stdout:3/239: rename d4/d22/c2c to d4/de/d16/d1c/c4a 0 2024-03-14T13:02:13.618 INFO:tasks.workunit.client.1.smithi137.stdout:0/277: mknod d1/d5/d1b/c64 0 2024-03-14T13:02:13.618 INFO:tasks.workunit.client.1.smithi137.stdout:9/357: dread - d6/df/d19/d1f/f7e zero size 2024-03-14T13:02:13.619 INFO:tasks.workunit.client.1.smithi137.stdout:5/183: dread d7/d16/f1a [0,4194304] 0 2024-03-14T13:02:13.620 INFO:tasks.workunit.client.1.smithi137.stdout:6/275: mknod d2/d6/de/c63 0 2024-03-14T13:02:13.621 INFO:tasks.workunit.client.1.smithi137.stdout:3/240: dread d4/de/d16/f20 [0,4194304] 0 2024-03-14T13:02:13.623 INFO:tasks.workunit.client.1.smithi137.stdout:4/304: rename d4/d10/d3d/d54 to d4/d27/d5c/d6d 0 2024-03-14T13:02:13.623 INFO:tasks.workunit.client.1.smithi137.stdout:8/190: dread d1/d17/d34/f47 [0,4194304] 0 2024-03-14T13:02:13.623 INFO:tasks.workunit.client.1.smithi137.stdout:7/255: mkdir d3/d54 0 2024-03-14T13:02:13.624 INFO:tasks.workunit.client.1.smithi137.stdout:0/278: creat d1/d44/f65 x:0 0 0 2024-03-14T13:02:13.625 INFO:tasks.workunit.client.0.smithi111.stdout:2/295: rename dd/d11/f5a to dd/d11/d48/d49/f5c 0 2024-03-14T13:02:13.625 INFO:tasks.workunit.client.0.smithi111.stdout:5/438: dread d2/d13/f29 [0,4194304] 0 2024-03-14T13:02:13.626 INFO:tasks.workunit.client.1.smithi137.stdout:5/184: mkdir d7/d16/d41 0 2024-03-14T13:02:13.626 INFO:tasks.workunit.client.1.smithi137.stdout:4/305: write d4/d9/d3a/d41/d38/d55/f59 [642620,117020] 0 2024-03-14T13:02:13.628 INFO:tasks.workunit.client.0.smithi111.stdout:3/409: dread d0/d2d/d35/d50/d5e/d40/d6f/f79 [0,4194304] 0 2024-03-14T13:02:13.629 INFO:tasks.workunit.client.1.smithi137.stdout:3/241: unlink d4/de/d13/f40 0 2024-03-14T13:02:13.630 INFO:tasks.workunit.client.1.smithi137.stdout:9/358: creat d6/df/d19/d33/f80 x:0 0 0 2024-03-14T13:02:13.630 INFO:tasks.workunit.client.0.smithi111.stdout:1/382: dwrite d20/d21/d3f/d59/f6d [0,4194304] 0 2024-03-14T13:02:13.631 INFO:tasks.workunit.client.0.smithi111.stdout:3/410: chown d0/d2d/d35/d50/d5e 1895065 1 2024-03-14T13:02:13.631 INFO:tasks.workunit.client.0.smithi111.stdout:9/396: dread d5/d6/f2f [0,4194304] 0 2024-03-14T13:02:13.631 INFO:tasks.workunit.client.1.smithi137.stdout:6/276: creat d2/d54/d55/d33/f64 x:0 0 0 2024-03-14T13:02:13.633 INFO:tasks.workunit.client.1.smithi137.stdout:8/191: write d1/f6 [756101,35012] 0 2024-03-14T13:02:13.634 INFO:tasks.workunit.client.1.smithi137.stdout:7/256: symlink d3/d5/d23/l55 0 2024-03-14T13:02:13.635 INFO:tasks.workunit.client.1.smithi137.stdout:7/257: stat d3/d6/f15 0 2024-03-14T13:02:13.636 INFO:tasks.workunit.client.1.smithi137.stdout:0/279: mknod d1/d1f/c66 0 2024-03-14T13:02:13.637 INFO:tasks.workunit.client.1.smithi137.stdout:7/258: dread d3/d6/d2e/f49 [0,4194304] 0 2024-03-14T13:02:13.638 INFO:tasks.workunit.client.1.smithi137.stdout:5/185: symlink d7/d16/l42 0 2024-03-14T13:02:13.638 INFO:tasks.workunit.client.0.smithi111.stdout:4/597: write d1d/d22/d28/f4b [609944,109253] 0 2024-03-14T13:02:13.640 INFO:tasks.workunit.client.1.smithi137.stdout:4/306: write d4/d10/f3f [2165107,128511] 0 2024-03-14T13:02:13.641 INFO:tasks.workunit.client.0.smithi111.stdout:5/439: mknod d2/d1f/d6a/c94 0 2024-03-14T13:02:13.642 INFO:tasks.workunit.client.1.smithi137.stdout:4/307: chown d4/d9/l4a 462 1 2024-03-14T13:02:13.643 INFO:tasks.workunit.client.0.smithi111.stdout:3/411: creat d0/d91/f9e x:0 0 0 2024-03-14T13:02:13.644 INFO:tasks.workunit.client.0.smithi111.stdout:4/598: truncate d1d/d22/d30/f32 1761271 0 2024-03-14T13:02:13.645 INFO:tasks.workunit.client.1.smithi137.stdout:4/308: read d4/d10/d42/f52 [206840,109003] 0 2024-03-14T13:02:13.645 INFO:tasks.workunit.client.1.smithi137.stdout:3/242: mkdir d4/df/d35/d4b 0 2024-03-14T13:02:13.645 INFO:tasks.workunit.client.1.smithi137.stdout:9/359: readlink d6/df/l42 0 2024-03-14T13:02:13.645 INFO:tasks.workunit.client.0.smithi111.stdout:1/383: symlink d20/d25/d75/l85 0 2024-03-14T13:02:13.645 INFO:tasks.workunit.client.0.smithi111.stdout:8/541: dread d3/d1b/d20/d35/d40/d48/f68 [0,4194304] 0 2024-03-14T13:02:13.647 INFO:tasks.workunit.client.0.smithi111.stdout:7/308: mkdir d0/d1/d21/d5d 0 2024-03-14T13:02:13.647 INFO:tasks.workunit.client.0.smithi111.stdout:6/302: dwrite d0/f1a [4194304,4194304] 0 2024-03-14T13:02:13.648 INFO:tasks.workunit.client.0.smithi111.stdout:1/384: fdatasync d20/d21/d3f/f83 0 2024-03-14T13:02:13.650 INFO:tasks.workunit.client.0.smithi111.stdout:2/296: write dd/d11/d1a/f1c [1514402,85356] 0 2024-03-14T13:02:13.651 INFO:tasks.workunit.client.0.smithi111.stdout:6/303: write d0/f4 [3627805,34230] 0 2024-03-14T13:02:13.654 INFO:tasks.workunit.client.0.smithi111.stdout:3/412: symlink d0/d2d/d35/d50/d6c/l9f 0 2024-03-14T13:02:13.656 INFO:tasks.workunit.client.0.smithi111.stdout:8/542: creat d3/d1b/d20/d35/d40/faa x:0 0 0 2024-03-14T13:02:13.658 INFO:tasks.workunit.client.1.smithi137.stdout:4/309: dwrite d4/d9/d3a/d41/d38/f4c [0,4194304] 0 2024-03-14T13:02:13.660 INFO:tasks.workunit.client.0.smithi111.stdout:3/413: creat d0/d2d/d35/d69/fa0 x:0 0 0 2024-03-14T13:02:13.661 INFO:tasks.workunit.client.0.smithi111.stdout:8/543: read d3/d1b/f92 [3447340,15080] 0 2024-03-14T13:02:13.661 INFO:tasks.workunit.client.0.smithi111.stdout:7/309: rename d0/d1/d5/d38/d9/f12 to d0/d1/d5/d56/f5e 0 2024-03-14T13:02:13.664 INFO:tasks.workunit.client.1.smithi137.stdout:7/259: rename d3/d6/d2e/c31 to d3/d6/c56 0 2024-03-14T13:02:13.665 INFO:tasks.workunit.client.1.smithi137.stdout:9/360: dwrite d6/df/d19/d3a/d40/f41 [0,4194304] 0 2024-03-14T13:02:13.665 INFO:tasks.workunit.client.0.smithi111.stdout:9/397: dwrite d5/d42/f79 [0,4194304] 0 2024-03-14T13:02:13.666 INFO:tasks.workunit.client.1.smithi137.stdout:4/310: read f2 [819776,71146] 0 2024-03-14T13:02:13.668 INFO:tasks.workunit.client.1.smithi137.stdout:7/260: write d3/d5/f46 [322797,12516] 0 2024-03-14T13:02:13.670 INFO:tasks.workunit.client.0.smithi111.stdout:7/310: chown d0/d1/d5/d38/d9/fb 6026525 1 2024-03-14T13:02:13.675 INFO:tasks.workunit.client.0.smithi111.stdout:2/297: dwrite dd/d17/d34/f42 [0,4194304] 0 2024-03-14T13:02:13.676 INFO:tasks.workunit.client.0.smithi111.stdout:6/304: dwrite d0/d17/d21/f2d [0,4194304] 0 2024-03-14T13:02:13.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/398: write d5/d6/d14/f81 [87629,112696] 0 2024-03-14T13:02:13.679 INFO:tasks.workunit.client.0.smithi111.stdout:4/599: dwrite d1d/f26 [0,4194304] 0 2024-03-14T13:02:13.680 INFO:tasks.workunit.client.0.smithi111.stdout:3/414: dread d0/d30/f48 [0,4194304] 0 2024-03-14T13:02:13.681 INFO:tasks.workunit.client.0.smithi111.stdout:4/600: read - d1d/d22/d30/d5e/d6f/d7e/fc8 zero size 2024-03-14T13:02:13.682 INFO:tasks.workunit.client.0.smithi111.stdout:2/298: creat dd/d17/d34/d4f/d55/f5d x:0 0 0 2024-03-14T13:02:13.683 INFO:tasks.workunit.client.0.smithi111.stdout:2/299: readlink dd/l29 0 2024-03-14T13:02:13.684 INFO:tasks.workunit.client.0.smithi111.stdout:4/601: fsync d1d/d22/d40/d69/d86/db1/fbf 0 2024-03-14T13:02:13.687 INFO:tasks.workunit.client.0.smithi111.stdout:2/300: rename dd/d17/d34/f58 to dd/d11/d48/f5e 0 2024-03-14T13:02:13.688 INFO:tasks.workunit.client.0.smithi111.stdout:4/602: mkdir d1d/d22/d30/d87/dd1 0 2024-03-14T13:02:13.689 INFO:tasks.workunit.client.0.smithi111.stdout:3/415: dread d0/f1 [0,4194304] 0 2024-03-14T13:02:13.691 INFO:tasks.workunit.client.1.smithi137.stdout:9/361: mknod d6/d53/c81 0 2024-03-14T13:02:13.692 INFO:tasks.workunit.client.1.smithi137.stdout:4/311: read f3 [1685735,9854] 0 2024-03-14T13:02:13.701 INFO:tasks.workunit.client.1.smithi137.stdout:5/186: rename d7/fa to d7/d16/d18/d25/d28/f43 0 2024-03-14T13:02:13.702 INFO:tasks.workunit.client.1.smithi137.stdout:8/192: write d1/dc/d11/d13/f45 [315516,81353] 0 2024-03-14T13:02:13.702 INFO:tasks.workunit.client.1.smithi137.stdout:7/261: creat d3/d6/d14/f57 x:0 0 0 2024-03-14T13:02:13.702 INFO:tasks.workunit.client.0.smithi111.stdout:4/603: dread d1d/d22/d28/fb6 [0,4194304] 0 2024-03-14T13:02:13.703 INFO:tasks.workunit.client.1.smithi137.stdout:9/362: mknod d6/df/d19/d1f/c82 0 2024-03-14T13:02:13.704 INFO:tasks.workunit.client.0.smithi111.stdout:4/604: chown d1d/d22/d40/fcf 204 1 2024-03-14T13:02:13.704 INFO:tasks.workunit.client.1.smithi137.stdout:6/277: write d2/d54/d55/f1f [2094739,16257] 0 2024-03-14T13:02:13.707 INFO:tasks.workunit.client.0.smithi111.stdout:4/605: getdents d1d/d22/d40/d69/d86 0 2024-03-14T13:02:13.708 INFO:tasks.workunit.client.1.smithi137.stdout:3/243: truncate f1 3727205 0 2024-03-14T13:02:13.708 INFO:tasks.workunit.client.1.smithi137.stdout:4/312: mkdir d4/d4d/d6e 0 2024-03-14T13:02:13.712 INFO:tasks.workunit.client.1.smithi137.stdout:9/363: rmdir d6/df/d71 39 2024-03-14T13:02:13.713 INFO:tasks.workunit.client.0.smithi111.stdout:6/305: truncate d0/f6e 2430631 0 2024-03-14T13:02:13.715 INFO:tasks.workunit.client.1.smithi137.stdout:3/244: creat d4/d22/f4c x:0 0 0 2024-03-14T13:02:13.715 INFO:tasks.workunit.client.0.smithi111.stdout:2/301: rmdir dd/d17/d34 39 2024-03-14T13:02:13.716 INFO:tasks.workunit.client.1.smithi137.stdout:5/187: mknod d7/c44 0 2024-03-14T13:02:13.717 INFO:tasks.workunit.client.0.smithi111.stdout:6/306: creat d0/f72 x:0 0 0 2024-03-14T13:02:13.718 INFO:tasks.workunit.client.0.smithi111.stdout:4/606: dwrite d1d/d22/d23/f67 [0,4194304] 0 2024-03-14T13:02:13.719 INFO:tasks.workunit.client.1.smithi137.stdout:3/245: rmdir d4/d19 39 2024-03-14T13:02:13.719 INFO:tasks.workunit.client.0.smithi111.stdout:3/416: write d0/d2d/d35/d50/d5e/f37 [615900,130335] 0 2024-03-14T13:02:13.721 INFO:tasks.workunit.client.0.smithi111.stdout:2/302: creat dd/d17/d53/f5f x:0 0 0 2024-03-14T13:02:13.721 INFO:tasks.workunit.client.0.smithi111.stdout:6/307: mknod d0/d17/c73 0 2024-03-14T13:02:13.722 INFO:tasks.workunit.client.1.smithi137.stdout:6/278: link d2/d6/f3d d2/d6/f65 0 2024-03-14T13:02:13.723 INFO:tasks.workunit.client.0.smithi111.stdout:7/311: dwrite d0/d1/d5/d38/d9/f3d [0,4194304] 0 2024-03-14T13:02:13.723 INFO:tasks.workunit.client.0.smithi111.stdout:2/303: chown dd/d17/f31 28109177 1 2024-03-14T13:02:13.724 INFO:tasks.workunit.client.0.smithi111.stdout:3/417: symlink d0/d2d/d35/la1 0 2024-03-14T13:02:13.727 INFO:tasks.workunit.client.0.smithi111.stdout:6/308: truncate d0/d50/f67 215139 0 2024-03-14T13:02:13.727 INFO:tasks.workunit.client.0.smithi111.stdout:2/304: dread - dd/d11/d1a/d56/f5b zero size 2024-03-14T13:02:13.727 INFO:tasks.workunit.client.1.smithi137.stdout:6/279: chown d2/d54/d55/d33/f64 32 1 2024-03-14T13:02:13.727 INFO:tasks.workunit.client.1.smithi137.stdout:9/364: creat d6/df/d19/f83 x:0 0 0 2024-03-14T13:02:13.729 INFO:tasks.workunit.client.1.smithi137.stdout:6/280: readlink d2/d6/l27 0 2024-03-14T13:02:13.730 INFO:tasks.workunit.client.1.smithi137.stdout:5/188: dwrite d7/d16/d18/f39 [0,4194304] 0 2024-03-14T13:02:13.731 INFO:tasks.workunit.client.0.smithi111.stdout:7/312: rename d0/d1/d5/d56/c59 to d0/d1/d4a/c5f 0 2024-03-14T13:02:13.731 INFO:tasks.workunit.client.0.smithi111.stdout:7/313: write d0/f2e [1185135,121606] 0 2024-03-14T13:02:13.731 INFO:tasks.workunit.client.0.smithi111.stdout:7/314: fdatasync d0/d1/d4a/f4b 0 2024-03-14T13:02:13.732 INFO:tasks.workunit.client.1.smithi137.stdout:5/189: chown d7/d16/d18/f34 1225 1 2024-03-14T13:02:13.732 INFO:tasks.workunit.client.0.smithi111.stdout:6/309: creat d0/f74 x:0 0 0 2024-03-14T13:02:13.732 INFO:tasks.workunit.client.0.smithi111.stdout:3/418: mkdir d0/d2d/d35/d50/da2 0 2024-03-14T13:02:13.733 INFO:tasks.workunit.client.1.smithi137.stdout:9/365: write d6/df/d19/d33/f80 [745866,26996] 0 2024-03-14T13:02:13.733 INFO:tasks.workunit.client.1.smithi137.stdout:6/281: dread - d2/d54/d55/d33/f64 zero size 2024-03-14T13:02:13.733 INFO:tasks.workunit.client.0.smithi111.stdout:7/315: write d0/d11/f29 [3998577,69252] 0 2024-03-14T13:02:13.737 INFO:tasks.workunit.client.1.smithi137.stdout:5/190: readlink d7/l12 0 2024-03-14T13:02:13.738 INFO:tasks.workunit.client.0.smithi111.stdout:3/419: mkdir d0/d2d/d35/d50/d6c/da3 0 2024-03-14T13:02:13.738 INFO:tasks.workunit.client.0.smithi111.stdout:2/305: dread dd/d11/f33 [0,4194304] 0 2024-03-14T13:02:13.741 INFO:tasks.workunit.client.0.smithi111.stdout:7/316: unlink d0/d1/d4a/f4b 0 2024-03-14T13:02:13.742 INFO:tasks.workunit.client.0.smithi111.stdout:6/310: symlink d0/d17/d20/l75 0 2024-03-14T13:02:13.744 INFO:tasks.workunit.client.1.smithi137.stdout:6/282: rmdir d2/d54/d55 39 2024-03-14T13:02:13.744 INFO:tasks.workunit.client.0.smithi111.stdout:2/306: creat dd/d11/d48/f60 x:0 0 0 2024-03-14T13:02:13.744 INFO:tasks.workunit.client.0.smithi111.stdout:7/317: write d0/d1/d22/f39 [1074739,22151] 0 2024-03-14T13:02:13.745 INFO:tasks.workunit.client.1.smithi137.stdout:5/191: symlink d7/d1d/l45 0 2024-03-14T13:02:13.746 INFO:tasks.workunit.client.0.smithi111.stdout:6/311: creat d0/d17/d21/d69/f76 x:0 0 0 2024-03-14T13:02:13.748 INFO:tasks.workunit.client.0.smithi111.stdout:3/420: rename d0/d2d/d35/d50/d5e/d40/d67/l68 to d0/d2d/d35/d50/d5e/d40/d71/la4 0 2024-03-14T13:02:13.748 INFO:tasks.workunit.client.1.smithi137.stdout:4/313: dread d4/f12 [0,4194304] 0 2024-03-14T13:02:13.749 INFO:tasks.workunit.client.1.smithi137.stdout:8/193: write d1/fd [849023,109133] 0 2024-03-14T13:02:13.751 INFO:tasks.workunit.client.0.smithi111.stdout:3/421: truncate d0/d2d/f86 193341 0 2024-03-14T13:02:13.752 INFO:tasks.workunit.client.0.smithi111.stdout:6/312: creat d0/d17/d21/d42/f77 x:0 0 0 2024-03-14T13:02:13.753 INFO:tasks.workunit.client.0.smithi111.stdout:7/318: getdents d0/d1/d21/d2a 0 2024-03-14T13:02:13.755 INFO:tasks.workunit.client.0.smithi111.stdout:2/307: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:13.755 INFO:tasks.workunit.client.0.smithi111.stdout:7/319: fdatasync d0/d1/d5/f4c 0 2024-03-14T13:02:13.757 INFO:tasks.workunit.client.1.smithi137.stdout:9/366: truncate d6/f2b 2500186 0 2024-03-14T13:02:13.757 INFO:tasks.workunit.client.1.smithi137.stdout:4/314: chown d4/d9/d3a/d41/d38/l43 1031196771 1 2024-03-14T13:02:13.757 INFO:tasks.workunit.client.0.smithi111.stdout:6/313: symlink d0/d17/d20/d65/l78 0 2024-03-14T13:02:13.757 INFO:tasks.workunit.client.0.smithi111.stdout:2/308: write dd/d11/d48/f60 [698144,86647] 0 2024-03-14T13:02:13.758 INFO:tasks.workunit.client.1.smithi137.stdout:6/283: getdents d2/d54/d55 0 2024-03-14T13:02:13.759 INFO:tasks.workunit.client.0.smithi111.stdout:3/422: rename d0/d2d/d35/d50/d5e/d19/l5c to d0/d2d/d35/d4c/la5 0 2024-03-14T13:02:13.760 INFO:tasks.workunit.client.1.smithi137.stdout:5/192: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:13.760 INFO:tasks.workunit.client.0.smithi111.stdout:7/320: creat d0/d1/d4a/f60 x:0 0 0 2024-03-14T13:02:13.762 INFO:tasks.workunit.client.1.smithi137.stdout:4/315: rmdir d4 39 2024-03-14T13:02:13.762 INFO:tasks.workunit.client.0.smithi111.stdout:7/321: readlink d0/d1/d5/d38/d1b/l2c 0 2024-03-14T13:02:13.764 INFO:tasks.workunit.client.1.smithi137.stdout:8/194: mknod d1/d4a/c4d 0 2024-03-14T13:02:13.764 INFO:tasks.workunit.client.1.smithi137.stdout:9/367: truncate d6/df/d19/d1f/f3b 5116127 0 2024-03-14T13:02:13.764 INFO:tasks.workunit.client.0.smithi111.stdout:3/423: creat d0/d2d/d35/d50/d5e/d40/d71/fa6 x:0 0 0 2024-03-14T13:02:13.765 INFO:tasks.workunit.client.0.smithi111.stdout:7/322: mkdir d0/d1/d5/d61 0 2024-03-14T13:02:13.767 INFO:tasks.workunit.client.0.smithi111.stdout:2/309: creat dd/f61 x:0 0 0 2024-03-14T13:02:13.767 INFO:tasks.workunit.client.0.smithi111.stdout:6/314: creat d0/d17/d21/f79 x:0 0 0 2024-03-14T13:02:13.768 INFO:tasks.workunit.client.1.smithi137.stdout:9/368: write d6/f69 [1224088,3289] 0 2024-03-14T13:02:13.769 INFO:tasks.workunit.client.0.smithi111.stdout:2/310: truncate dd/f4c 226859 0 2024-03-14T13:02:13.769 INFO:tasks.workunit.client.1.smithi137.stdout:8/195: mkdir d1/dc/d4e 0 2024-03-14T13:02:13.771 INFO:tasks.workunit.client.0.smithi111.stdout:6/315: mknod d0/d17/d21/d69/c7a 0 2024-03-14T13:02:13.772 INFO:tasks.workunit.client.1.smithi137.stdout:5/193: rename d7/d16/f26 to d7/d16/d18/d25/f46 0 2024-03-14T13:02:13.772 INFO:tasks.workunit.client.0.smithi111.stdout:7/323: creat d0/d1/d4a/d52/f62 x:0 0 0 2024-03-14T13:02:13.773 INFO:tasks.workunit.client.1.smithi137.stdout:8/196: creat d1/dc/d11/d13/f4f x:0 0 0 2024-03-14T13:02:13.773 INFO:tasks.workunit.client.0.smithi111.stdout:7/324: dread - d0/d11/f53 zero size 2024-03-14T13:02:13.774 INFO:tasks.workunit.client.1.smithi137.stdout:4/316: stat d4/d9/d3a/d41/c50 0 2024-03-14T13:02:13.774 INFO:tasks.workunit.client.0.smithi111.stdout:6/316: unlink d0/c16 0 2024-03-14T13:02:13.774 INFO:tasks.workunit.client.0.smithi111.stdout:7/325: chown d0/d1/d22/d3e 352139 1 2024-03-14T13:02:13.776 INFO:tasks.workunit.client.1.smithi137.stdout:8/197: fdatasync d1/dc/d10/f27 0 2024-03-14T13:02:13.776 INFO:tasks.workunit.client.1.smithi137.stdout:5/194: dread - d7/d16/d18/d25/f3f zero size 2024-03-14T13:02:13.778 INFO:tasks.workunit.client.1.smithi137.stdout:9/369: mknod d6/df/d71/c84 0 2024-03-14T13:02:13.781 INFO:tasks.workunit.client.1.smithi137.stdout:6/284: dwrite d2/d6/f34 [0,4194304] 0 2024-03-14T13:02:13.781 INFO:tasks.workunit.client.1.smithi137.stdout:8/198: dread - d1/dc/d10/f27 zero size 2024-03-14T13:02:13.782 INFO:tasks.workunit.client.1.smithi137.stdout:4/317: write d4/d9/d3a/d41/f21 [839415,45054] 0 2024-03-14T13:02:13.782 INFO:tasks.workunit.client.1.smithi137.stdout:5/195: mkdir d7/d16/d47 0 2024-03-14T13:02:13.786 INFO:tasks.workunit.client.1.smithi137.stdout:5/196: fdatasync d7/d16/d18/d25/f3f 0 2024-03-14T13:02:13.786 INFO:tasks.workunit.client.1.smithi137.stdout:3/246: dread d4/d19/f23 [4194304,4194304] 0 2024-03-14T13:02:13.787 INFO:tasks.workunit.client.1.smithi137.stdout:8/199: write d1/f44 [128293,53874] 0 2024-03-14T13:02:13.788 INFO:tasks.workunit.client.1.smithi137.stdout:9/370: dread d6/f69 [0,4194304] 0 2024-03-14T13:02:13.789 INFO:tasks.workunit.client.1.smithi137.stdout:6/285: stat d2/d6/de/d45/d59/d39/d3c/d43/d57 0 2024-03-14T13:02:13.789 INFO:tasks.workunit.client.1.smithi137.stdout:8/200: readlink d1/l1d 0 2024-03-14T13:02:13.789 INFO:tasks.workunit.client.0.smithi111.stdout:6/317: dwrite d0/d17/f1f [0,4194304] 0 2024-03-14T13:02:13.789 INFO:tasks.workunit.client.0.smithi111.stdout:2/311: dwrite dd/d17/d34/f2e [4194304,4194304] 0 2024-03-14T13:02:13.790 INFO:tasks.workunit.client.1.smithi137.stdout:5/197: fdatasync d7/fb 0 2024-03-14T13:02:13.791 INFO:tasks.workunit.client.1.smithi137.stdout:9/371: write d6/df/d19/d33/f46 [3822640,114946] 0 2024-03-14T13:02:13.792 INFO:tasks.workunit.client.0.smithi111.stdout:3/424: rmdir d0/d2d 39 2024-03-14T13:02:13.793 INFO:tasks.workunit.client.0.smithi111.stdout:7/326: dwrite d0/d11/f53 [0,4194304] 0 2024-03-14T13:02:13.793 INFO:tasks.workunit.client.0.smithi111.stdout:6/318: write d0/f4 [1671464,35593] 0 2024-03-14T13:02:13.796 INFO:tasks.workunit.client.1.smithi137.stdout:3/247: dread d4/df/f30 [0,4194304] 0 2024-03-14T13:02:13.796 INFO:tasks.workunit.client.1.smithi137.stdout:5/198: write d7/d16/d18/f39 [4265289,631] 0 2024-03-14T13:02:13.798 INFO:tasks.workunit.client.1.smithi137.stdout:4/318: read d4/f13 [810588,91959] 0 2024-03-14T13:02:13.800 INFO:tasks.workunit.client.0.smithi111.stdout:3/425: dread - d0/d2d/d35/d69/fa0 zero size 2024-03-14T13:02:13.802 INFO:tasks.workunit.client.0.smithi111.stdout:6/319: dread d0/ff [4194304,4194304] 0 2024-03-14T13:02:13.803 INFO:tasks.workunit.client.0.smithi111.stdout:7/327: link d0/d1/d21/d3a/l57 d0/d1/d5/d38/d5c/l63 0 2024-03-14T13:02:13.804 INFO:tasks.workunit.client.0.smithi111.stdout:7/328: dread - d0/d1/d4a/f60 zero size 2024-03-14T13:02:13.805 INFO:tasks.workunit.client.0.smithi111.stdout:3/426: mkdir d0/d2d/d35/d50/d5e/d40/da7 0 2024-03-14T13:02:13.805 INFO:tasks.workunit.client.0.smithi111.stdout:2/312: dread dd/d11/d48/d49/f5c [0,4194304] 0 2024-03-14T13:02:13.806 INFO:tasks.workunit.client.0.smithi111.stdout:6/320: mknod d0/d17/d21/d69/d3d/c7b 0 2024-03-14T13:02:13.807 INFO:tasks.workunit.client.0.smithi111.stdout:7/329: creat d0/d1/d22/d3e/f64 x:0 0 0 2024-03-14T13:02:13.809 INFO:tasks.workunit.client.0.smithi111.stdout:6/321: fsync d0/d17/d21/d69/d3d/f41 0 2024-03-14T13:02:13.810 INFO:tasks.workunit.client.1.smithi137.stdout:3/248: symlink d4/de/d16/d27/l4d 0 2024-03-14T13:02:13.810 INFO:tasks.workunit.client.1.smithi137.stdout:5/199: mkdir d7/d16/d18/d48 0 2024-03-14T13:02:13.811 INFO:tasks.workunit.client.1.smithi137.stdout:6/286: rename d2/d6/c19 to d2/d54/c66 0 2024-03-14T13:02:13.811 INFO:tasks.workunit.client.0.smithi111.stdout:7/330: mknod d0/d1/d21/d3a/c65 0 2024-03-14T13:02:13.814 INFO:tasks.workunit.client.0.smithi111.stdout:6/322: symlink d0/d17/d20/l7c 0 2024-03-14T13:02:13.816 INFO:tasks.workunit.client.1.smithi137.stdout:3/249: mkdir d4/de/d4e 0 2024-03-14T13:02:13.816 INFO:tasks.workunit.client.1.smithi137.stdout:5/200: mkdir d7/d49 0 2024-03-14T13:02:13.816 INFO:tasks.workunit.client.1.smithi137.stdout:6/287: read d2/f5a [2947488,43143] 0 2024-03-14T13:02:13.817 INFO:tasks.workunit.client.1.smithi137.stdout:5/201: stat d7/d16/d18/d25 0 2024-03-14T13:02:13.818 INFO:tasks.workunit.client.0.smithi111.stdout:7/331: truncate d0/d1/d5/d38/d9/fb 8504636 0 2024-03-14T13:02:13.818 INFO:tasks.workunit.client.1.smithi137.stdout:3/250: write d4/de/d16/d1c/f2d [1467725,46488] 0 2024-03-14T13:02:13.819 INFO:tasks.workunit.client.1.smithi137.stdout:4/319: dwrite d4/d4d/f6b [0,4194304] 0 2024-03-14T13:02:13.820 INFO:tasks.workunit.client.1.smithi137.stdout:6/288: fsync d2/d54/d55/f1e 0 2024-03-14T13:02:13.820 INFO:tasks.workunit.client.0.smithi111.stdout:2/313: dread dd/d11/d1a/f1b [0,4194304] 0 2024-03-14T13:02:13.821 INFO:tasks.workunit.client.1.smithi137.stdout:5/202: chown d7/d16/d18/l40 820366 1 2024-03-14T13:02:13.823 INFO:tasks.workunit.client.1.smithi137.stdout:4/320: creat d4/d10/d26/f6f x:0 0 0 2024-03-14T13:02:13.824 INFO:tasks.workunit.client.0.smithi111.stdout:2/314: creat dd/d11/d1a/f62 x:0 0 0 2024-03-14T13:02:13.825 INFO:tasks.workunit.client.1.smithi137.stdout:4/321: fsync d4/f11 0 2024-03-14T13:02:13.826 INFO:tasks.workunit.client.1.smithi137.stdout:5/203: symlink d7/d16/d18/d48/l4a 0 2024-03-14T13:02:13.826 INFO:tasks.workunit.client.1.smithi137.stdout:4/322: readlink d4/d9/d3a/d41/d38/l43 0 2024-03-14T13:02:13.828 INFO:tasks.workunit.client.1.smithi137.stdout:4/323: write d4/f8 [2093367,103088] 0 2024-03-14T13:02:13.828 INFO:tasks.workunit.client.1.smithi137.stdout:5/204: write d7/d16/d18/d2d/f2f [736504,56799] 0 2024-03-14T13:02:13.829 INFO:tasks.workunit.client.1.smithi137.stdout:3/251: getdents d4/df/d34 0 2024-03-14T13:02:13.830 INFO:tasks.workunit.client.0.smithi111.stdout:7/332: dread d0/d1/d5/d38/d9/f19 [0,4194304] 0 2024-03-14T13:02:13.830 INFO:tasks.workunit.client.0.smithi111.stdout:7/333: read d0/d1/f28 [2985580,49969] 0 2024-03-14T13:02:13.831 INFO:tasks.workunit.client.1.smithi137.stdout:3/252: dread - d4/de/f3f zero size 2024-03-14T13:02:13.831 INFO:tasks.workunit.client.0.smithi111.stdout:2/315: creat dd/d17/f63 x:0 0 0 2024-03-14T13:02:13.831 INFO:tasks.workunit.client.1.smithi137.stdout:4/324: mkdir d4/d9/d3a/d70 0 2024-03-14T13:02:13.833 INFO:tasks.workunit.client.0.smithi111.stdout:2/316: write dd/d11/d48/d49/f5c [1348087,34707] 0 2024-03-14T13:02:13.836 INFO:tasks.workunit.client.0.smithi111.stdout:7/334: dread d0/d1/d22/f39 [0,4194304] 0 2024-03-14T13:02:13.836 INFO:tasks.workunit.client.0.smithi111.stdout:2/317: creat dd/d17/d53/f64 x:0 0 0 2024-03-14T13:02:13.837 INFO:tasks.workunit.client.1.smithi137.stdout:5/205: mknod d7/d33/c4b 0 2024-03-14T13:02:13.840 INFO:tasks.workunit.client.1.smithi137.stdout:5/206: mknod d7/d1d/c4c 0 2024-03-14T13:02:13.841 INFO:tasks.workunit.client.1.smithi137.stdout:9/372: dread d6/fc [0,4194304] 0 2024-03-14T13:02:13.842 INFO:tasks.workunit.client.0.smithi111.stdout:2/318: creat dd/d17/d34/d4f/d55/d59/f65 x:0 0 0 2024-03-14T13:02:13.842 INFO:tasks.workunit.client.0.smithi111.stdout:7/335: dread d0/d1/f28 [4194304,4194304] 0 2024-03-14T13:02:13.842 INFO:tasks.workunit.client.1.smithi137.stdout:5/207: readlink d7/l21 0 2024-03-14T13:02:13.842 INFO:tasks.workunit.client.1.smithi137.stdout:9/373: chown d6/f1d 117 1 2024-03-14T13:02:13.843 INFO:tasks.workunit.client.0.smithi111.stdout:2/319: dread - dd/d11/d1a/d56/f5b zero size 2024-03-14T13:02:13.843 INFO:tasks.workunit.client.1.smithi137.stdout:6/289: write d2/d6/de/d45/d59/d39/d3c/f47 [1365782,130307] 0 2024-03-14T13:02:13.844 INFO:tasks.workunit.client.0.smithi111.stdout:2/320: dread - dd/d17/f37 zero size 2024-03-14T13:02:13.844 INFO:tasks.workunit.client.0.smithi111.stdout:7/336: chown d0/d1/d21/d3a/l57 204 1 2024-03-14T13:02:13.844 INFO:tasks.workunit.client.1.smithi137.stdout:6/290: chown d2/d6/de/d45/d59/d39/d3c/d43 592 1 2024-03-14T13:02:13.846 INFO:tasks.workunit.client.1.smithi137.stdout:5/208: truncate d7/f2c 566661 0 2024-03-14T13:02:13.846 INFO:tasks.workunit.client.0.smithi111.stdout:2/321: write dd/d11/d1a/f1b [335168,30418] 0 2024-03-14T13:02:13.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/291: write d2/d6/de/d45/d59/f5e [132256,112634] 0 2024-03-14T13:02:13.848 INFO:tasks.workunit.client.1.smithi137.stdout:9/374: rename d6/df/d19/c1e to d6/df/d19/d1f/c85 0 2024-03-14T13:02:13.849 INFO:tasks.workunit.client.1.smithi137.stdout:5/209: symlink d7/d36/l4d 0 2024-03-14T13:02:13.851 INFO:tasks.workunit.client.0.smithi111.stdout:6/323: dread d0/d17/f55 [0,4194304] 0 2024-03-14T13:02:13.852 INFO:tasks.workunit.client.0.smithi111.stdout:2/322: truncate dd/d11/d48/d49/f51 1231377 0 2024-03-14T13:02:13.852 INFO:tasks.workunit.client.1.smithi137.stdout:3/253: dwrite d4/fd [0,4194304] 0 2024-03-14T13:02:13.853 INFO:tasks.workunit.client.0.smithi111.stdout:7/337: dwrite d0/d1/f51 [0,4194304] 0 2024-03-14T13:02:13.854 INFO:tasks.workunit.client.0.smithi111.stdout:7/338: chown d0/d1/d22/f39 3 1 2024-03-14T13:02:13.855 INFO:tasks.workunit.client.0.smithi111.stdout:7/339: chown d0/d1/d5/d38/d5c 21 1 2024-03-14T13:02:13.857 INFO:tasks.workunit.client.0.smithi111.stdout:2/323: unlink dd/d11/d48/c54 0 2024-03-14T13:02:13.858 INFO:tasks.workunit.client.1.smithi137.stdout:5/210: write d7/d16/f1a [4803387,65881] 0 2024-03-14T13:02:13.859 INFO:tasks.workunit.client.0.smithi111.stdout:6/324: mknod d0/d17/d21/d69/d3d/d53/c7d 0 2024-03-14T13:02:13.863 INFO:tasks.workunit.client.0.smithi111.stdout:2/324: creat dd/d17/d34/d4f/f66 x:0 0 0 2024-03-14T13:02:13.865 INFO:tasks.workunit.client.0.smithi111.stdout:6/325: creat d0/d17/d20/d4a/d59/f7e x:0 0 0 2024-03-14T13:02:13.866 INFO:tasks.workunit.client.0.smithi111.stdout:6/326: fdatasync d0/f57 0 2024-03-14T13:02:13.867 INFO:tasks.workunit.client.1.smithi137.stdout:9/375: dwrite d6/fc [0,4194304] 0 2024-03-14T13:02:13.867 INFO:tasks.workunit.client.1.smithi137.stdout:6/292: dwrite d2/d6/f65 [4194304,4194304] 0 2024-03-14T13:02:13.868 INFO:tasks.workunit.client.0.smithi111.stdout:6/327: fdatasync d0/d50/f56 0 2024-03-14T13:02:13.868 INFO:tasks.workunit.client.0.smithi111.stdout:2/325: creat dd/d11/d48/d49/f67 x:0 0 0 2024-03-14T13:02:13.869 INFO:tasks.workunit.client.1.smithi137.stdout:5/211: dread d7/f15 [0,4194304] 0 2024-03-14T13:02:13.870 INFO:tasks.workunit.client.0.smithi111.stdout:7/340: dwrite d0/d1/d21/f15 [0,4194304] 0 2024-03-14T13:02:13.871 INFO:tasks.workunit.client.1.smithi137.stdout:9/376: readlink d6/df/d19/d3a/d40/l79 0 2024-03-14T13:02:13.872 INFO:tasks.workunit.client.1.smithi137.stdout:6/293: write d2/f5a [3664938,70500] 0 2024-03-14T13:02:13.872 INFO:tasks.workunit.client.1.smithi137.stdout:5/212: write d7/d16/f1a [5515817,120863] 0 2024-03-14T13:02:13.872 INFO:tasks.workunit.client.0.smithi111.stdout:2/326: creat dd/d17/d53/f68 x:0 0 0 2024-03-14T13:02:13.875 INFO:tasks.workunit.client.1.smithi137.stdout:9/377: creat d6/d26/f86 x:0 0 0 2024-03-14T13:02:13.876 INFO:tasks.workunit.client.1.smithi137.stdout:6/294: mkdir d2/d6/de/d45/d59/d39/d67 0 2024-03-14T13:02:13.877 INFO:tasks.workunit.client.0.smithi111.stdout:6/328: mknod d0/d17/d20/d4a/d68/c7f 0 2024-03-14T13:02:13.877 INFO:tasks.workunit.client.0.smithi111.stdout:2/327: truncate dd/d11/d48/d49/f67 328638 0 2024-03-14T13:02:13.879 INFO:tasks.workunit.client.0.smithi111.stdout:6/329: dread - d0/d17/d21/d42/f58 zero size 2024-03-14T13:02:13.881 INFO:tasks.workunit.client.1.smithi137.stdout:5/213: mknod d7/d49/c4e 0 2024-03-14T13:02:13.882 INFO:tasks.workunit.client.0.smithi111.stdout:6/330: mkdir d0/d17/d20/d4a/d59/d80 0 2024-03-14T13:02:13.882 INFO:tasks.workunit.client.1.smithi137.stdout:6/295: creat d2/d6/de/f68 x:0 0 0 2024-03-14T13:02:13.884 INFO:tasks.workunit.client.1.smithi137.stdout:5/214: write d7/d16/d18/f20 [2339391,8577] 0 2024-03-14T13:02:13.885 INFO:tasks.workunit.client.0.smithi111.stdout:2/328: rmdir dd/d17/d34/d4f 39 2024-03-14T13:02:13.888 INFO:tasks.workunit.client.1.smithi137.stdout:9/378: dwrite d6/df/f7d [0,4194304] 0 2024-03-14T13:02:13.888 INFO:tasks.workunit.client.0.smithi111.stdout:2/329: symlink dd/d17/l69 0 2024-03-14T13:02:13.889 INFO:tasks.workunit.client.1.smithi137.stdout:5/215: rmdir d7/d16/d18/d48 39 2024-03-14T13:02:13.890 INFO:tasks.workunit.client.1.smithi137.stdout:6/296: creat d2/d4c/f69 x:0 0 0 2024-03-14T13:02:13.890 INFO:tasks.workunit.client.0.smithi111.stdout:2/330: fsync dd/d17/f31 0 2024-03-14T13:02:13.890 INFO:tasks.workunit.client.0.smithi111.stdout:6/331: dwrite d0/f1c [0,4194304] 0 2024-03-14T13:02:13.891 INFO:tasks.workunit.client.1.smithi137.stdout:9/379: write d6/df/d19/d1f/f3b [511988,78871] 0 2024-03-14T13:02:13.894 INFO:tasks.workunit.client.1.smithi137.stdout:6/297: truncate d2/d6/de/d45/d59/f5e 360000 0 2024-03-14T13:02:13.896 INFO:tasks.workunit.client.1.smithi137.stdout:6/298: write d2/f2d [2095288,129400] 0 2024-03-14T13:02:13.896 INFO:tasks.workunit.client.1.smithi137.stdout:5/216: creat d7/d16/d18/d25/d28/d3c/f4f x:0 0 0 2024-03-14T13:02:13.896 INFO:tasks.workunit.client.0.smithi111.stdout:2/331: mknod dd/d17/d34/d4f/d55/c6a 0 2024-03-14T13:02:13.897 INFO:tasks.workunit.client.0.smithi111.stdout:6/332: mkdir d0/d17/d21/d69/d3d/d53/d81 0 2024-03-14T13:02:13.900 INFO:tasks.workunit.client.0.smithi111.stdout:6/333: mkdir d0/d17/d20/d4a/d59/d82 0 2024-03-14T13:02:13.900 INFO:tasks.workunit.client.1.smithi137.stdout:6/299: mknod d2/d4c/c6a 0 2024-03-14T13:02:13.902 INFO:tasks.workunit.client.1.smithi137.stdout:5/217: creat d7/d16/d18/d25/f50 x:0 0 0 2024-03-14T13:02:13.902 INFO:tasks.workunit.client.1.smithi137.stdout:6/300: chown d2/d54/d55/d33/l3a 5755 1 2024-03-14T13:02:13.903 INFO:tasks.workunit.client.0.smithi111.stdout:6/334: symlink d0/d17/d20/l83 0 2024-03-14T13:02:13.903 INFO:tasks.workunit.client.0.smithi111.stdout:6/335: chown d0/d50/f5f 842806 1 2024-03-14T13:02:13.904 INFO:tasks.workunit.client.0.smithi111.stdout:6/336: symlink d0/l84 0 2024-03-14T13:02:13.905 INFO:tasks.workunit.client.1.smithi137.stdout:5/218: symlink d7/d16/l51 0 2024-03-14T13:02:13.905 INFO:tasks.workunit.client.1.smithi137.stdout:6/301: write d2/d6/de/d45/d59/f37 [1294803,80839] 0 2024-03-14T13:02:13.905 INFO:tasks.workunit.client.1.smithi137.stdout:9/380: dwrite d6/df/d19/d33/f80 [0,4194304] 0 2024-03-14T13:02:13.906 INFO:tasks.workunit.client.0.smithi111.stdout:7/341: dread d0/d1/d5/f1a [0,4194304] 0 2024-03-14T13:02:13.906 INFO:tasks.workunit.client.0.smithi111.stdout:6/337: truncate d0/d17/f2b 4731805 0 2024-03-14T13:02:13.910 INFO:tasks.workunit.client.0.smithi111.stdout:6/338: mknod d0/d17/d21/d42/c85 0 2024-03-14T13:02:13.910 INFO:tasks.workunit.client.1.smithi137.stdout:9/381: unlink d6/d53/c81 0 2024-03-14T13:02:13.911 INFO:tasks.workunit.client.1.smithi137.stdout:6/302: rename d2/d6/de/d45/d59/d39/d3c/d43 to d2/d6/d1d/d6b 0 2024-03-14T13:02:13.912 INFO:tasks.workunit.client.0.smithi111.stdout:6/339: write d0/d17/d20/d4a/d59/f6f [742877,102601] 0 2024-03-14T13:02:13.912 INFO:tasks.workunit.client.0.smithi111.stdout:7/342: dread d0/d1/d5/d38/d9/f3d [0,4194304] 0 2024-03-14T13:02:13.926 INFO:tasks.workunit.client.0.smithi111.stdout:6/340: dread d0/f14 [0,4194304] 0 2024-03-14T13:02:13.928 INFO:tasks.workunit.client.0.smithi111.stdout:6/341: truncate d0/d17/d21/d69/f76 232430 0 2024-03-14T13:02:13.938 INFO:tasks.workunit.client.1.smithi137.stdout:2/258: sync 2024-03-14T13:02:13.940 INFO:tasks.workunit.client.1.smithi137.stdout:2/259: chown d2/f5 2267 1 2024-03-14T13:02:13.941 INFO:tasks.workunit.client.1.smithi137.stdout:2/260: dread - d2/d7/db/de/f4a zero size 2024-03-14T13:02:13.942 INFO:tasks.workunit.client.1.smithi137.stdout:9/382: mknod d6/d56/c87 0 2024-03-14T13:02:13.943 INFO:tasks.workunit.client.1.smithi137.stdout:6/303: mknod d2/d6/de/d45/d59/d39/c6c 0 2024-03-14T13:02:13.944 INFO:tasks.workunit.client.1.smithi137.stdout:9/383: write d6/df/d19/d3a/f47 [4611869,124476] 0 2024-03-14T13:02:13.945 INFO:tasks.workunit.client.1.smithi137.stdout:5/219: link d7/f30 d7/d16/d18/d2d/f52 0 2024-03-14T13:02:13.948 INFO:tasks.workunit.client.0.smithi111.stdout:7/343: dread d0/d1/d22/f42 [0,4194304] 0 2024-03-14T13:02:13.948 INFO:tasks.workunit.client.1.smithi137.stdout:6/304: mkdir d2/d54/d55/d6d 0 2024-03-14T13:02:13.950 INFO:tasks.workunit.client.0.smithi111.stdout:7/344: symlink d0/d1/d5/d38/d9/l66 0 2024-03-14T13:02:13.952 INFO:tasks.workunit.client.1.smithi137.stdout:6/305: write d2/d6/f65 [8347726,124290] 0 2024-03-14T13:02:13.953 INFO:tasks.workunit.client.0.smithi111.stdout:6/342: truncate d0/d17/d20/d4a/f51 345329 0 2024-03-14T13:02:13.953 INFO:tasks.workunit.client.1.smithi137.stdout:5/220: creat d7/d16/d18/d2d/f53 x:0 0 0 2024-03-14T13:02:13.953 INFO:tasks.workunit.client.0.smithi111.stdout:6/343: rmdir d0/d17/d20/d4a 39 2024-03-14T13:02:13.954 INFO:tasks.workunit.client.1.smithi137.stdout:2/261: dwrite d2/d7/db/f38 [0,4194304] 0 2024-03-14T13:02:13.955 INFO:tasks.workunit.client.1.smithi137.stdout:5/221: write d7/d16/f38 [733997,14538] 0 2024-03-14T13:02:13.956 INFO:tasks.workunit.client.0.smithi111.stdout:6/344: unlink d0/d17/d20/f32 0 2024-03-14T13:02:13.957 INFO:tasks.workunit.client.0.smithi111.stdout:6/345: write d0/d17/d21/f28 [3317547,102715] 0 2024-03-14T13:02:13.959 INFO:tasks.workunit.client.1.smithi137.stdout:5/222: fsync d7/fb 0 2024-03-14T13:02:13.960 INFO:tasks.workunit.client.0.smithi111.stdout:6/346: creat d0/d17/d21/d69/f86 x:0 0 0 2024-03-14T13:02:13.961 INFO:tasks.workunit.client.1.smithi137.stdout:2/262: write d2/d7/db/d2f/f45 [318788,27634] 0 2024-03-14T13:02:13.962 INFO:tasks.workunit.client.0.smithi111.stdout:6/347: write d0/d17/d21/d69/d3d/f71 [5367376,40981] 0 2024-03-14T13:02:13.964 INFO:tasks.workunit.client.1.smithi137.stdout:5/223: truncate d7/d16/d18/d25/f46 5900764 0 2024-03-14T13:02:13.964 INFO:tasks.workunit.client.1.smithi137.stdout:6/306: dwrite d2/d6/d1d/f4e [0,4194304] 0 2024-03-14T13:02:13.966 INFO:tasks.workunit.client.1.smithi137.stdout:2/263: mknod d2/d7/c59 0 2024-03-14T13:02:13.967 INFO:tasks.workunit.client.1.smithi137.stdout:6/307: write d2/d6/f34 [1703976,110776] 0 2024-03-14T13:02:13.971 INFO:tasks.workunit.client.1.smithi137.stdout:2/264: read d2/d7/f13 [1213529,125410] 0 2024-03-14T13:02:13.971 INFO:tasks.workunit.client.1.smithi137.stdout:5/224: unlink d7/l13 0 2024-03-14T13:02:13.972 INFO:tasks.workunit.client.1.smithi137.stdout:6/308: chown d2/d6/de/f4f 182066 1 2024-03-14T13:02:13.973 INFO:tasks.workunit.client.1.smithi137.stdout:9/384: dread d6/df/d19/d3a/f43 [0,4194304] 0 2024-03-14T13:02:13.974 INFO:tasks.workunit.client.1.smithi137.stdout:9/385: read - d6/df/f64 zero size 2024-03-14T13:02:13.975 INFO:tasks.workunit.client.1.smithi137.stdout:2/265: fdatasync d2/d7/db/de/f20 0 2024-03-14T13:02:13.981 INFO:tasks.workunit.client.0.smithi111.stdout:6/348: dread d0/d17/d20/d4a/f51 [0,4194304] 0 2024-03-14T13:02:13.985 INFO:tasks.workunit.client.0.smithi111.stdout:6/349: mknod d0/d17/d20/c87 0 2024-03-14T13:02:13.986 INFO:tasks.workunit.client.0.smithi111.stdout:6/350: creat d0/d17/d20/d4a/d68/f88 x:0 0 0 2024-03-14T13:02:13.989 INFO:tasks.workunit.client.0.smithi111.stdout:6/351: dread d0/ff [4194304,4194304] 0 2024-03-14T13:02:13.996 INFO:tasks.workunit.client.0.smithi111.stdout:6/352: dwrite d0/d17/d20/d4a/f51 [0,4194304] 0 2024-03-14T13:02:14.002 INFO:tasks.workunit.client.0.smithi111.stdout:6/353: rename d0/c3b to d0/d17/d21/d69/d3d/d53/c89 0 2024-03-14T13:02:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:13 smithi137 ceph-mon[50336]: pgmap v21: 65 pgs: 65 active+clean; 691 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 4.6 MiB/s rd, 56 MiB/s wr, 98 op/s 2024-03-14T13:02:14.018 INFO:tasks.workunit.client.0.smithi111.stdout:6/354: dread d0/f4 [0,4194304] 0 2024-03-14T13:02:14.023 INFO:tasks.workunit.client.0.smithi111.stdout:6/355: dwrite d0/f72 [0,4194304] 0 2024-03-14T13:02:14.027 INFO:tasks.workunit.client.0.smithi111.stdout:6/356: symlink d0/d17/d24/l8a 0 2024-03-14T13:02:14.034 INFO:tasks.workunit.client.0.smithi111.stdout:6/357: dwrite d0/d17/d20/d4a/d59/f7e [0,4194304] 0 2024-03-14T13:02:14.037 INFO:tasks.workunit.client.0.smithi111.stdout:6/358: creat d0/d17/d21/d69/d3d/d53/d81/f8b x:0 0 0 2024-03-14T13:02:14.039 INFO:tasks.workunit.client.0.smithi111.stdout:6/359: rename d0/d17/l6a to d0/d17/d20/d4a/d59/d82/l8c 0 2024-03-14T13:02:14.040 INFO:tasks.workunit.client.0.smithi111.stdout:6/360: write d0/d17/f2b [4481124,59626] 0 2024-03-14T13:02:14.041 INFO:tasks.workunit.client.0.smithi111.stdout:6/361: fdatasync d0/d17/d21/d69/d3d/f5c 0 2024-03-14T13:02:14.043 INFO:tasks.workunit.client.0.smithi111.stdout:6/362: symlink d0/d17/d24/l8d 0 2024-03-14T13:02:14.046 INFO:tasks.workunit.client.0.smithi111.stdout:6/363: unlink d0/d17/d21/d69/f86 0 2024-03-14T13:02:14.047 INFO:tasks.workunit.client.0.smithi111.stdout:6/364: write d0/d17/d21/f79 [537074,122426] 0 2024-03-14T13:02:14.048 INFO:tasks.workunit.client.0.smithi111.stdout:6/365: truncate d0/d17/d21/d69/f39 1749038 0 2024-03-14T13:02:14.053 INFO:tasks.workunit.client.1.smithi137.stdout:5/225: mknod d7/d16/d41/c54 0 2024-03-14T13:02:14.054 INFO:tasks.workunit.client.1.smithi137.stdout:9/386: truncate d6/df/d19/f62 368212 0 2024-03-14T13:02:14.054 INFO:tasks.workunit.client.1.smithi137.stdout:5/226: dread - d7/d16/d18/d25/f50 zero size 2024-03-14T13:02:14.055 INFO:tasks.workunit.client.0.smithi111.stdout:6/366: dwrite d0/d17/d21/f22 [0,4194304] 0 2024-03-14T13:02:14.055 INFO:tasks.workunit.client.1.smithi137.stdout:6/309: mknod d2/d6/d1d/d6b/d57/d58/c6e 0 2024-03-14T13:02:14.055 INFO:tasks.workunit.client.0.smithi111.stdout:6/367: write d0/d17/d21/d42/f58 [207997,49275] 0 2024-03-14T13:02:14.055 INFO:tasks.workunit.client.1.smithi137.stdout:9/387: write d6/df/f13 [1184753,71865] 0 2024-03-14T13:02:14.058 INFO:tasks.workunit.client.0.smithi111.stdout:6/368: symlink d0/d17/l8e 0 2024-03-14T13:02:14.059 INFO:tasks.workunit.client.1.smithi137.stdout:5/227: rename d7/f2c to d7/d16/d18/d25/f55 0 2024-03-14T13:02:14.059 INFO:tasks.workunit.client.1.smithi137.stdout:2/266: dread d2/f14 [0,4194304] 0 2024-03-14T13:02:14.062 INFO:tasks.workunit.client.1.smithi137.stdout:5/228: fdatasync d7/f15 0 2024-03-14T13:02:14.062 INFO:tasks.workunit.client.1.smithi137.stdout:5/229: chown d7/d16/d18 15 1 2024-03-14T13:02:14.064 INFO:tasks.workunit.client.1.smithi137.stdout:2/267: creat d2/d7/db/de/d22/f5a x:0 0 0 2024-03-14T13:02:14.065 INFO:tasks.workunit.client.0.smithi111.stdout:6/369: dwrite d0/d17/d20/d4a/d59/f6f [0,4194304] 0 2024-03-14T13:02:14.068 INFO:tasks.workunit.client.1.smithi137.stdout:2/268: stat d2/d7/db/de/d22/d4d/d57 0 2024-03-14T13:02:14.069 INFO:tasks.workunit.client.1.smithi137.stdout:9/388: dwrite d6/df/d19/d33/f75 [0,4194304] 0 2024-03-14T13:02:14.073 INFO:tasks.workunit.client.1.smithi137.stdout:5/230: dwrite d7/f15 [4194304,4194304] 0 2024-03-14T13:02:14.082 INFO:tasks.workunit.client.1.smithi137.stdout:5/231: rename d7/d16/d18/d25/d28/c2e to d7/d33/c56 0 2024-03-14T13:02:14.083 INFO:tasks.workunit.client.1.smithi137.stdout:9/389: dwrite d6/df/f51 [4194304,4194304] 0 2024-03-14T13:02:14.084 INFO:tasks.workunit.client.1.smithi137.stdout:2/269: dwrite d2/d7/d2e/f3d [0,4194304] 0 2024-03-14T13:02:14.084 INFO:tasks.workunit.client.1.smithi137.stdout:5/232: fsync d7/d16/f3e 0 2024-03-14T13:02:14.085 INFO:tasks.workunit.client.1.smithi137.stdout:9/390: write d6/df/f51 [7360587,80010] 0 2024-03-14T13:02:14.086 INFO:tasks.workunit.client.1.smithi137.stdout:9/391: chown d6/d26 121 1 2024-03-14T13:02:14.088 INFO:tasks.workunit.client.1.smithi137.stdout:2/270: symlink d2/d34/d3c/d53/l5b 0 2024-03-14T13:02:14.089 INFO:tasks.workunit.client.1.smithi137.stdout:5/233: rename d7/d16/d18/d2d/f52 to d7/d1d/f57 0 2024-03-14T13:02:14.093 INFO:tasks.workunit.client.1.smithi137.stdout:5/234: rename c1 to d7/d16/c58 0 2024-03-14T13:02:14.095 INFO:tasks.workunit.client.1.smithi137.stdout:5/235: mknod d7/d36/c59 0 2024-03-14T13:02:14.098 INFO:tasks.workunit.client.1.smithi137.stdout:2/271: getdents d2/d7/d2e/d3e 0 2024-03-14T13:02:14.099 INFO:tasks.workunit.client.1.smithi137.stdout:9/392: write d6/df/d19/d3a/d40/f4b [634702,63341] 0 2024-03-14T13:02:14.100 INFO:tasks.workunit.client.1.smithi137.stdout:5/236: mknod d7/d16/d41/c5a 0 2024-03-14T13:02:14.100 INFO:tasks.workunit.client.1.smithi137.stdout:9/393: fsync d6/df/d19/d3a/f67 0 2024-03-14T13:02:14.101 INFO:tasks.workunit.client.1.smithi137.stdout:5/237: chown d7/d16/d18/d25 0 1 2024-03-14T13:02:14.102 INFO:tasks.workunit.client.1.smithi137.stdout:2/272: rename d2/d7/d2e/d3e/d44/f56 to d2/d7/db/de/d22/d26/f5c 0 2024-03-14T13:02:14.103 INFO:tasks.workunit.client.1.smithi137.stdout:2/273: write d2/d7/db/de/f39 [949556,28991] 0 2024-03-14T13:02:14.105 INFO:tasks.workunit.client.1.smithi137.stdout:5/238: creat d7/d16/d18/d25/f5b x:0 0 0 2024-03-14T13:02:14.106 INFO:tasks.workunit.client.1.smithi137.stdout:5/239: fsync d7/d16/d18/d2d/f53 0 2024-03-14T13:02:14.106 INFO:tasks.workunit.client.1.smithi137.stdout:5/240: dread - d7/d16/d18/d2d/f53 zero size 2024-03-14T13:02:14.107 INFO:tasks.workunit.client.1.smithi137.stdout:2/274: creat d2/d34/d3c/f5d x:0 0 0 2024-03-14T13:02:14.109 INFO:tasks.workunit.client.1.smithi137.stdout:9/394: rmdir d6/d56/d7a 0 2024-03-14T13:02:14.109 INFO:tasks.workunit.client.1.smithi137.stdout:5/241: mknod d7/d36/c5c 0 2024-03-14T13:02:14.115 INFO:tasks.workunit.client.1.smithi137.stdout:1/358: sync 2024-03-14T13:02:14.115 INFO:tasks.workunit.client.1.smithi137.stdout:9/395: mkdir d6/df/d88 0 2024-03-14T13:02:14.118 INFO:tasks.workunit.client.1.smithi137.stdout:2/275: rename d2/d7/f13 to d2/d34/d3c/d53/f5e 0 2024-03-14T13:02:14.118 INFO:tasks.workunit.client.1.smithi137.stdout:1/359: write d7/d9/d19/f74 [580476,10967] 0 2024-03-14T13:02:14.118 INFO:tasks.workunit.client.1.smithi137.stdout:5/242: mknod d7/d16/d18/d2d/c5d 0 2024-03-14T13:02:14.122 INFO:tasks.workunit.client.1.smithi137.stdout:5/243: fsync d7/d16/f38 0 2024-03-14T13:02:14.122 INFO:tasks.workunit.client.1.smithi137.stdout:9/396: truncate d6/f69 1623729 0 2024-03-14T13:02:14.125 INFO:tasks.workunit.client.1.smithi137.stdout:1/360: mkdir d7/d9/d19/d40/d7a 0 2024-03-14T13:02:14.125 INFO:tasks.workunit.client.1.smithi137.stdout:9/397: truncate d6/df/d19/d33/f3e 443688 0 2024-03-14T13:02:14.127 INFO:tasks.workunit.client.1.smithi137.stdout:1/361: write d7/d31/d54/f62 [589848,94147] 0 2024-03-14T13:02:14.133 INFO:tasks.workunit.client.1.smithi137.stdout:9/398: creat d6/d56/f89 x:0 0 0 2024-03-14T13:02:14.133 INFO:tasks.workunit.client.1.smithi137.stdout:2/276: getdents d2/d7/db/de 0 2024-03-14T13:02:14.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/362: creat d7/d9/d19/d5f/f7b x:0 0 0 2024-03-14T13:02:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:13 smithi111 ceph-mon[39688]: pgmap v21: 65 pgs: 65 active+clean; 691 MiB data, 3.6 GiB used, 533 GiB / 536 GiB avail; 4.6 MiB/s rd, 56 MiB/s wr, 98 op/s 2024-03-14T13:02:14.140 INFO:tasks.workunit.client.1.smithi137.stdout:2/277: readlink d2/d7/l15 0 2024-03-14T13:02:14.144 INFO:tasks.workunit.client.1.smithi137.stdout:2/278: creat d2/d7/db/d2f/f5f x:0 0 0 2024-03-14T13:02:14.144 INFO:tasks.workunit.client.0.smithi111.stdout:6/370: truncate d0/d17/d24/f38 3915480 0 2024-03-14T13:02:14.150 INFO:tasks.workunit.client.1.smithi137.stdout:2/279: link d2/d7/db/de/d22/f4c d2/d7/d2e/d3e/d44/f60 0 2024-03-14T13:02:14.153 INFO:tasks.workunit.client.1.smithi137.stdout:5/244: write d7/d1d/f57 [994739,124237] 0 2024-03-14T13:02:14.155 INFO:tasks.workunit.client.1.smithi137.stdout:5/245: fdatasync d7/d16/d18/d2d/f2f 0 2024-03-14T13:02:14.157 INFO:tasks.workunit.client.1.smithi137.stdout:2/280: creat d2/d7/db/de/d1d/d2d/f61 x:0 0 0 2024-03-14T13:02:14.158 INFO:tasks.workunit.client.1.smithi137.stdout:2/281: dread - d2/d7/db/de/d22/d26/f5c zero size 2024-03-14T13:02:14.158 INFO:tasks.workunit.client.1.smithi137.stdout:5/246: creat d7/d33/f5e x:0 0 0 2024-03-14T13:02:14.171 INFO:tasks.workunit.client.1.smithi137.stdout:9/399: dread d6/df/f13 [0,4194304] 0 2024-03-14T13:02:14.173 INFO:tasks.workunit.client.1.smithi137.stdout:1/363: dwrite d7/d9/d19/d1f/d2e/f34 [0,4194304] 0 2024-03-14T13:02:14.174 INFO:tasks.workunit.client.0.smithi111.stdout:6/371: dwrite d0/d17/d20/f26 [0,4194304] 0 2024-03-14T13:02:14.176 INFO:tasks.workunit.client.1.smithi137.stdout:9/400: chown d6/df/d19/l23 234 1 2024-03-14T13:02:14.183 INFO:tasks.workunit.client.0.smithi111.stdout:6/372: dwrite d0/d17/d21/d69/d3d/f41 [0,4194304] 0 2024-03-14T13:02:14.185 INFO:tasks.workunit.client.1.smithi137.stdout:1/364: symlink d7/d9/d19/d1f/d26/d6c/l7c 0 2024-03-14T13:02:14.187 INFO:tasks.workunit.client.1.smithi137.stdout:9/401: rmdir d6/df/d19/d3a/d40 39 2024-03-14T13:02:14.191 INFO:tasks.workunit.client.0.smithi111.stdout:6/373: dwrite d0/d17/d21/f79 [0,4194304] 0 2024-03-14T13:02:14.193 INFO:tasks.workunit.client.1.smithi137.stdout:9/402: unlink d6/df/l2e 0 2024-03-14T13:02:14.194 INFO:tasks.workunit.client.1.smithi137.stdout:2/282: dread d2/d7/db/de/f1b [0,4194304] 0 2024-03-14T13:02:14.196 INFO:tasks.workunit.client.1.smithi137.stdout:1/365: dread d7/d9/d19/d1f/d2e/f34 [0,4194304] 0 2024-03-14T13:02:14.197 INFO:tasks.workunit.client.1.smithi137.stdout:1/366: chown d7/d31 888345168 1 2024-03-14T13:02:14.197 INFO:tasks.workunit.client.1.smithi137.stdout:9/403: symlink d6/l8a 0 2024-03-14T13:02:14.198 INFO:tasks.workunit.client.0.smithi111.stdout:6/374: link d0/f3e d0/d17/d20/d4a/d59/d82/f8f 0 2024-03-14T13:02:14.200 INFO:tasks.workunit.client.0.smithi111.stdout:6/375: rename d0/d17/d21/d69/d3d/l54 to d0/l90 0 2024-03-14T13:02:14.201 INFO:tasks.workunit.client.1.smithi137.stdout:9/404: creat d6/df/d71/f8b x:0 0 0 2024-03-14T13:02:14.202 INFO:tasks.workunit.client.0.smithi111.stdout:6/376: mkdir d0/d91 0 2024-03-14T13:02:14.209 INFO:tasks.workunit.client.1.smithi137.stdout:9/405: mknod d6/d56/c8c 0 2024-03-14T13:02:14.209 INFO:tasks.workunit.client.1.smithi137.stdout:2/283: link d2/d7/db/de/d22/l50 d2/d34/l62 0 2024-03-14T13:02:14.209 INFO:tasks.workunit.client.1.smithi137.stdout:5/247: dread d7/d16/d18/f2a [0,4194304] 0 2024-03-14T13:02:14.211 INFO:tasks.workunit.client.1.smithi137.stdout:5/248: read d7/f10 [4000896,127905] 0 2024-03-14T13:02:14.215 INFO:tasks.workunit.client.1.smithi137.stdout:2/284: fsync d2/f14 0 2024-03-14T13:02:14.215 INFO:tasks.workunit.client.1.smithi137.stdout:2/285: chown d2/d7/d2e 24687558 1 2024-03-14T13:02:14.215 INFO:tasks.workunit.client.1.smithi137.stdout:5/249: write d7/d16/d18/d2d/f53 [688587,118403] 0 2024-03-14T13:02:14.215 INFO:tasks.workunit.client.1.smithi137.stdout:9/406: mknod d6/df/d19/c8d 0 2024-03-14T13:02:14.215 INFO:tasks.workunit.client.1.smithi137.stdout:5/250: dread - d7/d16/d18/d25/d28/d3c/f4f zero size 2024-03-14T13:02:14.218 INFO:tasks.workunit.client.1.smithi137.stdout:5/251: write d7/f30 [2102631,116799] 0 2024-03-14T13:02:14.219 INFO:tasks.workunit.client.1.smithi137.stdout:9/407: write d6/df/d19/d3a/d40/f7f [566557,125856] 0 2024-03-14T13:02:14.221 INFO:tasks.workunit.client.1.smithi137.stdout:9/408: chown d6/df/f51 66 1 2024-03-14T13:02:14.223 INFO:tasks.workunit.client.1.smithi137.stdout:2/286: dwrite d2/d7/db/de/f20 [4194304,4194304] 0 2024-03-14T13:02:14.223 INFO:tasks.workunit.client.1.smithi137.stdout:5/252: chown d7/d16/d18/d25/f50 669923 1 2024-03-14T13:02:14.224 INFO:tasks.workunit.client.1.smithi137.stdout:9/409: chown d6/df/d19/d33/f65 853 1 2024-03-14T13:02:14.226 INFO:tasks.workunit.client.1.smithi137.stdout:9/410: write d6/df/d19/d1f/f35 [3545370,50870] 0 2024-03-14T13:02:14.228 INFO:tasks.workunit.client.1.smithi137.stdout:5/253: creat d7/d33/f5f x:0 0 0 2024-03-14T13:02:14.230 INFO:tasks.workunit.client.1.smithi137.stdout:1/367: truncate d7/d4d/f56 5409904 0 2024-03-14T13:02:14.232 INFO:tasks.workunit.client.1.smithi137.stdout:9/411: mknod d6/df/d71/c8e 0 2024-03-14T13:02:14.232 INFO:tasks.workunit.client.1.smithi137.stdout:1/368: dread - d7/d4d/f64 zero size 2024-03-14T13:02:14.234 INFO:tasks.workunit.client.1.smithi137.stdout:1/369: truncate d7/d9/d19/d1f/f69 1030279 0 2024-03-14T13:02:14.235 INFO:tasks.workunit.client.1.smithi137.stdout:5/254: creat d7/d16/d18/d25/d28/f60 x:0 0 0 2024-03-14T13:02:14.236 INFO:tasks.workunit.client.1.smithi137.stdout:9/412: rmdir d6/d56 39 2024-03-14T13:02:14.237 INFO:tasks.workunit.client.1.smithi137.stdout:5/255: stat d7/d49 0 2024-03-14T13:02:14.237 INFO:tasks.workunit.client.1.smithi137.stdout:1/370: chown d7/d9/d19/d1f/d26/l4e 133866 1 2024-03-14T13:02:14.238 INFO:tasks.workunit.client.0.smithi111.stdout:6/377: dread d0/d17/d21/d69/f52 [0,4194304] 0 2024-03-14T13:02:14.240 INFO:tasks.workunit.client.0.smithi111.stdout:6/378: creat d0/d17/d20/d65/f92 x:0 0 0 2024-03-14T13:02:14.241 INFO:tasks.workunit.client.1.smithi137.stdout:2/287: truncate d2/d7/db/de/f20 5646029 0 2024-03-14T13:02:14.243 INFO:tasks.workunit.client.1.smithi137.stdout:9/413: rmdir d6/df 39 2024-03-14T13:02:14.244 INFO:tasks.workunit.client.1.smithi137.stdout:1/371: mkdir d7/d9/d19/d1f/d26/d7d 0 2024-03-14T13:02:14.245 INFO:tasks.workunit.client.1.smithi137.stdout:2/288: creat d2/d7/db/de/d22/d4d/f63 x:0 0 0 2024-03-14T13:02:14.246 INFO:tasks.workunit.client.0.smithi111.stdout:6/379: dwrite d0/d17/d21/d42/f58 [0,4194304] 0 2024-03-14T13:02:14.249 INFO:tasks.workunit.client.0.smithi111.stdout:6/380: mkdir d0/d17/d20/d65/d93 0 2024-03-14T13:02:14.249 INFO:tasks.workunit.client.1.smithi137.stdout:2/289: fdatasync d2/f4b 0 2024-03-14T13:02:14.250 INFO:tasks.workunit.client.1.smithi137.stdout:5/256: dread d7/d16/d18/f20 [0,4194304] 0 2024-03-14T13:02:14.250 INFO:tasks.workunit.client.1.smithi137.stdout:9/414: mkdir d6/df/d19/d33/d8f 0 2024-03-14T13:02:14.251 INFO:tasks.workunit.client.0.smithi111.stdout:6/381: mkdir d0/d17/d20/d94 0 2024-03-14T13:02:14.254 INFO:tasks.workunit.client.0.smithi111.stdout:6/382: rmdir d0/d91 0 2024-03-14T13:02:14.255 INFO:tasks.workunit.client.1.smithi137.stdout:1/372: dwrite d7/d9/d19/d1f/f69 [0,4194304] 0 2024-03-14T13:02:14.255 INFO:tasks.workunit.client.1.smithi137.stdout:7/262: sync 2024-03-14T13:02:14.255 INFO:tasks.workunit.client.1.smithi137.stdout:0/280: sync 2024-03-14T13:02:14.256 INFO:tasks.workunit.client.0.smithi111.stdout:6/383: creat d0/d17/d21/d69/d3d/d53/d81/f95 x:0 0 0 2024-03-14T13:02:14.259 INFO:tasks.workunit.client.1.smithi137.stdout:9/415: chown d6/df/d71/f8b 43830 1 2024-03-14T13:02:14.259 INFO:tasks.workunit.client.1.smithi137.stdout:2/290: write d2/d7/db/de/f39 [2436827,48835] 0 2024-03-14T13:02:14.260 INFO:tasks.workunit.client.0.smithi111.stdout:6/384: dread d0/d17/d21/d69/f39 [0,4194304] 0 2024-03-14T13:02:14.260 INFO:tasks.workunit.client.1.smithi137.stdout:2/291: dread d2/f14 [0,4194304] 0 2024-03-14T13:02:14.261 INFO:tasks.workunit.client.1.smithi137.stdout:7/263: rename d3/d5/l17 to d3/d4a/l58 0 2024-03-14T13:02:14.261 INFO:tasks.workunit.client.1.smithi137.stdout:0/281: fsync d1/f13 0 2024-03-14T13:02:14.263 INFO:tasks.workunit.client.1.smithi137.stdout:1/373: write d7/d9/d19/d1f/d22/f25 [2734070,101545] 0 2024-03-14T13:02:14.263 INFO:tasks.workunit.client.1.smithi137.stdout:0/282: stat d1/d5/d1b/d42/f32 0 2024-03-14T13:02:14.265 INFO:tasks.workunit.client.1.smithi137.stdout:7/264: mkdir d3/d6/d59 0 2024-03-14T13:02:14.266 INFO:tasks.workunit.client.1.smithi137.stdout:2/292: mkdir d2/d7/db/de/d22/d4d/d57/d64 0 2024-03-14T13:02:14.267 INFO:tasks.workunit.client.1.smithi137.stdout:0/283: unlink d1/d11/f19 0 2024-03-14T13:02:14.269 INFO:tasks.workunit.client.1.smithi137.stdout:9/416: rename d6/df/d19/f4c to d6/df/f90 0 2024-03-14T13:02:14.269 INFO:tasks.workunit.client.1.smithi137.stdout:9/417: read - d6/f6d zero size 2024-03-14T13:02:14.270 INFO:tasks.workunit.client.1.smithi137.stdout:1/374: rmdir d7/d9/d19/d1f/d26/d6c 39 2024-03-14T13:02:14.272 INFO:tasks.workunit.client.1.smithi137.stdout:1/375: write d7/d9/d19/d1f/f23 [744805,122933] 0 2024-03-14T13:02:14.272 INFO:tasks.workunit.client.1.smithi137.stdout:7/265: mkdir d3/d5/d5a 0 2024-03-14T13:02:14.273 INFO:tasks.workunit.client.1.smithi137.stdout:7/266: readlink d3/d5/la 0 2024-03-14T13:02:14.274 INFO:tasks.workunit.client.1.smithi137.stdout:7/267: chown d3/f1c 356211 1 2024-03-14T13:02:14.275 INFO:tasks.workunit.client.1.smithi137.stdout:7/268: readlink d3/d6/d14/d22/l41 0 2024-03-14T13:02:14.275 INFO:tasks.workunit.client.1.smithi137.stdout:2/293: symlink d2/d7/d2e/l65 0 2024-03-14T13:02:14.277 INFO:tasks.workunit.client.1.smithi137.stdout:5/257: dread d7/f1b [0,4194304] 0 2024-03-14T13:02:14.277 INFO:tasks.workunit.client.1.smithi137.stdout:7/269: truncate d3/d5/f46 400006 0 2024-03-14T13:02:14.278 INFO:tasks.workunit.client.1.smithi137.stdout:0/284: creat d1/d5/d1b/d1e/d25/d55/f67 x:0 0 0 2024-03-14T13:02:14.281 INFO:tasks.workunit.client.1.smithi137.stdout:2/294: dread d2/d7/db/f38 [0,4194304] 0 2024-03-14T13:02:14.282 INFO:tasks.workunit.client.1.smithi137.stdout:0/285: dread d1/d11/f21 [0,4194304] 0 2024-03-14T13:02:14.284 INFO:tasks.workunit.client.1.smithi137.stdout:9/418: mkdir d6/d91 0 2024-03-14T13:02:14.285 INFO:tasks.workunit.client.1.smithi137.stdout:5/258: unlink d7/d36/c59 0 2024-03-14T13:02:14.288 INFO:tasks.workunit.client.1.smithi137.stdout:5/259: chown d7/d16/d18/d2d/c5d 265685 1 2024-03-14T13:02:14.291 INFO:tasks.workunit.client.1.smithi137.stdout:2/295: dwrite d2/d7/db/d2f/f45 [0,4194304] 0 2024-03-14T13:02:14.293 INFO:tasks.workunit.client.1.smithi137.stdout:7/270: unlink d3/d6/d14/d27/c4c 0 2024-03-14T13:02:14.293 INFO:tasks.workunit.client.1.smithi137.stdout:0/286: dread d1/d11/d4a/d54/f5b [0,4194304] 0 2024-03-14T13:02:14.293 INFO:tasks.workunit.client.1.smithi137.stdout:9/419: chown d6/df/c30 103300254 1 2024-03-14T13:02:14.296 INFO:tasks.workunit.client.1.smithi137.stdout:5/260: write d7/d16/d18/d25/d28/f3d [739345,101198] 0 2024-03-14T13:02:14.298 INFO:tasks.workunit.client.1.smithi137.stdout:0/287: mknod d1/d44/c68 0 2024-03-14T13:02:14.298 INFO:tasks.workunit.client.1.smithi137.stdout:5/261: write d7/d16/d18/d2d/f53 [682452,64497] 0 2024-03-14T13:02:14.298 INFO:tasks.workunit.client.1.smithi137.stdout:5/262: readlink d7/l21 0 2024-03-14T13:02:14.300 INFO:tasks.workunit.client.1.smithi137.stdout:5/263: readlink d7/d16/d18/l40 0 2024-03-14T13:02:14.300 INFO:tasks.workunit.client.1.smithi137.stdout:2/296: rename d2/d34/d3c/d53/d54 to d2/d7/db/d2f/d66 0 2024-03-14T13:02:14.301 INFO:tasks.workunit.client.1.smithi137.stdout:0/288: truncate d1/d5/d1b/d1e/d25/d26/f5d 4845507 0 2024-03-14T13:02:14.302 INFO:tasks.workunit.client.1.smithi137.stdout:5/264: dread - d7/d16/d18/d25/d28/d3c/f4f zero size 2024-03-14T13:02:14.303 INFO:tasks.workunit.client.1.smithi137.stdout:9/420: creat d6/df/f92 x:0 0 0 2024-03-14T13:02:14.304 INFO:tasks.workunit.client.1.smithi137.stdout:0/289: truncate d1/d44/d56/f5f 786004 0 2024-03-14T13:02:14.304 INFO:tasks.workunit.client.1.smithi137.stdout:5/265: creat d7/d33/f61 x:0 0 0 2024-03-14T13:02:14.305 INFO:tasks.workunit.client.1.smithi137.stdout:9/421: write d6/df/d19/d1f/f7e [914508,39738] 0 2024-03-14T13:02:14.307 INFO:tasks.workunit.client.1.smithi137.stdout:7/271: dread d3/fe [0,4194304] 0 2024-03-14T13:02:14.307 INFO:tasks.workunit.client.1.smithi137.stdout:0/290: unlink d1/d11/c52 0 2024-03-14T13:02:14.309 INFO:tasks.workunit.client.1.smithi137.stdout:9/422: fsync d6/df/d71/f72 0 2024-03-14T13:02:14.311 INFO:tasks.workunit.client.1.smithi137.stdout:7/272: mkdir d3/d5/d23/d3f/d5b 0 2024-03-14T13:02:14.311 INFO:tasks.workunit.client.1.smithi137.stdout:0/291: write d1/d11/d4a/d54/f5b [3270281,100542] 0 2024-03-14T13:02:14.312 INFO:tasks.workunit.client.1.smithi137.stdout:7/273: write d3/d6/d14/f2f [364002,62903] 0 2024-03-14T13:02:14.313 INFO:tasks.workunit.client.1.smithi137.stdout:7/274: truncate d3/d5/f19 713055 0 2024-03-14T13:02:14.317 INFO:tasks.workunit.client.1.smithi137.stdout:0/292: mknod d1/d11/c69 0 2024-03-14T13:02:14.318 INFO:tasks.workunit.client.1.smithi137.stdout:5/266: getdents d7/d16/d18/d25/d28 0 2024-03-14T13:02:14.319 INFO:tasks.workunit.client.1.smithi137.stdout:5/267: chown d7/d16/l51 0 1 2024-03-14T13:02:14.322 INFO:tasks.workunit.client.1.smithi137.stdout:9/423: getdents d6/df/d88 0 2024-03-14T13:02:14.323 INFO:tasks.workunit.client.1.smithi137.stdout:9/424: write d6/df/f5e [336672,4325] 0 2024-03-14T13:02:14.324 INFO:tasks.workunit.client.1.smithi137.stdout:9/425: write d6/df/f5e [162852,45105] 0 2024-03-14T13:02:14.326 INFO:tasks.workunit.client.1.smithi137.stdout:0/293: truncate d1/f46 2361599 0 2024-03-14T13:02:14.326 INFO:tasks.workunit.client.1.smithi137.stdout:7/275: creat d3/d54/f5c x:0 0 0 2024-03-14T13:02:14.326 INFO:tasks.workunit.client.1.smithi137.stdout:0/294: chown d1/d5/d1b/d1e/d53 2086507776 1 2024-03-14T13:02:14.326 INFO:tasks.workunit.client.1.smithi137.stdout:4/325: sync 2024-03-14T13:02:14.327 INFO:tasks.workunit.client.1.smithi137.stdout:8/201: sync 2024-03-14T13:02:14.327 INFO:tasks.workunit.client.1.smithi137.stdout:3/254: sync 2024-03-14T13:02:14.328 INFO:tasks.workunit.client.1.smithi137.stdout:8/202: dread - d1/dc/d11/f2d zero size 2024-03-14T13:02:14.329 INFO:tasks.workunit.client.1.smithi137.stdout:5/268: creat d7/d36/f62 x:0 0 0 2024-03-14T13:02:14.330 INFO:tasks.workunit.client.1.smithi137.stdout:5/269: chown d7/f10 219 1 2024-03-14T13:02:14.331 INFO:tasks.workunit.client.1.smithi137.stdout:9/426: rename d6/df/d19/d1f/f7e to d6/d26/f93 0 2024-03-14T13:02:14.337 INFO:tasks.workunit.client.1.smithi137.stdout:4/326: dwrite d4/d9/f18 [0,4194304] 0 2024-03-14T13:02:14.343 INFO:tasks.workunit.client.1.smithi137.stdout:5/270: rename d7/d33/f5e to d7/d16/d18/d2d/f63 0 2024-03-14T13:02:14.344 INFO:tasks.workunit.client.1.smithi137.stdout:3/255: mknod d4/df/d35/d4b/c4f 0 2024-03-14T13:02:14.346 INFO:tasks.workunit.client.1.smithi137.stdout:9/427: dwrite d6/df/f92 [0,4194304] 0 2024-03-14T13:02:14.349 INFO:tasks.workunit.client.1.smithi137.stdout:5/271: truncate d7/d16/d18/d25/f3f 574481 0 2024-03-14T13:02:14.350 INFO:tasks.workunit.client.1.smithi137.stdout:9/428: write d6/df/d19/d1f/f4a [1285389,771] 0 2024-03-14T13:02:14.352 INFO:tasks.workunit.client.1.smithi137.stdout:2/297: fsync d2/f1e 0 2024-03-14T13:02:14.353 INFO:tasks.workunit.client.1.smithi137.stdout:8/203: dwrite d1/d17/d34/f47 [4194304,4194304] 0 2024-03-14T13:02:14.354 INFO:tasks.workunit.client.0.smithi111.stdout:5/440: sync 2024-03-14T13:02:14.354 INFO:tasks.workunit.client.0.smithi111.stdout:0/314: sync 2024-03-14T13:02:14.355 INFO:tasks.workunit.client.0.smithi111.stdout:0/315: chown d6/d5a 156 1 2024-03-14T13:02:14.356 INFO:tasks.workunit.client.1.smithi137.stdout:7/276: dwrite d3/d6/f9 [0,4194304] 0 2024-03-14T13:02:14.358 INFO:tasks.workunit.client.0.smithi111.stdout:5/441: unlink d2/d45/d76/d8d/c86 0 2024-03-14T13:02:14.358 INFO:tasks.workunit.client.0.smithi111.stdout:5/442: readlink d2/d45/l69 0 2024-03-14T13:02:14.359 INFO:tasks.workunit.client.0.smithi111.stdout:0/316: mknod d6/d3d/c63 0 2024-03-14T13:02:14.362 INFO:tasks.workunit.client.0.smithi111.stdout:0/317: mkdir d6/d9/de/d32/d64 0 2024-03-14T13:02:14.364 INFO:tasks.workunit.client.0.smithi111.stdout:0/318: symlink d6/d9/l65 0 2024-03-14T13:02:14.364 INFO:tasks.workunit.client.0.smithi111.stdout:0/319: stat d6/d9/d15/d49/c51 0 2024-03-14T13:02:14.365 INFO:tasks.workunit.client.0.smithi111.stdout:0/320: dread - d6/d45/f4b zero size 2024-03-14T13:02:14.366 INFO:tasks.workunit.client.1.smithi137.stdout:1/376: fsync d7/d9/d19/d1f/f69 0 2024-03-14T13:02:14.368 INFO:tasks.workunit.client.1.smithi137.stdout:9/429: dwrite d6/df/d19/d1f/f3b [0,4194304] 0 2024-03-14T13:02:14.371 INFO:tasks.workunit.client.0.smithi111.stdout:0/321: dwrite f3 [8388608,4194304] 0 2024-03-14T13:02:14.373 INFO:tasks.workunit.client.1.smithi137.stdout:8/204: dwrite d1/d17/d34/f40 [0,4194304] 0 2024-03-14T13:02:14.379 INFO:tasks.workunit.client.1.smithi137.stdout:8/205: dread d1/d17/f1e [0,4194304] 0 2024-03-14T13:02:14.384 INFO:tasks.workunit.client.1.smithi137.stdout:8/206: dread d1/d17/f1c [0,4194304] 0 2024-03-14T13:02:14.385 INFO:tasks.workunit.client.1.smithi137.stdout:8/207: dread - d1/dc/d11/d13/f4f zero size 2024-03-14T13:02:14.386 INFO:tasks.workunit.client.0.smithi111.stdout:0/322: dwrite d6/d21/f42 [0,4194304] 0 2024-03-14T13:02:14.390 INFO:tasks.workunit.client.0.smithi111.stdout:0/323: creat d6/d9/de/d3a/d46/f66 x:0 0 0 2024-03-14T13:02:14.393 INFO:tasks.workunit.client.0.smithi111.stdout:5/443: truncate d2/d13/f4f 8359 0 2024-03-14T13:02:14.394 INFO:tasks.workunit.client.0.smithi111.stdout:0/324: dread - d6/d9/de/d3a/f61 zero size 2024-03-14T13:02:14.394 INFO:tasks.workunit.client.0.smithi111.stdout:0/325: readlink d6/d9/de/d3a/d46/d4a/l59 0 2024-03-14T13:02:14.394 INFO:tasks.workunit.client.0.smithi111.stdout:0/326: stat d6/d9/de/d3a/d46/f66 0 2024-03-14T13:02:14.396 INFO:tasks.workunit.client.0.smithi111.stdout:0/327: symlink d6/d9/d15/d49/d56/l67 0 2024-03-14T13:02:14.396 INFO:tasks.workunit.client.0.smithi111.stdout:5/444: read d2/d16/d49/d4c/f73 [2559948,14431] 0 2024-03-14T13:02:14.397 INFO:tasks.workunit.client.0.smithi111.stdout:5/445: fdatasync d2/d16/d49/d4c/f55 0 2024-03-14T13:02:14.398 INFO:tasks.workunit.client.0.smithi111.stdout:0/328: unlink d6/d9/d15/l3e 0 2024-03-14T13:02:14.400 INFO:tasks.workunit.client.0.smithi111.stdout:5/446: rmdir d2/d16/d49/d7c 39 2024-03-14T13:02:14.401 INFO:tasks.workunit.client.0.smithi111.stdout:5/447: readlink d2/l85 0 2024-03-14T13:02:14.404 INFO:tasks.workunit.client.0.smithi111.stdout:5/448: rename d2/d1f to d2/d16/d49/d7c/d95 0 2024-03-14T13:02:14.417 INFO:tasks.workunit.client.0.smithi111.stdout:5/449: truncate d2/d16/d49/f12 2460189 0 2024-03-14T13:02:14.417 INFO:tasks.workunit.client.0.smithi111.stdout:5/450: stat d2/d16/d49/d33/l5f 0 2024-03-14T13:02:14.420 INFO:tasks.workunit.client.0.smithi111.stdout:5/451: unlink d2/d16/d49/d33/f40 0 2024-03-14T13:02:14.420 INFO:tasks.workunit.client.0.smithi111.stdout:0/329: dread d6/d9/de/f18 [0,4194304] 0 2024-03-14T13:02:14.421 INFO:tasks.workunit.client.0.smithi111.stdout:5/452: chown d2/d45/d79/c89 18796 1 2024-03-14T13:02:14.425 INFO:tasks.workunit.client.0.smithi111.stdout:5/453: unlink d2/d16/d49/d7c/d95/d6a/c94 0 2024-03-14T13:02:14.425 INFO:tasks.workunit.client.0.smithi111.stdout:0/330: dread d6/fd [8388608,4194304] 0 2024-03-14T13:02:14.425 INFO:tasks.workunit.client.0.smithi111.stdout:8/544: sync 2024-03-14T13:02:14.425 INFO:tasks.workunit.client.0.smithi111.stdout:9/399: sync 2024-03-14T13:02:14.425 INFO:tasks.workunit.client.0.smithi111.stdout:1/385: sync 2024-03-14T13:02:14.427 INFO:tasks.workunit.client.0.smithi111.stdout:5/454: write d2/d16/d49/d44/d4a/f60 [3514156,27016] 0 2024-03-14T13:02:14.427 INFO:tasks.workunit.client.0.smithi111.stdout:9/400: chown d5/d6/d2a 198748 1 2024-03-14T13:02:14.428 INFO:tasks.workunit.client.0.smithi111.stdout:0/331: creat d6/d9/de/d32/d64/f68 x:0 0 0 2024-03-14T13:02:14.428 INFO:tasks.workunit.client.0.smithi111.stdout:5/455: readlink d2/d16/d49/d44/l61 0 2024-03-14T13:02:14.429 INFO:tasks.workunit.client.0.smithi111.stdout:1/386: symlink d20/l86 0 2024-03-14T13:02:14.430 INFO:tasks.workunit.client.0.smithi111.stdout:9/401: dread - d5/d6/d8/d5b/f6e zero size 2024-03-14T13:02:14.430 INFO:tasks.workunit.client.0.smithi111.stdout:8/545: symlink d3/dc/d45/d8d/lab 0 2024-03-14T13:02:14.431 INFO:tasks.workunit.client.0.smithi111.stdout:1/387: fdatasync f11 0 2024-03-14T13:02:14.432 INFO:tasks.workunit.client.0.smithi111.stdout:8/546: readlink d3/d1b/d20/d35/l4f 0 2024-03-14T13:02:14.433 INFO:tasks.workunit.client.0.smithi111.stdout:9/402: chown d5/c28 5 1 2024-03-14T13:02:14.433 INFO:tasks.workunit.client.0.smithi111.stdout:1/388: chown l8 12187666 1 2024-03-14T13:02:14.436 INFO:tasks.workunit.client.1.smithi137.stdout:0/295: link d1/d1f/l2c d1/d1f/l6a 0 2024-03-14T13:02:14.437 INFO:tasks.workunit.client.0.smithi111.stdout:5/456: unlink d2/d16/d49/f12 0 2024-03-14T13:02:14.438 INFO:tasks.workunit.client.0.smithi111.stdout:0/332: dwrite d6/d9/de/d3a/d46/d4a/f5c [0,4194304] 0 2024-03-14T13:02:14.438 INFO:tasks.workunit.client.1.smithi137.stdout:3/256: symlink d4/de/d13/l50 0 2024-03-14T13:02:14.440 INFO:tasks.workunit.client.0.smithi111.stdout:8/547: readlink d3/d1b/d20/d35/d40/d48/l5d 0 2024-03-14T13:02:14.440 INFO:tasks.workunit.client.0.smithi111.stdout:0/333: fdatasync f1 0 2024-03-14T13:02:14.441 INFO:tasks.workunit.client.0.smithi111.stdout:8/548: dread - d3/d5/f9c zero size 2024-03-14T13:02:14.441 INFO:tasks.workunit.client.0.smithi111.stdout:1/389: mkdir d20/d25/d87 0 2024-03-14T13:02:14.441 INFO:tasks.workunit.client.0.smithi111.stdout:0/334: stat d6/d9/de/d32/c39 0 2024-03-14T13:02:14.443 INFO:tasks.workunit.client.0.smithi111.stdout:5/457: dread d2/d16/d49/d33/f7b [0,4194304] 0 2024-03-14T13:02:14.443 INFO:tasks.workunit.client.0.smithi111.stdout:1/390: write f9 [354561,84044] 0 2024-03-14T13:02:14.444 INFO:tasks.workunit.client.0.smithi111.stdout:9/403: symlink d5/d6/d8/d5b/d84/l85 0 2024-03-14T13:02:14.444 INFO:tasks.workunit.client.1.smithi137.stdout:7/277: mkdir d3/d5/d23/d2c/d5d 0 2024-03-14T13:02:14.445 INFO:tasks.workunit.client.0.smithi111.stdout:9/404: chown d5/d42/l78 659156 1 2024-03-14T13:02:14.446 INFO:tasks.workunit.client.1.smithi137.stdout:2/298: symlink d2/d34/d3c/d53/l67 0 2024-03-14T13:02:14.447 INFO:tasks.workunit.client.1.smithi137.stdout:7/278: dread - d3/d54/f5c zero size 2024-03-14T13:02:14.447 INFO:tasks.workunit.client.1.smithi137.stdout:5/272: dwrite d7/d16/d18/d25/f55 [0,4194304] 0 2024-03-14T13:02:14.448 INFO:tasks.workunit.client.0.smithi111.stdout:0/335: readlink d6/d9/de/l2d 0 2024-03-14T13:02:14.449 INFO:tasks.workunit.client.1.smithi137.stdout:1/377: rmdir d7 39 2024-03-14T13:02:14.449 INFO:tasks.workunit.client.1.smithi137.stdout:6/310: sync 2024-03-14T13:02:14.452 INFO:tasks.workunit.client.1.smithi137.stdout:9/430: mkdir d6/df/d71/d94 0 2024-03-14T13:02:14.454 INFO:tasks.workunit.client.1.smithi137.stdout:8/208: symlink d1/d17/d34/l50 0 2024-03-14T13:02:14.455 INFO:tasks.workunit.client.1.smithi137.stdout:8/209: fsync d1/f49 0 2024-03-14T13:02:14.456 INFO:tasks.workunit.client.0.smithi111.stdout:5/458: dwrite d2/d16/d49/d7c/d95/d6a/f71 [0,4194304] 0 2024-03-14T13:02:14.456 INFO:tasks.workunit.client.0.smithi111.stdout:0/336: dread d6/f1a [0,4194304] 0 2024-03-14T13:02:14.456 INFO:tasks.workunit.client.1.smithi137.stdout:3/257: chown f1 15010574 1 2024-03-14T13:02:14.457 INFO:tasks.workunit.client.1.smithi137.stdout:7/279: creat d3/d4a/f5e x:0 0 0 2024-03-14T13:02:14.457 INFO:tasks.workunit.client.1.smithi137.stdout:5/273: chown d7/d33/c56 415314 1 2024-03-14T13:02:14.460 INFO:tasks.workunit.client.0.smithi111.stdout:0/337: chown d6/d9/de/d32/l2b 19 1 2024-03-14T13:02:14.460 INFO:tasks.workunit.client.1.smithi137.stdout:9/431: creat d6/df/d19/d1f/f95 x:0 0 0 2024-03-14T13:02:14.462 INFO:tasks.workunit.client.0.smithi111.stdout:9/405: mkdir d5/d6/d2a/d86 0 2024-03-14T13:02:14.462 INFO:tasks.workunit.client.0.smithi111.stdout:1/391: symlink d20/d21/d28/l88 0 2024-03-14T13:02:14.463 INFO:tasks.workunit.client.0.smithi111.stdout:0/338: write d6/d9/de/d3a/f5d [269150,16988] 0 2024-03-14T13:02:14.464 INFO:tasks.workunit.client.1.smithi137.stdout:0/296: rename d1/d5/l1a to d1/d5/d1b/d1e/d25/l6b 0 2024-03-14T13:02:14.465 INFO:tasks.workunit.client.0.smithi111.stdout:5/459: chown d2/d45/d79/c89 19 1 2024-03-14T13:02:14.466 INFO:tasks.workunit.client.1.smithi137.stdout:9/432: dwrite d6/df/d19/d3a/f47 [0,4194304] 0 2024-03-14T13:02:14.468 INFO:tasks.workunit.client.0.smithi111.stdout:0/339: symlink d6/d45/l69 0 2024-03-14T13:02:14.468 INFO:tasks.workunit.client.0.smithi111.stdout:1/392: symlink d20/d25/l89 0 2024-03-14T13:02:14.468 INFO:tasks.workunit.client.0.smithi111.stdout:9/406: mkdir d5/d6/d8/d5b/d87 0 2024-03-14T13:02:14.468 INFO:tasks.workunit.client.0.smithi111.stdout:5/460: fsync d2/d16/d49/d44/d72/f3e 0 2024-03-14T13:02:14.469 INFO:tasks.workunit.client.1.smithi137.stdout:8/210: mkdir d1/dc/d11/d13/d51 0 2024-03-14T13:02:14.470 INFO:tasks.workunit.client.0.smithi111.stdout:0/340: chown d6/d9/de/d32/f31 44982 1 2024-03-14T13:02:14.471 INFO:tasks.workunit.client.0.smithi111.stdout:1/393: chown d20/d40/c6e 15 1 2024-03-14T13:02:14.471 INFO:tasks.workunit.client.0.smithi111.stdout:8/549: dread d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:14.471 INFO:tasks.workunit.client.0.smithi111.stdout:9/407: chown d5/d6/d14/f54 8912 1 2024-03-14T13:02:14.472 INFO:tasks.workunit.client.1.smithi137.stdout:3/258: mknod d4/df/d35/d4b/c51 0 2024-03-14T13:02:14.473 INFO:tasks.workunit.client.0.smithi111.stdout:8/550: chown d3/d1b/d20/d35/d40/d48/d7e 60 1 2024-03-14T13:02:14.475 INFO:tasks.workunit.client.0.smithi111.stdout:5/461: rename d2/d16/d49/d7c/d95/c32 to d2/d13/c96 0 2024-03-14T13:02:14.475 INFO:tasks.workunit.client.1.smithi137.stdout:1/378: unlink d7/d9/d19/f74 0 2024-03-14T13:02:14.477 INFO:tasks.workunit.client.0.smithi111.stdout:5/462: chown d2/c9 50717695 1 2024-03-14T13:02:14.478 INFO:tasks.workunit.client.1.smithi137.stdout:2/299: rename d2/d7/f10 to d2/d7/d2e/d3e/d44/f68 0 2024-03-14T13:02:14.479 INFO:tasks.workunit.client.0.smithi111.stdout:0/341: dwrite d6/d9/de/f27 [0,4194304] 0 2024-03-14T13:02:14.479 INFO:tasks.workunit.client.0.smithi111.stdout:5/463: mknod d2/d16/d49/d33/c97 0 2024-03-14T13:02:14.480 INFO:tasks.workunit.client.1.smithi137.stdout:7/280: unlink d3/d6/d2e/f4f 0 2024-03-14T13:02:14.480 INFO:tasks.workunit.client.0.smithi111.stdout:5/464: write d2/d16/f70 [6476288,69301] 0 2024-03-14T13:02:14.481 INFO:tasks.workunit.client.1.smithi137.stdout:2/300: read d2/d7/d2e/f3d [1878424,19226] 0 2024-03-14T13:02:14.481 INFO:tasks.workunit.client.1.smithi137.stdout:8/211: creat d1/dc/f52 x:0 0 0 2024-03-14T13:02:14.484 INFO:tasks.workunit.client.1.smithi137.stdout:3/259: write d4/d19/f23 [1332438,76492] 0 2024-03-14T13:02:14.484 INFO:tasks.workunit.client.1.smithi137.stdout:1/379: rename d7/d31/f3a to d7/d4d/f7e 0 2024-03-14T13:02:14.487 INFO:tasks.workunit.client.1.smithi137.stdout:9/433: creat d6/f96 x:0 0 0 2024-03-14T13:02:14.488 INFO:tasks.workunit.client.1.smithi137.stdout:5/274: dwrite d7/d16/d18/d25/d28/f43 [4194304,4194304] 0 2024-03-14T13:02:14.489 INFO:tasks.workunit.client.1.smithi137.stdout:2/301: creat d2/d34/f69 x:0 0 0 2024-03-14T13:02:14.489 INFO:tasks.workunit.client.1.smithi137.stdout:8/212: symlink d1/d17/d34/l53 0 2024-03-14T13:02:14.491 INFO:tasks.workunit.client.1.smithi137.stdout:3/260: symlink d4/d22/l52 0 2024-03-14T13:02:14.492 INFO:tasks.workunit.client.1.smithi137.stdout:9/434: write d6/f1d [76091,1959] 0 2024-03-14T13:02:14.495 INFO:tasks.workunit.client.1.smithi137.stdout:1/380: unlink d7/d9/d19/d1f/l24 0 2024-03-14T13:02:14.495 INFO:tasks.workunit.client.1.smithi137.stdout:7/281: dwrite d3/d5/f37 [0,4194304] 0 2024-03-14T13:02:14.495 INFO:tasks.workunit.client.1.smithi137.stdout:5/275: mkdir d7/d33/d64 0 2024-03-14T13:02:14.496 INFO:tasks.workunit.client.1.smithi137.stdout:9/435: chown d6/df/f5e 49776 1 2024-03-14T13:02:14.497 INFO:tasks.workunit.client.1.smithi137.stdout:7/282: chown d3 6 1 2024-03-14T13:02:14.497 INFO:tasks.workunit.client.1.smithi137.stdout:8/213: write d1/dc/d11/d13/f4f [902174,94655] 0 2024-03-14T13:02:14.506 INFO:tasks.workunit.client.1.smithi137.stdout:6/311: truncate d2/d6/f3d 1371294 0 2024-03-14T13:02:14.506 INFO:tasks.workunit.client.1.smithi137.stdout:9/436: dread - d6/d56/f89 zero size 2024-03-14T13:02:14.509 INFO:tasks.workunit.client.1.smithi137.stdout:3/261: mknod d4/c53 0 2024-03-14T13:02:14.509 INFO:tasks.workunit.client.1.smithi137.stdout:2/302: link d2/d7/db/de/d22/f5a d2/d7/d2e/f6a 0 2024-03-14T13:02:14.509 INFO:tasks.workunit.client.1.smithi137.stdout:9/437: stat d6/f69 0 2024-03-14T13:02:14.510 INFO:tasks.workunit.client.1.smithi137.stdout:0/297: dread d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:14.510 INFO:tasks.workunit.client.1.smithi137.stdout:5/276: rmdir d7/d16/d18/d25/d28 39 2024-03-14T13:02:14.515 INFO:tasks.workunit.client.1.smithi137.stdout:3/262: creat d4/d19/f54 x:0 0 0 2024-03-14T13:02:14.516 INFO:tasks.workunit.client.1.smithi137.stdout:1/381: dwrite f5 [0,4194304] 0 2024-03-14T13:02:14.518 INFO:tasks.workunit.client.1.smithi137.stdout:6/312: symlink d2/d54/l6f 0 2024-03-14T13:02:14.518 INFO:tasks.workunit.client.1.smithi137.stdout:3/263: chown d4/d19/f23 740 1 2024-03-14T13:02:14.519 INFO:tasks.workunit.client.1.smithi137.stdout:2/303: fdatasync d2/d7/db/de/d22/f5a 0 2024-03-14T13:02:14.520 INFO:tasks.workunit.client.0.smithi111.stdout:8/551: write d3/d5/fd [7736172,36054] 0 2024-03-14T13:02:14.522 INFO:tasks.workunit.client.1.smithi137.stdout:9/438: dwrite d6/df/f51 [0,4194304] 0 2024-03-14T13:02:14.523 INFO:tasks.workunit.client.0.smithi111.stdout:5/465: truncate d2/d16/d49/d44/d4a/f60 2415422 0 2024-03-14T13:02:14.524 INFO:tasks.workunit.client.1.smithi137.stdout:0/298: mkdir d1/d5/d1b/d1e/d25/d26/d6c 0 2024-03-14T13:02:14.524 INFO:tasks.workunit.client.1.smithi137.stdout:2/304: dread - d2/d7/d2e/f47 zero size 2024-03-14T13:02:14.525 INFO:tasks.workunit.client.1.smithi137.stdout:8/214: rename d1/d17 to d1/dc/d3c/d54 0 2024-03-14T13:02:14.526 INFO:tasks.workunit.client.1.smithi137.stdout:1/382: creat d7/d9/d19/d5f/f7f x:0 0 0 2024-03-14T13:02:14.527 INFO:tasks.workunit.client.1.smithi137.stdout:8/215: dread - d1/dc/d11/f2a zero size 2024-03-14T13:02:14.528 INFO:tasks.workunit.client.1.smithi137.stdout:6/313: creat d2/d6/d1d/d6b/d57/f70 x:0 0 0 2024-03-14T13:02:14.530 INFO:tasks.workunit.client.1.smithi137.stdout:3/264: creat d4/df/d35/d4b/f55 x:0 0 0 2024-03-14T13:02:14.530 INFO:tasks.workunit.client.0.smithi111.stdout:8/552: dwrite d3/d1b/d20/d83/f63 [0,4194304] 0 2024-03-14T13:02:14.532 INFO:tasks.workunit.client.1.smithi137.stdout:0/299: mkdir d1/d5/d1b/d1e/d53/d6d 0 2024-03-14T13:02:14.533 INFO:tasks.workunit.client.1.smithi137.stdout:0/300: chown d1/d44/f65 94 1 2024-03-14T13:02:14.536 INFO:tasks.workunit.client.1.smithi137.stdout:0/301: write d1/d5/d1b/d1e/d25/d55/f67 [966761,104123] 0 2024-03-14T13:02:14.539 INFO:tasks.workunit.client.1.smithi137.stdout:8/216: dread d1/f49 [0,4194304] 0 2024-03-14T13:02:14.540 INFO:tasks.workunit.client.1.smithi137.stdout:6/314: symlink d2/d6/de/d45/d59/l71 0 2024-03-14T13:02:14.542 INFO:tasks.workunit.client.1.smithi137.stdout:7/283: dread d3/f18 [0,4194304] 0 2024-03-14T13:02:14.542 INFO:tasks.workunit.client.1.smithi137.stdout:8/217: dread - d1/dc/d3c/d54/f48 zero size 2024-03-14T13:02:14.544 INFO:tasks.workunit.client.1.smithi137.stdout:2/305: rename d2/d7/db/de to d2/d7/d2e/d3e/d44/d6b 0 2024-03-14T13:02:14.544 INFO:tasks.workunit.client.1.smithi137.stdout:9/439: symlink d6/df/d19/d3a/l97 0 2024-03-14T13:02:14.546 INFO:tasks.workunit.client.1.smithi137.stdout:8/218: write d1/dc/d10/f27 [865667,13013] 0 2024-03-14T13:02:14.546 INFO:tasks.workunit.client.1.smithi137.stdout:1/383: creat d7/d9/d19/d40/d50/f80 x:0 0 0 2024-03-14T13:02:14.548 INFO:tasks.workunit.client.1.smithi137.stdout:9/440: chown d6/c3f 20191644 1 2024-03-14T13:02:14.549 INFO:tasks.workunit.client.1.smithi137.stdout:8/219: read d1/f49 [404125,32322] 0 2024-03-14T13:02:14.550 INFO:tasks.workunit.client.1.smithi137.stdout:8/220: fsync d1/f44 0 2024-03-14T13:02:14.550 INFO:tasks.workunit.client.1.smithi137.stdout:3/265: rename d4/de/d13/c3a to d4/df/d34/c56 0 2024-03-14T13:02:14.551 INFO:tasks.workunit.client.1.smithi137.stdout:9/441: dread - d6/df/d19/d33/f65 zero size 2024-03-14T13:02:14.551 INFO:tasks.workunit.client.1.smithi137.stdout:1/384: fdatasync d7/d9/d19/d40/d50/f80 0 2024-03-14T13:02:14.552 INFO:tasks.workunit.client.1.smithi137.stdout:8/221: rename d1/dc to d1/dc/d55 22 2024-03-14T13:02:14.552 INFO:tasks.workunit.client.1.smithi137.stdout:2/306: symlink d2/d7/db/d2f/l6c 0 2024-03-14T13:02:14.552 INFO:tasks.workunit.client.1.smithi137.stdout:3/266: dread - d4/d22/f39 zero size 2024-03-14T13:02:14.555 INFO:tasks.workunit.client.1.smithi137.stdout:1/385: write d7/d9/f3e [390849,38049] 0 2024-03-14T13:02:14.555 INFO:tasks.workunit.client.1.smithi137.stdout:7/284: truncate d3/f18 2534999 0 2024-03-14T13:02:14.558 INFO:tasks.workunit.client.1.smithi137.stdout:9/442: symlink d6/df/d19/d3a/d40/l98 0 2024-03-14T13:02:14.559 INFO:tasks.workunit.client.1.smithi137.stdout:2/307: creat d2/d34/d3c/f6d x:0 0 0 2024-03-14T13:02:14.561 INFO:tasks.workunit.client.1.smithi137.stdout:5/277: dwrite d7/d16/d18/f20 [0,4194304] 0 2024-03-14T13:02:14.562 INFO:tasks.workunit.client.1.smithi137.stdout:8/222: rename d1/dc/d10/d46 to d1/dc/d11/d56 0 2024-03-14T13:02:14.567 INFO:tasks.workunit.client.1.smithi137.stdout:6/315: symlink d2/d54/d55/d38/l72 0 2024-03-14T13:02:14.571 INFO:tasks.workunit.client.1.smithi137.stdout:9/443: creat d6/d53/f99 x:0 0 0 2024-03-14T13:02:14.575 INFO:tasks.workunit.client.1.smithi137.stdout:8/223: dwrite d1/dc/d3c/d54/f48 [0,4194304] 0 2024-03-14T13:02:14.575 INFO:tasks.workunit.client.0.smithi111.stdout:4/607: sync 2024-03-14T13:02:14.576 INFO:tasks.workunit.client.0.smithi111.stdout:2/332: sync 2024-03-14T13:02:14.576 INFO:tasks.workunit.client.0.smithi111.stdout:7/345: sync 2024-03-14T13:02:14.576 INFO:tasks.workunit.client.0.smithi111.stdout:6/385: sync 2024-03-14T13:02:14.576 INFO:tasks.workunit.client.0.smithi111.stdout:3/427: sync 2024-03-14T13:02:14.577 INFO:tasks.workunit.client.1.smithi137.stdout:9/444: dread - d6/df/d19/d3a/f5b zero size 2024-03-14T13:02:14.578 INFO:tasks.workunit.client.0.smithi111.stdout:4/608: write d1d/f26 [471379,728] 0 2024-03-14T13:02:14.578 INFO:tasks.workunit.client.0.smithi111.stdout:2/333: stat dd/d17/d34/d4f/d55/c6a 0 2024-03-14T13:02:14.583 INFO:tasks.workunit.client.1.smithi137.stdout:2/308: truncate d2/d7/db/f1c 226885 0 2024-03-14T13:02:14.586 INFO:tasks.workunit.client.1.smithi137.stdout:1/386: creat d7/d9/d19/d1f/f81 x:0 0 0 2024-03-14T13:02:14.586 INFO:tasks.workunit.client.1.smithi137.stdout:7/285: dread d3/f18 [0,4194304] 0 2024-03-14T13:02:14.587 INFO:tasks.workunit.client.1.smithi137.stdout:5/278: write d7/fb [499497,113500] 0 2024-03-14T13:02:14.589 INFO:tasks.workunit.client.1.smithi137.stdout:0/302: dwrite d1/d5/d1b/d1e/d25/d26/f2d [0,4194304] 0 2024-03-14T13:02:14.591 INFO:tasks.workunit.client.1.smithi137.stdout:6/316: link d2/d6/d1d/f4e d2/d6/de/d45/d59/d39/f73 0 2024-03-14T13:02:14.596 INFO:tasks.workunit.client.1.smithi137.stdout:0/303: chown d1/d11/f50 124548 1 2024-03-14T13:02:14.596 INFO:tasks.workunit.client.1.smithi137.stdout:9/445: creat d6/df/d19/d33/d8f/f9a x:0 0 0 2024-03-14T13:02:14.596 INFO:tasks.workunit.client.0.smithi111.stdout:7/346: rmdir d0/d1/d22/d3e 39 2024-03-14T13:02:14.597 INFO:tasks.workunit.client.0.smithi111.stdout:5/466: dread d2/f6 [0,4194304] 0 2024-03-14T13:02:14.598 INFO:tasks.workunit.client.1.smithi137.stdout:8/224: write d1/dc/d3c/d54/f1e [965765,9871] 0 2024-03-14T13:02:14.598 INFO:tasks.workunit.client.0.smithi111.stdout:3/428: symlink d0/d2d/d35/d50/d5e/d40/da7/la8 0 2024-03-14T13:02:14.600 INFO:tasks.workunit.client.0.smithi111.stdout:5/467: readlink d2/d16/d49/d44/l61 0 2024-03-14T13:02:14.600 INFO:tasks.workunit.client.0.smithi111.stdout:6/386: dread d0/d17/d21/d69/f39 [0,4194304] 0 2024-03-14T13:02:14.601 INFO:tasks.workunit.client.1.smithi137.stdout:5/279: write d7/d16/d18/d25/f46 [1658406,9086] 0 2024-03-14T13:02:14.601 INFO:tasks.workunit.client.1.smithi137.stdout:6/317: creat d2/d6/d1d/d6b/d57/f74 x:0 0 0 2024-03-14T13:02:14.602 INFO:tasks.workunit.client.0.smithi111.stdout:4/609: link d1d/d22/d30/d5e/d6f/fc3 d1d/d22/d40/d69/d86/db1/fd2 0 2024-03-14T13:02:14.603 INFO:tasks.workunit.client.0.smithi111.stdout:5/468: stat d2/d45/f22 0 2024-03-14T13:02:14.603 INFO:tasks.workunit.client.0.smithi111.stdout:8/553: truncate d3/d22/d30/f97 1165557 0 2024-03-14T13:02:14.604 INFO:tasks.workunit.client.0.smithi111.stdout:5/469: readlink d2/d16/d49/d7c/d95/d36/l4d 0 2024-03-14T13:02:14.604 INFO:tasks.workunit.client.0.smithi111.stdout:7/347: mknod d0/d1/c67 0 2024-03-14T13:02:14.606 INFO:tasks.workunit.client.0.smithi111.stdout:3/429: rmdir d0/d2d/d35/d50/d5e/d19 39 2024-03-14T13:02:14.609 INFO:tasks.workunit.client.0.smithi111.stdout:2/334: getdents dd/d17/d34/d4f 0 2024-03-14T13:02:14.611 INFO:tasks.workunit.client.0.smithi111.stdout:8/554: rmdir d3/d5 39 2024-03-14T13:02:14.612 INFO:tasks.workunit.client.0.smithi111.stdout:8/555: write d3/d1b/d20/d83/f63 [3924298,37022] 0 2024-03-14T13:02:14.615 INFO:tasks.workunit.client.0.smithi111.stdout:8/556: dread d3/d1b/d20/d35/d40/d48/f68 [0,4194304] 0 2024-03-14T13:02:14.615 INFO:tasks.workunit.client.0.smithi111.stdout:6/387: symlink d0/d17/d20/l96 0 2024-03-14T13:02:14.616 INFO:tasks.workunit.client.0.smithi111.stdout:8/557: stat d3/d1b/d20/d35/l4f 0 2024-03-14T13:02:14.617 INFO:tasks.workunit.client.0.smithi111.stdout:8/558: write d3/f6b [3125518,96624] 0 2024-03-14T13:02:14.618 INFO:tasks.workunit.client.0.smithi111.stdout:4/610: symlink d1d/d22/d40/ld3 0 2024-03-14T13:02:14.619 INFO:tasks.workunit.client.0.smithi111.stdout:4/611: write d1d/d22/f37 [2068276,119642] 0 2024-03-14T13:02:14.620 INFO:tasks.workunit.client.0.smithi111.stdout:5/470: mknod d2/d16/d49/c98 0 2024-03-14T13:02:14.627 INFO:tasks.workunit.client.0.smithi111.stdout:3/430: truncate d0/f8 464688 0 2024-03-14T13:02:14.627 INFO:tasks.workunit.client.0.smithi111.stdout:4/612: mknod d1d/cd4 0 2024-03-14T13:02:14.628 INFO:tasks.workunit.client.0.smithi111.stdout:7/348: rename d0/d1/f51 to d0/d1/d5/d38/f68 0 2024-03-14T13:02:14.629 INFO:tasks.workunit.client.0.smithi111.stdout:5/471: creat d2/d13/f99 x:0 0 0 2024-03-14T13:02:14.647 INFO:tasks.workunit.client.0.smithi111.stdout:2/335: rename dd/d11/f2c to dd/d17/d34/f6b 0 2024-03-14T13:02:14.648 INFO:tasks.workunit.client.0.smithi111.stdout:4/613: getdents d1d/d22/d40 0 2024-03-14T13:02:14.648 INFO:tasks.workunit.client.0.smithi111.stdout:5/472: getdents d2/d45/d76 0 2024-03-14T13:02:14.651 INFO:tasks.workunit.client.0.smithi111.stdout:8/559: rename d3/d49/f53 to d3/d1b/d20/d35/d40/fac 0 2024-03-14T13:02:14.652 INFO:tasks.workunit.client.0.smithi111.stdout:2/336: rmdir dd/d17/d34 39 2024-03-14T13:02:14.652 INFO:tasks.workunit.client.0.smithi111.stdout:2/337: chown l5 3 1 2024-03-14T13:02:14.654 INFO:tasks.workunit.client.0.smithi111.stdout:5/473: readlink d2/l85 0 2024-03-14T13:02:14.656 INFO:tasks.workunit.client.0.smithi111.stdout:7/349: rename d0/d1/d5/d38/l36 to d0/d1/d5/l69 0 2024-03-14T13:02:14.657 INFO:tasks.workunit.client.0.smithi111.stdout:8/560: chown d3/d1b/c8a 1 1 2024-03-14T13:02:14.659 INFO:tasks.workunit.client.0.smithi111.stdout:5/474: mkdir d2/d45/d76/d9a 0 2024-03-14T13:02:14.661 INFO:tasks.workunit.client.0.smithi111.stdout:6/388: truncate d0/f1a 5271829 0 2024-03-14T13:02:14.662 INFO:tasks.workunit.client.0.smithi111.stdout:6/389: chown d0/l84 0 1 2024-03-14T13:02:14.662 INFO:tasks.workunit.client.0.smithi111.stdout:6/390: stat d0/d17/d21/d42/c62 0 2024-03-14T13:02:14.663 INFO:tasks.workunit.client.0.smithi111.stdout:5/475: mknod d2/d16/d49/d33/c9b 0 2024-03-14T13:02:14.664 INFO:tasks.workunit.client.0.smithi111.stdout:5/476: write d2/d16/d49/d4c/f55 [676074,26324] 0 2024-03-14T13:02:14.668 INFO:tasks.workunit.client.0.smithi111.stdout:6/391: creat d0/d17/d20/d4a/d59/f97 x:0 0 0 2024-03-14T13:02:14.671 INFO:tasks.workunit.client.0.smithi111.stdout:3/431: dwrite d0/d30/f95 [4194304,4194304] 0 2024-03-14T13:02:14.673 INFO:tasks.workunit.client.0.smithi111.stdout:6/392: creat d0/f98 x:0 0 0 2024-03-14T13:02:14.675 INFO:tasks.workunit.client.0.smithi111.stdout:5/477: rename d2/d16/d49/d4c/d88 to d2/d16/d49/d4c/d9c 0 2024-03-14T13:02:14.675 INFO:tasks.workunit.client.1.smithi137.stdout:4/327: sync 2024-03-14T13:02:14.676 INFO:tasks.workunit.client.0.smithi111.stdout:2/338: symlink dd/d11/l6c 0 2024-03-14T13:02:14.677 INFO:tasks.workunit.client.0.smithi111.stdout:5/478: write d2/d13/f15 [712336,123835] 0 2024-03-14T13:02:14.678 INFO:tasks.workunit.client.1.smithi137.stdout:8/225: write d1/dc/f39 [248951,110177] 0 2024-03-14T13:02:14.679 INFO:tasks.workunit.client.0.smithi111.stdout:7/350: truncate d0/d1/d5/d38/d9/fb 2628928 0 2024-03-14T13:02:14.680 INFO:tasks.workunit.client.0.smithi111.stdout:4/614: dwrite fb [0,4194304] 0 2024-03-14T13:02:14.680 INFO:tasks.workunit.client.1.smithi137.stdout:3/267: dwrite d4/de/d16/f20 [0,4194304] 0 2024-03-14T13:02:14.682 INFO:tasks.workunit.client.0.smithi111.stdout:6/393: unlink d0/d17/f55 0 2024-03-14T13:02:14.683 INFO:tasks.workunit.client.1.smithi137.stdout:8/226: truncate d1/f44 2239335 0 2024-03-14T13:02:14.684 INFO:tasks.workunit.client.1.smithi137.stdout:6/318: read d2/d54/d55/d38/f51 [496940,87639] 0 2024-03-14T13:02:14.684 INFO:tasks.workunit.client.1.smithi137.stdout:1/387: truncate d7/d9/d19/d40/f46 183620 0 2024-03-14T13:02:14.686 INFO:tasks.workunit.client.0.smithi111.stdout:3/432: mknod d0/d2d/d35/d50/d6c/da3/ca9 0 2024-03-14T13:02:14.688 INFO:tasks.workunit.client.1.smithi137.stdout:8/227: chown d1/dc/d3c/d54/l3f 19557104 1 2024-03-14T13:02:14.689 INFO:tasks.workunit.client.1.smithi137.stdout:0/304: getdents d1/d5/d1b/d1e 0 2024-03-14T13:02:14.690 INFO:tasks.workunit.client.0.smithi111.stdout:8/561: dwrite d3/d5/f12 [0,4194304] 0 2024-03-14T13:02:14.691 INFO:tasks.workunit.client.1.smithi137.stdout:5/280: chown d7/d16/d18/d25/d28/f60 228 1 2024-03-14T13:02:14.692 INFO:tasks.workunit.client.1.smithi137.stdout:3/268: symlink d4/df/d34/l57 0 2024-03-14T13:02:14.693 INFO:tasks.workunit.client.1.smithi137.stdout:4/328: getdents d4/d10/d5a 0 2024-03-14T13:02:14.693 INFO:tasks.workunit.client.1.smithi137.stdout:6/319: chown d2/d6/de/l36 5655943 1 2024-03-14T13:02:14.695 INFO:tasks.workunit.client.0.smithi111.stdout:3/433: fdatasync d0/d2d/d35/d69/d84/f99 0 2024-03-14T13:02:14.695 INFO:tasks.workunit.client.1.smithi137.stdout:8/228: symlink d1/dc/l57 0 2024-03-14T13:02:14.696 INFO:tasks.workunit.client.0.smithi111.stdout:7/351: creat d0/d1/d4a/f6a x:0 0 0 2024-03-14T13:02:14.700 INFO:tasks.workunit.client.0.smithi111.stdout:8/562: write d3/d1b/d20/d35/fa0 [385107,15271] 0 2024-03-14T13:02:14.700 INFO:tasks.workunit.client.0.smithi111.stdout:1/394: sync 2024-03-14T13:02:14.700 INFO:tasks.workunit.client.0.smithi111.stdout:0/342: sync 2024-03-14T13:02:14.700 INFO:tasks.workunit.client.0.smithi111.stdout:9/408: sync 2024-03-14T13:02:14.700 INFO:tasks.workunit.client.1.smithi137.stdout:0/305: read d1/d44/d56/f5f [631830,127522] 0 2024-03-14T13:02:14.701 INFO:tasks.workunit.client.0.smithi111.stdout:6/394: dread d0/d17/d21/d69/f39 [0,4194304] 0 2024-03-14T13:02:14.701 INFO:tasks.workunit.client.1.smithi137.stdout:3/269: rmdir d4/de/d16/d1c 39 2024-03-14T13:02:14.702 INFO:tasks.workunit.client.1.smithi137.stdout:7/286: dwrite d3/d6/d14/f57 [0,4194304] 0 2024-03-14T13:02:14.702 INFO:tasks.workunit.client.1.smithi137.stdout:5/281: rename d7/d33/d64 to d7/d33/d65 0 2024-03-14T13:02:14.703 INFO:tasks.workunit.client.0.smithi111.stdout:2/339: dwrite dd/d11/d1a/f21 [4194304,4194304] 0 2024-03-14T13:02:14.703 INFO:tasks.workunit.client.0.smithi111.stdout:0/343: chown d6/d21/f42 5 1 2024-03-14T13:02:14.704 INFO:tasks.workunit.client.0.smithi111.stdout:4/615: write d1d/d22/d40/fa0 [332339,111224] 0 2024-03-14T13:02:14.705 INFO:tasks.workunit.client.0.smithi111.stdout:1/395: mknod d20/d25/c8a 0 2024-03-14T13:02:14.705 INFO:tasks.workunit.client.1.smithi137.stdout:6/320: chown d2/d6/de/f68 432497 1 2024-03-14T13:02:14.706 INFO:tasks.workunit.client.0.smithi111.stdout:8/563: mkdir d3/d1b/d20/d35/d40/dad 0 2024-03-14T13:02:14.706 INFO:tasks.workunit.client.0.smithi111.stdout:7/352: dread d0/d1/d22/f42 [0,4194304] 0 2024-03-14T13:02:14.707 INFO:tasks.workunit.client.1.smithi137.stdout:2/309: dwrite d2/d7/db/f38 [0,4194304] 0 2024-03-14T13:02:14.708 INFO:tasks.workunit.client.0.smithi111.stdout:1/396: readlink d20/d21/l32 0 2024-03-14T13:02:14.711 INFO:tasks.workunit.client.0.smithi111.stdout:6/395: mkdir d0/d17/d99 0 2024-03-14T13:02:14.711 INFO:tasks.workunit.client.0.smithi111.stdout:4/616: truncate d1d/d22/d30/d7c/f93 1807760 0 2024-03-14T13:02:14.711 INFO:tasks.workunit.client.0.smithi111.stdout:8/564: write d3/d1b/d20/d35/d40/faa [305743,95366] 0 2024-03-14T13:02:14.711 INFO:tasks.workunit.client.1.smithi137.stdout:6/321: chown d2/d54/d55 125589 1 2024-03-14T13:02:14.711 INFO:tasks.workunit.client.1.smithi137.stdout:0/306: unlink d1/lf 0 2024-03-14T13:02:14.712 INFO:tasks.workunit.client.0.smithi111.stdout:3/434: getdents d0 0 2024-03-14T13:02:14.714 INFO:tasks.workunit.client.0.smithi111.stdout:9/409: creat d5/d42/d52/d83/f88 x:0 0 0 2024-03-14T13:02:14.715 INFO:tasks.workunit.client.0.smithi111.stdout:2/340: truncate dd/d11/d48/f5e 1086609 0 2024-03-14T13:02:14.715 INFO:tasks.workunit.client.0.smithi111.stdout:8/565: chown d3/d22/f72 3442 1 2024-03-14T13:02:14.715 INFO:tasks.workunit.client.0.smithi111.stdout:7/353: mknod d0/d1/d21/d3a/c6b 0 2024-03-14T13:02:14.715 INFO:tasks.workunit.client.1.smithi137.stdout:3/270: write d4/d22/f4c [64421,123718] 0 2024-03-14T13:02:14.719 INFO:tasks.workunit.client.0.smithi111.stdout:7/354: dread - d0/d1/d4a/d52/f62 zero size 2024-03-14T13:02:14.719 INFO:tasks.workunit.client.0.smithi111.stdout:8/566: dread - d3/d1b/d20/d35/d40/f90 zero size 2024-03-14T13:02:14.719 INFO:tasks.workunit.client.0.smithi111.stdout:4/617: mknod d1d/d22/d30/cd5 0 2024-03-14T13:02:14.719 INFO:tasks.workunit.client.0.smithi111.stdout:6/396: dread d0/f18 [0,4194304] 0 2024-03-14T13:02:14.722 INFO:tasks.workunit.client.1.smithi137.stdout:1/388: dwrite d7/d9/d19/d40/f5b [0,4194304] 0 2024-03-14T13:02:14.722 INFO:tasks.workunit.client.0.smithi111.stdout:3/435: mkdir d0/d2d/d35/d50/d6c/da3/daa 0 2024-03-14T13:02:14.723 INFO:tasks.workunit.client.0.smithi111.stdout:1/397: creat d20/d21/f8b x:0 0 0 2024-03-14T13:02:14.724 INFO:tasks.workunit.client.1.smithi137.stdout:3/271: unlink d4/d22/f39 0 2024-03-14T13:02:14.724 INFO:tasks.workunit.client.1.smithi137.stdout:0/307: mknod d1/d1f/c6e 0 2024-03-14T13:02:14.725 INFO:tasks.workunit.client.0.smithi111.stdout:8/567: chown d3/d1b/d20/d35/d40/d48 1321945850 1 2024-03-14T13:02:14.725 INFO:tasks.workunit.client.0.smithi111.stdout:5/479: write d2/d16/d49/d44/d4a/f60 [3293901,60427] 0 2024-03-14T13:02:14.726 INFO:tasks.workunit.client.0.smithi111.stdout:7/355: fdatasync d0/d1/d5/f1a 0 2024-03-14T13:02:14.728 INFO:tasks.workunit.client.0.smithi111.stdout:2/341: creat dd/d11/f6d x:0 0 0 2024-03-14T13:02:14.729 INFO:tasks.workunit.client.0.smithi111.stdout:4/618: fdatasync d1d/d22/d30/d5e/f7a 0 2024-03-14T13:02:14.731 INFO:tasks.workunit.client.0.smithi111.stdout:3/436: creat d0/d2d/d35/d4c/d82/fab x:0 0 0 2024-03-14T13:02:14.731 INFO:tasks.workunit.client.1.smithi137.stdout:0/308: dread - d1/d44/f65 zero size 2024-03-14T13:02:14.732 INFO:tasks.workunit.client.0.smithi111.stdout:5/480: unlink d2/d16/d49/d33/f5b 0 2024-03-14T13:02:14.733 INFO:tasks.workunit.client.1.smithi137.stdout:1/389: mkdir d7/d31/d82 0 2024-03-14T13:02:14.733 INFO:tasks.workunit.client.0.smithi111.stdout:6/397: dread - d0/d50/f66 zero size 2024-03-14T13:02:14.733 INFO:tasks.workunit.client.1.smithi137.stdout:4/329: dwrite d4/d10/d26/f40 [0,4194304] 0 2024-03-14T13:02:14.734 INFO:tasks.workunit.client.0.smithi111.stdout:7/356: creat d0/d1/d4a/d52/f6c x:0 0 0 2024-03-14T13:02:14.734 INFO:tasks.workunit.client.0.smithi111.stdout:0/344: dread d6/d9/d15/f1b [0,4194304] 0 2024-03-14T13:02:14.735 INFO:tasks.workunit.client.1.smithi137.stdout:6/322: truncate d2/d54/d55/f1e 4361380 0 2024-03-14T13:02:14.736 INFO:tasks.workunit.client.1.smithi137.stdout:5/282: rmdir d7/d16/d47 0 2024-03-14T13:02:14.736 INFO:tasks.workunit.client.1.smithi137.stdout:0/309: chown d1/d5/d1b/d1e/d25/d26/f2d 868 1 2024-03-14T13:02:14.736 INFO:tasks.workunit.client.0.smithi111.stdout:8/568: rename d3/d5/l15 to d3/dc/lae 0 2024-03-14T13:02:14.737 INFO:tasks.workunit.client.1.smithi137.stdout:6/323: fdatasync d2/d6/f5c 0 2024-03-14T13:02:14.737 INFO:tasks.workunit.client.1.smithi137.stdout:4/330: creat d4/d10/d5a/f71 x:0 0 0 2024-03-14T13:02:14.738 INFO:tasks.workunit.client.1.smithi137.stdout:1/390: fdatasync d7/d9/d19/d1f/d2e/f34 0 2024-03-14T13:02:14.738 INFO:tasks.workunit.client.0.smithi111.stdout:6/398: write d0/f74 [256978,48765] 0 2024-03-14T13:02:14.739 INFO:tasks.workunit.client.1.smithi137.stdout:6/324: chown d2/c21 1500498 1 2024-03-14T13:02:14.739 INFO:tasks.workunit.client.0.smithi111.stdout:0/345: unlink f3 0 2024-03-14T13:02:14.740 INFO:tasks.workunit.client.0.smithi111.stdout:9/410: dwrite d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:14.741 INFO:tasks.workunit.client.1.smithi137.stdout:5/283: readlink d7/l31 0 2024-03-14T13:02:14.741 INFO:tasks.workunit.client.1.smithi137.stdout:6/325: chown d2/d4c/c6a 12226 1 2024-03-14T13:02:14.741 INFO:tasks.workunit.client.1.smithi137.stdout:0/310: creat d1/d11/d4a/d54/f6f x:0 0 0 2024-03-14T13:02:14.742 INFO:tasks.workunit.client.1.smithi137.stdout:3/272: getdents d4/df 0 2024-03-14T13:02:14.744 INFO:tasks.workunit.client.0.smithi111.stdout:4/619: symlink d1d/d22/d40/d69/d86/db1/ld6 0 2024-03-14T13:02:14.744 INFO:tasks.workunit.client.0.smithi111.stdout:1/398: link d20/l86 d20/d21/l8c 0 2024-03-14T13:02:14.745 INFO:tasks.workunit.client.0.smithi111.stdout:7/357: mkdir d0/d1/d5/d38/d5c/d6d 0 2024-03-14T13:02:14.746 INFO:tasks.workunit.client.1.smithi137.stdout:4/331: creat d4/d27/d5c/d6d/f72 x:0 0 0 2024-03-14T13:02:14.748 INFO:tasks.workunit.client.0.smithi111.stdout:3/437: rename d0/c21 to d0/d2d/d35/d50/d5e/d40/d6f/cac 0 2024-03-14T13:02:14.748 INFO:tasks.workunit.client.1.smithi137.stdout:6/326: creat d2/d6/de/d45/d59/f75 x:0 0 0 2024-03-14T13:02:14.748 INFO:tasks.workunit.client.1.smithi137.stdout:5/284: write d7/d16/d18/d2d/f63 [75730,117508] 0 2024-03-14T13:02:14.749 INFO:tasks.workunit.client.0.smithi111.stdout:5/481: readlink d2/d16/d49/d44/d72/l81 0 2024-03-14T13:02:14.749 INFO:tasks.workunit.client.0.smithi111.stdout:8/569: fdatasync d3/d1b/d20/f37 0 2024-03-14T13:02:14.751 INFO:tasks.workunit.client.0.smithi111.stdout:9/411: unlink d5/d42/d52/f6c 0 2024-03-14T13:02:14.752 INFO:tasks.workunit.client.1.smithi137.stdout:4/332: mkdir d4/d10/d42/d73 0 2024-03-14T13:02:14.754 INFO:tasks.workunit.client.0.smithi111.stdout:5/482: dread - d2/d16/d49/d4c/f7d zero size 2024-03-14T13:02:14.754 INFO:tasks.workunit.client.0.smithi111.stdout:1/399: rmdir d20/d21/d3f 39 2024-03-14T13:02:14.756 INFO:tasks.workunit.client.0.smithi111.stdout:0/346: dread d6/d9/d15/f37 [0,4194304] 0 2024-03-14T13:02:14.759 INFO:tasks.workunit.client.0.smithi111.stdout:8/570: write d3/d1b/d20/f2b [4825905,60575] 0 2024-03-14T13:02:14.759 INFO:tasks.workunit.client.0.smithi111.stdout:4/620: creat d1d/d22/fd7 x:0 0 0 2024-03-14T13:02:14.760 INFO:tasks.workunit.client.0.smithi111.stdout:2/342: dwrite dd/d17/d34/d4f/d55/f5d [0,4194304] 0 2024-03-14T13:02:14.762 INFO:tasks.workunit.client.1.smithi137.stdout:6/327: rename d2/d6/d1d/d6b to d2/d54/d55/d6d/d76 0 2024-03-14T13:02:14.762 INFO:tasks.workunit.client.1.smithi137.stdout:4/333: write f3 [744912,44298] 0 2024-03-14T13:02:14.765 INFO:tasks.workunit.client.1.smithi137.stdout:3/273: link d4/df/d34/f49 d4/f58 0 2024-03-14T13:02:14.766 INFO:tasks.workunit.client.1.smithi137.stdout:5/285: dread d7/d16/d18/f2a [0,4194304] 0 2024-03-14T13:02:14.768 INFO:tasks.workunit.client.1.smithi137.stdout:4/334: symlink d4/d9/d3a/d41/d38/l74 0 2024-03-14T13:02:14.768 INFO:tasks.workunit.client.0.smithi111.stdout:1/400: truncate d20/d21/f41 185929 0 2024-03-14T13:02:14.769 INFO:tasks.workunit.client.1.smithi137.stdout:8/229: truncate d1/dc/d3c/d54/d34/f40 2533545 0 2024-03-14T13:02:14.769 INFO:tasks.workunit.client.0.smithi111.stdout:5/483: dread d2/d13/f6c [0,4194304] 0 2024-03-14T13:02:14.769 INFO:tasks.workunit.client.1.smithi137.stdout:5/286: read d7/d1d/f57 [1032225,1538] 0 2024-03-14T13:02:14.770 INFO:tasks.workunit.client.0.smithi111.stdout:8/571: unlink d3/dc/c5a 0 2024-03-14T13:02:14.771 INFO:tasks.workunit.client.1.smithi137.stdout:6/328: rename d2/d6/d1d/f4e to d2/d6/de/d45/f77 0 2024-03-14T13:02:14.772 INFO:tasks.workunit.client.0.smithi111.stdout:7/358: dwrite d0/d1/d21/f3f [4194304,4194304] 0 2024-03-14T13:02:14.772 INFO:tasks.workunit.client.1.smithi137.stdout:4/335: write d4/f8 [379854,113491] 0 2024-03-14T13:02:14.773 INFO:tasks.workunit.client.0.smithi111.stdout:6/399: dread d0/d17/f1f [0,4194304] 0 2024-03-14T13:02:14.774 INFO:tasks.workunit.client.0.smithi111.stdout:9/412: dwrite d5/d6/d8/d74/f77 [0,4194304] 0 2024-03-14T13:02:14.777 INFO:tasks.workunit.client.0.smithi111.stdout:9/413: fdatasync d5/fa 0 2024-03-14T13:02:14.778 INFO:tasks.workunit.client.0.smithi111.stdout:2/343: creat dd/d17/d34/d4f/d55/f6e x:0 0 0 2024-03-14T13:02:14.779 INFO:tasks.workunit.client.0.smithi111.stdout:7/359: truncate d0/d1/d4a/f60 1030549 0 2024-03-14T13:02:14.779 INFO:tasks.workunit.client.0.smithi111.stdout:5/484: fdatasync d2/d16/d49/d7c/d95/d6a/f6e 0 2024-03-14T13:02:14.780 INFO:tasks.workunit.client.0.smithi111.stdout:1/401: rmdir d20/d21/d28/d5c 39 2024-03-14T13:02:14.781 INFO:tasks.workunit.client.1.smithi137.stdout:2/310: dwrite d2/d7/d2e/d3e/d44/d6b/f3f [0,4194304] 0 2024-03-14T13:02:14.781 INFO:tasks.workunit.client.0.smithi111.stdout:6/400: read d0/d17/d20/d4a/d59/f6f [3638839,105883] 0 2024-03-14T13:02:14.783 INFO:tasks.workunit.client.0.smithi111.stdout:2/344: chown dd/d17/d53/f68 111253 1 2024-03-14T13:02:14.785 INFO:tasks.workunit.client.1.smithi137.stdout:8/230: rename d1/dc/d11/l28 to d1/dc/d11/d56/l58 0 2024-03-14T13:02:14.785 INFO:tasks.workunit.client.0.smithi111.stdout:1/402: write d20/d21/f8b [779461,66136] 0 2024-03-14T13:02:14.786 INFO:tasks.workunit.client.0.smithi111.stdout:9/414: read d5/fa [735290,9786] 0 2024-03-14T13:02:14.789 INFO:tasks.workunit.client.0.smithi111.stdout:0/347: rename d6/d9/d15/f1b to d6/d9/de/f6a 0 2024-03-14T13:02:14.789 INFO:tasks.workunit.client.1.smithi137.stdout:8/231: fsync d1/dc/d3c/d54/f1e 0 2024-03-14T13:02:14.791 INFO:tasks.workunit.client.1.smithi137.stdout:5/287: dwrite d7/d16/d18/f39 [0,4194304] 0 2024-03-14T13:02:14.793 INFO:tasks.workunit.client.0.smithi111.stdout:5/485: fdatasync d2/d13/f29 0 2024-03-14T13:02:14.793 INFO:tasks.workunit.client.0.smithi111.stdout:5/486: write d2/d13/f99 [236816,20225] 0 2024-03-14T13:02:14.795 INFO:tasks.workunit.client.0.smithi111.stdout:2/345: write dd/d17/d34/d4f/d55/f5d [3394540,75389] 0 2024-03-14T13:02:14.795 INFO:tasks.workunit.client.0.smithi111.stdout:3/438: unlink d0/d2d/d35/d50/d5e/d40/da7/la8 0 2024-03-14T13:02:14.797 INFO:tasks.workunit.client.1.smithi137.stdout:6/329: link d2/d6/de/d45/d59/f5e d2/d54/d55/f78 0 2024-03-14T13:02:14.797 INFO:tasks.workunit.client.0.smithi111.stdout:8/572: dwrite d3/dc/f99 [0,4194304] 0 2024-03-14T13:02:14.798 INFO:tasks.workunit.client.0.smithi111.stdout:4/621: getdents d1d/d22/d30 0 2024-03-14T13:02:14.802 INFO:tasks.workunit.client.1.smithi137.stdout:1/391: dread d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:14.803 INFO:tasks.workunit.client.0.smithi111.stdout:4/622: stat d1d/d22/d40/d69/d86/faa 0 2024-03-14T13:02:14.803 INFO:tasks.workunit.client.1.smithi137.stdout:4/336: rename d4/d9/f5b to d4/d10/d42/f75 0 2024-03-14T13:02:14.804 INFO:tasks.workunit.client.0.smithi111.stdout:2/346: creat dd/d17/d34/f6f x:0 0 0 2024-03-14T13:02:14.805 INFO:tasks.workunit.client.1.smithi137.stdout:6/330: creat d2/d6/de/d45/d59/f79 x:0 0 0 2024-03-14T13:02:14.807 INFO:tasks.workunit.client.0.smithi111.stdout:1/403: symlink d20/d25/d87/l8d 0 2024-03-14T13:02:14.808 INFO:tasks.workunit.client.0.smithi111.stdout:4/623: write d1d/d22/d56/f7f [296313,125932] 0 2024-03-14T13:02:14.809 INFO:tasks.workunit.client.0.smithi111.stdout:3/439: chown d0/d2d/d35/d50/d5e/d19/l62 263809075 1 2024-03-14T13:02:14.812 INFO:tasks.workunit.client.0.smithi111.stdout:6/401: dread d0/d17/d20/d4a/f51 [0,4194304] 0 2024-03-14T13:02:14.812 INFO:tasks.workunit.client.1.smithi137.stdout:6/331: creat d2/d6/de/d45/d59/d39/f7a x:0 0 0 2024-03-14T13:02:14.814 INFO:tasks.workunit.client.1.smithi137.stdout:5/288: link d7/d16/f38 d7/d16/d18/f66 0 2024-03-14T13:02:14.814 INFO:tasks.workunit.client.1.smithi137.stdout:8/232: getdents d1/dc/d3c/d54 0 2024-03-14T13:02:14.816 INFO:tasks.workunit.client.1.smithi137.stdout:8/233: fsync d1/dc/d10/f32 0 2024-03-14T13:02:14.818 INFO:tasks.workunit.client.1.smithi137.stdout:1/392: symlink d7/d9/d19/d1f/d26/d6c/l83 0 2024-03-14T13:02:14.819 INFO:tasks.workunit.client.1.smithi137.stdout:6/332: rename d2/d6/de/d45/d59/l71 to d2/d6/de/l7b 0 2024-03-14T13:02:14.819 INFO:tasks.workunit.client.0.smithi111.stdout:5/487: dwrite d2/d13/f6c [0,4194304] 0 2024-03-14T13:02:14.819 INFO:tasks.workunit.client.0.smithi111.stdout:6/402: read - d0/d17/d21/d69/f45 zero size 2024-03-14T13:02:14.820 INFO:tasks.workunit.client.0.smithi111.stdout:0/348: read d6/d9/de/d3a/d46/d4a/f5c [3169281,42142] 0 2024-03-14T13:02:14.821 INFO:tasks.workunit.client.0.smithi111.stdout:3/440: mkdir d0/d2d/d35/d4c/dad 0 2024-03-14T13:02:14.822 INFO:tasks.workunit.client.0.smithi111.stdout:1/404: symlink d20/d25/d31/l8e 0 2024-03-14T13:02:14.822 INFO:tasks.workunit.client.1.smithi137.stdout:6/333: fsync d2/d6/f5c 0 2024-03-14T13:02:14.823 INFO:tasks.workunit.client.0.smithi111.stdout:8/573: getdents d3/d22 0 2024-03-14T13:02:14.825 INFO:tasks.workunit.client.0.smithi111.stdout:4/624: creat d1d/d22/fd8 x:0 0 0 2024-03-14T13:02:14.825 INFO:tasks.workunit.client.1.smithi137.stdout:4/337: dread d4/f32 [0,4194304] 0 2024-03-14T13:02:14.826 INFO:tasks.workunit.client.0.smithi111.stdout:3/441: fsync d0/d2d/d35/d50/f88 0 2024-03-14T13:02:14.827 INFO:tasks.workunit.client.0.smithi111.stdout:2/347: dwrite dd/d11/d1a/f38 [0,4194304] 0 2024-03-14T13:02:14.827 INFO:tasks.workunit.client.1.smithi137.stdout:1/393: mkdir d7/d9/d19/d1f/d26/d84 0 2024-03-14T13:02:14.829 INFO:tasks.workunit.client.1.smithi137.stdout:2/311: dwrite d2/f4b [0,4194304] 0 2024-03-14T13:02:14.829 INFO:tasks.workunit.client.1.smithi137.stdout:1/394: chown d7/d9/d19/d40/d50 332476 1 2024-03-14T13:02:14.830 INFO:tasks.workunit.client.0.smithi111.stdout:5/488: symlink d2/d16/d49/d44/l9d 0 2024-03-14T13:02:14.831 INFO:tasks.workunit.client.1.smithi137.stdout:8/234: truncate d1/dc/d3c/d54/d34/f47 1958351 0 2024-03-14T13:02:14.833 INFO:tasks.workunit.client.0.smithi111.stdout:1/405: creat d20/d25/d87/f8f x:0 0 0 2024-03-14T13:02:14.836 INFO:tasks.workunit.client.0.smithi111.stdout:4/625: unlink d1d/d22/d30/d7c/f93 0 2024-03-14T13:02:14.836 INFO:tasks.workunit.client.0.smithi111.stdout:2/348: mkdir dd/d11/d1a/d70 0 2024-03-14T13:02:14.837 INFO:tasks.workunit.client.1.smithi137.stdout:4/338: creat d4/d27/d5c/d6d/d58/f76 x:0 0 0 2024-03-14T13:02:14.837 INFO:tasks.workunit.client.1.smithi137.stdout:6/334: symlink d2/d54/d55/d33/l7c 0 2024-03-14T13:02:14.838 INFO:tasks.workunit.client.0.smithi111.stdout:4/626: dread - d1d/d22/fd8 zero size 2024-03-14T13:02:14.839 INFO:tasks.workunit.client.1.smithi137.stdout:1/395: creat d7/d4d/f85 x:0 0 0 2024-03-14T13:02:14.840 INFO:tasks.workunit.client.0.smithi111.stdout:5/489: dread d2/d16/d49/d44/f50 [0,4194304] 0 2024-03-14T13:02:14.841 INFO:tasks.workunit.client.0.smithi111.stdout:5/490: chown d2/d16/d49/d7c/d95/d36/l4d 55 1 2024-03-14T13:02:14.842 INFO:tasks.workunit.client.0.smithi111.stdout:4/627: symlink d1d/d22/d30/d5e/d6f/d7e/d66/ld9 0 2024-03-14T13:02:14.842 INFO:tasks.workunit.client.0.smithi111.stdout:5/491: stat d2/d45/d79 0 2024-03-14T13:02:14.843 INFO:tasks.workunit.client.1.smithi137.stdout:4/339: mkdir d4/d27/d5c/d77 0 2024-03-14T13:02:14.843 INFO:tasks.workunit.client.1.smithi137.stdout:5/289: dwrite d7/d1d/f57 [0,4194304] 0 2024-03-14T13:02:14.843 INFO:tasks.workunit.client.0.smithi111.stdout:2/349: write dd/d11/d1a/f62 [582003,111959] 0 2024-03-14T13:02:14.844 INFO:tasks.workunit.client.1.smithi137.stdout:2/312: creat d2/d7/d2e/d3e/d44/d6b/d1d/d41/f6e x:0 0 0 2024-03-14T13:02:14.845 INFO:tasks.workunit.client.0.smithi111.stdout:0/349: dwrite d6/d9/de/f3c [0,4194304] 0 2024-03-14T13:02:14.846 INFO:tasks.workunit.client.1.smithi137.stdout:1/396: rmdir d7 39 2024-03-14T13:02:14.847 INFO:tasks.workunit.client.0.smithi111.stdout:3/442: dread d0/d2d/d35/d50/d5e/d19/f1a [0,4194304] 0 2024-03-14T13:02:14.848 INFO:tasks.workunit.client.0.smithi111.stdout:5/492: write d2/d13/f29 [3118175,105162] 0 2024-03-14T13:02:14.849 INFO:tasks.workunit.client.1.smithi137.stdout:2/313: unlink d2/d7/d2e/d3e/d44/d6b/d1d/c55 0 2024-03-14T13:02:14.849 INFO:tasks.workunit.client.0.smithi111.stdout:8/574: dread d3/d1b/d20/d35/d40/d48/f50 [0,4194304] 0 2024-03-14T13:02:14.850 INFO:tasks.workunit.client.0.smithi111.stdout:2/350: creat dd/d17/f71 x:0 0 0 2024-03-14T13:02:14.851 INFO:tasks.workunit.client.1.smithi137.stdout:2/314: chown d2/d7/d2e/d3e/d44/d6b/d22 187662651 1 2024-03-14T13:02:14.851 INFO:tasks.workunit.client.1.smithi137.stdout:8/235: link d1/dc/d3c/d54/d34/l50 d1/dc/d11/l59 0 2024-03-14T13:02:14.852 INFO:tasks.workunit.client.1.smithi137.stdout:2/315: readlink d2/d34/d3c/d53/l58 0 2024-03-14T13:02:14.852 INFO:tasks.workunit.client.1.smithi137.stdout:6/335: mknod d2/d54/d55/c7d 0 2024-03-14T13:02:14.853 INFO:tasks.workunit.client.0.smithi111.stdout:2/351: readlink dd/l1d 0 2024-03-14T13:02:14.854 INFO:tasks.workunit.client.0.smithi111.stdout:0/350: write d6/d3d/f52 [350360,131052] 0 2024-03-14T13:02:14.855 INFO:tasks.workunit.client.0.smithi111.stdout:5/493: fsync d2/d16/d49/d7c/d95/f24 0 2024-03-14T13:02:14.855 INFO:tasks.workunit.client.1.smithi137.stdout:1/397: symlink d7/d9/d19/d1f/d2e/l86 0 2024-03-14T13:02:14.856 INFO:tasks.workunit.client.0.smithi111.stdout:8/575: mkdir d3/d1b/d20/d35/d40/d48/daf 0 2024-03-14T13:02:14.859 INFO:tasks.workunit.client.0.smithi111.stdout:2/352: chown dd/c22 244399578 1 2024-03-14T13:02:14.859 INFO:tasks.workunit.client.0.smithi111.stdout:4/628: dwrite d1d/d22/fd7 [0,4194304] 0 2024-03-14T13:02:14.859 INFO:tasks.workunit.client.0.smithi111.stdout:5/494: mkdir d2/d16/d49/d44/d4a/d9e 0 2024-03-14T13:02:14.860 INFO:tasks.workunit.client.0.smithi111.stdout:2/353: readlink dd/l14 0 2024-03-14T13:02:14.862 INFO:tasks.workunit.client.0.smithi111.stdout:5/495: read d2/d16/d49/d7c/d95/f24 [2808361,127] 0 2024-03-14T13:02:14.865 INFO:tasks.workunit.client.1.smithi137.stdout:6/336: dwrite d2/d54/d55/d6d/d76/d57/f74 [0,4194304] 0 2024-03-14T13:02:14.867 INFO:tasks.workunit.client.1.smithi137.stdout:2/316: rename d2/d7/d2e/d3e/d44/d6b/c11 to d2/d7/d2e/d3e/d44/d6b/d1d/c6f 0 2024-03-14T13:02:14.868 INFO:tasks.workunit.client.1.smithi137.stdout:6/337: mknod d2/d4c/c7e 0 2024-03-14T13:02:14.869 INFO:tasks.workunit.client.0.smithi111.stdout:2/354: dwrite dd/d11/d48/d49/f67 [0,4194304] 0 2024-03-14T13:02:14.873 INFO:tasks.workunit.client.1.smithi137.stdout:8/236: chown d1/dc/d3c/d54/d34/f40 99 1 2024-03-14T13:02:14.875 INFO:tasks.workunit.client.0.smithi111.stdout:5/496: dwrite d2/d16/d49/d7c/d8b/f91 [0,4194304] 0 2024-03-14T13:02:14.875 INFO:tasks.workunit.client.1.smithi137.stdout:6/338: rename d2/d6/cb to d2/d6/de/c7f 0 2024-03-14T13:02:14.878 INFO:tasks.workunit.client.1.smithi137.stdout:6/339: write d2/d6/de/d45/d59/f37 [825326,130016] 0 2024-03-14T13:02:14.885 INFO:tasks.workunit.client.1.smithi137.stdout:5/290: truncate d7/d16/d18/d25/d28/f3d 472290 0 2024-03-14T13:02:14.886 INFO:tasks.workunit.client.1.smithi137.stdout:8/237: dwrite d1/fd [0,4194304] 0 2024-03-14T13:02:14.887 INFO:tasks.workunit.client.1.smithi137.stdout:6/340: symlink d2/d54/d55/d6d/d76/d57/d58/l80 0 2024-03-14T13:02:14.890 INFO:tasks.workunit.client.1.smithi137.stdout:8/238: dread - d1/dc/d10/f21 zero size 2024-03-14T13:02:14.891 INFO:tasks.workunit.client.1.smithi137.stdout:5/291: unlink d7/d36/f62 0 2024-03-14T13:02:14.894 INFO:tasks.workunit.client.1.smithi137.stdout:2/317: write d2/d7/d2e/d3e/d44/d6b/f4a [96509,44134] 0 2024-03-14T13:02:14.895 INFO:tasks.workunit.client.1.smithi137.stdout:1/398: dwrite d7/d9/d19/f2f [0,4194304] 0 2024-03-14T13:02:14.896 INFO:tasks.workunit.client.1.smithi137.stdout:4/340: dread d4/d9/d3a/d41/d38/f4c [0,4194304] 0 2024-03-14T13:02:14.898 INFO:tasks.workunit.client.1.smithi137.stdout:4/341: dread - d4/d9/d3a/f56 zero size 2024-03-14T13:02:14.901 INFO:tasks.workunit.client.1.smithi137.stdout:8/239: dwrite d1/dc/d11/f2d [0,4194304] 0 2024-03-14T13:02:14.904 INFO:tasks.workunit.client.0.smithi111.stdout:1/406: dread f19 [0,4194304] 0 2024-03-14T13:02:14.906 INFO:tasks.workunit.client.1.smithi137.stdout:9/446: sync 2024-03-14T13:02:14.915 INFO:tasks.workunit.client.1.smithi137.stdout:1/399: mknod d7/d9/d19/d1f/d2e/c87 0 2024-03-14T13:02:14.916 INFO:tasks.workunit.client.1.smithi137.stdout:4/342: symlink d4/d10/d5a/l78 0 2024-03-14T13:02:14.917 INFO:tasks.workunit.client.1.smithi137.stdout:9/447: dwrite d6/d56/f89 [0,4194304] 0 2024-03-14T13:02:14.920 INFO:tasks.workunit.client.1.smithi137.stdout:6/341: link d2/d6/de/d45/d59/f40 d2/d54/d55/d6d/d76/d57/f81 0 2024-03-14T13:02:14.920 INFO:tasks.workunit.client.1.smithi137.stdout:1/400: read d7/d9/d19/d40/d42/f49 [363931,1701] 0 2024-03-14T13:02:14.926 INFO:tasks.workunit.client.1.smithi137.stdout:1/401: stat d7/d4d/f64 0 2024-03-14T13:02:14.927 INFO:tasks.workunit.client.1.smithi137.stdout:9/448: rmdir d6/d26 39 2024-03-14T13:02:14.929 INFO:tasks.workunit.client.1.smithi137.stdout:9/449: readlink d6/df/d19/d3a/l97 0 2024-03-14T13:02:14.929 INFO:tasks.workunit.client.1.smithi137.stdout:6/342: truncate d2/d54/d55/d6d/d76/d57/f70 600173 0 2024-03-14T13:02:14.929 INFO:tasks.workunit.client.1.smithi137.stdout:4/343: dwrite d4/d9/f18 [0,4194304] 0 2024-03-14T13:02:14.930 INFO:tasks.workunit.client.1.smithi137.stdout:1/402: rmdir d7/d9/d19/d40/d58 39 2024-03-14T13:02:14.932 INFO:tasks.workunit.client.1.smithi137.stdout:9/450: fdatasync d6/f49 0 2024-03-14T13:02:14.935 INFO:tasks.workunit.client.1.smithi137.stdout:4/344: mkdir d4/d9/d3a/d41/d38/d79 0 2024-03-14T13:02:14.939 INFO:tasks.workunit.client.1.smithi137.stdout:1/403: dwrite d7/d9/d19/f2f [4194304,4194304] 0 2024-03-14T13:02:14.942 INFO:tasks.workunit.client.1.smithi137.stdout:9/451: dread d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:14.942 INFO:tasks.workunit.client.1.smithi137.stdout:2/318: write d2/d34/d3c/d53/f5e [1334913,120860] 0 2024-03-14T13:02:14.943 INFO:tasks.workunit.client.1.smithi137.stdout:1/404: write d7/d9/d19/d40/d50/f80 [14772,98618] 0 2024-03-14T13:02:14.945 INFO:tasks.workunit.client.1.smithi137.stdout:9/452: readlink d6/l8a 0 2024-03-14T13:02:14.945 INFO:tasks.workunit.client.1.smithi137.stdout:4/345: link d4/c2a d4/d4d/c7a 0 2024-03-14T13:02:14.946 INFO:tasks.workunit.client.1.smithi137.stdout:9/453: fdatasync d6/f57 0 2024-03-14T13:02:14.946 INFO:tasks.workunit.client.1.smithi137.stdout:8/240: dread d1/dc/d3c/d54/f1c [0,4194304] 0 2024-03-14T13:02:14.949 INFO:tasks.workunit.client.1.smithi137.stdout:9/454: symlink d6/df/d19/d1f/l9b 0 2024-03-14T13:02:14.951 INFO:tasks.workunit.client.1.smithi137.stdout:2/319: readlink d2/d7/d2e/d3e/d44/d6b/d22/l50 0 2024-03-14T13:02:14.954 INFO:tasks.workunit.client.1.smithi137.stdout:2/320: read - d2/d34/f69 zero size 2024-03-14T13:02:14.955 INFO:tasks.workunit.client.1.smithi137.stdout:9/455: truncate d6/df/d71/f8b 868597 0 2024-03-14T13:02:14.956 INFO:tasks.workunit.client.1.smithi137.stdout:2/321: read - d2/f28 zero size 2024-03-14T13:02:14.958 INFO:tasks.workunit.client.1.smithi137.stdout:1/405: dwrite d7/d4d/f7e [0,4194304] 0 2024-03-14T13:02:14.959 INFO:tasks.workunit.client.1.smithi137.stdout:9/456: rmdir d6/d26 39 2024-03-14T13:02:14.960 INFO:tasks.workunit.client.1.smithi137.stdout:8/241: dwrite d1/dc/d3c/d54/f48 [4194304,4194304] 0 2024-03-14T13:02:14.962 INFO:tasks.workunit.client.1.smithi137.stdout:9/457: dread - d6/df/d19/d33/d8f/f9a zero size 2024-03-14T13:02:14.963 INFO:tasks.workunit.client.1.smithi137.stdout:1/406: write d7/d4d/f56 [6571000,7408] 0 2024-03-14T13:02:14.967 INFO:tasks.workunit.client.1.smithi137.stdout:2/322: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/c70 0 2024-03-14T13:02:14.971 INFO:tasks.workunit.client.0.smithi111.stdout:3/443: symlink d0/d2d/d35/d50/d5e/d40/d67/lae 0 2024-03-14T13:02:14.973 INFO:tasks.workunit.client.0.smithi111.stdout:0/351: mknod d6/c6b 0 2024-03-14T13:02:14.975 INFO:tasks.workunit.client.1.smithi137.stdout:2/323: fsync d2/d7/d2e/d3e/d44/d6b/f1b 0 2024-03-14T13:02:14.975 INFO:tasks.workunit.client.1.smithi137.stdout:8/242: dwrite d1/dc/d11/d13/f45 [0,4194304] 0 2024-03-14T13:02:14.976 INFO:tasks.workunit.client.1.smithi137.stdout:8/243: readlink d1/l2f 0 2024-03-14T13:02:14.980 INFO:tasks.workunit.client.1.smithi137.stdout:2/324: creat d2/d7/d2e/f71 x:0 0 0 2024-03-14T13:02:14.980 INFO:tasks.workunit.client.1.smithi137.stdout:8/244: mknod d1/dc/d3c/c5a 0 2024-03-14T13:02:14.984 INFO:tasks.workunit.client.1.smithi137.stdout:9/458: dwrite d6/df/f64 [0,4194304] 0 2024-03-14T13:02:14.984 INFO:tasks.workunit.client.1.smithi137.stdout:8/245: unlink d1/fd 0 2024-03-14T13:02:14.985 INFO:tasks.workunit.client.0.smithi111.stdout:3/444: dwrite d0/d2d/d35/d50/f88 [0,4194304] 0 2024-03-14T13:02:14.987 INFO:tasks.workunit.client.0.smithi111.stdout:8/576: mknod d3/d1b/d20/d35/d40/dad/cb0 0 2024-03-14T13:02:14.990 INFO:tasks.workunit.client.0.smithi111.stdout:2/355: symlink dd/l72 0 2024-03-14T13:02:14.990 INFO:tasks.workunit.client.0.smithi111.stdout:0/352: read - d6/d9/de/f41 zero size 2024-03-14T13:02:14.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/325: write d2/d7/d2e/d3e/d44/d6b/f1b [1126760,22255] 0 2024-03-14T13:02:14.992 INFO:tasks.workunit.client.1.smithi137.stdout:1/407: dwrite d7/d9/d19/d40/f46 [0,4194304] 0 2024-03-14T13:02:14.993 INFO:tasks.workunit.client.0.smithi111.stdout:3/445: stat d0/d30/l53 0 2024-03-14T13:02:14.995 INFO:tasks.workunit.client.0.smithi111.stdout:2/356: write dd/d17/f37 [1033479,96664] 0 2024-03-14T13:02:14.996 INFO:tasks.workunit.client.0.smithi111.stdout:5/497: truncate d2/d45/f20 1626282 0 2024-03-14T13:02:14.996 INFO:tasks.workunit.client.0.smithi111.stdout:1/407: dread d20/f44 [0,4194304] 0 2024-03-14T13:02:14.996 INFO:tasks.workunit.client.1.smithi137.stdout:9/459: dread - d6/df/d19/d3a/d40/f5c zero size 2024-03-14T13:02:14.996 INFO:tasks.workunit.client.1.smithi137.stdout:8/246: read d1/dc/d11/d13/f3e [813750,78528] 0 2024-03-14T13:02:14.999 INFO:tasks.workunit.client.1.smithi137.stdout:1/408: chown d7/c1a 51718 1 2024-03-14T13:02:14.999 INFO:tasks.workunit.client.1.smithi137.stdout:8/247: fdatasync d1/dc/d11/d13/f26 0 2024-03-14T13:02:14.999 INFO:tasks.workunit.client.0.smithi111.stdout:4/629: dwrite d1d/d22/d40/fcf [0,4194304] 0 2024-03-14T13:02:15.003 INFO:tasks.workunit.client.1.smithi137.stdout:8/248: chown d1/dc/lf 79455320 1 2024-03-14T13:02:15.004 INFO:tasks.workunit.client.1.smithi137.stdout:2/326: dread - d2/d7/d2e/d3e/d44/f60 zero size 2024-03-14T13:02:15.005 INFO:tasks.workunit.client.0.smithi111.stdout:0/353: rmdir d6/d9/de/d32 39 2024-03-14T13:02:15.008 INFO:tasks.workunit.client.0.smithi111.stdout:3/446: mknod d0/d2d/d35/d50/d6c/da3/caf 0 2024-03-14T13:02:15.010 INFO:tasks.workunit.client.1.smithi137.stdout:9/460: getdents d6/df/d19/d3a/d40 0 2024-03-14T13:02:15.011 INFO:tasks.workunit.client.0.smithi111.stdout:1/408: rename d20/c53 to d20/d40/c90 0 2024-03-14T13:02:15.012 INFO:tasks.workunit.client.0.smithi111.stdout:8/577: getdents d3/d1b/d20/d35 0 2024-03-14T13:02:15.014 INFO:tasks.workunit.client.0.smithi111.stdout:3/447: mkdir d0/d2d/d35/d4c/db0 0 2024-03-14T13:02:15.015 INFO:tasks.workunit.client.1.smithi137.stdout:1/409: dwrite d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:15.016 INFO:tasks.workunit.client.0.smithi111.stdout:3/448: write d0/d2d/d35/d69/d84/f99 [544129,55679] 0 2024-03-14T13:02:15.017 INFO:tasks.workunit.client.1.smithi137.stdout:1/410: write d7/d31/d54/f62 [243279,79631] 0 2024-03-14T13:02:15.018 INFO:tasks.workunit.client.0.smithi111.stdout:0/354: symlink d6/d9/de/d32/d64/l6c 0 2024-03-14T13:02:15.019 INFO:tasks.workunit.client.0.smithi111.stdout:4/630: link d1d/d22/d28/ca4 d1d/d22/d30/d87/dd1/cda 0 2024-03-14T13:02:15.019 INFO:tasks.workunit.client.0.smithi111.stdout:2/357: dwrite dd/d17/f63 [0,4194304] 0 2024-03-14T13:02:15.020 INFO:tasks.workunit.client.0.smithi111.stdout:2/358: stat dd/c15 0 2024-03-14T13:02:15.020 INFO:tasks.workunit.client.0.smithi111.stdout:4/631: write d1d/d22/f55 [2023274,59765] 0 2024-03-14T13:02:15.023 INFO:tasks.workunit.client.0.smithi111.stdout:2/359: chown dd/d11/d48/d49/f5c 7252 1 2024-03-14T13:02:15.023 INFO:tasks.workunit.client.0.smithi111.stdout:1/409: read d20/d21/f42 [1543495,15669] 0 2024-03-14T13:02:15.025 INFO:tasks.workunit.client.0.smithi111.stdout:3/449: creat d0/d2d/d35/d50/d6c/da3/daa/fb1 x:0 0 0 2024-03-14T13:02:15.027 INFO:tasks.workunit.client.1.smithi137.stdout:2/327: getdents d2/d34/d3c/d53 0 2024-03-14T13:02:15.027 INFO:tasks.workunit.client.1.smithi137.stdout:9/461: dread d6/f1d [0,4194304] 0 2024-03-14T13:02:15.028 INFO:tasks.workunit.client.0.smithi111.stdout:4/632: rename d1d/d22/d40/fa0 to d1d/d22/d30/d7c/d59/dc4/fdb 0 2024-03-14T13:02:15.030 INFO:tasks.workunit.client.1.smithi137.stdout:8/249: fdatasync d1/f31 0 2024-03-14T13:02:15.030 INFO:tasks.workunit.client.0.smithi111.stdout:1/410: symlink d20/d25/d31/d80/l91 0 2024-03-14T13:02:15.030 INFO:tasks.workunit.client.0.smithi111.stdout:4/633: readlink d1d/d22/d28/l8a 0 2024-03-14T13:02:15.031 INFO:tasks.workunit.client.1.smithi137.stdout:1/411: creat d7/f88 x:0 0 0 2024-03-14T13:02:15.032 INFO:tasks.workunit.client.1.smithi137.stdout:9/462: mkdir d6/d56/d9c 0 2024-03-14T13:02:15.033 INFO:tasks.workunit.client.0.smithi111.stdout:2/360: dwrite dd/d11/f6d [0,4194304] 0 2024-03-14T13:02:15.034 INFO:tasks.workunit.client.1.smithi137.stdout:9/463: chown d6/df/f70 711 1 2024-03-14T13:02:15.034 INFO:tasks.workunit.client.0.smithi111.stdout:4/634: write d1d/d22/d56/f5f [3844007,38416] 0 2024-03-14T13:02:15.035 INFO:tasks.workunit.client.1.smithi137.stdout:9/464: dread - d6/df/f70 zero size 2024-03-14T13:02:15.036 INFO:tasks.workunit.client.0.smithi111.stdout:1/411: creat d20/d25/d31/d80/f92 x:0 0 0 2024-03-14T13:02:15.037 INFO:tasks.workunit.client.1.smithi137.stdout:9/465: write d6/df/d19/d33/f46 [2732373,111395] 0 2024-03-14T13:02:15.037 INFO:tasks.workunit.client.1.smithi137.stdout:1/412: mkdir d7/d9/d19/d1f/d89 0 2024-03-14T13:02:15.040 INFO:tasks.workunit.client.0.smithi111.stdout:2/361: rename dd/d11/d48 to dd/d17/d34/d73 0 2024-03-14T13:02:15.044 INFO:tasks.workunit.client.0.smithi111.stdout:4/635: fdatasync d1d/d22/d30/d5e/d6f/d7e/d66/f94 0 2024-03-14T13:02:15.045 INFO:tasks.workunit.client.1.smithi137.stdout:9/466: unlink d6/df/d19/f83 0 2024-03-14T13:02:15.045 INFO:tasks.workunit.client.0.smithi111.stdout:4/636: stat d1d/d22/d30/d87 0 2024-03-14T13:02:15.047 INFO:tasks.workunit.client.1.smithi137.stdout:9/467: fdatasync d6/df/d19/d3a/d40/f7f 0 2024-03-14T13:02:15.047 INFO:tasks.workunit.client.0.smithi111.stdout:2/362: mkdir dd/d17/d34/d74 0 2024-03-14T13:02:15.048 INFO:tasks.workunit.client.0.smithi111.stdout:1/412: dwrite d20/f23 [0,4194304] 0 2024-03-14T13:02:15.048 INFO:tasks.workunit.client.1.smithi137.stdout:8/250: getdents d1/dc/d3c/d54/d34 0 2024-03-14T13:02:15.049 INFO:tasks.workunit.client.1.smithi137.stdout:1/413: dwrite d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:15.049 INFO:tasks.workunit.client.0.smithi111.stdout:4/637: symlink d1d/d22/d30/d5e/d6f/d7e/ldc 0 2024-03-14T13:02:15.051 INFO:tasks.workunit.client.0.smithi111.stdout:1/413: write d20/f57 [982280,2226] 0 2024-03-14T13:02:15.052 INFO:tasks.workunit.client.0.smithi111.stdout:1/414: stat l1 0 2024-03-14T13:02:15.052 INFO:tasks.workunit.client.1.smithi137.stdout:8/251: getdents d1/dc/d11/d13/d51 0 2024-03-14T13:02:15.054 INFO:tasks.workunit.client.0.smithi111.stdout:5/498: dread d2/d45/f34 [0,4194304] 0 2024-03-14T13:02:15.055 INFO:tasks.workunit.client.0.smithi111.stdout:2/363: rename dd/d17/d34/d4f/d55 to dd/d17/d34/d73/d75 0 2024-03-14T13:02:15.056 INFO:tasks.workunit.client.0.smithi111.stdout:1/415: read d20/d25/f7c [58017,17976] 0 2024-03-14T13:02:15.056 INFO:tasks.workunit.client.0.smithi111.stdout:4/638: symlink d1d/d22/d30/d5e/db8/ldd 0 2024-03-14T13:02:15.057 INFO:tasks.workunit.client.1.smithi137.stdout:8/252: write d1/f31 [1569765,82457] 0 2024-03-14T13:02:15.060 INFO:tasks.workunit.client.0.smithi111.stdout:4/639: mknod d1d/d22/d30/d7c/cde 0 2024-03-14T13:02:15.060 INFO:tasks.workunit.client.1.smithi137.stdout:8/253: symlink d1/dc/l5b 0 2024-03-14T13:02:15.062 INFO:tasks.workunit.client.0.smithi111.stdout:1/416: creat d20/d21/d28/d5c/f93 x:0 0 0 2024-03-14T13:02:15.062 INFO:tasks.workunit.client.1.smithi137.stdout:8/254: creat d1/dc/f5c x:0 0 0 2024-03-14T13:02:15.063 INFO:tasks.workunit.client.0.smithi111.stdout:1/417: write d20/d40/f54 [450801,127473] 0 2024-03-14T13:02:15.064 INFO:tasks.workunit.client.1.smithi137.stdout:8/255: fsync d1/dc/d3c/d54/f38 0 2024-03-14T13:02:15.064 INFO:tasks.workunit.client.0.smithi111.stdout:2/364: rename dd/d17/d53/f5f to dd/f76 0 2024-03-14T13:02:15.064 INFO:tasks.workunit.client.0.smithi111.stdout:1/418: write d20/d25/d75/f5e [236996,87693] 0 2024-03-14T13:02:15.067 INFO:tasks.workunit.client.1.smithi137.stdout:8/256: symlink d1/dc/l5d 0 2024-03-14T13:02:15.068 INFO:tasks.workunit.client.1.smithi137.stdout:8/257: write d1/dc/d11/d13/f4f [964624,96186] 0 2024-03-14T13:02:15.071 INFO:tasks.workunit.client.0.smithi111.stdout:0/355: dread d6/d9/f12 [0,4194304] 0 2024-03-14T13:02:15.075 INFO:tasks.workunit.client.1.smithi137.stdout:8/258: dwrite d1/dc/d10/f21 [0,4194304] 0 2024-03-14T13:02:15.077 INFO:tasks.workunit.client.1.smithi137.stdout:8/259: mkdir d1/d5e 0 2024-03-14T13:02:15.083 INFO:tasks.workunit.client.1.smithi137.stdout:8/260: dread d1/dc/f39 [0,4194304] 0 2024-03-14T13:02:15.083 INFO:tasks.workunit.client.1.smithi137.stdout:9/468: dread d6/f49 [0,4194304] 0 2024-03-14T13:02:15.083 INFO:tasks.workunit.client.1.smithi137.stdout:2/328: truncate d2/d7/db/f38 389528 0 2024-03-14T13:02:15.084 INFO:tasks.workunit.client.1.smithi137.stdout:7/287: sync 2024-03-14T13:02:15.085 INFO:tasks.workunit.client.1.smithi137.stdout:9/469: fdatasync d6/df/f51 0 2024-03-14T13:02:15.086 INFO:tasks.workunit.client.0.smithi111.stdout:4/640: getdents d1d/d22/d40 0 2024-03-14T13:02:15.087 INFO:tasks.workunit.client.0.smithi111.stdout:4/641: chown d1d/d22/d23/f67 442742710 1 2024-03-14T13:02:15.087 INFO:tasks.workunit.client.1.smithi137.stdout:9/470: write d6/d56/f73 [702719,29699] 0 2024-03-14T13:02:15.088 INFO:tasks.workunit.client.1.smithi137.stdout:8/261: unlink d1/ce 0 2024-03-14T13:02:15.092 INFO:tasks.workunit.client.0.smithi111.stdout:3/450: dwrite d0/d2d/d35/d50/d5e/f97 [0,4194304] 0 2024-03-14T13:02:15.093 INFO:tasks.workunit.client.1.smithi137.stdout:2/329: dwrite d2/d7/db/d2f/f40 [0,4194304] 0 2024-03-14T13:02:15.093 INFO:tasks.workunit.client.0.smithi111.stdout:2/365: rmdir dd/d17/d34/d73/d75 39 2024-03-14T13:02:15.095 INFO:tasks.workunit.client.1.smithi137.stdout:9/471: unlink d6/df/f64 0 2024-03-14T13:02:15.098 INFO:tasks.workunit.client.1.smithi137.stdout:7/288: unlink d3/d6/d14/d27/c4d 0 2024-03-14T13:02:15.098 INFO:tasks.workunit.client.0.smithi111.stdout:3/451: creat d0/d2d/d35/d69/d84/fb2 x:0 0 0 2024-03-14T13:02:15.101 INFO:tasks.workunit.client.1.smithi137.stdout:2/330: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/l72 0 2024-03-14T13:02:15.103 INFO:tasks.workunit.client.1.smithi137.stdout:8/262: dwrite d1/f49 [0,4194304] 0 2024-03-14T13:02:15.104 INFO:tasks.workunit.client.1.smithi137.stdout:7/289: creat d3/d6/d59/f5f x:0 0 0 2024-03-14T13:02:15.106 INFO:tasks.workunit.client.0.smithi111.stdout:0/356: dwrite d6/d9/de/f2c [0,4194304] 0 2024-03-14T13:02:15.110 INFO:tasks.workunit.client.0.smithi111.stdout:0/357: creat d6/d9/d5b/f6d x:0 0 0 2024-03-14T13:02:15.112 INFO:tasks.workunit.client.1.smithi137.stdout:8/263: link d1/dc/d3c/d54/f1e d1/dc/f5f 0 2024-03-14T13:02:15.112 INFO:tasks.workunit.client.0.smithi111.stdout:0/358: creat d6/d5a/f6e x:0 0 0 2024-03-14T13:02:15.115 INFO:tasks.workunit.client.0.smithi111.stdout:0/359: write d6/d9/de/f18 [2605790,20501] 0 2024-03-14T13:02:15.115 INFO:tasks.workunit.client.1.smithi137.stdout:7/290: link d3/d6/d14/d27/c36 d3/d52/c60 0 2024-03-14T13:02:15.115 INFO:tasks.workunit.client.0.smithi111.stdout:2/366: truncate dd/d17/d34/d73/d49/f5c 1177916 0 2024-03-14T13:02:15.119 INFO:tasks.workunit.client.1.smithi137.stdout:2/331: dwrite d2/d7/d2e/d3e/d44/d6b/f39 [0,4194304] 0 2024-03-14T13:02:15.119 INFO:tasks.workunit.client.0.smithi111.stdout:2/367: rename dd/l72 to dd/d17/d34/d4f/l77 0 2024-03-14T13:02:15.121 INFO:tasks.workunit.client.1.smithi137.stdout:8/264: creat d1/f60 x:0 0 0 2024-03-14T13:02:15.123 INFO:tasks.workunit.client.0.smithi111.stdout:0/360: link d6/d9/d15/c50 d6/d5a/c6f 0 2024-03-14T13:02:15.124 INFO:tasks.workunit.client.0.smithi111.stdout:2/368: creat dd/d11/d1a/d56/f78 x:0 0 0 2024-03-14T13:02:15.125 INFO:tasks.workunit.client.1.smithi137.stdout:8/265: creat d1/d5e/f61 x:0 0 0 2024-03-14T13:02:15.125 INFO:tasks.workunit.client.1.smithi137.stdout:2/332: fsync d2/d7/d2e/d3e/d44/d6b/d1d/f48 0 2024-03-14T13:02:15.126 INFO:tasks.workunit.client.0.smithi111.stdout:0/361: symlink d6/d9/d15/l70 0 2024-03-14T13:02:15.126 INFO:tasks.workunit.client.1.smithi137.stdout:2/333: write d2/f4b [2878399,69512] 0 2024-03-14T13:02:15.127 INFO:tasks.workunit.client.1.smithi137.stdout:8/266: write d1/f6 [1088839,121966] 0 2024-03-14T13:02:15.130 INFO:tasks.workunit.client.1.smithi137.stdout:2/334: unlink d2/d7/d2e/d3e/d44/d6b/d22/d26/f5c 0 2024-03-14T13:02:15.131 INFO:tasks.workunit.client.1.smithi137.stdout:2/335: chown d2/d7/db/d2f/d66 284724 1 2024-03-14T13:02:15.131 INFO:tasks.workunit.client.0.smithi111.stdout:4/642: dread d1d/d22/d23/f67 [0,4194304] 0 2024-03-14T13:02:15.132 INFO:tasks.workunit.client.1.smithi137.stdout:8/267: creat d1/dc/d3c/d54/d34/f62 x:0 0 0 2024-03-14T13:02:15.133 INFO:tasks.workunit.client.0.smithi111.stdout:0/362: rename d6/d9/cf to d6/c71 0 2024-03-14T13:02:15.134 INFO:tasks.workunit.client.1.smithi137.stdout:8/268: creat d1/d5e/f63 x:0 0 0 2024-03-14T13:02:15.135 INFO:tasks.workunit.client.0.smithi111.stdout:4/643: mkdir d1d/d22/d30/d5e/db8/ddf 0 2024-03-14T13:02:15.136 INFO:tasks.workunit.client.0.smithi111.stdout:2/369: dwrite dd/f10 [4194304,4194304] 0 2024-03-14T13:02:15.136 INFO:tasks.workunit.client.1.smithi137.stdout:8/269: write d1/dc/d3c/d54/f48 [6192297,13700] 0 2024-03-14T13:02:15.136 INFO:tasks.workunit.client.0.smithi111.stdout:4/644: chown d1d/c4a 792 1 2024-03-14T13:02:15.137 INFO:tasks.workunit.client.1.smithi137.stdout:8/270: chown d1/dc/d11/d13/f16 876 1 2024-03-14T13:02:15.138 INFO:tasks.workunit.client.0.smithi111.stdout:0/363: creat d6/f72 x:0 0 0 2024-03-14T13:02:15.139 INFO:tasks.workunit.client.1.smithi137.stdout:7/291: dread d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:15.140 INFO:tasks.workunit.client.1.smithi137.stdout:8/271: rmdir d1/dc/d11/d56 39 2024-03-14T13:02:15.141 INFO:tasks.workunit.client.0.smithi111.stdout:2/370: symlink dd/d17/d34/l79 0 2024-03-14T13:02:15.142 INFO:tasks.workunit.client.0.smithi111.stdout:2/371: dread - dd/d17/d34/f36 zero size 2024-03-14T13:02:15.143 INFO:tasks.workunit.client.0.smithi111.stdout:0/364: read f5 [1255585,77960] 0 2024-03-14T13:02:15.144 INFO:tasks.workunit.client.0.smithi111.stdout:2/372: symlink dd/d11/d1a/l7a 0 2024-03-14T13:02:15.147 INFO:tasks.workunit.client.1.smithi137.stdout:7/292: dread d3/d6/d14/f2f [0,4194304] 0 2024-03-14T13:02:15.147 INFO:tasks.workunit.client.0.smithi111.stdout:0/365: rename d6/d9/de/f26 to d6/d21/f73 0 2024-03-14T13:02:15.148 INFO:tasks.workunit.client.0.smithi111.stdout:2/373: unlink dd/d17/l69 0 2024-03-14T13:02:15.149 INFO:tasks.workunit.client.1.smithi137.stdout:8/272: creat d1/dc/d10/f64 x:0 0 0 2024-03-14T13:02:15.150 INFO:tasks.workunit.client.1.smithi137.stdout:8/273: write d1/dc/d11/d13/f26 [78774,34029] 0 2024-03-14T13:02:15.151 INFO:tasks.workunit.client.1.smithi137.stdout:7/293: rmdir d3/d5/d23/d2c 39 2024-03-14T13:02:15.154 INFO:tasks.workunit.client.1.smithi137.stdout:7/294: write d3/d6/d14/d22/f43 [798815,36115] 0 2024-03-14T13:02:15.156 INFO:tasks.workunit.client.0.smithi111.stdout:2/374: rename dd/d17/d34/c19 to dd/d11/c7b 0 2024-03-14T13:02:15.156 INFO:tasks.workunit.client.0.smithi111.stdout:2/375: chown dd 8112 1 2024-03-14T13:02:15.158 INFO:tasks.workunit.client.0.smithi111.stdout:2/376: write dd/d11/f33 [3800690,1477] 0 2024-03-14T13:02:15.159 INFO:tasks.workunit.client.1.smithi137.stdout:7/295: rename d3/d6/f9 to d3/d54/f61 0 2024-03-14T13:02:15.160 INFO:tasks.workunit.client.1.smithi137.stdout:7/296: fsync d3/d6/d59/f5f 0 2024-03-14T13:02:15.160 INFO:tasks.workunit.client.0.smithi111.stdout:0/366: getdents d6 0 2024-03-14T13:02:15.160 INFO:tasks.workunit.client.0.smithi111.stdout:4/645: truncate d1d/d22/d30/d5e/d6f/d7e/f65 319207 0 2024-03-14T13:02:15.162 INFO:tasks.workunit.client.1.smithi137.stdout:7/297: symlink d3/d6/d59/l62 0 2024-03-14T13:02:15.163 INFO:tasks.workunit.client.1.smithi137.stdout:7/298: write d3/d6/d14/f57 [4706198,13392] 0 2024-03-14T13:02:15.163 INFO:tasks.workunit.client.0.smithi111.stdout:0/367: mkdir d6/d21/d74 0 2024-03-14T13:02:15.164 INFO:tasks.workunit.client.0.smithi111.stdout:4/646: creat d1d/d22/dc6/fe0 x:0 0 0 2024-03-14T13:02:15.165 INFO:tasks.workunit.client.1.smithi137.stdout:2/336: write d2/d7/d2e/d3e/d44/f68 [1093084,49768] 0 2024-03-14T13:02:15.168 INFO:tasks.workunit.client.0.smithi111.stdout:0/368: symlink d6/d9/de/d3a/d46/l75 0 2024-03-14T13:02:15.169 INFO:tasks.workunit.client.0.smithi111.stdout:2/377: dwrite dd/d11/d1a/d56/f78 [0,4194304] 0 2024-03-14T13:02:15.172 INFO:tasks.workunit.client.1.smithi137.stdout:2/337: truncate d2/d7/d2e/f6a 343798 0 2024-03-14T13:02:15.173 INFO:tasks.workunit.client.1.smithi137.stdout:7/299: dwrite d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:15.173 INFO:tasks.workunit.client.0.smithi111.stdout:0/369: symlink d6/d9/d15/d55/l76 0 2024-03-14T13:02:15.176 INFO:tasks.workunit.client.1.smithi137.stdout:8/274: dwrite d1/f44 [0,4194304] 0 2024-03-14T13:02:15.177 INFO:tasks.workunit.client.0.smithi111.stdout:4/647: dwrite d1d/d22/d56/f7f [0,4194304] 0 2024-03-14T13:02:15.177 INFO:tasks.workunit.client.0.smithi111.stdout:0/370: fdatasync d6/d9/de/d3a/d46/f66 0 2024-03-14T13:02:15.177 INFO:tasks.workunit.client.0.smithi111.stdout:2/378: mknod dd/d11/d1a/d70/c7c 0 2024-03-14T13:02:15.178 INFO:tasks.workunit.client.0.smithi111.stdout:0/371: write d6/d3d/f52 [1335169,85611] 0 2024-03-14T13:02:15.180 INFO:tasks.workunit.client.0.smithi111.stdout:2/379: write dd/d11/d1a/f52 [356194,87734] 0 2024-03-14T13:02:15.182 INFO:tasks.workunit.client.0.smithi111.stdout:0/372: mkdir d6/d9/de/d77 0 2024-03-14T13:02:15.184 INFO:tasks.workunit.client.0.smithi111.stdout:2/380: rmdir dd/d11/d1a/d56 39 2024-03-14T13:02:15.185 INFO:tasks.workunit.client.1.smithi137.stdout:2/338: dwrite d2/d7/db/d2f/f45 [0,4194304] 0 2024-03-14T13:02:15.188 INFO:tasks.workunit.client.1.smithi137.stdout:7/300: read d3/d6/f15 [4125047,74400] 0 2024-03-14T13:02:15.188 INFO:tasks.workunit.client.0.smithi111.stdout:0/373: creat d6/d9/de/d3a/d46/f78 x:0 0 0 2024-03-14T13:02:15.188 INFO:tasks.workunit.client.0.smithi111.stdout:4/648: truncate d1d/d22/d30/d5e/d6f/d7e/d66/f85 3455059 0 2024-03-14T13:02:15.189 INFO:tasks.workunit.client.1.smithi137.stdout:8/275: rmdir d1/dc/d11/d13/d51 0 2024-03-14T13:02:15.190 INFO:tasks.workunit.client.0.smithi111.stdout:2/381: creat dd/d17/d34/d74/f7d x:0 0 0 2024-03-14T13:02:15.190 INFO:tasks.workunit.client.1.smithi137.stdout:8/276: write d1/f44 [4499721,17103] 0 2024-03-14T13:02:15.191 INFO:tasks.workunit.client.1.smithi137.stdout:2/339: rename d2/d7/d2e/f71 to d2/d7/db/d2f/d66/f73 0 2024-03-14T13:02:15.192 INFO:tasks.workunit.client.0.smithi111.stdout:0/374: dread d6/d9/de/f1d [0,4194304] 0 2024-03-14T13:02:15.194 INFO:tasks.workunit.client.1.smithi137.stdout:2/340: fsync d2/d7/d2e/d3e/d44/d6b/d22/f5a 0 2024-03-14T13:02:15.194 INFO:tasks.workunit.client.0.smithi111.stdout:4/649: unlink d1d/d22/d40/fcf 0 2024-03-14T13:02:15.197 INFO:tasks.workunit.client.0.smithi111.stdout:2/382: creat dd/d17/d34/d4f/f7e x:0 0 0 2024-03-14T13:02:15.198 INFO:tasks.workunit.client.1.smithi137.stdout:7/301: creat d3/d6/f63 x:0 0 0 2024-03-14T13:02:15.198 INFO:tasks.workunit.client.1.smithi137.stdout:7/302: chown d3/d6/d14/d22/l41 8018 1 2024-03-14T13:02:15.198 INFO:tasks.workunit.client.0.smithi111.stdout:4/650: dread fb [0,4194304] 0 2024-03-14T13:02:15.203 INFO:tasks.workunit.client.1.smithi137.stdout:2/341: dwrite d2/d7/d2e/d3e/d44/d6b/d22/f5a [0,4194304] 0 2024-03-14T13:02:15.204 INFO:tasks.workunit.client.1.smithi137.stdout:2/342: chown d2/d7 203486 1 2024-03-14T13:02:15.205 INFO:tasks.workunit.client.0.smithi111.stdout:4/651: mkdir d1d/d22/d30/d87/dd1/de1 0 2024-03-14T13:02:15.205 INFO:tasks.workunit.client.0.smithi111.stdout:0/375: getdents d6/d9/d5b 0 2024-03-14T13:02:15.208 INFO:tasks.workunit.client.0.smithi111.stdout:2/383: getdents dd/d11 0 2024-03-14T13:02:15.208 INFO:tasks.workunit.client.1.smithi137.stdout:7/303: dwrite d3/d6/d14/f2f [0,4194304] 0 2024-03-14T13:02:15.209 INFO:tasks.workunit.client.0.smithi111.stdout:2/384: dread - dd/d11/d1a/f3d zero size 2024-03-14T13:02:15.215 INFO:tasks.workunit.client.1.smithi137.stdout:2/343: mkdir d2/d7/db/d74 0 2024-03-14T13:02:15.215 INFO:tasks.workunit.client.0.smithi111.stdout:2/385: mknod dd/d17/d34/d73/c7f 0 2024-03-14T13:02:15.216 INFO:tasks.workunit.client.0.smithi111.stdout:4/652: link d1d/d22/d30/d5e/db8/cb9 d1d/d22/d30/d5e/db8/ce2 0 2024-03-14T13:02:15.218 INFO:tasks.workunit.client.0.smithi111.stdout:2/386: mknod dd/d11/d1a/d70/c80 0 2024-03-14T13:02:15.219 INFO:tasks.workunit.client.1.smithi137.stdout:7/304: getdents d3/d5/d23/d2c 0 2024-03-14T13:02:15.219 INFO:tasks.workunit.client.0.smithi111.stdout:2/387: write dd/d11/d1a/f62 [1011103,123917] 0 2024-03-14T13:02:15.220 INFO:tasks.workunit.client.1.smithi137.stdout:2/344: dread d2/d7/d2e/d3e/d44/f68 [0,4194304] 0 2024-03-14T13:02:15.220 INFO:tasks.workunit.client.1.smithi137.stdout:8/277: write d1/dc/d3c/d54/d34/f47 [1880323,12037] 0 2024-03-14T13:02:15.220 INFO:tasks.workunit.client.0.smithi111.stdout:4/653: mkdir d1d/d22/d30/d5e/db8/ddf/de3 0 2024-03-14T13:02:15.220 INFO:tasks.workunit.client.0.smithi111.stdout:2/388: chown dd/d17/d34/f6f 7 1 2024-03-14T13:02:15.223 INFO:tasks.workunit.client.1.smithi137.stdout:7/305: truncate d3/d5/f46 588806 0 2024-03-14T13:02:15.223 INFO:tasks.workunit.client.0.smithi111.stdout:0/376: write d6/d9/de/f1d [30798,40289] 0 2024-03-14T13:02:15.223 INFO:tasks.workunit.client.1.smithi137.stdout:2/345: truncate d2/d7/d2e/d3e/d44/f68 1419690 0 2024-03-14T13:02:15.225 INFO:tasks.workunit.client.1.smithi137.stdout:2/346: write d2/d7/db/f16 [4144510,78173] 0 2024-03-14T13:02:15.225 INFO:tasks.workunit.client.0.smithi111.stdout:0/377: mkdir d6/d9/de/d32/d79 0 2024-03-14T13:02:15.227 INFO:tasks.workunit.client.1.smithi137.stdout:8/278: rename d1/dc/d3c/d54/d34/f62 to d1/dc/d3c/d54/d34/f65 0 2024-03-14T13:02:15.230 INFO:tasks.workunit.client.0.smithi111.stdout:2/389: dread dd/d11/d1a/f62 [0,4194304] 0 2024-03-14T13:02:15.231 INFO:tasks.workunit.client.1.smithi137.stdout:8/279: write d1/dc/d3c/d54/f1c [697363,111265] 0 2024-03-14T13:02:15.232 INFO:tasks.workunit.client.0.smithi111.stdout:0/378: dwrite f1 [0,4194304] 0 2024-03-14T13:02:15.232 INFO:tasks.workunit.client.0.smithi111.stdout:2/390: chown dd/d11/c7b 3 1 2024-03-14T13:02:15.233 INFO:tasks.workunit.client.1.smithi137.stdout:7/306: dwrite d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:15.234 INFO:tasks.workunit.client.0.smithi111.stdout:0/379: read - d6/d9/de/d32/f5e zero size 2024-03-14T13:02:15.235 INFO:tasks.workunit.client.1.smithi137.stdout:2/347: dread d2/d7/d2e/f6a [0,4194304] 0 2024-03-14T13:02:15.236 INFO:tasks.workunit.client.1.smithi137.stdout:8/280: truncate d1/dc/d11/d13/f16 3621541 0 2024-03-14T13:02:15.237 INFO:tasks.workunit.client.1.smithi137.stdout:2/348: stat d2/d7/d2e/f47 0 2024-03-14T13:02:15.237 INFO:tasks.workunit.client.0.smithi111.stdout:2/391: symlink dd/d17/d34/d73/d75/d59/l81 0 2024-03-14T13:02:15.237 INFO:tasks.workunit.client.1.smithi137.stdout:8/281: dread - d1/dc/d3c/d54/d34/f65 zero size 2024-03-14T13:02:15.237 INFO:tasks.workunit.client.0.smithi111.stdout:0/380: creat d6/d9/de/d3a/d46/f7a x:0 0 0 2024-03-14T13:02:15.238 INFO:tasks.workunit.client.0.smithi111.stdout:4/654: dread d1d/d22/d30/d5e/fb3 [0,4194304] 0 2024-03-14T13:02:15.238 INFO:tasks.workunit.client.0.smithi111.stdout:0/381: fdatasync d6/d3d/f5f 0 2024-03-14T13:02:15.240 INFO:tasks.workunit.client.0.smithi111.stdout:4/655: fdatasync d1d/d22/f37 0 2024-03-14T13:02:15.243 INFO:tasks.workunit.client.1.smithi137.stdout:2/349: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f75 x:0 0 0 2024-03-14T13:02:15.243 INFO:tasks.workunit.client.0.smithi111.stdout:2/392: rename dd/d17/f31 to dd/d17/d34/d73/d49/f82 0 2024-03-14T13:02:15.243 INFO:tasks.workunit.client.0.smithi111.stdout:2/393: dread - dd/f61 zero size 2024-03-14T13:02:15.243 INFO:tasks.workunit.client.0.smithi111.stdout:0/382: write d6/d9/de/f36 [1844266,92637] 0 2024-03-14T13:02:15.244 INFO:tasks.workunit.client.0.smithi111.stdout:4/656: creat d1d/d22/d28/fe4 x:0 0 0 2024-03-14T13:02:15.245 INFO:tasks.workunit.client.0.smithi111.stdout:0/383: chown d6/d9/de/d3a/d46/f7a 2052 1 2024-03-14T13:02:15.245 INFO:tasks.workunit.client.1.smithi137.stdout:8/282: creat d1/d4a/f66 x:0 0 0 2024-03-14T13:02:15.246 INFO:tasks.workunit.client.0.smithi111.stdout:0/384: rename d6 to d6/d9/de/d77/d7b 22 2024-03-14T13:02:15.247 INFO:tasks.workunit.client.1.smithi137.stdout:2/350: write d2/d7/d2e/d3e/d44/d6b/f20 [4008924,83743] 0 2024-03-14T13:02:15.247 INFO:tasks.workunit.client.0.smithi111.stdout:4/657: unlink d1d/d22/d23/fc0 0 2024-03-14T13:02:15.248 INFO:tasks.workunit.client.0.smithi111.stdout:2/394: creat dd/d17/f83 x:0 0 0 2024-03-14T13:02:15.249 INFO:tasks.workunit.client.1.smithi137.stdout:2/351: write d2/d7/d2e/f6a [1934495,62096] 0 2024-03-14T13:02:15.250 INFO:tasks.workunit.client.0.smithi111.stdout:0/385: unlink d6/d45/f4b 0 2024-03-14T13:02:15.251 INFO:tasks.workunit.client.1.smithi137.stdout:2/352: symlink d2/d7/d2e/d3e/d44/d6b/d1d/d41/l76 0 2024-03-14T13:02:15.251 INFO:tasks.workunit.client.1.smithi137.stdout:2/353: read d2/f1e [895340,78226] 0 2024-03-14T13:02:15.253 INFO:tasks.workunit.client.0.smithi111.stdout:0/386: chown d6/d5a/c6f 217146 1 2024-03-14T13:02:15.255 INFO:tasks.workunit.client.0.smithi111.stdout:4/658: dwrite d1d/d22/fd7 [0,4194304] 0 2024-03-14T13:02:15.257 INFO:tasks.workunit.client.0.smithi111.stdout:0/387: creat d6/d9/de/d77/f7c x:0 0 0 2024-03-14T13:02:15.257 INFO:tasks.workunit.client.0.smithi111.stdout:4/659: write d1d/d22/d30/d7c/d59/dc4/fce [300264,3801] 0 2024-03-14T13:02:15.258 INFO:tasks.workunit.client.0.smithi111.stdout:0/388: read d6/d9/de/f3c [2847977,106314] 0 2024-03-14T13:02:15.263 INFO:tasks.workunit.client.0.smithi111.stdout:0/389: link d6/d9/de/f27 d6/f7d 0 2024-03-14T13:02:15.263 INFO:tasks.workunit.client.1.smithi137.stdout:2/354: dread d2/d7/d2e/d3e/d44/d6b/f1b [0,4194304] 0 2024-03-14T13:02:15.264 INFO:tasks.workunit.client.0.smithi111.stdout:4/660: dwrite d1d/d22/d28/fb0 [0,4194304] 0 2024-03-14T13:02:15.267 INFO:tasks.workunit.client.1.smithi137.stdout:8/283: truncate d1/dc/d11/d13/f3e 764234 0 2024-03-14T13:02:15.267 INFO:tasks.workunit.client.1.smithi137.stdout:8/284: stat d1/dc/d3c/d54 0 2024-03-14T13:02:15.270 INFO:tasks.workunit.client.1.smithi137.stdout:2/355: rename d2/d7/d2e/d3e/d44/f68 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f77 0 2024-03-14T13:02:15.271 INFO:tasks.workunit.client.1.smithi137.stdout:8/285: write d1/dc/d3c/d54/d34/f65 [282861,113013] 0 2024-03-14T13:02:15.271 INFO:tasks.workunit.client.0.smithi111.stdout:4/661: unlink d1d/d22/d56/f5f 0 2024-03-14T13:02:15.274 INFO:tasks.workunit.client.0.smithi111.stdout:0/390: symlink d6/d3d/l7e 0 2024-03-14T13:02:15.279 INFO:tasks.workunit.client.1.smithi137.stdout:8/286: dwrite d1/d5e/f61 [0,4194304] 0 2024-03-14T13:02:15.279 INFO:tasks.workunit.client.0.smithi111.stdout:0/391: getdents d6/d9/d15/d49 0 2024-03-14T13:02:15.281 INFO:tasks.workunit.client.0.smithi111.stdout:4/662: rename d1d/d22/d28 to d1d/d22/d30/de5 0 2024-03-14T13:02:15.286 INFO:tasks.workunit.client.0.smithi111.stdout:4/663: read - d1d/d22/d40/d69/d86/db1/fbf zero size 2024-03-14T13:02:15.287 INFO:tasks.workunit.client.0.smithi111.stdout:2/395: truncate dd/d17/d34/d73/f3c 2482939 0 2024-03-14T13:02:15.287 INFO:tasks.workunit.client.0.smithi111.stdout:0/392: fsync d6/d9/de/d77/f7c 0 2024-03-14T13:02:15.288 INFO:tasks.workunit.client.0.smithi111.stdout:4/664: mkdir d1d/d22/dc6/de6 0 2024-03-14T13:02:15.289 INFO:tasks.workunit.client.0.smithi111.stdout:0/393: read - d6/d9/de/d3a/f61 zero size 2024-03-14T13:02:15.289 INFO:tasks.workunit.client.1.smithi137.stdout:8/287: write d1/dc/d3c/d54/d34/f40 [484137,93701] 0 2024-03-14T13:02:15.290 INFO:tasks.workunit.client.0.smithi111.stdout:2/396: creat dd/d17/d53/f84 x:0 0 0 2024-03-14T13:02:15.290 INFO:tasks.workunit.client.0.smithi111.stdout:4/665: readlink d1d/d22/d40/l99 0 2024-03-14T13:02:15.292 INFO:tasks.workunit.client.1.smithi137.stdout:8/288: write d1/dc/f5c [456189,130196] 0 2024-03-14T13:02:15.292 INFO:tasks.workunit.client.0.smithi111.stdout:0/394: mknod d6/d9/d15/d49/c7f 0 2024-03-14T13:02:15.295 INFO:tasks.workunit.client.1.smithi137.stdout:2/356: dwrite d2/d7/db/f1c [0,4194304] 0 2024-03-14T13:02:15.297 INFO:tasks.workunit.client.0.smithi111.stdout:0/395: dread d6/d9/d15/f16 [0,4194304] 0 2024-03-14T13:02:15.298 INFO:tasks.workunit.client.1.smithi137.stdout:2/357: getdents d2/d7/d2e/d3e/d4e 0 2024-03-14T13:02:15.299 INFO:tasks.workunit.client.1.smithi137.stdout:8/289: truncate d1/dc/d11/d13/f16 263380 0 2024-03-14T13:02:15.299 INFO:tasks.workunit.client.1.smithi137.stdout:8/290: chown d1/dc/d10/f32 909 1 2024-03-14T13:02:15.300 INFO:tasks.workunit.client.1.smithi137.stdout:8/291: chown d1/dc/l5d 65502 1 2024-03-14T13:02:15.300 INFO:tasks.workunit.client.1.smithi137.stdout:8/292: stat d1/d4a/f66 0 2024-03-14T13:02:15.301 INFO:tasks.workunit.client.0.smithi111.stdout:0/396: symlink d6/d45/l80 0 2024-03-14T13:02:15.302 INFO:tasks.workunit.client.0.smithi111.stdout:4/666: write d1d/d22/d30/d5e/f7a [10255,84859] 0 2024-03-14T13:02:15.302 INFO:tasks.workunit.client.1.smithi137.stdout:8/293: creat d1/dc/d11/d13/f67 x:0 0 0 2024-03-14T13:02:15.303 INFO:tasks.workunit.client.0.smithi111.stdout:0/397: truncate d6/d9/de/f41 842336 0 2024-03-14T13:02:15.306 INFO:tasks.workunit.client.0.smithi111.stdout:4/667: fsync d1d/d22/d30/d7c/d59/dc4/fce 0 2024-03-14T13:02:15.306 INFO:tasks.workunit.client.0.smithi111.stdout:4/668: fsync d1d/f2e 0 2024-03-14T13:02:15.307 INFO:tasks.workunit.client.0.smithi111.stdout:2/397: write dd/d17/d34/d73/d49/f5c [847725,66095] 0 2024-03-14T13:02:15.307 INFO:tasks.workunit.client.1.smithi137.stdout:2/358: write d2/f1e [1126257,28124] 0 2024-03-14T13:02:15.308 INFO:tasks.workunit.client.1.smithi137.stdout:8/294: dread d1/dc/d11/d13/f4f [0,4194304] 0 2024-03-14T13:02:15.309 INFO:tasks.workunit.client.0.smithi111.stdout:0/398: creat d6/d9/de/d3a/d46/d4a/f81 x:0 0 0 2024-03-14T13:02:15.311 INFO:tasks.workunit.client.0.smithi111.stdout:4/669: creat d1d/d22/d23/fe7 x:0 0 0 2024-03-14T13:02:15.311 INFO:tasks.workunit.client.1.smithi137.stdout:8/295: write d1/dc/f52 [824172,79422] 0 2024-03-14T13:02:15.313 INFO:tasks.workunit.client.1.smithi137.stdout:2/359: unlink d2/d7/d2e/d3e/d44/d6b/c2b 0 2024-03-14T13:02:15.313 INFO:tasks.workunit.client.0.smithi111.stdout:0/399: dread d6/d9/d15/f2e [0,4194304] 0 2024-03-14T13:02:15.313 INFO:tasks.workunit.client.0.smithi111.stdout:0/400: stat d6/d9/de 0 2024-03-14T13:02:15.313 INFO:tasks.workunit.client.0.smithi111.stdout:4/670: write d1d/d22/d40/d69/d86/faa [292078,93882] 0 2024-03-14T13:02:15.314 INFO:tasks.workunit.client.1.smithi137.stdout:2/360: dread d2/d7/db/f2c [0,4194304] 0 2024-03-14T13:02:15.315 INFO:tasks.workunit.client.0.smithi111.stdout:2/398: dread dd/d17/d34/d73/f5e [0,4194304] 0 2024-03-14T13:02:15.316 INFO:tasks.workunit.client.1.smithi137.stdout:8/296: rmdir d1/dc/d11/d13 39 2024-03-14T13:02:15.317 INFO:tasks.workunit.client.0.smithi111.stdout:2/399: chown dd/d17/d34/d73/d49/f51 4 1 2024-03-14T13:02:15.318 INFO:tasks.workunit.client.0.smithi111.stdout:0/401: rename d6/f7d to d6/d9/d15/d55/f82 0 2024-03-14T13:02:15.318 INFO:tasks.workunit.client.1.smithi137.stdout:2/361: stat d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 0 2024-03-14T13:02:15.319 INFO:tasks.workunit.client.0.smithi111.stdout:0/402: chown d6/d9/d15/d55 71 1 2024-03-14T13:02:15.319 INFO:tasks.workunit.client.1.smithi137.stdout:8/297: unlink d1/dc/d3c/c5a 0 2024-03-14T13:02:15.322 INFO:tasks.workunit.client.0.smithi111.stdout:0/403: creat d6/d9/de/d3a/d46/f83 x:0 0 0 2024-03-14T13:02:15.323 INFO:tasks.workunit.client.0.smithi111.stdout:4/671: link d1d/d22/d40/c57 d1d/d22/d30/d87/dd1/de1/ce8 0 2024-03-14T13:02:15.323 INFO:tasks.workunit.client.0.smithi111.stdout:2/400: dwrite dd/d17/f63 [0,4194304] 0 2024-03-14T13:02:15.324 INFO:tasks.workunit.client.1.smithi137.stdout:2/362: link d2/f5 d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 0 2024-03-14T13:02:15.325 INFO:tasks.workunit.client.0.smithi111.stdout:0/404: read d6/d9/de/d32/f31 [826804,78580] 0 2024-03-14T13:02:15.326 INFO:tasks.workunit.client.1.smithi137.stdout:2/363: write d2/f12 [3774033,24832] 0 2024-03-14T13:02:15.328 INFO:tasks.workunit.client.1.smithi137.stdout:8/298: getdents d1/dc/d11/d13 0 2024-03-14T13:02:15.329 INFO:tasks.workunit.client.1.smithi137.stdout:8/299: chown d1/dc/d3c/d54/d34/l53 3984 1 2024-03-14T13:02:15.331 INFO:tasks.workunit.client.1.smithi137.stdout:2/364: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 x:0 0 0 2024-03-14T13:02:15.331 INFO:tasks.workunit.client.0.smithi111.stdout:2/401: rename dd/d17/d34/d74/f7d to dd/d11/d1a/d56/f85 0 2024-03-14T13:02:15.332 INFO:tasks.workunit.client.1.smithi137.stdout:8/300: creat d1/dc/d3c/d54/d34/f68 x:0 0 0 2024-03-14T13:02:15.332 INFO:tasks.workunit.client.1.smithi137.stdout:2/365: write d2/d34/d3c/f6d [984049,20100] 0 2024-03-14T13:02:15.332 INFO:tasks.workunit.client.1.smithi137.stdout:8/301: chown d1/dc/f39 492732 1 2024-03-14T13:02:15.333 INFO:tasks.workunit.client.1.smithi137.stdout:2/366: chown d2/d7/db/d2f/d66 58 1 2024-03-14T13:02:15.334 INFO:tasks.workunit.client.0.smithi111.stdout:2/402: mkdir dd/d11/d86 0 2024-03-14T13:02:15.335 INFO:tasks.workunit.client.1.smithi137.stdout:8/302: symlink d1/dc/d11/d13/l69 0 2024-03-14T13:02:15.335 INFO:tasks.workunit.client.0.smithi111.stdout:0/405: link d6/c6b d6/c84 0 2024-03-14T13:02:15.336 INFO:tasks.workunit.client.1.smithi137.stdout:2/367: creat d2/d7/db/d2f/d66/f7a x:0 0 0 2024-03-14T13:02:15.336 INFO:tasks.workunit.client.1.smithi137.stdout:8/303: truncate d1/dc/f4b 521747 0 2024-03-14T13:02:15.336 INFO:tasks.workunit.client.0.smithi111.stdout:4/672: dwrite d1d/f43 [0,4194304] 0 2024-03-14T13:02:15.337 INFO:tasks.workunit.client.1.smithi137.stdout:8/304: write d1/dc/d3c/d54/f48 [3196343,38448] 0 2024-03-14T13:02:15.338 INFO:tasks.workunit.client.0.smithi111.stdout:4/673: write d1d/d22/fd7 [1513235,56799] 0 2024-03-14T13:02:15.339 INFO:tasks.workunit.client.0.smithi111.stdout:2/403: creat dd/f87 x:0 0 0 2024-03-14T13:02:15.339 INFO:tasks.workunit.client.1.smithi137.stdout:2/368: creat d2/d7/d2e/d3e/d44/d6b/d1d/f7b x:0 0 0 2024-03-14T13:02:15.340 INFO:tasks.workunit.client.0.smithi111.stdout:4/674: dread - d1d/d22/d30/d5e/d6f/d7e/fc8 zero size 2024-03-14T13:02:15.341 INFO:tasks.workunit.client.0.smithi111.stdout:0/406: unlink d6/d9/de/d3a/d46/f7a 0 2024-03-14T13:02:15.342 INFO:tasks.workunit.client.1.smithi137.stdout:3/274: sync 2024-03-14T13:02:15.342 INFO:tasks.workunit.client.1.smithi137.stdout:0/311: sync 2024-03-14T13:02:15.342 INFO:tasks.workunit.client.0.smithi111.stdout:4/675: read d1d/d22/d30/d5e/fb3 [2740111,86191] 0 2024-03-14T13:02:15.343 INFO:tasks.workunit.client.0.smithi111.stdout:0/407: read d6/d9/d15/f37 [793247,82348] 0 2024-03-14T13:02:15.346 INFO:tasks.workunit.client.0.smithi111.stdout:4/676: chown d1d/d22/d30/d5e/d6f/d7e/d66/f85 2929929 1 2024-03-14T13:02:15.350 INFO:tasks.workunit.client.0.smithi111.stdout:4/677: mknod d1d/d22/d30/d5e/db8/ddf/de3/ce9 0 2024-03-14T13:02:15.350 INFO:tasks.workunit.client.1.smithi137.stdout:3/275: symlink d4/de/d13/l59 0 2024-03-14T13:02:15.350 INFO:tasks.workunit.client.0.smithi111.stdout:0/408: dwrite d6/d9/f58 [0,4194304] 0 2024-03-14T13:02:15.351 INFO:tasks.workunit.client.1.smithi137.stdout:3/276: fdatasync f1 0 2024-03-14T13:02:15.351 INFO:tasks.workunit.client.1.smithi137.stdout:0/312: dread d1/d11/f21 [0,4194304] 0 2024-03-14T13:02:15.352 INFO:tasks.workunit.client.1.smithi137.stdout:3/277: dread - d4/d19/f54 zero size 2024-03-14T13:02:15.353 INFO:tasks.workunit.client.1.smithi137.stdout:0/313: fsync d1/d11/d4a/d54/f6f 0 2024-03-14T13:02:15.353 INFO:tasks.workunit.client.0.smithi111.stdout:4/678: mknod d1d/d22/d30/d87/cea 0 2024-03-14T13:02:15.354 INFO:tasks.workunit.client.1.smithi137.stdout:0/314: chown d1/d11/f21 589731588 1 2024-03-14T13:02:15.356 INFO:tasks.workunit.client.1.smithi137.stdout:3/278: creat d4/df/d34/f5a x:0 0 0 2024-03-14T13:02:15.356 INFO:tasks.workunit.client.1.smithi137.stdout:2/369: truncate d2/d7/d2e/f6a 574147 0 2024-03-14T13:02:15.358 INFO:tasks.workunit.client.1.smithi137.stdout:0/315: creat d1/d44/d56/f70 x:0 0 0 2024-03-14T13:02:15.359 INFO:tasks.workunit.client.1.smithi137.stdout:3/279: write d4/de/d16/f1a [1691370,53257] 0 2024-03-14T13:02:15.360 INFO:tasks.workunit.client.0.smithi111.stdout:4/679: dwrite d1d/d22/d23/fc7 [0,4194304] 0 2024-03-14T13:02:15.362 INFO:tasks.workunit.client.1.smithi137.stdout:2/370: symlink d2/d34/d3c/l7c 0 2024-03-14T13:02:15.362 INFO:tasks.workunit.client.1.smithi137.stdout:8/305: dwrite d1/dc/f5f [0,4194304] 0 2024-03-14T13:02:15.364 INFO:tasks.workunit.client.1.smithi137.stdout:0/316: chown d1/d5/d1b/d1e/d25/d26/f41 0 1 2024-03-14T13:02:15.367 INFO:tasks.workunit.client.1.smithi137.stdout:8/306: chown d1/c3b 14 1 2024-03-14T13:02:15.368 INFO:tasks.workunit.client.1.smithi137.stdout:2/371: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 129690 0 2024-03-14T13:02:15.371 INFO:tasks.workunit.client.1.smithi137.stdout:3/280: creat d4/de/d16/f5b x:0 0 0 2024-03-14T13:02:15.376 INFO:tasks.workunit.client.1.smithi137.stdout:3/281: mkdir d4/d19/d5c 0 2024-03-14T13:02:15.377 INFO:tasks.workunit.client.1.smithi137.stdout:8/307: unlink d1/dc/d11/d56/l58 0 2024-03-14T13:02:15.380 INFO:tasks.workunit.client.1.smithi137.stdout:8/308: mknod d1/d5e/c6a 0 2024-03-14T13:02:15.381 INFO:tasks.workunit.client.0.smithi111.stdout:7/360: sync 2024-03-14T13:02:15.382 INFO:tasks.workunit.client.1.smithi137.stdout:2/372: write d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 [128202,121618] 0 2024-03-14T13:02:15.384 INFO:tasks.workunit.client.1.smithi137.stdout:2/373: dread - d2/d7/d2e/d3e/d44/f60 zero size 2024-03-14T13:02:15.385 INFO:tasks.workunit.client.1.smithi137.stdout:4/346: sync 2024-03-14T13:02:15.386 INFO:tasks.workunit.client.1.smithi137.stdout:6/343: sync 2024-03-14T13:02:15.386 INFO:tasks.workunit.client.1.smithi137.stdout:5/292: sync 2024-03-14T13:02:15.386 INFO:tasks.workunit.client.1.smithi137.stdout:1/414: sync 2024-03-14T13:02:15.386 INFO:tasks.workunit.client.1.smithi137.stdout:3/282: write d4/de/f2a [2122592,36261] 0 2024-03-14T13:02:15.389 INFO:tasks.workunit.client.1.smithi137.stdout:8/309: write d1/dc/d11/d13/f3e [1073531,111696] 0 2024-03-14T13:02:15.390 INFO:tasks.workunit.client.1.smithi137.stdout:0/317: dread d1/d11/d4a/d54/f5b [0,4194304] 0 2024-03-14T13:02:15.392 INFO:tasks.workunit.client.1.smithi137.stdout:6/344: symlink d2/d54/d55/l82 0 2024-03-14T13:02:15.394 INFO:tasks.workunit.client.1.smithi137.stdout:4/347: rename d4/d9/f5d to d4/d27/d5c/d6d/d58/d65/f7b 0 2024-03-14T13:02:15.396 INFO:tasks.workunit.client.1.smithi137.stdout:2/374: getdents d2/d7/d2e/d3e/d44/d6b/d1d/d3a 0 2024-03-14T13:02:15.403 INFO:tasks.workunit.client.1.smithi137.stdout:5/293: dwrite d7/d16/d18/d25/f50 [0,4194304] 0 2024-03-14T13:02:15.404 INFO:tasks.workunit.client.1.smithi137.stdout:4/348: truncate d4/d9/d3a/f56 817027 0 2024-03-14T13:02:15.404 INFO:tasks.workunit.client.1.smithi137.stdout:2/375: read - d2/d7/db/d2f/f5f zero size 2024-03-14T13:02:15.409 INFO:tasks.workunit.client.1.smithi137.stdout:3/283: dread d4/de/d16/d1c/f41 [0,4194304] 0 2024-03-14T13:02:15.410 INFO:tasks.workunit.client.1.smithi137.stdout:5/294: dread - d7/d33/f5f zero size 2024-03-14T13:02:15.410 INFO:tasks.workunit.client.1.smithi137.stdout:1/415: dwrite d7/f1d [0,4194304] 0 2024-03-14T13:02:15.414 INFO:tasks.workunit.client.1.smithi137.stdout:0/318: dwrite d1/d5/d1b/d42/f4f [0,4194304] 0 2024-03-14T13:02:15.416 INFO:tasks.workunit.client.1.smithi137.stdout:3/284: unlink d4/d22/f45 0 2024-03-14T13:02:15.416 INFO:tasks.workunit.client.1.smithi137.stdout:6/345: dwrite d2/f7 [4194304,4194304] 0 2024-03-14T13:02:15.416 INFO:tasks.workunit.client.1.smithi137.stdout:5/295: write d7/d16/d18/d2d/f2f [416503,96528] 0 2024-03-14T13:02:15.417 INFO:tasks.workunit.client.1.smithi137.stdout:2/376: symlink d2/d7/d2e/d3e/d44/d6b/l7d 0 2024-03-14T13:02:15.418 INFO:tasks.workunit.client.1.smithi137.stdout:1/416: unlink d7/d9/d19/c3b 0 2024-03-14T13:02:15.420 INFO:tasks.workunit.client.1.smithi137.stdout:1/417: fdatasync d7/d9/d19/d1f/d26/f48 0 2024-03-14T13:02:15.426 INFO:tasks.workunit.client.1.smithi137.stdout:4/349: dread d4/f48 [0,4194304] 0 2024-03-14T13:02:15.426 INFO:tasks.workunit.client.1.smithi137.stdout:1/418: mknod d7/d9/d19/d1f/c8a 0 2024-03-14T13:02:15.427 INFO:tasks.workunit.client.1.smithi137.stdout:0/319: unlink d1/d11/l38 0 2024-03-14T13:02:15.428 INFO:tasks.workunit.client.0.smithi111.stdout:4/680: mknod d1d/d22/d30/d5e/d6f/d7e/d66/dba/ceb 0 2024-03-14T13:02:15.430 INFO:tasks.workunit.client.0.smithi111.stdout:2/404: dread dd/d17/d34/f6b [0,4194304] 0 2024-03-14T13:02:15.431 INFO:tasks.workunit.client.0.smithi111.stdout:7/361: symlink d0/d1/d5/d38/d9/l6e 0 2024-03-14T13:02:15.432 INFO:tasks.workunit.client.1.smithi137.stdout:5/296: write d7/d1d/f57 [343636,10052] 0 2024-03-14T13:02:15.432 INFO:tasks.workunit.client.1.smithi137.stdout:0/320: read - d1/d5/d1b/d42/f58 zero size 2024-03-14T13:02:15.432 INFO:tasks.workunit.client.0.smithi111.stdout:7/362: write d0/d1/d5/d38/d9/f19 [8489819,10410] 0 2024-03-14T13:02:15.432 INFO:tasks.workunit.client.1.smithi137.stdout:4/350: creat d4/d27/d5c/d77/f7c x:0 0 0 2024-03-14T13:02:15.433 INFO:tasks.workunit.client.0.smithi111.stdout:4/681: creat d1d/d22/d30/fec x:0 0 0 2024-03-14T13:02:15.435 INFO:tasks.workunit.client.1.smithi137.stdout:3/285: dwrite d4/de/d13/f36 [8388608,4194304] 0 2024-03-14T13:02:15.436 INFO:tasks.workunit.client.1.smithi137.stdout:5/297: chown d7/c44 4489 1 2024-03-14T13:02:15.437 INFO:tasks.workunit.client.1.smithi137.stdout:2/377: dread d2/f1e [0,4194304] 0 2024-03-14T13:02:15.440 INFO:tasks.workunit.client.1.smithi137.stdout:5/298: mknod d7/c67 0 2024-03-14T13:02:15.442 INFO:tasks.workunit.client.1.smithi137.stdout:2/378: dread - d2/d7/d2e/d3e/d44/d6b/d22/f4c zero size 2024-03-14T13:02:15.444 INFO:tasks.workunit.client.1.smithi137.stdout:1/419: dwrite d7/d9/d19/d1f/d2e/f34 [0,4194304] 0 2024-03-14T13:02:15.445 INFO:tasks.workunit.client.1.smithi137.stdout:3/286: rename d4/df/d35 to d4/de/d16/d1c/d5d 0 2024-03-14T13:02:15.448 INFO:tasks.workunit.client.1.smithi137.stdout:4/351: getdents d4/d10 0 2024-03-14T13:02:15.450 INFO:tasks.workunit.client.1.smithi137.stdout:5/299: unlink d7/d16/d18/d25/l2b 0 2024-03-14T13:02:15.453 INFO:tasks.workunit.client.1.smithi137.stdout:0/321: getdents d1/d5/d1b/d1e/d53 0 2024-03-14T13:02:15.454 INFO:tasks.workunit.client.1.smithi137.stdout:4/352: truncate d4/d9/d3a/d41/f49 5155681 0 2024-03-14T13:02:15.455 INFO:tasks.workunit.client.1.smithi137.stdout:2/379: dwrite d2/f4b [0,4194304] 0 2024-03-14T13:02:15.458 INFO:tasks.workunit.client.1.smithi137.stdout:5/300: creat d7/d16/f68 x:0 0 0 2024-03-14T13:02:15.462 INFO:tasks.workunit.client.1.smithi137.stdout:3/287: link d4/df/d34/f5a d4/de/d16/d1c/d5d/d4b/f5e 0 2024-03-14T13:02:15.464 INFO:tasks.workunit.client.1.smithi137.stdout:1/420: dread d7/d29/f2b [0,4194304] 0 2024-03-14T13:02:15.466 INFO:tasks.workunit.client.1.smithi137.stdout:6/346: dread d2/f25 [0,4194304] 0 2024-03-14T13:02:15.466 INFO:tasks.workunit.client.1.smithi137.stdout:4/353: symlink d4/d10/d3d/l7d 0 2024-03-14T13:02:15.472 INFO:tasks.workunit.client.1.smithi137.stdout:2/380: mknod d2/d7/db/d74/c7e 0 2024-03-14T13:02:15.473 INFO:tasks.workunit.client.1.smithi137.stdout:1/421: dwrite d7/d9/d19/f2f [0,4194304] 0 2024-03-14T13:02:15.473 INFO:tasks.workunit.client.1.smithi137.stdout:4/354: creat d4/d27/d5c/d6d/d58/f7e x:0 0 0 2024-03-14T13:02:15.475 INFO:tasks.workunit.client.1.smithi137.stdout:4/355: stat d4/d4d/l62 0 2024-03-14T13:02:15.475 INFO:tasks.workunit.client.1.smithi137.stdout:5/301: mknod d7/d16/d18/d25/c69 0 2024-03-14T13:02:15.476 INFO:tasks.workunit.client.1.smithi137.stdout:1/422: write d7/d9/d19/d40/f46 [5135437,11226] 0 2024-03-14T13:02:15.478 INFO:tasks.workunit.client.1.smithi137.stdout:2/381: readlink d2/l6 0 2024-03-14T13:02:15.478 INFO:tasks.workunit.client.1.smithi137.stdout:4/356: truncate d4/d9/f1f 369706 0 2024-03-14T13:02:15.479 INFO:tasks.workunit.client.1.smithi137.stdout:5/302: unlink d7/d33/f5f 0 2024-03-14T13:02:15.481 INFO:tasks.workunit.client.1.smithi137.stdout:1/423: write d7/d9/d19/d1f/d2e/f79 [978278,28092] 0 2024-03-14T13:02:15.485 INFO:tasks.workunit.client.1.smithi137.stdout:2/382: fsync d2/d7/d2e/d3e/d44/d6b/f20 0 2024-03-14T13:02:15.487 INFO:tasks.workunit.client.1.smithi137.stdout:0/322: write d1/f46 [2748979,124938] 0 2024-03-14T13:02:15.488 INFO:tasks.workunit.client.1.smithi137.stdout:4/357: unlink d4/d27/d5c/d6d/d58/l69 0 2024-03-14T13:02:15.491 INFO:tasks.workunit.client.1.smithi137.stdout:4/358: read d4/f32 [854105,42350] 0 2024-03-14T13:02:15.492 INFO:tasks.workunit.client.1.smithi137.stdout:2/383: write d2/d34/d3c/d53/f5e [1308798,54471] 0 2024-03-14T13:02:15.493 INFO:tasks.workunit.client.1.smithi137.stdout:0/323: chown d1/d5/d1b/d1e/f30 2 1 2024-03-14T13:02:15.494 INFO:tasks.workunit.client.1.smithi137.stdout:3/288: dread d4/df/d34/f42 [0,4194304] 0 2024-03-14T13:02:15.494 INFO:tasks.workunit.client.1.smithi137.stdout:5/303: dwrite d7/d16/d18/d2d/f53 [0,4194304] 0 2024-03-14T13:02:15.498 INFO:tasks.workunit.client.1.smithi137.stdout:4/359: truncate d4/d9/d3a/d41/f21 4362377 0 2024-03-14T13:02:15.499 INFO:tasks.workunit.client.1.smithi137.stdout:6/347: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:15.505 INFO:tasks.workunit.client.0.smithi111.stdout:2/405: mkdir dd/d11/d1a/d88 0 2024-03-14T13:02:15.505 INFO:tasks.workunit.client.1.smithi137.stdout:2/384: rename d2/d7/db/d2f/f40 to d2/d34/f7f 0 2024-03-14T13:02:15.506 INFO:tasks.workunit.client.1.smithi137.stdout:1/424: link d7/d9/d19/d1f/d22/l75 d7/d9/d19/l8b 0 2024-03-14T13:02:15.508 INFO:tasks.workunit.client.0.smithi111.stdout:2/406: rmdir dd/d17/d34 39 2024-03-14T13:02:15.508 INFO:tasks.workunit.client.1.smithi137.stdout:3/289: symlink d4/de/d16/d1c/d5d/d4b/l5f 0 2024-03-14T13:02:15.511 INFO:tasks.workunit.client.1.smithi137.stdout:6/348: mkdir d2/d6/de/d45/d59/d39/d3c/d83 0 2024-03-14T13:02:15.513 INFO:tasks.workunit.client.1.smithi137.stdout:5/304: dwrite d7/d16/d18/f39 [0,4194304] 0 2024-03-14T13:02:15.513 INFO:tasks.workunit.client.0.smithi111.stdout:7/363: dwrite d0/d1/d22/f39 [0,4194304] 0 2024-03-14T13:02:15.515 INFO:tasks.workunit.client.1.smithi137.stdout:3/290: stat d4/de/d16/d1c/d5d/d4b/f55 0 2024-03-14T13:02:15.515 INFO:tasks.workunit.client.0.smithi111.stdout:4/682: dwrite d1d/d22/d30/de5/fb0 [0,4194304] 0 2024-03-14T13:02:15.517 INFO:tasks.workunit.client.0.smithi111.stdout:7/364: readlink d0/d1/l48 0 2024-03-14T13:02:15.517 INFO:tasks.workunit.client.1.smithi137.stdout:6/349: write d2/d6/de/f5d [391968,28489] 0 2024-03-14T13:02:15.517 INFO:tasks.workunit.client.1.smithi137.stdout:5/305: rename d7 to d7/d16/d18/d2d/d6a 22 2024-03-14T13:02:15.518 INFO:tasks.workunit.client.1.smithi137.stdout:4/360: dread d4/d9/d3a/d41/d38/d55/f64 [0,4194304] 0 2024-03-14T13:02:15.518 INFO:tasks.workunit.client.1.smithi137.stdout:2/385: write d2/d7/d2e/d3e/d44/d6b/d1d/f48 [608800,56793] 0 2024-03-14T13:02:15.524 INFO:tasks.workunit.client.1.smithi137.stdout:5/306: chown d7/f15 123743 1 2024-03-14T13:02:15.524 INFO:tasks.workunit.client.1.smithi137.stdout:3/291: mkdir d4/df/d60 0 2024-03-14T13:02:15.525 INFO:tasks.workunit.client.0.smithi111.stdout:0/409: dread d6/d9/de/f3b [0,4194304] 0 2024-03-14T13:02:15.526 INFO:tasks.workunit.client.0.smithi111.stdout:4/683: symlink d1d/d22/d30/d7c/d59/dc4/led 0 2024-03-14T13:02:15.526 INFO:tasks.workunit.client.0.smithi111.stdout:0/410: readlink d6/d9/d15/d55/l76 0 2024-03-14T13:02:15.526 INFO:tasks.workunit.client.1.smithi137.stdout:0/324: dwrite d1/d11/f50 [4194304,4194304] 0 2024-03-14T13:02:15.528 INFO:tasks.workunit.client.0.smithi111.stdout:4/684: mknod d1d/d22/d40/d69/cee 0 2024-03-14T13:02:15.531 INFO:tasks.workunit.client.1.smithi137.stdout:2/386: chown d2/d7/l15 7 1 2024-03-14T13:02:15.532 INFO:tasks.workunit.client.0.smithi111.stdout:0/411: rename d6/d9/de/f2c to d6/d9/de/f85 0 2024-03-14T13:02:15.533 INFO:tasks.workunit.client.1.smithi137.stdout:6/350: write d2/f25 [4227829,40450] 0 2024-03-14T13:02:15.534 INFO:tasks.workunit.client.0.smithi111.stdout:4/685: creat d1d/d22/d40/d69/fef x:0 0 0 2024-03-14T13:02:15.535 INFO:tasks.workunit.client.0.smithi111.stdout:0/412: creat d6/d9/d15/d55/f86 x:0 0 0 2024-03-14T13:02:15.535 INFO:tasks.workunit.client.1.smithi137.stdout:6/351: chown d2/d54/d55 2028 1 2024-03-14T13:02:15.536 INFO:tasks.workunit.client.1.smithi137.stdout:4/361: dwrite d4/f31 [0,4194304] 0 2024-03-14T13:02:15.536 INFO:tasks.workunit.client.1.smithi137.stdout:2/387: creat d2/d7/db/d2f/d66/f80 x:0 0 0 2024-03-14T13:02:15.539 INFO:tasks.workunit.client.1.smithi137.stdout:3/292: link d4/de/f28 d4/de/d13/f61 0 2024-03-14T13:02:15.539 INFO:tasks.workunit.client.1.smithi137.stdout:6/352: write d2/d6/de/f32 [4928137,123116] 0 2024-03-14T13:02:15.539 INFO:tasks.workunit.client.1.smithi137.stdout:7/307: sync 2024-03-14T13:02:15.539 INFO:tasks.workunit.client.1.smithi137.stdout:9/472: sync 2024-03-14T13:02:15.540 INFO:tasks.workunit.client.0.smithi111.stdout:4/686: link fe d1d/d22/d30/d87/dd1/ff0 0 2024-03-14T13:02:15.541 INFO:tasks.workunit.client.1.smithi137.stdout:0/325: getdents d1/d5/d1b/d1e/d25/d26/d6c 0 2024-03-14T13:02:15.542 INFO:tasks.workunit.client.1.smithi137.stdout:7/308: stat d3/d5/d23/d3f/c50 0 2024-03-14T13:02:15.542 INFO:tasks.workunit.client.1.smithi137.stdout:6/353: chown d2/d6/c12 63796 1 2024-03-14T13:02:15.542 INFO:tasks.workunit.client.1.smithi137.stdout:2/388: fdatasync d2/d7/db/d2f/d66/f80 0 2024-03-14T13:02:15.543 INFO:tasks.workunit.client.1.smithi137.stdout:9/473: truncate d6/d53/f99 114790 0 2024-03-14T13:02:15.543 INFO:tasks.workunit.client.1.smithi137.stdout:3/293: rmdir d4/de/d16/d1c 39 2024-03-14T13:02:15.544 INFO:tasks.workunit.client.1.smithi137.stdout:0/326: write d1/d1f/f3b [602585,72190] 0 2024-03-14T13:02:15.544 INFO:tasks.workunit.client.0.smithi111.stdout:0/413: dwrite d6/d9/f58 [0,4194304] 0 2024-03-14T13:02:15.544 INFO:tasks.workunit.client.0.smithi111.stdout:4/687: mknod d1d/d22/d40/d69/cf1 0 2024-03-14T13:02:15.546 INFO:tasks.workunit.client.0.smithi111.stdout:0/414: chown d6/d9/d15/d55 31 1 2024-03-14T13:02:15.551 INFO:tasks.workunit.client.0.smithi111.stdout:2/407: truncate dd/d11/f6d 1884050 0 2024-03-14T13:02:15.551 INFO:tasks.workunit.client.0.smithi111.stdout:7/365: write d0/d1/d5/f1a [50385,52857] 0 2024-03-14T13:02:15.553 INFO:tasks.workunit.client.0.smithi111.stdout:4/688: link d1d/d22/d56/f7f d1d/d22/d40/d69/d86/db1/ff2 0 2024-03-14T13:02:15.554 INFO:tasks.workunit.client.0.smithi111.stdout:4/689: symlink d1d/lf3 0 2024-03-14T13:02:15.556 INFO:tasks.workunit.client.0.smithi111.stdout:2/408: rename dd/d11/f33 to dd/d11/d1a/d88/f89 0 2024-03-14T13:02:15.557 INFO:tasks.workunit.client.0.smithi111.stdout:2/409: dread - dd/d17/d53/f68 zero size 2024-03-14T13:02:15.558 INFO:tasks.workunit.client.0.smithi111.stdout:2/410: stat dd/d11/d1a 0 2024-03-14T13:02:15.561 INFO:tasks.workunit.client.0.smithi111.stdout:4/690: dwrite d1d/d22/d23/fc7 [0,4194304] 0 2024-03-14T13:02:15.562 INFO:tasks.workunit.client.0.smithi111.stdout:4/691: stat d1d/d22/d30/de5/f4c 0 2024-03-14T13:02:15.573 INFO:tasks.workunit.client.0.smithi111.stdout:9/415: sync 2024-03-14T13:02:15.575 INFO:tasks.workunit.client.0.smithi111.stdout:9/416: symlink d5/d42/d52/l89 0 2024-03-14T13:02:15.577 INFO:tasks.workunit.client.0.smithi111.stdout:9/417: mknod d5/d6/d8/c8a 0 2024-03-14T13:02:15.577 INFO:tasks.workunit.client.0.smithi111.stdout:9/418: chown d5/d6/d8/d1a 47646 1 2024-03-14T13:02:15.583 INFO:tasks.workunit.client.0.smithi111.stdout:9/419: dwrite d5/d6/f7e [0,4194304] 0 2024-03-14T13:02:15.585 INFO:tasks.workunit.client.0.smithi111.stdout:9/420: mkdir d5/d6/d2a/d2d/d8b 0 2024-03-14T13:02:15.589 INFO:tasks.workunit.client.0.smithi111.stdout:9/421: getdents d5/d6/d8/d5b/d7c 0 2024-03-14T13:02:15.592 INFO:tasks.workunit.client.0.smithi111.stdout:9/422: creat d5/d6/d8/d5b/d87/f8c x:0 0 0 2024-03-14T13:02:15.593 INFO:tasks.workunit.client.1.smithi137.stdout:9/474: dread d6/df/d19/d33/f3e [0,4194304] 0 2024-03-14T13:02:15.594 INFO:tasks.workunit.client.1.smithi137.stdout:4/362: dread d4/f31 [0,4194304] 0 2024-03-14T13:02:15.594 INFO:tasks.workunit.client.0.smithi111.stdout:9/423: mkdir d5/d6/d8/d5b/d87/d8d 0 2024-03-14T13:02:15.596 INFO:tasks.workunit.client.0.smithi111.stdout:9/424: unlink d5/f50 0 2024-03-14T13:02:15.596 INFO:tasks.workunit.client.0.smithi111.stdout:9/425: fsync d5/d6/d14/f81 0 2024-03-14T13:02:15.601 INFO:tasks.workunit.client.1.smithi137.stdout:1/425: truncate d7/d9/d19/d1f/d22/f25 2393918 0 2024-03-14T13:02:15.603 INFO:tasks.workunit.client.1.smithi137.stdout:0/327: symlink d1/d5/d1b/d1e/d53/d6d/l71 0 2024-03-14T13:02:15.603 INFO:tasks.workunit.client.1.smithi137.stdout:3/294: symlink d4/de/d16/d1c/d5d/l62 0 2024-03-14T13:02:15.604 INFO:tasks.workunit.client.0.smithi111.stdout:9/426: read d5/d6/d14/f54 [980079,18448] 0 2024-03-14T13:02:15.606 INFO:tasks.workunit.client.0.smithi111.stdout:9/427: chown d5/d42/c7b 1 1 2024-03-14T13:02:15.606 INFO:tasks.workunit.client.1.smithi137.stdout:1/426: stat d7/d9/d19/d1f/d26/d6c 0 2024-03-14T13:02:15.606 INFO:tasks.workunit.client.1.smithi137.stdout:2/389: getdents d2/d7/db 0 2024-03-14T13:02:15.607 INFO:tasks.workunit.client.1.smithi137.stdout:9/475: symlink d6/df/d19/d3a/l9d 0 2024-03-14T13:02:15.607 INFO:tasks.workunit.client.1.smithi137.stdout:0/328: read - d1/d44/d56/f70 zero size 2024-03-14T13:02:15.608 INFO:tasks.workunit.client.0.smithi111.stdout:9/428: creat d5/d6/d2a/d2d/d8b/f8e x:0 0 0 2024-03-14T13:02:15.609 INFO:tasks.workunit.client.0.smithi111.stdout:9/429: fdatasync d5/d6/d2a/d2d/f4a 0 2024-03-14T13:02:15.609 INFO:tasks.workunit.client.0.smithi111.stdout:9/430: chown d5/d6/d2a/d2d/f66 0 1 2024-03-14T13:02:15.612 INFO:tasks.workunit.client.1.smithi137.stdout:4/363: dwrite f3 [0,4194304] 0 2024-03-14T13:02:15.612 INFO:tasks.workunit.client.1.smithi137.stdout:9/476: fsync d6/df/d19/d33/f65 0 2024-03-14T13:02:15.617 INFO:tasks.workunit.client.1.smithi137.stdout:2/390: fsync d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 0 2024-03-14T13:02:15.617 INFO:tasks.workunit.client.0.smithi111.stdout:0/415: write d6/d9/de/f3b [4750584,99379] 0 2024-03-14T13:02:15.619 INFO:tasks.workunit.client.1.smithi137.stdout:6/354: dwrite d2/d54/d55/f78 [0,4194304] 0 2024-03-14T13:02:15.621 INFO:tasks.workunit.client.0.smithi111.stdout:2/411: truncate dd/f10 3305402 0 2024-03-14T13:02:15.621 INFO:tasks.workunit.client.0.smithi111.stdout:4/692: write d1d/d22/f44 [496463,123529] 0 2024-03-14T13:02:15.623 INFO:tasks.workunit.client.1.smithi137.stdout:2/391: chown d2/d7/d2e/d3e/d44/d6b/d1d 3636882 1 2024-03-14T13:02:15.625 INFO:tasks.workunit.client.0.smithi111.stdout:0/416: dread d6/d9/d15/d55/f82 [0,4194304] 0 2024-03-14T13:02:15.626 INFO:tasks.workunit.client.1.smithi137.stdout:4/364: rename d4/d9/d3a/d41/c2f to d4/d9/d3a/c7f 0 2024-03-14T13:02:15.626 INFO:tasks.workunit.client.1.smithi137.stdout:2/392: fdatasync d2/d7/db/ff 0 2024-03-14T13:02:15.633 INFO:tasks.workunit.client.1.smithi137.stdout:9/477: unlink d6/df/d19/d1f/c59 0 2024-03-14T13:02:15.634 INFO:tasks.workunit.client.1.smithi137.stdout:1/427: dread f3 [0,4194304] 0 2024-03-14T13:02:15.635 INFO:tasks.workunit.client.1.smithi137.stdout:5/307: dread d7/d16/d18/f20 [0,4194304] 0 2024-03-14T13:02:15.637 INFO:tasks.workunit.client.0.smithi111.stdout:2/412: dwrite dd/d17/f37 [0,4194304] 0 2024-03-14T13:02:15.638 INFO:tasks.workunit.client.1.smithi137.stdout:2/393: getdents d2/d34 0 2024-03-14T13:02:15.640 INFO:tasks.workunit.client.0.smithi111.stdout:4/693: mkdir d1d/d22/d30/d5e/d6f/d7e/d66/df4 0 2024-03-14T13:02:15.641 INFO:tasks.workunit.client.0.smithi111.stdout:9/431: dwrite d5/d6/d8/f12 [0,4194304] 0 2024-03-14T13:02:15.641 INFO:tasks.workunit.client.0.smithi111.stdout:0/417: unlink d6/d9/de/l1c 0 2024-03-14T13:02:15.642 INFO:tasks.workunit.client.0.smithi111.stdout:9/432: chown d5/d6/l4d 211 1 2024-03-14T13:02:15.645 INFO:tasks.workunit.client.0.smithi111.stdout:2/413: creat dd/d11/d1a/d88/f8a x:0 0 0 2024-03-14T13:02:15.645 INFO:tasks.workunit.client.0.smithi111.stdout:0/418: stat d6/d21 0 2024-03-14T13:02:15.648 INFO:tasks.workunit.client.0.smithi111.stdout:4/694: mkdir d1d/d22/d30/d5e/d6f/d7e/d66/df4/df5 0 2024-03-14T13:02:15.651 INFO:tasks.workunit.client.0.smithi111.stdout:2/414: mkdir dd/d11/d8b 0 2024-03-14T13:02:15.652 INFO:tasks.workunit.client.1.smithi137.stdout:1/428: getdents d7/d9/d19/d5f 0 2024-03-14T13:02:15.652 INFO:tasks.workunit.client.1.smithi137.stdout:9/478: dread d6/df/d19/d3a/f43 [0,4194304] 0 2024-03-14T13:02:15.653 INFO:tasks.workunit.client.1.smithi137.stdout:5/308: dread d7/fb [0,4194304] 0 2024-03-14T13:02:15.655 INFO:tasks.workunit.client.0.smithi111.stdout:9/433: link d5/l19 d5/d6/d8/d5b/d84/l8f 0 2024-03-14T13:02:15.656 INFO:tasks.workunit.client.1.smithi137.stdout:2/394: dwrite d2/d7/db/ff [0,4194304] 0 2024-03-14T13:02:15.658 INFO:tasks.workunit.client.1.smithi137.stdout:0/329: truncate d1/d1f/f3b 57155 0 2024-03-14T13:02:15.658 INFO:tasks.workunit.client.1.smithi137.stdout:4/365: truncate d4/d9/d3a/d41/f49 3800033 0 2024-03-14T13:02:15.659 INFO:tasks.workunit.client.0.smithi111.stdout:9/434: unlink d5/fe 0 2024-03-14T13:02:15.659 INFO:tasks.workunit.client.0.smithi111.stdout:2/415: dread dd/d11/d1a/f52 [0,4194304] 0 2024-03-14T13:02:15.659 INFO:tasks.workunit.client.1.smithi137.stdout:1/429: rename d7/d4d/f7e to d7/d9/d19/d1f/d26/d84/f8c 0 2024-03-14T13:02:15.660 INFO:tasks.workunit.client.0.smithi111.stdout:2/416: chown dd/d17/d53/c57 186193650 1 2024-03-14T13:02:15.662 INFO:tasks.workunit.client.0.smithi111.stdout:9/435: creat d5/d6/d8/d5b/d7c/f90 x:0 0 0 2024-03-14T13:02:15.664 INFO:tasks.workunit.client.1.smithi137.stdout:6/355: dread d2/f17 [4194304,4194304] 0 2024-03-14T13:02:15.664 INFO:tasks.workunit.client.1.smithi137.stdout:1/430: symlink d7/d9/d19/d1f/d22/l8d 0 2024-03-14T13:02:15.667 INFO:tasks.workunit.client.1.smithi137.stdout:5/309: creat d7/d16/d18/d25/f6b x:0 0 0 2024-03-14T13:02:15.667 INFO:tasks.workunit.client.1.smithi137.stdout:4/366: mkdir d4/d9/d3a/d70/d80 0 2024-03-14T13:02:15.668 INFO:tasks.workunit.client.0.smithi111.stdout:2/417: rmdir dd/d11/d86 0 2024-03-14T13:02:15.670 INFO:tasks.workunit.client.1.smithi137.stdout:0/330: dwrite d1/f13 [4194304,4194304] 0 2024-03-14T13:02:15.671 INFO:tasks.workunit.client.1.smithi137.stdout:6/356: write d2/d54/d55/d6d/d76/d57/f74 [3209368,49025] 0 2024-03-14T13:02:15.671 INFO:tasks.workunit.client.0.smithi111.stdout:2/418: chown dd/d11/l6c 1 1 2024-03-14T13:02:15.673 INFO:tasks.workunit.client.0.smithi111.stdout:9/436: rmdir d5/d6/d8/d5b/d87/d8d 0 2024-03-14T13:02:15.676 INFO:tasks.workunit.client.0.smithi111.stdout:9/437: rmdir d5/d6/d8/d5b/d84 39 2024-03-14T13:02:15.677 INFO:tasks.workunit.client.0.smithi111.stdout:4/695: dread d1d/d22/f54 [0,4194304] 0 2024-03-14T13:02:15.678 INFO:tasks.workunit.client.0.smithi111.stdout:0/419: dwrite d6/d9/d15/d49/f62 [0,4194304] 0 2024-03-14T13:02:15.679 INFO:tasks.workunit.client.1.smithi137.stdout:0/331: mkdir d1/d44/d72 0 2024-03-14T13:02:15.680 INFO:tasks.workunit.client.0.smithi111.stdout:9/438: creat d5/d6/d14/f91 x:0 0 0 2024-03-14T13:02:15.681 INFO:tasks.workunit.client.1.smithi137.stdout:5/310: link d7/d33/c56 d7/d36/c6c 0 2024-03-14T13:02:15.681 INFO:tasks.workunit.client.1.smithi137.stdout:6/357: dwrite d2/d6/de/f5d [0,4194304] 0 2024-03-14T13:02:15.682 INFO:tasks.workunit.client.0.smithi111.stdout:0/420: write d6/d9/de/f30 [169605,129122] 0 2024-03-14T13:02:15.682 INFO:tasks.workunit.client.1.smithi137.stdout:4/367: getdents d4/d9 0 2024-03-14T13:02:15.685 INFO:tasks.workunit.client.0.smithi111.stdout:4/696: creat d1d/d22/d30/d7c/ff6 x:0 0 0 2024-03-14T13:02:15.685 INFO:tasks.workunit.client.1.smithi137.stdout:0/332: mkdir d1/d5/d1b/d1e/d25/d55/d73 0 2024-03-14T13:02:15.688 INFO:tasks.workunit.client.1.smithi137.stdout:4/368: truncate d4/d10/d42/f52 660809 0 2024-03-14T13:02:15.688 INFO:tasks.workunit.client.0.smithi111.stdout:2/419: dwrite dd/d17/d34/f2e [8388608,4194304] 0 2024-03-14T13:02:15.690 INFO:tasks.workunit.client.1.smithi137.stdout:0/333: fsync d1/f13 0 2024-03-14T13:02:15.692 INFO:tasks.workunit.client.0.smithi111.stdout:2/420: creat dd/d17/d53/f8c x:0 0 0 2024-03-14T13:02:15.692 INFO:tasks.workunit.client.1.smithi137.stdout:4/369: readlink d4/l19 0 2024-03-14T13:02:15.693 INFO:tasks.workunit.client.0.smithi111.stdout:2/421: write dd/f87 [254044,83161] 0 2024-03-14T13:02:15.694 INFO:tasks.workunit.client.1.smithi137.stdout:6/358: getdents d2/d6/de/d45/d59/d39/d3c/d83 0 2024-03-14T13:02:15.695 INFO:tasks.workunit.client.1.smithi137.stdout:4/370: mknod d4/d10/c81 0 2024-03-14T13:02:15.697 INFO:tasks.workunit.client.1.smithi137.stdout:6/359: creat d2/d4c/f84 x:0 0 0 2024-03-14T13:02:15.700 INFO:tasks.workunit.client.1.smithi137.stdout:0/334: link d1/d5/d1b/d1e/d25/d55/f60 d1/d5/d1b/d1e/d53/d6d/f74 0 2024-03-14T13:02:15.702 INFO:tasks.workunit.client.1.smithi137.stdout:2/395: dwrite d2/d7/d2e/d3e/d44/d6b/d22/f5a [0,4194304] 0 2024-03-14T13:02:15.702 INFO:tasks.workunit.client.1.smithi137.stdout:6/360: rename d2/d54/d55/d33/f44 to d2/d54/d55/d38/f85 0 2024-03-14T13:02:15.703 INFO:tasks.workunit.client.1.smithi137.stdout:6/361: fdatasync d2/d6/fc 0 2024-03-14T13:02:15.706 INFO:tasks.workunit.client.1.smithi137.stdout:8/310: sync 2024-03-14T13:02:15.707 INFO:tasks.workunit.client.0.smithi111.stdout:9/439: write d5/f20 [349627,22355] 0 2024-03-14T13:02:15.707 INFO:tasks.workunit.client.0.smithi111.stdout:0/421: write d6/d9/de/f6a [412862,13252] 0 2024-03-14T13:02:15.708 INFO:tasks.workunit.client.1.smithi137.stdout:8/311: readlink d1/dc/d3c/d54/l3f 0 2024-03-14T13:02:15.709 INFO:tasks.workunit.client.1.smithi137.stdout:8/312: chown d1/dc/l25 14291 1 2024-03-14T13:02:15.709 INFO:tasks.workunit.client.1.smithi137.stdout:2/396: dwrite d2/f4b [4194304,4194304] 0 2024-03-14T13:02:15.711 INFO:tasks.workunit.client.1.smithi137.stdout:8/313: write d1/d5e/f61 [1473989,23545] 0 2024-03-14T13:02:15.712 INFO:tasks.workunit.client.1.smithi137.stdout:8/314: fsync d1/dc/f52 0 2024-03-14T13:02:15.713 INFO:tasks.workunit.client.0.smithi111.stdout:0/422: mkdir d6/d45/d87 0 2024-03-14T13:02:15.715 INFO:tasks.workunit.client.1.smithi137.stdout:8/315: stat d1/dc/d10/l36 0 2024-03-14T13:02:15.718 INFO:tasks.workunit.client.0.smithi111.stdout:0/423: dread d6/d9/de/f85 [0,4194304] 0 2024-03-14T13:02:15.718 INFO:tasks.workunit.client.0.smithi111.stdout:4/697: dread d1d/d22/d23/fa2 [0,4194304] 0 2024-03-14T13:02:15.723 INFO:tasks.workunit.client.1.smithi137.stdout:2/397: rename d2/f5 to d2/d7/d2e/d3e/d44/f81 0 2024-03-14T13:02:15.723 INFO:tasks.workunit.client.0.smithi111.stdout:4/698: read d1d/d22/d56/f7b [24624,71029] 0 2024-03-14T13:02:15.724 INFO:tasks.workunit.client.0.smithi111.stdout:0/424: dread d6/d9/de/f3c [0,4194304] 0 2024-03-14T13:02:15.724 INFO:tasks.workunit.client.1.smithi137.stdout:1/431: truncate d7/f10 4597536 0 2024-03-14T13:02:15.726 INFO:tasks.workunit.client.1.smithi137.stdout:1/432: readlink d7/d9/l13 0 2024-03-14T13:02:15.727 INFO:tasks.workunit.client.1.smithi137.stdout:2/398: symlink d2/d7/db/d74/l82 0 2024-03-14T13:02:15.728 INFO:tasks.workunit.client.0.smithi111.stdout:2/422: dread dd/d17/d34/d73/d49/f82 [0,4194304] 0 2024-03-14T13:02:15.729 INFO:tasks.workunit.client.1.smithi137.stdout:8/316: dwrite d1/dc/d3c/d54/d34/f47 [0,4194304] 0 2024-03-14T13:02:15.729 INFO:tasks.workunit.client.0.smithi111.stdout:6/403: sync 2024-03-14T13:02:15.729 INFO:tasks.workunit.client.0.smithi111.stdout:5/499: sync 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:7/366: sync 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:8/578: sync 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:1/419: sync 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:3/452: sync 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:0/425: rename d6/d9/de/l2d to d6/d9/l88 0 2024-03-14T13:02:15.730 INFO:tasks.workunit.client.0.smithi111.stdout:4/699: dwrite d1d/d22/f37 [0,4194304] 0 2024-03-14T13:02:15.731 INFO:tasks.workunit.client.0.smithi111.stdout:2/423: chown dd/d17/d34/d73/d49/f67 8379701 1 2024-03-14T13:02:15.732 INFO:tasks.workunit.client.1.smithi137.stdout:1/433: symlink d7/d9/d19/d1f/l8e 0 2024-03-14T13:02:15.733 INFO:tasks.workunit.client.0.smithi111.stdout:1/420: fdatasync f3 0 2024-03-14T13:02:15.735 INFO:tasks.workunit.client.1.smithi137.stdout:2/399: creat d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f83 x:0 0 0 2024-03-14T13:02:15.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/500: chown d2/d16/d49 111 1 2024-03-14T13:02:15.738 INFO:tasks.workunit.client.0.smithi111.stdout:8/579: symlink d3/d1b/d34/lb1 0 2024-03-14T13:02:15.739 INFO:tasks.workunit.client.0.smithi111.stdout:4/700: fdatasync d1d/d22/d30/f3d 0 2024-03-14T13:02:15.740 INFO:tasks.workunit.client.0.smithi111.stdout:7/367: rename d0/d1/d5/d38/d9/c32 to d0/d11/c6f 0 2024-03-14T13:02:15.743 INFO:tasks.workunit.client.0.smithi111.stdout:2/424: truncate dd/d17/d53/f84 274770 0 2024-03-14T13:02:15.744 INFO:tasks.workunit.client.1.smithi137.stdout:2/400: mkdir d2/d7/db/d74/d84 0 2024-03-14T13:02:15.744 INFO:tasks.workunit.client.1.smithi137.stdout:8/317: rename d1/dc/d11/d13/l33 to d1/l6b 0 2024-03-14T13:02:15.745 INFO:tasks.workunit.client.0.smithi111.stdout:5/501: mkdir d2/d45/d76/d9a/d9f 0 2024-03-14T13:02:15.747 INFO:tasks.workunit.client.1.smithi137.stdout:2/401: write d2/d7/db/f16 [2927364,91920] 0 2024-03-14T13:02:15.751 INFO:tasks.workunit.client.0.smithi111.stdout:1/421: truncate d20/d21/d3f/f66 1392123 0 2024-03-14T13:02:15.751 INFO:tasks.workunit.client.1.smithi137.stdout:1/434: dread d7/d9/d19/d1f/d2e/f79 [0,4194304] 0 2024-03-14T13:02:15.754 INFO:tasks.workunit.client.0.smithi111.stdout:7/368: mkdir d0/d1/d5/d56/d70 0 2024-03-14T13:02:15.754 INFO:tasks.workunit.client.1.smithi137.stdout:1/435: chown d7/d31/l35 108 1 2024-03-14T13:02:15.754 INFO:tasks.workunit.client.1.smithi137.stdout:6/362: dread d2/d6/de/f13 [0,4194304] 0 2024-03-14T13:02:15.755 INFO:tasks.workunit.client.0.smithi111.stdout:3/453: rename d0/d2d/d35/d50/d5e/d40 to d0/d2d/d35/d50/da2/db3 0 2024-03-14T13:02:15.755 INFO:tasks.workunit.client.0.smithi111.stdout:6/404: dread d0/d17/d21/d69/f52 [0,4194304] 0 2024-03-14T13:02:15.757 INFO:tasks.workunit.client.0.smithi111.stdout:9/440: dwrite d5/d6/f22 [0,4194304] 0 2024-03-14T13:02:15.757 INFO:tasks.workunit.client.0.smithi111.stdout:5/502: creat d2/d16/d49/d33/d68/fa0 x:0 0 0 2024-03-14T13:02:15.757 INFO:tasks.workunit.client.0.smithi111.stdout:2/425: chown dd/d17/d34/d73/f3c 1 1 2024-03-14T13:02:15.758 INFO:tasks.workunit.client.0.smithi111.stdout:0/426: dwrite d6/d9/de/f6a [0,4194304] 0 2024-03-14T13:02:15.759 INFO:tasks.workunit.client.1.smithi137.stdout:1/436: truncate d7/d4d/f85 340375 0 2024-03-14T13:02:15.759 INFO:tasks.workunit.client.1.smithi137.stdout:2/402: unlink d2/d7/d2e/d3e/d44/f81 0 2024-03-14T13:02:15.760 INFO:tasks.workunit.client.0.smithi111.stdout:7/369: dread - d0/d1/d5/d38/d9/f49 zero size 2024-03-14T13:02:15.765 INFO:tasks.workunit.client.0.smithi111.stdout:1/422: rename d20/d21/c6b to d20/d25/d75/c94 0 2024-03-14T13:02:15.767 INFO:tasks.workunit.client.0.smithi111.stdout:7/370: fsync d0/d1/d4a/d52/f62 0 2024-03-14T13:02:15.767 INFO:tasks.workunit.client.0.smithi111.stdout:3/454: write d0/d2d/d35/d50/da2/db3/d6f/f79 [1132782,61650] 0 2024-03-14T13:02:15.767 INFO:tasks.workunit.client.1.smithi137.stdout:8/318: dwrite d1/dc/f39 [0,4194304] 0 2024-03-14T13:02:15.768 INFO:tasks.workunit.client.0.smithi111.stdout:3/455: chown d0/d2d 14487733 1 2024-03-14T13:02:15.770 INFO:tasks.workunit.client.1.smithi137.stdout:1/437: unlink d7/d9/c17 0 2024-03-14T13:02:15.771 INFO:tasks.workunit.client.0.smithi111.stdout:9/441: rmdir d5/d6/d8/d1a 39 2024-03-14T13:02:15.771 INFO:tasks.workunit.client.0.smithi111.stdout:5/503: unlink d2/d45/f82 0 2024-03-14T13:02:15.772 INFO:tasks.workunit.client.0.smithi111.stdout:0/427: mkdir d6/d9/d15/d55/d89 0 2024-03-14T13:02:15.773 INFO:tasks.workunit.client.0.smithi111.stdout:3/456: mknod d0/d2d/d35/d50/da2/db3/d71/cb4 0 2024-03-14T13:02:15.774 INFO:tasks.workunit.client.0.smithi111.stdout:2/426: write dd/d11/d1a/d88/f89 [1735913,58721] 0 2024-03-14T13:02:15.775 INFO:tasks.workunit.client.0.smithi111.stdout:6/405: creat d0/d17/d20/f9a x:0 0 0 2024-03-14T13:02:15.778 INFO:tasks.workunit.client.0.smithi111.stdout:1/423: symlink d20/l95 0 2024-03-14T13:02:15.778 INFO:tasks.workunit.client.0.smithi111.stdout:2/427: fsync dd/d17/d34/f42 0 2024-03-14T13:02:15.780 INFO:tasks.workunit.client.1.smithi137.stdout:8/319: symlink d1/dc/l6c 0 2024-03-14T13:02:15.780 INFO:tasks.workunit.client.1.smithi137.stdout:8/320: fdatasync d1/dc/d10/f27 0 2024-03-14T13:02:15.782 INFO:tasks.workunit.client.0.smithi111.stdout:4/701: dread d1d/d22/d30/d87/dd1/ff0 [0,4194304] 0 2024-03-14T13:02:15.783 INFO:tasks.workunit.client.0.smithi111.stdout:1/424: creat d20/d21/d3f/f96 x:0 0 0 2024-03-14T13:02:15.784 INFO:tasks.workunit.client.0.smithi111.stdout:3/457: symlink d0/d2d/d35/d50/d5e/d19/lb5 0 2024-03-14T13:02:15.785 INFO:tasks.workunit.client.1.smithi137.stdout:8/321: mkdir d1/dc/d4e/d6d 0 2024-03-14T13:02:15.786 INFO:tasks.workunit.client.0.smithi111.stdout:9/442: dwrite d5/d6/fc [4194304,4194304] 0 2024-03-14T13:02:15.786 INFO:tasks.workunit.client.0.smithi111.stdout:0/428: rmdir d6/d9/de/d32/d79 0 2024-03-14T13:02:15.787 INFO:tasks.workunit.client.1.smithi137.stdout:1/438: rename d7/d9/d19/d1f/d2e/l86 to d7/d9/d19/d1f/l8f 0 2024-03-14T13:02:15.788 INFO:tasks.workunit.client.0.smithi111.stdout:9/443: write d5/d6/f56 [784106,43075] 0 2024-03-14T13:02:15.789 INFO:tasks.workunit.client.1.smithi137.stdout:1/439: write d7/d9/d19/d1f/d2e/f34 [2603937,40996] 0 2024-03-14T13:02:15.790 INFO:tasks.workunit.client.0.smithi111.stdout:1/425: mknod d20/d21/d3f/c97 0 2024-03-14T13:02:15.792 INFO:tasks.workunit.client.1.smithi137.stdout:6/363: dread d2/d6/de/d45/d59/d39/f73 [0,4194304] 0 2024-03-14T13:02:15.792 INFO:tasks.workunit.client.0.smithi111.stdout:9/444: write d5/d6/d2a/d5a/f5d [2200127,37975] 0 2024-03-14T13:02:15.793 INFO:tasks.workunit.client.0.smithi111.stdout:3/458: read d0/f23 [205782,63898] 0 2024-03-14T13:02:15.793 INFO:tasks.workunit.client.0.smithi111.stdout:4/702: read d1d/d22/d30/d5e/f8d [3308994,93515] 0 2024-03-14T13:02:15.793 INFO:tasks.workunit.client.0.smithi111.stdout:2/428: getdents dd/d17/d34/d73/d75/d59 0 2024-03-14T13:02:15.794 INFO:tasks.workunit.client.1.smithi137.stdout:6/364: chown d2/d4c/c6a 782543 1 2024-03-14T13:02:15.796 INFO:tasks.workunit.client.0.smithi111.stdout:1/426: mknod d20/d25/d31/d80/c98 0 2024-03-14T13:02:15.797 INFO:tasks.workunit.client.1.smithi137.stdout:1/440: dwrite d7/d4d/f56 [4194304,4194304] 0 2024-03-14T13:02:15.798 INFO:tasks.workunit.client.1.smithi137.stdout:6/365: stat d2/d6/c24 0 2024-03-14T13:02:15.799 INFO:tasks.workunit.client.0.smithi111.stdout:2/429: mkdir dd/d8d 0 2024-03-14T13:02:15.803 INFO:tasks.workunit.client.0.smithi111.stdout:0/429: dwrite f5 [0,4194304] 0 2024-03-14T13:02:15.804 INFO:tasks.workunit.client.0.smithi111.stdout:1/427: stat d20/d25/d31/d80 0 2024-03-14T13:02:15.805 INFO:tasks.workunit.client.1.smithi137.stdout:2/403: dwrite d2/d7/db/f38 [0,4194304] 0 2024-03-14T13:02:15.806 INFO:tasks.workunit.client.1.smithi137.stdout:6/366: mkdir d2/d6/d86 0 2024-03-14T13:02:15.807 INFO:tasks.workunit.client.0.smithi111.stdout:6/406: truncate d0/d17/d21/f2d 3759439 0 2024-03-14T13:02:15.808 INFO:tasks.workunit.client.1.smithi137.stdout:6/367: dread - d2/d6/de/d45/d59/f75 zero size 2024-03-14T13:02:15.811 INFO:tasks.workunit.client.0.smithi111.stdout:9/445: dread d5/d6/d2a/d2d/f66 [0,4194304] 0 2024-03-14T13:02:15.811 INFO:tasks.workunit.client.1.smithi137.stdout:6/368: dread - d2/d6/de/d45/f60 zero size 2024-03-14T13:02:15.817 INFO:tasks.workunit.client.0.smithi111.stdout:1/428: symlink d20/d21/d28/l99 0 2024-03-14T13:02:15.818 INFO:tasks.workunit.client.0.smithi111.stdout:7/371: dread d0/d1/d21/d3a/f5b [0,4194304] 0 2024-03-14T13:02:15.818 INFO:tasks.workunit.client.0.smithi111.stdout:6/407: dread - d0/d17/d20/d65/f92 zero size 2024-03-14T13:02:15.819 INFO:tasks.workunit.client.0.smithi111.stdout:4/703: dwrite d1d/d22/d40/d69/d86/f9f [0,4194304] 0 2024-03-14T13:02:15.819 INFO:tasks.workunit.client.0.smithi111.stdout:2/430: chown dd/c15 25352 1 2024-03-14T13:02:15.821 INFO:tasks.workunit.client.0.smithi111.stdout:5/504: dwrite d2/f6 [0,4194304] 0 2024-03-14T13:02:15.821 INFO:tasks.workunit.client.0.smithi111.stdout:1/429: fdatasync d20/d25/d75/f7d 0 2024-03-14T13:02:15.822 INFO:tasks.workunit.client.1.smithi137.stdout:2/404: unlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/l72 0 2024-03-14T13:02:15.823 INFO:tasks.workunit.client.1.smithi137.stdout:8/322: dread d1/dc/d11/d13/f45 [0,4194304] 0 2024-03-14T13:02:15.823 INFO:tasks.workunit.client.1.smithi137.stdout:2/405: fsync d2/d34/d3c/d53/f5e 0 2024-03-14T13:02:15.824 INFO:tasks.workunit.client.0.smithi111.stdout:1/430: write d20/f23 [2773145,97778] 0 2024-03-14T13:02:15.825 INFO:tasks.workunit.client.1.smithi137.stdout:8/323: stat d1/dc/lf 0 2024-03-14T13:02:15.826 INFO:tasks.workunit.client.0.smithi111.stdout:4/704: symlink d1d/d22/dc6/lf7 0 2024-03-14T13:02:15.827 INFO:tasks.workunit.client.0.smithi111.stdout:2/431: creat dd/d17/d34/d73/d75/f8e x:0 0 0 2024-03-14T13:02:15.827 INFO:tasks.workunit.client.1.smithi137.stdout:6/369: link d2/d54/d55/f78 d2/d54/d55/d6d/f87 0 2024-03-14T13:02:15.828 INFO:tasks.workunit.client.1.smithi137.stdout:8/324: chown d1/dc/d11/c15 694666 1 2024-03-14T13:02:15.831 INFO:tasks.workunit.client.0.smithi111.stdout:4/705: readlink d1d/d22/d30/d5e/db8/ldd 0 2024-03-14T13:02:15.831 INFO:tasks.workunit.client.0.smithi111.stdout:5/505: mknod d2/d16/d49/d4c/ca1 0 2024-03-14T13:02:15.831 INFO:tasks.workunit.client.1.smithi137.stdout:1/441: dread d7/d9/d19/d1f/d26/d6c/f6f [0,4194304] 0 2024-03-14T13:02:15.831 INFO:tasks.workunit.client.1.smithi137.stdout:8/325: chown d1/dc/d3c/d54/c23 4523 1 2024-03-14T13:02:15.831 INFO:tasks.workunit.client.1.smithi137.stdout:6/370: write d2/d54/d55/d38/f51 [4089331,22178] 0 2024-03-14T13:02:15.832 INFO:tasks.workunit.client.0.smithi111.stdout:2/432: write dd/f76 [103036,64188] 0 2024-03-14T13:02:15.833 INFO:tasks.workunit.client.0.smithi111.stdout:2/433: chown dd/d17/d34/d73/d75/f6e 226363 1 2024-03-14T13:02:15.833 INFO:tasks.workunit.client.1.smithi137.stdout:1/442: dread - d7/d9/d19/d1f/f78 zero size 2024-03-14T13:02:15.834 INFO:tasks.workunit.client.0.smithi111.stdout:2/434: chown dd/d17/d53/f8c 109602 1 2024-03-14T13:02:15.835 INFO:tasks.workunit.client.1.smithi137.stdout:2/406: dwrite d2/d7/db/f38 [0,4194304] 0 2024-03-14T13:02:15.836 INFO:tasks.workunit.client.0.smithi111.stdout:6/408: creat d0/d17/d20/d5a/f9b x:0 0 0 2024-03-14T13:02:15.836 INFO:tasks.workunit.client.1.smithi137.stdout:8/326: rmdir d1/dc/d11 39 2024-03-14T13:02:15.838 INFO:tasks.workunit.client.0.smithi111.stdout:9/446: mknod d5/d6/d14/d7a/c92 0 2024-03-14T13:02:15.839 INFO:tasks.workunit.client.1.smithi137.stdout:8/327: dread - d1/d5e/f63 zero size 2024-03-14T13:02:15.839 INFO:tasks.workunit.client.1.smithi137.stdout:6/371: write d2/d6/de/f13 [2411647,56494] 0 2024-03-14T13:02:15.840 INFO:tasks.workunit.client.0.smithi111.stdout:1/431: readlink d20/d25/d75/l85 0 2024-03-14T13:02:15.842 INFO:tasks.workunit.client.0.smithi111.stdout:0/430: write d6/d9/de/d32/f31 [1184711,15863] 0 2024-03-14T13:02:15.843 INFO:tasks.workunit.client.0.smithi111.stdout:4/706: mkdir d1d/d22/d30/d7c/df8 0 2024-03-14T13:02:15.843 INFO:tasks.workunit.client.1.smithi137.stdout:1/443: mkdir d7/d9/d19/d1f/d89/d90 0 2024-03-14T13:02:15.844 INFO:tasks.workunit.client.1.smithi137.stdout:6/372: mkdir d2/d6/d1d/d88 0 2024-03-14T13:02:15.845 INFO:tasks.workunit.client.0.smithi111.stdout:4/707: chown d1d/d22/d30/d87 9061334 1 2024-03-14T13:02:15.845 INFO:tasks.workunit.client.0.smithi111.stdout:1/432: creat d20/d21/d28/d5c/f9a x:0 0 0 2024-03-14T13:02:15.845 INFO:tasks.workunit.client.0.smithi111.stdout:2/435: symlink dd/d17/d34/d73/l8f 0 2024-03-14T13:02:15.846 INFO:tasks.workunit.client.1.smithi137.stdout:8/328: rmdir d1/dc/d11/d13 39 2024-03-14T13:02:15.847 INFO:tasks.workunit.client.0.smithi111.stdout:5/506: mknod d2/d16/d49/d44/d4a/d9e/ca2 0 2024-03-14T13:02:15.847 INFO:tasks.workunit.client.0.smithi111.stdout:0/431: dread - d6/d5a/f6e zero size 2024-03-14T13:02:15.847 INFO:tasks.workunit.client.1.smithi137.stdout:1/444: chown d7/d9/d19/d1f/d22/l75 1040 1 2024-03-14T13:02:15.847 INFO:tasks.workunit.client.1.smithi137.stdout:2/407: link d2/d7/d2e/c32 d2/d7/db/d2f/c85 0 2024-03-14T13:02:15.848 INFO:tasks.workunit.client.1.smithi137.stdout:6/373: symlink d2/d54/d55/d6d/d76/l89 0 2024-03-14T13:02:15.849 INFO:tasks.workunit.client.0.smithi111.stdout:0/432: stat d6/d3d 0 2024-03-14T13:02:15.849 INFO:tasks.workunit.client.0.smithi111.stdout:2/436: chown dd/d17/d34/d4f/f7e 61144000 1 2024-03-14T13:02:15.850 INFO:tasks.workunit.client.0.smithi111.stdout:9/447: creat d5/d6/d8/f93 x:0 0 0 2024-03-14T13:02:15.851 INFO:tasks.workunit.client.0.smithi111.stdout:1/433: creat d20/d40/f9b x:0 0 0 2024-03-14T13:02:15.851 INFO:tasks.workunit.client.1.smithi137.stdout:1/445: mkdir d7/d9/d19/d1f/d26/d7d/d91 0 2024-03-14T13:02:15.852 INFO:tasks.workunit.client.0.smithi111.stdout:2/437: write dd/f4c [564317,86549] 0 2024-03-14T13:02:15.853 INFO:tasks.workunit.client.0.smithi111.stdout:7/372: dread d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:15.854 INFO:tasks.workunit.client.1.smithi137.stdout:6/374: mkdir d2/d6/d86/d8a 0 2024-03-14T13:02:15.854 INFO:tasks.workunit.client.0.smithi111.stdout:0/433: mknod d6/d9/de/d32/c8a 0 2024-03-14T13:02:15.855 INFO:tasks.workunit.client.1.smithi137.stdout:6/375: chown d2/d6/f34 26 1 2024-03-14T13:02:15.856 INFO:tasks.workunit.client.1.smithi137.stdout:8/329: dread d1/dc/d11/d13/f3e [0,4194304] 0 2024-03-14T13:02:15.856 INFO:tasks.workunit.client.0.smithi111.stdout:9/448: write d5/d6/d14/f54 [1543226,127667] 0 2024-03-14T13:02:15.857 INFO:tasks.workunit.client.0.smithi111.stdout:0/434: stat d6/d9/d15/d49/d56 0 2024-03-14T13:02:15.858 INFO:tasks.workunit.client.0.smithi111.stdout:4/708: rename d1d/d22/d30/de5/f4c to d1d/d22/d30/d7c/ff9 0 2024-03-14T13:02:15.858 INFO:tasks.workunit.client.0.smithi111.stdout:9/449: write d5/d6/d8/f16 [4694240,111243] 0 2024-03-14T13:02:15.859 INFO:tasks.workunit.client.1.smithi137.stdout:6/376: fdatasync d2/d54/d55/d38/f85 0 2024-03-14T13:02:15.859 INFO:tasks.workunit.client.1.smithi137.stdout:8/330: write d1/dc/d3c/d54/f48 [7340644,81737] 0 2024-03-14T13:02:15.860 INFO:tasks.workunit.client.1.smithi137.stdout:6/377: readlink d2/d6/l42 0 2024-03-14T13:02:15.860 INFO:tasks.workunit.client.1.smithi137.stdout:1/446: dwrite d7/d9/d19/f2f [4194304,4194304] 0 2024-03-14T13:02:15.860 INFO:tasks.workunit.client.0.smithi111.stdout:2/438: creat dd/d17/d34/d73/d49/f90 x:0 0 0 2024-03-14T13:02:15.861 INFO:tasks.workunit.client.0.smithi111.stdout:7/373: write d0/d1/d22/d3e/f64 [653044,79497] 0 2024-03-14T13:02:15.862 INFO:tasks.workunit.client.0.smithi111.stdout:4/709: chown d1d/d22/d30/d5e/d6f 72 1 2024-03-14T13:02:15.863 INFO:tasks.workunit.client.1.smithi137.stdout:8/331: rmdir d1 39 2024-03-14T13:02:15.865 INFO:tasks.workunit.client.0.smithi111.stdout:1/434: rename d20/d21/d3f/l4b to d20/d25/d31/d80/l9c 0 2024-03-14T13:02:15.867 INFO:tasks.workunit.client.1.smithi137.stdout:1/447: fsync d7/d9/d19/d1f/d22/f25 0 2024-03-14T13:02:15.868 INFO:tasks.workunit.client.0.smithi111.stdout:7/374: chown d0/d1/d21/d3a/c6b 163571 1 2024-03-14T13:02:15.869 INFO:tasks.workunit.client.1.smithi137.stdout:2/408: dread d2/d7/d2e/d3e/d44/d6b/d1d/f48 [0,4194304] 0 2024-03-14T13:02:15.870 INFO:tasks.workunit.client.1.smithi137.stdout:6/378: write d2/d6/de/d45/f77 [847917,48564] 0 2024-03-14T13:02:15.871 INFO:tasks.workunit.client.0.smithi111.stdout:2/439: truncate dd/d17/d34/d73/d49/f90 336158 0 2024-03-14T13:02:15.872 INFO:tasks.workunit.client.1.smithi137.stdout:2/409: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f75 0 2024-03-14T13:02:15.873 INFO:tasks.workunit.client.0.smithi111.stdout:2/440: fdatasync dd/fe 0 2024-03-14T13:02:15.873 INFO:tasks.workunit.client.0.smithi111.stdout:9/450: truncate d5/d6/d2a/f75 5064913 0 2024-03-14T13:02:15.873 INFO:tasks.workunit.client.1.smithi137.stdout:6/379: truncate d2/d6/de/f13 4742104 0 2024-03-14T13:02:15.874 INFO:tasks.workunit.client.0.smithi111.stdout:1/435: creat d20/f9d x:0 0 0 2024-03-14T13:02:15.874 INFO:tasks.workunit.client.1.smithi137.stdout:6/380: dread - d2/d4c/f69 zero size 2024-03-14T13:02:15.875 INFO:tasks.workunit.client.1.smithi137.stdout:2/410: readlink d2/d34/d3c/l42 0 2024-03-14T13:02:15.876 INFO:tasks.workunit.client.1.smithi137.stdout:8/332: getdents d1/dc/d4e 0 2024-03-14T13:02:15.877 INFO:tasks.workunit.client.1.smithi137.stdout:1/448: truncate d7/d29/f2b 146057 0 2024-03-14T13:02:15.877 INFO:tasks.workunit.client.1.smithi137.stdout:8/333: chown d1/dc/c30 199911 1 2024-03-14T13:02:15.878 INFO:tasks.workunit.client.0.smithi111.stdout:7/375: symlink d0/d1/d21/d5d/l71 0 2024-03-14T13:02:15.879 INFO:tasks.workunit.client.0.smithi111.stdout:9/451: chown f4 258323 1 2024-03-14T13:02:15.880 INFO:tasks.workunit.client.1.smithi137.stdout:8/334: read d1/d5e/f61 [2634944,116078] 0 2024-03-14T13:02:15.880 INFO:tasks.workunit.client.1.smithi137.stdout:2/411: write d2/d34/f7f [2620049,13927] 0 2024-03-14T13:02:15.882 INFO:tasks.workunit.client.0.smithi111.stdout:2/441: mknod dd/c91 0 2024-03-14T13:02:15.882 INFO:tasks.workunit.client.1.smithi137.stdout:8/335: stat d1/dc/l5d 0 2024-03-14T13:02:15.886 INFO:tasks.workunit.client.1.smithi137.stdout:2/412: mknod d2/d7/db/d2f/c86 0 2024-03-14T13:02:15.886 INFO:tasks.workunit.client.0.smithi111.stdout:1/436: rmdir d20/d21/d3f/d59 39 2024-03-14T13:02:15.887 INFO:tasks.workunit.client.0.smithi111.stdout:5/507: dwrite d2/d16/d49/d44/f50 [0,4194304] 0 2024-03-14T13:02:15.888 INFO:tasks.workunit.client.1.smithi137.stdout:6/381: dwrite d2/d6/de/d45/d59/f37 [0,4194304] 0 2024-03-14T13:02:15.888 INFO:tasks.workunit.client.1.smithi137.stdout:8/336: creat d1/dc/d3c/d54/f6e x:0 0 0 2024-03-14T13:02:15.890 INFO:tasks.workunit.client.0.smithi111.stdout:7/376: creat d0/f72 x:0 0 0 2024-03-14T13:02:15.890 INFO:tasks.workunit.client.1.smithi137.stdout:1/449: write f6 [1538283,1185] 0 2024-03-14T13:02:15.891 INFO:tasks.workunit.client.1.smithi137.stdout:8/337: symlink d1/d5e/l6f 0 2024-03-14T13:02:15.891 INFO:tasks.workunit.client.0.smithi111.stdout:0/435: dwrite d6/d9/de/d3a/d46/d4a/f5c [0,4194304] 0 2024-03-14T13:02:15.891 INFO:tasks.workunit.client.0.smithi111.stdout:6/409: dwrite d0/d17/d21/d69/d3d/f71 [0,4194304] 0 2024-03-14T13:02:15.891 INFO:tasks.workunit.client.1.smithi137.stdout:1/450: chown d7/d9/c5c 3 1 2024-03-14T13:02:15.895 INFO:tasks.workunit.client.1.smithi137.stdout:2/413: chown d2/d7/c30 64709 1 2024-03-14T13:02:15.897 INFO:tasks.workunit.client.1.smithi137.stdout:2/414: dread - d2/d34/f69 zero size 2024-03-14T13:02:15.897 INFO:tasks.workunit.client.0.smithi111.stdout:9/452: creat d5/d6/d8/d5b/d7c/f94 x:0 0 0 2024-03-14T13:02:15.897 INFO:tasks.workunit.client.0.smithi111.stdout:2/442: dread - dd/d17/d34/d73/d75/f6e zero size 2024-03-14T13:02:15.898 INFO:tasks.workunit.client.1.smithi137.stdout:6/382: dread d2/d6/de/f13 [0,4194304] 0 2024-03-14T13:02:15.900 INFO:tasks.workunit.client.1.smithi137.stdout:8/338: dread d1/dc/d3c/d54/f1c [0,4194304] 0 2024-03-14T13:02:15.902 INFO:tasks.workunit.client.0.smithi111.stdout:7/377: creat d0/d11/f73 x:0 0 0 2024-03-14T13:02:15.902 INFO:tasks.workunit.client.0.smithi111.stdout:4/710: dwrite d1d/d22/d30/d5e/d6f/d7e/fc8 [0,4194304] 0 2024-03-14T13:02:15.903 INFO:tasks.workunit.client.0.smithi111.stdout:7/378: chown d0/d1/d21 1276033 1 2024-03-14T13:02:15.904 INFO:tasks.workunit.client.1.smithi137.stdout:8/339: symlink d1/dc/d3c/d54/l70 0 2024-03-14T13:02:15.905 INFO:tasks.workunit.client.0.smithi111.stdout:6/410: creat d0/d17/d20/d4a/d59/d80/f9c x:0 0 0 2024-03-14T13:02:15.905 INFO:tasks.workunit.client.1.smithi137.stdout:2/415: creat d2/d7/f87 x:0 0 0 2024-03-14T13:02:15.906 INFO:tasks.workunit.client.0.smithi111.stdout:5/508: rename d2/d16/d49/d4c/f55 to d2/d16/d49/d7c/d95/fa3 0 2024-03-14T13:02:15.908 INFO:tasks.workunit.client.0.smithi111.stdout:0/436: dread f5 [0,4194304] 0 2024-03-14T13:02:15.909 INFO:tasks.workunit.client.0.smithi111.stdout:7/379: unlink d0/d1/d5/d38/l3b 0 2024-03-14T13:02:15.910 INFO:tasks.workunit.client.0.smithi111.stdout:1/437: dwrite f1a [0,4194304] 0 2024-03-14T13:02:15.910 INFO:tasks.workunit.client.1.smithi137.stdout:8/340: dread d1/f44 [0,4194304] 0 2024-03-14T13:02:15.917 INFO:tasks.workunit.client.1.smithi137.stdout:8/341: truncate d1/dc/f39 4575692 0 2024-03-14T13:02:15.917 INFO:tasks.workunit.client.1.smithi137.stdout:6/383: getdents d2/d54/d55/d38 0 2024-03-14T13:02:15.917 INFO:tasks.workunit.client.1.smithi137.stdout:8/342: chown d1/dc/d10/l41 47092 1 2024-03-14T13:02:15.918 INFO:tasks.workunit.client.1.smithi137.stdout:6/384: mkdir d2/d54/d8b 0 2024-03-14T13:02:15.918 INFO:tasks.workunit.client.1.smithi137.stdout:2/416: dread d2/d7/d2e/d3e/d44/d6b/f4a [0,4194304] 0 2024-03-14T13:02:15.919 INFO:tasks.workunit.client.1.smithi137.stdout:8/343: mkdir d1/dc/d71 0 2024-03-14T13:02:15.921 INFO:tasks.workunit.client.1.smithi137.stdout:2/417: symlink d2/d7/db/d74/l88 0 2024-03-14T13:02:15.922 INFO:tasks.workunit.client.1.smithi137.stdout:8/344: creat d1/dc/d71/f72 x:0 0 0 2024-03-14T13:02:15.923 INFO:tasks.workunit.client.1.smithi137.stdout:2/418: creat d2/d7/db/d2f/f89 x:0 0 0 2024-03-14T13:02:15.924 INFO:tasks.workunit.client.1.smithi137.stdout:8/345: write d1/dc/d11/d13/f22 [1717925,69823] 0 2024-03-14T13:02:15.927 INFO:tasks.workunit.client.1.smithi137.stdout:8/346: creat d1/dc/d11/d13/f73 x:0 0 0 2024-03-14T13:02:15.928 INFO:tasks.workunit.client.1.smithi137.stdout:8/347: read - d1/dc/d10/f32 zero size 2024-03-14T13:02:15.931 INFO:tasks.workunit.client.1.smithi137.stdout:1/451: dread d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:15.931 INFO:tasks.workunit.client.1.smithi137.stdout:2/419: dread d2/d7/d2e/d3e/d44/d6b/f39 [0,4194304] 0 2024-03-14T13:02:15.933 INFO:tasks.workunit.client.1.smithi137.stdout:1/452: write f5 [1213852,47665] 0 2024-03-14T13:02:15.935 INFO:tasks.workunit.client.1.smithi137.stdout:7/309: sync 2024-03-14T13:02:15.936 INFO:tasks.workunit.client.1.smithi137.stdout:1/453: chown d7/d9/d19/d1f/d22/f45 319 1 2024-03-14T13:02:15.936 INFO:tasks.workunit.client.1.smithi137.stdout:2/420: write d2/d7/d2e/d3e/d44/d6b/f4a [924186,88169] 0 2024-03-14T13:02:15.946 INFO:tasks.workunit.client.1.smithi137.stdout:6/385: dread f0 [0,4194304] 0 2024-03-14T13:02:15.946 INFO:tasks.workunit.client.1.smithi137.stdout:7/310: dwrite d3/f32 [0,4194304] 0 2024-03-14T13:02:15.948 INFO:tasks.workunit.client.1.smithi137.stdout:2/421: dwrite d2/f4b [8388608,4194304] 0 2024-03-14T13:02:15.949 INFO:tasks.workunit.client.1.smithi137.stdout:8/348: dread d1/dc/d11/f2d [0,4194304] 0 2024-03-14T13:02:15.951 INFO:tasks.workunit.client.1.smithi137.stdout:6/386: write d2/d6/de/d45/f77 [3822658,70656] 0 2024-03-14T13:02:15.954 INFO:tasks.workunit.client.1.smithi137.stdout:8/349: creat d1/dc/d4e/d6d/f74 x:0 0 0 2024-03-14T13:02:15.954 INFO:tasks.workunit.client.1.smithi137.stdout:7/311: truncate d3/d6/f45 818414 0 2024-03-14T13:02:15.959 INFO:tasks.workunit.client.1.smithi137.stdout:2/422: dwrite d2/d7/d2e/d3e/d44/d6b/f3f [0,4194304] 0 2024-03-14T13:02:15.961 INFO:tasks.workunit.client.1.smithi137.stdout:1/454: dread d7/d9/d19/d1f/d22/f25 [0,4194304] 0 2024-03-14T13:02:15.962 INFO:tasks.workunit.client.1.smithi137.stdout:2/423: write d2/d7/db/d2f/f89 [6759,80936] 0 2024-03-14T13:02:15.963 INFO:tasks.workunit.client.1.smithi137.stdout:8/350: link d1/dc/d11/l59 d1/dc/d4e/d6d/l75 0 2024-03-14T13:02:15.964 INFO:tasks.workunit.client.1.smithi137.stdout:1/455: rmdir d7/d9/d19/d40/d42 39 2024-03-14T13:02:15.965 INFO:tasks.workunit.client.1.smithi137.stdout:8/351: chown d1/dc/d3c/d54/l3f 10518779 1 2024-03-14T13:02:15.969 INFO:tasks.workunit.client.1.smithi137.stdout:2/424: creat d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f8a x:0 0 0 2024-03-14T13:02:15.969 INFO:tasks.workunit.client.1.smithi137.stdout:8/352: symlink d1/dc/d4e/l76 0 2024-03-14T13:02:15.972 INFO:tasks.workunit.client.1.smithi137.stdout:7/312: read d3/f1c [3392343,130700] 0 2024-03-14T13:02:15.972 INFO:tasks.workunit.client.1.smithi137.stdout:1/456: dwrite d7/d9/f3e [0,4194304] 0 2024-03-14T13:02:15.972 INFO:tasks.workunit.client.1.smithi137.stdout:2/425: fsync d2/d7/d2e/d3e/d44/d6b/d22/f5a 0 2024-03-14T13:02:15.974 INFO:tasks.workunit.client.1.smithi137.stdout:7/313: stat d3/d5/d23/d2c/d5d 0 2024-03-14T13:02:15.974 INFO:tasks.workunit.client.1.smithi137.stdout:2/426: dread - d2/d34/d3c/f5d zero size 2024-03-14T13:02:15.974 INFO:tasks.workunit.client.1.smithi137.stdout:1/457: fdatasync d7/d9/d19/d40/d50/f80 0 2024-03-14T13:02:15.976 INFO:tasks.workunit.client.1.smithi137.stdout:8/353: dread d1/dc/f52 [0,4194304] 0 2024-03-14T13:02:15.978 INFO:tasks.workunit.client.1.smithi137.stdout:1/458: chown d7/d9/d19/d40/d50/l66 509782895 1 2024-03-14T13:02:15.980 INFO:tasks.workunit.client.1.smithi137.stdout:8/354: mkdir d1/dc/d77 0 2024-03-14T13:02:15.981 INFO:tasks.workunit.client.1.smithi137.stdout:2/427: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b 0 2024-03-14T13:02:15.982 INFO:tasks.workunit.client.0.smithi111.stdout:2/443: rename dd/fe to dd/d17/d53/f92 0 2024-03-14T13:02:15.983 INFO:tasks.workunit.client.1.smithi137.stdout:2/428: chown d2/d34/f69 4858 1 2024-03-14T13:02:15.983 INFO:tasks.workunit.client.1.smithi137.stdout:7/314: dwrite d3/f32 [4194304,4194304] 0 2024-03-14T13:02:15.983 INFO:tasks.workunit.client.0.smithi111.stdout:6/411: truncate d0/f35 853335 0 2024-03-14T13:02:15.983 INFO:tasks.workunit.client.0.smithi111.stdout:4/711: unlink d1d/d22/d30/d5e/d6f/d7e/d66/dba/ceb 0 2024-03-14T13:02:15.984 INFO:tasks.workunit.client.1.smithi137.stdout:8/355: creat d1/dc/d77/f78 x:0 0 0 2024-03-14T13:02:15.986 INFO:tasks.workunit.client.0.smithi111.stdout:2/444: stat dd/d17/d34/d73/d75/d59 0 2024-03-14T13:02:15.988 INFO:tasks.workunit.client.0.smithi111.stdout:4/712: readlink d1d/d22/d30/d5e/d6f/d7e/d66/l6d 0 2024-03-14T13:02:15.988 INFO:tasks.workunit.client.0.smithi111.stdout:5/509: symlink d2/la4 0 2024-03-14T13:02:15.990 INFO:tasks.workunit.client.1.smithi137.stdout:8/356: unlink d1/dc/d10/f21 0 2024-03-14T13:02:15.992 INFO:tasks.workunit.client.0.smithi111.stdout:9/453: dwrite d5/d6/d2a/d5a/f76 [0,4194304] 0 2024-03-14T13:02:15.994 INFO:tasks.workunit.client.1.smithi137.stdout:1/459: dwrite f3 [4194304,4194304] 0 2024-03-14T13:02:15.995 INFO:tasks.workunit.client.0.smithi111.stdout:0/437: symlink d6/d9/de/d32/l8b 0 2024-03-14T13:02:15.995 INFO:tasks.workunit.client.0.smithi111.stdout:2/445: write dd/d17/d34/d73/d49/f51 [774956,90745] 0 2024-03-14T13:02:15.995 INFO:tasks.workunit.client.1.smithi137.stdout:8/357: truncate d1/f60 909212 0 2024-03-14T13:02:16.001 INFO:tasks.workunit.client.0.smithi111.stdout:9/454: creat d5/d42/f95 x:0 0 0 2024-03-14T13:02:16.003 INFO:tasks.workunit.client.0.smithi111.stdout:6/412: rename d0/d17/d24/l64 to d0/d17/d20/d4a/d68/l9d 0 2024-03-14T13:02:16.004 INFO:tasks.workunit.client.0.smithi111.stdout:0/438: symlink d6/d9/d15/d49/l8c 0 2024-03-14T13:02:16.004 INFO:tasks.workunit.client.1.smithi137.stdout:7/315: dwrite d3/d5/d23/f2b [0,4194304] 0 2024-03-14T13:02:16.004 INFO:tasks.workunit.client.0.smithi111.stdout:9/455: chown d5/d42/d52/d83 9494 1 2024-03-14T13:02:16.005 INFO:tasks.workunit.client.0.smithi111.stdout:7/380: getdents d0/d1/d21 0 2024-03-14T13:02:16.006 INFO:tasks.workunit.client.1.smithi137.stdout:8/358: mknod d1/dc/d11/c79 0 2024-03-14T13:02:16.006 INFO:tasks.workunit.client.0.smithi111.stdout:2/446: symlink dd/d17/d34/d4f/l93 0 2024-03-14T13:02:16.008 INFO:tasks.workunit.client.0.smithi111.stdout:2/447: read dd/d17/f63 [2213099,116553] 0 2024-03-14T13:02:16.008 INFO:tasks.workunit.client.0.smithi111.stdout:7/381: write d0/d1/d4a/f60 [1981343,91849] 0 2024-03-14T13:02:16.009 INFO:tasks.workunit.client.0.smithi111.stdout:1/438: rename l16 to d20/d21/d3f/d59/l9e 0 2024-03-14T13:02:16.010 INFO:tasks.workunit.client.1.smithi137.stdout:7/316: unlink d3/d5/d23/l29 0 2024-03-14T13:02:16.011 INFO:tasks.workunit.client.0.smithi111.stdout:7/382: chown d0/d1/d5/d38/d5c/l63 5587 1 2024-03-14T13:02:16.012 INFO:tasks.workunit.client.0.smithi111.stdout:9/456: mkdir d5/d42/d71/d96 0 2024-03-14T13:02:16.012 INFO:tasks.workunit.client.1.smithi137.stdout:2/429: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 [0,4194304] 0 2024-03-14T13:02:16.015 INFO:tasks.workunit.client.1.smithi137.stdout:2/430: dread d2/d7/db/f2c [0,4194304] 0 2024-03-14T13:02:16.018 INFO:tasks.workunit.client.1.smithi137.stdout:2/431: mknod d2/d7/d2e/d3e/c8c 0 2024-03-14T13:02:16.020 INFO:tasks.workunit.client.0.smithi111.stdout:1/439: creat d20/d25/f9f x:0 0 0 2024-03-14T13:02:16.022 INFO:tasks.workunit.client.0.smithi111.stdout:7/383: chown d0/d1/d5/d38/d9/l31 209974 1 2024-03-14T13:02:16.023 INFO:tasks.workunit.client.0.smithi111.stdout:4/713: dwrite d1d/d22/d30/d5e/fb3 [0,4194304] 0 2024-03-14T13:02:16.023 INFO:tasks.workunit.client.0.smithi111.stdout:9/457: symlink d5/d42/d71/l97 0 2024-03-14T13:02:16.028 INFO:tasks.workunit.client.1.smithi137.stdout:2/432: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/f48 [4194304,4194304] 0 2024-03-14T13:02:16.030 INFO:tasks.workunit.client.0.smithi111.stdout:5/510: rename d2/d16/d49/d44/c65 to d2/d16/d49/d4c/ca5 0 2024-03-14T13:02:16.030 INFO:tasks.workunit.client.1.smithi137.stdout:2/433: write d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 [2645572,125547] 0 2024-03-14T13:02:16.034 INFO:tasks.workunit.client.1.smithi137.stdout:2/434: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d 0 2024-03-14T13:02:16.038 INFO:tasks.workunit.client.0.smithi111.stdout:1/440: unlink d20/d21/d28/f7a 0 2024-03-14T13:02:16.041 INFO:tasks.workunit.client.0.smithi111.stdout:4/714: symlink d1d/d22/dc6/de6/lfa 0 2024-03-14T13:02:16.042 INFO:tasks.workunit.client.0.smithi111.stdout:9/458: symlink d5/l98 0 2024-03-14T13:02:16.042 INFO:tasks.workunit.client.0.smithi111.stdout:1/441: read d20/d21/f8b [319054,97704] 0 2024-03-14T13:02:16.043 INFO:tasks.workunit.client.0.smithi111.stdout:6/413: link d0/d17/d24/l8a d0/d17/d21/l9e 0 2024-03-14T13:02:16.043 INFO:tasks.workunit.client.1.smithi137.stdout:1/460: dread d7/d9/d19/d1f/f69 [0,4194304] 0 2024-03-14T13:02:16.044 INFO:tasks.workunit.client.1.smithi137.stdout:1/461: stat d7/d9/d19/d1f/f2a 0 2024-03-14T13:02:16.045 INFO:tasks.workunit.client.0.smithi111.stdout:4/715: creat d1d/d22/d56/ffb x:0 0 0 2024-03-14T13:02:16.046 INFO:tasks.workunit.client.0.smithi111.stdout:9/459: mknod d5/d42/c99 0 2024-03-14T13:02:16.046 INFO:tasks.workunit.client.1.smithi137.stdout:1/462: mkdir d7/d92 0 2024-03-14T13:02:16.046 INFO:tasks.workunit.client.1.smithi137.stdout:1/463: stat d7/d9/d19/d1f/d22 0 2024-03-14T13:02:16.047 INFO:tasks.workunit.client.0.smithi111.stdout:6/414: chown d0/d17/d20/d4a/d59/f97 688 1 2024-03-14T13:02:16.047 INFO:tasks.workunit.client.0.smithi111.stdout:1/442: write d20/d25/d87/f8f [393821,33291] 0 2024-03-14T13:02:16.050 INFO:tasks.workunit.client.0.smithi111.stdout:5/511: getdents d2/d45/d76/d8d 0 2024-03-14T13:02:16.051 INFO:tasks.workunit.client.0.smithi111.stdout:7/384: link d0/d1/d5/l69 d0/d1/d5/d38/l74 0 2024-03-14T13:02:16.051 INFO:tasks.workunit.client.0.smithi111.stdout:0/439: dwrite d6/f1f [0,4194304] 0 2024-03-14T13:02:16.052 INFO:tasks.workunit.client.1.smithi137.stdout:1/464: dwrite d7/d9/d19/d1f/d26/d84/f8c [0,4194304] 0 2024-03-14T13:02:16.052 INFO:tasks.workunit.client.1.smithi137.stdout:1/465: fsync d7/d9/d19/d1f/f23 0 2024-03-14T13:02:16.053 INFO:tasks.workunit.client.0.smithi111.stdout:4/716: creat d1d/d22/d30/d87/ffc x:0 0 0 2024-03-14T13:02:16.055 INFO:tasks.workunit.client.0.smithi111.stdout:6/415: rmdir d0/d17/d20/d4a/d59 39 2024-03-14T13:02:16.056 INFO:tasks.workunit.client.0.smithi111.stdout:5/512: truncate d2/d16/d49/d44/d4a/f60 4235439 0 2024-03-14T13:02:16.059 INFO:tasks.workunit.client.0.smithi111.stdout:1/443: mkdir d20/d21/d28/da0 0 2024-03-14T13:02:16.063 INFO:tasks.workunit.client.0.smithi111.stdout:7/385: write d0/d1/d5/d38/f68 [2954339,44938] 0 2024-03-14T13:02:16.065 INFO:tasks.workunit.client.0.smithi111.stdout:6/416: rename d0/d50/l70 to d0/d17/d20/d94/l9f 0 2024-03-14T13:02:16.065 INFO:tasks.workunit.client.0.smithi111.stdout:1/444: dread d20/d21/f41 [0,4194304] 0 2024-03-14T13:02:16.068 INFO:tasks.workunit.client.0.smithi111.stdout:4/717: dwrite d1d/f43 [4194304,4194304] 0 2024-03-14T13:02:16.069 INFO:tasks.workunit.client.0.smithi111.stdout:0/440: unlink d6/d9/d15/c35 0 2024-03-14T13:02:16.071 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:15 smithi111 ceph-mon[39688]: pgmap v22: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 21 MiB/s rd, 169 MiB/s wr, 264 op/s 2024-03-14T13:02:16.072 INFO:tasks.workunit.client.1.smithi137.stdout:7/317: truncate d3/f1c 806008 0 2024-03-14T13:02:16.075 INFO:tasks.workunit.client.0.smithi111.stdout:8/580: sync 2024-03-14T13:02:16.078 INFO:tasks.workunit.client.1.smithi137.stdout:7/318: mkdir d3/d6/d64 0 2024-03-14T13:02:16.078 INFO:tasks.workunit.client.1.smithi137.stdout:8/359: dwrite d1/d5e/f61 [0,4194304] 0 2024-03-14T13:02:16.079 INFO:tasks.workunit.client.0.smithi111.stdout:7/386: mkdir d0/d1/d4a/d75 0 2024-03-14T13:02:16.079 INFO:tasks.workunit.client.0.smithi111.stdout:1/445: creat d20/d25/fa1 x:0 0 0 2024-03-14T13:02:16.080 INFO:tasks.workunit.client.1.smithi137.stdout:8/360: write d1/f6 [5187137,7709] 0 2024-03-14T13:02:16.080 INFO:tasks.workunit.client.0.smithi111.stdout:5/513: rename d2/d16/d49/d33/d68/fa0 to d2/d16/d49/d33/fa6 0 2024-03-14T13:02:16.082 INFO:tasks.workunit.client.0.smithi111.stdout:6/417: fsync d0/d17/d20/f30 0 2024-03-14T13:02:16.083 INFO:tasks.workunit.client.0.smithi111.stdout:0/441: mknod d6/d5a/c8d 0 2024-03-14T13:02:16.084 INFO:tasks.workunit.client.0.smithi111.stdout:9/460: link d5/cd d5/d6/d8/d5b/d84/c9a 0 2024-03-14T13:02:16.086 INFO:tasks.workunit.client.0.smithi111.stdout:4/718: symlink d1d/d22/d40/d69/d86/lfd 0 2024-03-14T13:02:16.088 INFO:tasks.workunit.client.0.smithi111.stdout:8/581: creat d3/d1b/d20/d35/fb2 x:0 0 0 2024-03-14T13:02:16.090 INFO:tasks.workunit.client.1.smithi137.stdout:7/319: dwrite d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:16.090 INFO:tasks.workunit.client.0.smithi111.stdout:1/446: unlink d20/d21/d3f/d59/f6d 0 2024-03-14T13:02:16.091 INFO:tasks.workunit.client.0.smithi111.stdout:5/514: creat d2/d16/d49/d44/fa7 x:0 0 0 2024-03-14T13:02:16.092 INFO:tasks.workunit.client.0.smithi111.stdout:7/387: dwrite d0/d1/d5/d38/d9/f49 [0,4194304] 0 2024-03-14T13:02:16.094 INFO:tasks.workunit.client.0.smithi111.stdout:1/447: stat d20/d21/l2f 0 2024-03-14T13:02:16.096 INFO:tasks.workunit.client.1.smithi137.stdout:7/320: mknod d3/d54/c65 0 2024-03-14T13:02:16.097 INFO:tasks.workunit.client.0.smithi111.stdout:5/515: chown d2/d45/f34 1574 1 2024-03-14T13:02:16.097 INFO:tasks.workunit.client.0.smithi111.stdout:0/442: write d6/d9/de/f36 [431898,100485] 0 2024-03-14T13:02:16.097 INFO:tasks.workunit.client.1.smithi137.stdout:8/361: dwrite d1/dc/d3c/d54/d34/f68 [0,4194304] 0 2024-03-14T13:02:16.098 INFO:tasks.workunit.client.0.smithi111.stdout:4/719: fdatasync d1d/d22/d23/fa2 0 2024-03-14T13:02:16.100 INFO:tasks.workunit.client.1.smithi137.stdout:2/435: dwrite d2/d7/d2e/d3e/d44/f60 [0,4194304] 0 2024-03-14T13:02:16.101 INFO:tasks.workunit.client.0.smithi111.stdout:9/461: mkdir d5/d6/d2a/d2d/d9b 0 2024-03-14T13:02:16.101 INFO:tasks.workunit.client.0.smithi111.stdout:0/443: fsync d6/d9/d5b/f6d 0 2024-03-14T13:02:16.103 INFO:tasks.workunit.client.0.smithi111.stdout:8/582: dwrite d3/d1b/f1d [4194304,4194304] 0 2024-03-14T13:02:16.103 INFO:tasks.workunit.client.0.smithi111.stdout:1/448: symlink d20/d25/la2 0 2024-03-14T13:02:16.104 INFO:tasks.workunit.client.0.smithi111.stdout:9/462: chown d5/d6/d2a/d2d/c2e 190101 1 2024-03-14T13:02:16.105 INFO:tasks.workunit.client.1.smithi137.stdout:1/466: dwrite d7/d31/d54/f60 [0,4194304] 0 2024-03-14T13:02:16.105 INFO:tasks.workunit.client.1.smithi137.stdout:2/436: read - d2/d7/d2e/d3e/d44/d6b/d1d/d2d/f61 zero size 2024-03-14T13:02:16.106 INFO:tasks.workunit.client.1.smithi137.stdout:8/362: stat d1/dc/d3c/d54 0 2024-03-14T13:02:16.112 INFO:tasks.workunit.client.1.smithi137.stdout:7/321: rename d3/d5/d23/d3f/c50 to d3/d5/c66 0 2024-03-14T13:02:16.115 INFO:tasks.workunit.client.0.smithi111.stdout:4/720: mkdir d1d/dfe 0 2024-03-14T13:02:16.115 INFO:tasks.workunit.client.0.smithi111.stdout:6/418: rename d0/d17/d20/c87 to d0/d17/d21/d42/ca0 0 2024-03-14T13:02:16.116 INFO:tasks.workunit.client.1.smithi137.stdout:8/363: creat d1/dc/d4e/f7a x:0 0 0 2024-03-14T13:02:16.116 INFO:tasks.workunit.client.1.smithi137.stdout:8/364: rename d1/dc/d3c to d1/dc/d3c/d7b 22 2024-03-14T13:02:16.117 INFO:tasks.workunit.client.0.smithi111.stdout:7/388: symlink d0/d1/d5/d56/d70/l76 0 2024-03-14T13:02:16.118 INFO:tasks.workunit.client.0.smithi111.stdout:5/516: dwrite d2/d45/f34 [4194304,4194304] 0 2024-03-14T13:02:16.119 INFO:tasks.workunit.client.0.smithi111.stdout:0/444: symlink d6/d5a/l8e 0 2024-03-14T13:02:16.119 INFO:tasks.workunit.client.1.smithi137.stdout:1/467: fdatasync d7/d9/d19/d40/d42/f49 0 2024-03-14T13:02:16.121 INFO:tasks.workunit.client.1.smithi137.stdout:8/365: read d1/f6 [3851697,14898] 0 2024-03-14T13:02:16.121 INFO:tasks.workunit.client.1.smithi137.stdout:2/437: getdents d2/d7/db/d74 0 2024-03-14T13:02:16.123 INFO:tasks.workunit.client.0.smithi111.stdout:5/517: chown d2/d16/d49/d7c/d95/d6a 31700713 1 2024-03-14T13:02:16.123 INFO:tasks.workunit.client.1.smithi137.stdout:8/366: write d1/dc/d11/d13/f22 [1346969,120336] 0 2024-03-14T13:02:16.125 INFO:tasks.workunit.client.1.smithi137.stdout:7/322: getdents d3/d6/d14 0 2024-03-14T13:02:16.126 INFO:tasks.workunit.client.0.smithi111.stdout:4/721: truncate d1d/d22/d40/d69/d86/db1/fbf 1002481 0 2024-03-14T13:02:16.127 INFO:tasks.workunit.client.1.smithi137.stdout:1/468: symlink d7/d9/d19/d1f/d26/l93 0 2024-03-14T13:02:16.127 INFO:tasks.workunit.client.1.smithi137.stdout:2/438: read d2/d7/db/f1c [308265,20189] 0 2024-03-14T13:02:16.128 INFO:tasks.workunit.client.1.smithi137.stdout:3/295: sync 2024-03-14T13:02:16.128 INFO:tasks.workunit.client.1.smithi137.stdout:5/311: sync 2024-03-14T13:02:16.128 INFO:tasks.workunit.client.1.smithi137.stdout:9/479: sync 2024-03-14T13:02:16.128 INFO:tasks.workunit.client.0.smithi111.stdout:8/583: rename d3/d1b/d20/d35/d40/dad to d3/d1b/d20/d35/d40/d48/d7e/db3 0 2024-03-14T13:02:16.128 INFO:tasks.workunit.client.0.smithi111.stdout:1/449: dwrite d20/d40/f4c [0,4194304] 0 2024-03-14T13:02:16.130 INFO:tasks.workunit.client.1.smithi137.stdout:7/323: creat d3/d5/f67 x:0 0 0 2024-03-14T13:02:16.131 INFO:tasks.workunit.client.1.smithi137.stdout:8/367: link d1/f31 d1/dc/d11/d13/f7c 0 2024-03-14T13:02:16.131 INFO:tasks.workunit.client.0.smithi111.stdout:5/518: fdatasync d2/d16/d49/d7c/f90 0 2024-03-14T13:02:16.131 INFO:tasks.workunit.client.0.smithi111.stdout:0/445: unlink d6/d9/de/d32/c2f 0 2024-03-14T13:02:16.133 INFO:tasks.workunit.client.1.smithi137.stdout:2/439: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d8e 0 2024-03-14T13:02:16.135 INFO:tasks.workunit.client.0.smithi111.stdout:9/463: rename d5/d6/d2a/d2d/l5e to d5/d6/d8/d5b/d87/l9c 0 2024-03-14T13:02:16.136 INFO:tasks.workunit.client.1.smithi137.stdout:5/312: write d7/d16/f68 [271962,111794] 0 2024-03-14T13:02:16.137 INFO:tasks.workunit.client.1.smithi137.stdout:9/480: write d6/df/d19/d3a/f43 [462498,19225] 0 2024-03-14T13:02:16.137 INFO:tasks.workunit.client.1.smithi137.stdout:8/368: dread - d1/dc/d11/d13/f67 zero size 2024-03-14T13:02:16.138 INFO:tasks.workunit.client.1.smithi137.stdout:1/469: dread d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:16.139 INFO:tasks.workunit.client.0.smithi111.stdout:1/450: write d20/d25/d31/f45 [639877,34797] 0 2024-03-14T13:02:16.140 INFO:tasks.workunit.client.0.smithi111.stdout:7/389: getdents d0/d1/d21/d5d 0 2024-03-14T13:02:16.142 INFO:tasks.workunit.client.1.smithi137.stdout:9/481: dread - d6/df/d19/d3a/f67 zero size 2024-03-14T13:02:16.143 INFO:tasks.workunit.client.0.smithi111.stdout:0/446: creat d6/d9/de/d3a/d46/d4a/f8f x:0 0 0 2024-03-14T13:02:16.143 INFO:tasks.workunit.client.1.smithi137.stdout:5/313: write d7/d16/d18/d25/f5b [126029,22377] 0 2024-03-14T13:02:16.143 INFO:tasks.workunit.client.1.smithi137.stdout:2/440: chown d2/d7/d2e/d3e/d44/d6b/d1d/d41/f6e 1 1 2024-03-14T13:02:16.146 INFO:tasks.workunit.client.0.smithi111.stdout:1/451: chown d20/d21/d3f/d59/f76 849678930 1 2024-03-14T13:02:16.147 INFO:tasks.workunit.client.1.smithi137.stdout:3/296: dwrite d4/d22/f4c [0,4194304] 0 2024-03-14T13:02:16.148 INFO:tasks.workunit.client.0.smithi111.stdout:5/519: creat d2/d16/d49/d7c/d95/d6a/fa8 x:0 0 0 2024-03-14T13:02:16.148 INFO:tasks.workunit.client.0.smithi111.stdout:4/722: dwrite d1d/d22/d23/fa2 [0,4194304] 0 2024-03-14T13:02:16.149 INFO:tasks.workunit.client.1.smithi137.stdout:9/482: creat d6/df/d19/d3a/d40/f9e x:0 0 0 2024-03-14T13:02:16.151 INFO:tasks.workunit.client.1.smithi137.stdout:5/314: write d7/d16/d18/d25/f5b [846167,102448] 0 2024-03-14T13:02:16.155 INFO:tasks.workunit.client.0.smithi111.stdout:1/452: dread - d20/d25/d31/f7f zero size 2024-03-14T13:02:16.156 INFO:tasks.workunit.client.0.smithi111.stdout:5/520: readlink d2/d16/d49/d33/l39 0 2024-03-14T13:02:16.156 INFO:tasks.workunit.client.0.smithi111.stdout:8/584: link d3/d1b/d20/d35/d40/d48/l5d d3/d1b/d20/d35/d40/d48/d77/lb4 0 2024-03-14T13:02:16.158 INFO:tasks.workunit.client.0.smithi111.stdout:5/521: readlink d2/d45/l69 0 2024-03-14T13:02:16.159 INFO:tasks.workunit.client.1.smithi137.stdout:8/369: rename d1/dc/d3c/d54/d34/l50 to d1/dc/d3c/d54/d34/l7d 0 2024-03-14T13:02:16.159 INFO:tasks.workunit.client.1.smithi137.stdout:2/441: creat d2/d7/d2e/d3e/d44/d6b/d22/f8f x:0 0 0 2024-03-14T13:02:16.160 INFO:tasks.workunit.client.1.smithi137.stdout:1/470: getdents d7/d9/d19/d1f/d89/d90 0 2024-03-14T13:02:16.160 INFO:tasks.workunit.client.1.smithi137.stdout:5/315: chown d7/f1b 8437415 1 2024-03-14T13:02:16.161 INFO:tasks.workunit.client.0.smithi111.stdout:6/419: write d0/f3e [22307,15259] 0 2024-03-14T13:02:16.163 INFO:tasks.workunit.client.0.smithi111.stdout:0/447: dwrite d6/d9/de/d3a/d46/f78 [0,4194304] 0 2024-03-14T13:02:16.164 INFO:tasks.workunit.client.1.smithi137.stdout:3/297: symlink d4/d19/d5c/l63 0 2024-03-14T13:02:16.164 INFO:tasks.workunit.client.1.smithi137.stdout:4/371: sync 2024-03-14T13:02:16.165 INFO:tasks.workunit.client.0.smithi111.stdout:9/464: getdents d5/d6/d8/d5b/d7c 0 2024-03-14T13:02:16.165 INFO:tasks.workunit.client.0.smithi111.stdout:7/390: link d0/d11/l35 d0/d1/d22/l77 0 2024-03-14T13:02:16.165 INFO:tasks.workunit.client.1.smithi137.stdout:9/483: symlink d6/df/d88/l9f 0 2024-03-14T13:02:16.165 INFO:tasks.workunit.client.0.smithi111.stdout:4/723: dread d1d/d22/d56/f7f [0,4194304] 0 2024-03-14T13:02:16.166 INFO:tasks.workunit.client.1.smithi137.stdout:2/442: truncate d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f8a 310293 0 2024-03-14T13:02:16.166 INFO:tasks.workunit.client.1.smithi137.stdout:6/387: sync 2024-03-14T13:02:16.166 INFO:tasks.workunit.client.1.smithi137.stdout:0/335: sync 2024-03-14T13:02:16.168 INFO:tasks.workunit.client.1.smithi137.stdout:8/370: mknod d1/dc/c7e 0 2024-03-14T13:02:16.168 INFO:tasks.workunit.client.1.smithi137.stdout:7/324: dread d3/f1c [0,4194304] 0 2024-03-14T13:02:16.169 INFO:tasks.workunit.client.0.smithi111.stdout:6/420: write d0/f98 [72828,40232] 0 2024-03-14T13:02:16.169 INFO:tasks.workunit.client.0.smithi111.stdout:5/522: mknod d2/ca9 0 2024-03-14T13:02:16.169 INFO:tasks.workunit.client.1.smithi137.stdout:9/484: chown d6/df/d19/d3a/l9d 3329104 1 2024-03-14T13:02:16.171 INFO:tasks.workunit.client.0.smithi111.stdout:7/391: dread - d0/d1/d4a/d52/f62 zero size 2024-03-14T13:02:16.172 INFO:tasks.workunit.client.0.smithi111.stdout:8/585: rename d3/d1b/d20/d83 to d3/d1b/d20/d35/d40/d48/d7e/db3/db5 0 2024-03-14T13:02:16.172 INFO:tasks.workunit.client.0.smithi111.stdout:9/465: symlink d5/d6/d14/l9d 0 2024-03-14T13:02:16.173 INFO:tasks.workunit.client.0.smithi111.stdout:4/724: mknod d1d/d48/cff 0 2024-03-14T13:02:16.173 INFO:tasks.workunit.client.1.smithi137.stdout:6/388: creat d2/d6/de/f8c x:0 0 0 2024-03-14T13:02:16.173 INFO:tasks.workunit.client.1.smithi137.stdout:7/325: fdatasync d3/d6/d14/f57 0 2024-03-14T13:02:16.174 INFO:tasks.workunit.client.1.smithi137.stdout:1/471: dread d7/d9/d19/d40/d42/f49 [0,4194304] 0 2024-03-14T13:02:16.175 INFO:tasks.workunit.client.1.smithi137.stdout:1/472: fsync d7/fb 0 2024-03-14T13:02:16.177 INFO:tasks.workunit.client.1.smithi137.stdout:9/485: read d6/f57 [574618,90460] 0 2024-03-14T13:02:16.177 INFO:tasks.workunit.client.0.smithi111.stdout:8/586: readlink d3/dc/lae 0 2024-03-14T13:02:16.177 INFO:tasks.workunit.client.0.smithi111.stdout:5/523: mknod d2/d45/d76/d8d/caa 0 2024-03-14T13:02:16.177 INFO:tasks.workunit.client.1.smithi137.stdout:0/336: symlink d1/d11/l75 0 2024-03-14T13:02:16.178 INFO:tasks.workunit.client.1.smithi137.stdout:3/298: rmdir d4/de/d4e 0 2024-03-14T13:02:16.179 INFO:tasks.workunit.client.0.smithi111.stdout:0/448: dread d6/f1a [0,4194304] 0 2024-03-14T13:02:16.180 INFO:tasks.workunit.client.1.smithi137.stdout:8/371: mkdir d1/dc/d11/d7f 0 2024-03-14T13:02:16.182 INFO:tasks.workunit.client.0.smithi111.stdout:0/449: write d6/d9/de/d77/f7c [164232,114023] 0 2024-03-14T13:02:16.183 INFO:tasks.workunit.client.0.smithi111.stdout:9/466: symlink d5/d6/d8/d74/l9e 0 2024-03-14T13:02:16.183 INFO:tasks.workunit.client.0.smithi111.stdout:7/392: symlink d0/d1/d5/d38/d5c/d6d/l78 0 2024-03-14T13:02:16.184 INFO:tasks.workunit.client.1.smithi137.stdout:7/326: truncate d3/d6/d2e/f3e 626045 0 2024-03-14T13:02:16.185 INFO:tasks.workunit.client.1.smithi137.stdout:1/473: truncate d7/d9/d19/d1f/f78 331077 0 2024-03-14T13:02:16.186 INFO:tasks.workunit.client.1.smithi137.stdout:6/389: dwrite d2/d6/de/f68 [0,4194304] 0 2024-03-14T13:02:16.187 INFO:tasks.workunit.client.0.smithi111.stdout:0/450: readlink d6/d9/d15/d49/d56/l67 0 2024-03-14T13:02:16.188 INFO:tasks.workunit.client.1.smithi137.stdout:2/443: getdents d2/d34/d3c/d53 0 2024-03-14T13:02:16.188 INFO:tasks.workunit.client.1.smithi137.stdout:9/486: chown d6/df/d19/d1f/c85 7211 1 2024-03-14T13:02:16.189 INFO:tasks.workunit.client.0.smithi111.stdout:5/524: rename d2/d16/d49/f5d to d2/d45/d76/fab 0 2024-03-14T13:02:16.190 INFO:tasks.workunit.client.0.smithi111.stdout:8/587: creat d3/d22/d4a/fb6 x:0 0 0 2024-03-14T13:02:16.190 INFO:tasks.workunit.client.1.smithi137.stdout:8/372: write d1/dc/d3c/d54/f1e [3957134,26768] 0 2024-03-14T13:02:16.190 INFO:tasks.workunit.client.1.smithi137.stdout:3/299: creat d4/d22/f64 x:0 0 0 2024-03-14T13:02:16.193 INFO:tasks.workunit.client.1.smithi137.stdout:2/444: write d2/f28 [401735,109979] 0 2024-03-14T13:02:16.194 INFO:tasks.workunit.client.1.smithi137.stdout:9/487: dread - d6/d53/f74 zero size 2024-03-14T13:02:16.194 INFO:tasks.workunit.client.1.smithi137.stdout:8/373: fsync d1/f60 0 2024-03-14T13:02:16.195 INFO:tasks.workunit.client.1.smithi137.stdout:7/327: mkdir d3/d5/d68 0 2024-03-14T13:02:16.195 INFO:tasks.workunit.client.0.smithi111.stdout:7/393: mknod d0/d1/d21/d3a/c79 0 2024-03-14T13:02:16.197 INFO:tasks.workunit.client.1.smithi137.stdout:5/316: link d7/l27 d7/d33/d65/l6d 0 2024-03-14T13:02:16.198 INFO:tasks.workunit.client.0.smithi111.stdout:7/394: read d0/f7 [144421,66682] 0 2024-03-14T13:02:16.198 INFO:tasks.workunit.client.1.smithi137.stdout:8/374: write d1/d5e/f61 [1617803,44639] 0 2024-03-14T13:02:16.198 INFO:tasks.workunit.client.1.smithi137.stdout:0/337: mkdir d1/d5/d1b/d1e/d25/d55/d73/d76 0 2024-03-14T13:02:16.200 INFO:tasks.workunit.client.0.smithi111.stdout:5/525: mkdir d2/d45/d79/dac 0 2024-03-14T13:02:16.202 INFO:tasks.workunit.client.1.smithi137.stdout:3/300: rename d4/de/f28 to d4/de/d16/d27/f65 0 2024-03-14T13:02:16.203 INFO:tasks.workunit.client.0.smithi111.stdout:9/467: dwrite d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:16.203 INFO:tasks.workunit.client.0.smithi111.stdout:7/395: truncate d0/d11/f73 644247 0 2024-03-14T13:02:16.206 INFO:tasks.workunit.client.1.smithi137.stdout:2/445: symlink d2/d7/db/l90 0 2024-03-14T13:02:16.207 INFO:tasks.workunit.client.0.smithi111.stdout:5/526: mknod d2/d16/d49/d7c/d8b/cad 0 2024-03-14T13:02:16.208 INFO:tasks.workunit.client.1.smithi137.stdout:0/338: write d1/d5/d1b/d42/f58 [605826,107272] 0 2024-03-14T13:02:16.208 INFO:tasks.workunit.client.1.smithi137.stdout:1/474: mknod d7/d31/d82/c94 0 2024-03-14T13:02:16.209 INFO:tasks.workunit.client.0.smithi111.stdout:7/396: symlink d0/d11/l7a 0 2024-03-14T13:02:16.211 INFO:tasks.workunit.client.0.smithi111.stdout:0/451: dwrite d6/d9/de/d32/f31 [0,4194304] 0 2024-03-14T13:02:16.214 INFO:tasks.workunit.client.1.smithi137.stdout:6/390: creat d2/d54/d55/f8d x:0 0 0 2024-03-14T13:02:16.216 INFO:tasks.workunit.client.0.smithi111.stdout:9/468: creat d5/d6/d2a/d5a/f9f x:0 0 0 2024-03-14T13:02:16.217 INFO:tasks.workunit.client.1.smithi137.stdout:3/301: dread d4/de/d16/d1c/f2d [0,4194304] 0 2024-03-14T13:02:16.218 INFO:tasks.workunit.client.1.smithi137.stdout:6/391: read - d2/d4c/f69 zero size 2024-03-14T13:02:16.218 INFO:tasks.workunit.client.1.smithi137.stdout:7/328: symlink d3/d5/d23/d3f/d5b/l69 0 2024-03-14T13:02:16.221 INFO:tasks.workunit.client.0.smithi111.stdout:4/725: dwrite d1d/f20 [4194304,4194304] 0 2024-03-14T13:02:16.227 INFO:tasks.workunit.client.1.smithi137.stdout:8/375: link d1/d4a/c4d d1/dc/d10/c80 0 2024-03-14T13:02:16.229 INFO:tasks.workunit.client.0.smithi111.stdout:1/453: dread f19 [0,4194304] 0 2024-03-14T13:02:16.229 INFO:tasks.workunit.client.1.smithi137.stdout:4/372: dwrite d4/d27/d5c/d6d/d58/d65/f7b [0,4194304] 0 2024-03-14T13:02:16.231 INFO:tasks.workunit.client.0.smithi111.stdout:8/588: dwrite d3/dc/f11 [4194304,4194304] 0 2024-03-14T13:02:16.231 INFO:tasks.workunit.client.0.smithi111.stdout:7/397: creat d0/d1/d22/f7b x:0 0 0 2024-03-14T13:02:16.231 INFO:tasks.workunit.client.0.smithi111.stdout:6/421: dwrite d0/d17/d20/d4a/d59/f6f [0,4194304] 0 2024-03-14T13:02:16.231 INFO:tasks.workunit.client.0.smithi111.stdout:0/452: creat d6/f90 x:0 0 0 2024-03-14T13:02:16.233 INFO:tasks.workunit.client.1.smithi137.stdout:9/488: rename d6/d26/f86 to d6/df/fa0 0 2024-03-14T13:02:16.235 INFO:tasks.workunit.client.0.smithi111.stdout:5/527: truncate d2/d13/f2f 735210 0 2024-03-14T13:02:16.236 INFO:tasks.workunit.client.0.smithi111.stdout:9/469: symlink d5/d6/d2a/d2d/la0 0 2024-03-14T13:02:16.239 INFO:tasks.workunit.client.1.smithi137.stdout:6/392: chown d2/d6/de/d45/d59/f79 1355814 1 2024-03-14T13:02:16.239 INFO:tasks.workunit.client.0.smithi111.stdout:6/422: write d0/d17/d20/d65/f92 [979624,63787] 0 2024-03-14T13:02:16.240 INFO:tasks.workunit.client.0.smithi111.stdout:1/454: chown d20/l26 129854 1 2024-03-14T13:02:16.240 INFO:tasks.workunit.client.1.smithi137.stdout:0/339: dwrite d1/f46 [0,4194304] 0 2024-03-14T13:02:16.241 INFO:tasks.workunit.client.0.smithi111.stdout:7/398: creat d0/d1/d5/d56/f7c x:0 0 0 2024-03-14T13:02:16.241 INFO:tasks.workunit.client.1.smithi137.stdout:8/376: fsync d1/dc/d10/f32 0 2024-03-14T13:02:16.242 INFO:tasks.workunit.client.0.smithi111.stdout:9/470: unlink d5/d6/f7e 0 2024-03-14T13:02:16.242 INFO:tasks.workunit.client.1.smithi137.stdout:5/317: truncate d7/d16/d18/d25/d28/f43 5360847 0 2024-03-14T13:02:16.242 INFO:tasks.workunit.client.0.smithi111.stdout:8/589: creat d3/dc/d45/fb7 x:0 0 0 2024-03-14T13:02:16.243 INFO:tasks.workunit.client.0.smithi111.stdout:1/455: fdatasync d20/d25/d31/f3b 0 2024-03-14T13:02:16.245 INFO:tasks.workunit.client.1.smithi137.stdout:7/329: mkdir d3/d5/d23/d2c/d5d/d6a 0 2024-03-14T13:02:16.246 INFO:tasks.workunit.client.0.smithi111.stdout:7/399: dread - d0/d1/d5/f4c zero size 2024-03-14T13:02:16.246 INFO:tasks.workunit.client.0.smithi111.stdout:0/453: fsync d6/fd 0 2024-03-14T13:02:16.248 INFO:tasks.workunit.client.1.smithi137.stdout:4/373: mknod d4/d10/d42/c82 0 2024-03-14T13:02:16.248 INFO:tasks.workunit.client.0.smithi111.stdout:5/528: write d2/d16/d49/d7c/d95/d36/f3c [664060,108039] 0 2024-03-14T13:02:16.250 INFO:tasks.workunit.client.0.smithi111.stdout:6/423: mkdir d0/d17/d21/d69/d3d/d53/d81/da1 0 2024-03-14T13:02:16.250 INFO:tasks.workunit.client.0.smithi111.stdout:8/590: symlink d3/dc/d45/lb8 0 2024-03-14T13:02:16.250 INFO:tasks.workunit.client.0.smithi111.stdout:7/400: fsync d0/d1/d22/f42 0 2024-03-14T13:02:16.252 INFO:tasks.workunit.client.0.smithi111.stdout:8/591: dread - d3/d22/d30/f95 zero size 2024-03-14T13:02:16.254 INFO:tasks.workunit.client.1.smithi137.stdout:2/446: dwrite d2/f14 [0,4194304] 0 2024-03-14T13:02:16.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:15 smithi137 ceph-mon[50336]: pgmap v22: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 21 MiB/s rd, 169 MiB/s wr, 264 op/s 2024-03-14T13:02:16.256 INFO:tasks.workunit.client.0.smithi111.stdout:5/529: creat d2/d45/d76/d8d/fae x:0 0 0 2024-03-14T13:02:16.256 INFO:tasks.workunit.client.0.smithi111.stdout:0/454: creat d6/d9/de/d3a/d46/d4a/f91 x:0 0 0 2024-03-14T13:02:16.257 INFO:tasks.workunit.client.1.smithi137.stdout:7/330: dread d3/d5/f46 [0,4194304] 0 2024-03-14T13:02:16.257 INFO:tasks.workunit.client.1.smithi137.stdout:9/489: rmdir d6/df/d19 39 2024-03-14T13:02:16.258 INFO:tasks.workunit.client.1.smithi137.stdout:8/377: rmdir d1/dc/d77 39 2024-03-14T13:02:16.258 INFO:tasks.workunit.client.1.smithi137.stdout:1/475: dwrite d7/d9/d19/d1f/d22/f45 [0,4194304] 0 2024-03-14T13:02:16.258 INFO:tasks.workunit.client.1.smithi137.stdout:5/318: mknod d7/d16/c6e 0 2024-03-14T13:02:16.260 INFO:tasks.workunit.client.0.smithi111.stdout:1/456: rename d20/d40/f4c to d20/d21/d3f/fa3 0 2024-03-14T13:02:16.261 INFO:tasks.workunit.client.0.smithi111.stdout:6/424: creat d0/d17/d20/d5a/fa2 x:0 0 0 2024-03-14T13:02:16.261 INFO:tasks.workunit.client.0.smithi111.stdout:7/401: creat d0/d1/d21/d2a/f7d x:0 0 0 2024-03-14T13:02:16.261 INFO:tasks.workunit.client.1.smithi137.stdout:0/340: creat d1/d5/d1b/d42/f77 x:0 0 0 2024-03-14T13:02:16.261 INFO:tasks.workunit.client.1.smithi137.stdout:7/331: chown d3/d6/d59/f5f 226 1 2024-03-14T13:02:16.262 INFO:tasks.workunit.client.1.smithi137.stdout:3/302: dwrite d4/de/d16/f1a [4194304,4194304] 0 2024-03-14T13:02:16.263 INFO:tasks.workunit.client.0.smithi111.stdout:0/455: chown d6/d9/de/f27 98 1 2024-03-14T13:02:16.264 INFO:tasks.workunit.client.0.smithi111.stdout:8/592: fsync d3/f6d 0 2024-03-14T13:02:16.264 INFO:tasks.workunit.client.0.smithi111.stdout:7/402: write d0/d1/d5/d38/f68 [1929916,54763] 0 2024-03-14T13:02:16.267 INFO:tasks.workunit.client.0.smithi111.stdout:6/425: write d0/d17/d20/f26 [3798753,105537] 0 2024-03-14T13:02:16.268 INFO:tasks.workunit.client.0.smithi111.stdout:0/456: symlink d6/d9/d15/d49/l92 0 2024-03-14T13:02:16.268 INFO:tasks.workunit.client.0.smithi111.stdout:8/593: chown d3/d5/l93 0 1 2024-03-14T13:02:16.269 INFO:tasks.workunit.client.1.smithi137.stdout:7/332: chown d3/d5/d23/d2c/d5d 1225 1 2024-03-14T13:02:16.269 INFO:tasks.workunit.client.1.smithi137.stdout:4/374: mknod d4/d9/d3a/d70/c83 0 2024-03-14T13:02:16.269 INFO:tasks.workunit.client.1.smithi137.stdout:6/393: rmdir d2/d54/d55/d38 39 2024-03-14T13:02:16.269 INFO:tasks.workunit.client.1.smithi137.stdout:7/333: readlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:16.271 INFO:tasks.workunit.client.0.smithi111.stdout:8/594: truncate d3/dc/fa7 332253 0 2024-03-14T13:02:16.272 INFO:tasks.workunit.client.0.smithi111.stdout:5/530: dwrite d2/d13/f6c [0,4194304] 0 2024-03-14T13:02:16.275 INFO:tasks.workunit.client.0.smithi111.stdout:1/457: link d20/l27 d20/d21/la4 0 2024-03-14T13:02:16.275 INFO:tasks.workunit.client.0.smithi111.stdout:0/457: creat d6/d3d/f93 x:0 0 0 2024-03-14T13:02:16.276 INFO:tasks.workunit.client.1.smithi137.stdout:9/490: fdatasync d6/df/d19/d1f/f2a 0 2024-03-14T13:02:16.277 INFO:tasks.workunit.client.1.smithi137.stdout:3/303: creat d4/d19/f66 x:0 0 0 2024-03-14T13:02:16.278 INFO:tasks.workunit.client.1.smithi137.stdout:2/447: link d2/d7/db/d2f/d66/f7a d2/d7/d2e/d3e/d44/d6b/d22/f91 0 2024-03-14T13:02:16.278 INFO:tasks.workunit.client.0.smithi111.stdout:1/458: truncate d20/d25/d31/d80/f92 87872 0 2024-03-14T13:02:16.279 INFO:tasks.workunit.client.0.smithi111.stdout:1/459: fdatasync d20/d25/f3d 0 2024-03-14T13:02:16.279 INFO:tasks.workunit.client.1.smithi137.stdout:9/491: chown d6/df/d19/d3a/d40 6893 1 2024-03-14T13:02:16.280 INFO:tasks.workunit.client.1.smithi137.stdout:8/378: mknod d1/dc/d77/c81 0 2024-03-14T13:02:16.281 INFO:tasks.workunit.client.0.smithi111.stdout:7/403: rename d0/d1/d5/d38/d9 to d0/d7e 0 2024-03-14T13:02:16.282 INFO:tasks.workunit.client.1.smithi137.stdout:4/375: write d4/d10/d42/f52 [652531,106668] 0 2024-03-14T13:02:16.282 INFO:tasks.workunit.client.1.smithi137.stdout:4/376: stat d4/l57 0 2024-03-14T13:02:16.283 INFO:tasks.workunit.client.1.smithi137.stdout:5/319: rename d7/d16/d18/d2d/f2f to d7/d16/d18/d25/d28/d3c/f6f 0 2024-03-14T13:02:16.284 INFO:tasks.workunit.client.0.smithi111.stdout:6/426: dwrite d0/f57 [0,4194304] 0 2024-03-14T13:02:16.284 INFO:tasks.workunit.client.1.smithi137.stdout:3/304: fdatasync f1 0 2024-03-14T13:02:16.286 INFO:tasks.workunit.client.0.smithi111.stdout:1/460: dread d20/d25/f38 [0,4194304] 0 2024-03-14T13:02:16.287 INFO:tasks.workunit.client.1.smithi137.stdout:9/492: unlink d6/df/f5e 0 2024-03-14T13:02:16.288 INFO:tasks.workunit.client.0.smithi111.stdout:0/458: rename d6/f1a to d6/d3d/f94 0 2024-03-14T13:02:16.289 INFO:tasks.workunit.client.0.smithi111.stdout:5/531: dread d2/d45/f5c [0,4194304] 0 2024-03-14T13:02:16.290 INFO:tasks.workunit.client.0.smithi111.stdout:5/532: chown d2/d16/d49/cf 0 1 2024-03-14T13:02:16.293 INFO:tasks.workunit.client.0.smithi111.stdout:0/459: rename d6/d9/f12 to d6/d9/de/d3a/f95 0 2024-03-14T13:02:16.293 INFO:tasks.workunit.client.0.smithi111.stdout:5/533: creat d2/d16/d49/d44/d4a/faf x:0 0 0 2024-03-14T13:02:16.295 INFO:tasks.workunit.client.0.smithi111.stdout:7/404: creat d0/d1/d5/d38/f7f x:0 0 0 2024-03-14T13:02:16.296 INFO:tasks.workunit.client.1.smithi137.stdout:5/320: dwrite d7/d16/f1a [4194304,4194304] 0 2024-03-14T13:02:16.296 INFO:tasks.workunit.client.0.smithi111.stdout:8/595: link d3/d22/d4a/ca3 d3/d22/d4a/cb9 0 2024-03-14T13:02:16.297 INFO:tasks.workunit.client.0.smithi111.stdout:0/460: rename d6/d9 to d6/d9/de/d96 22 2024-03-14T13:02:16.297 INFO:tasks.workunit.client.1.smithi137.stdout:8/379: unlink d1/f60 0 2024-03-14T13:02:16.298 INFO:tasks.workunit.client.0.smithi111.stdout:5/534: write d2/d16/d49/d7c/d95/fa3 [1072064,77176] 0 2024-03-14T13:02:16.300 INFO:tasks.workunit.client.0.smithi111.stdout:8/596: truncate d3/d1b/d20/d35/d40/faa 461758 0 2024-03-14T13:02:16.300 INFO:tasks.workunit.client.1.smithi137.stdout:1/476: creat d7/d9/d19/d1f/f95 x:0 0 0 2024-03-14T13:02:16.301 INFO:tasks.workunit.client.1.smithi137.stdout:6/394: rename d2/d6/l23 to d2/d54/d55/d6d/d76/l8e 0 2024-03-14T13:02:16.301 INFO:tasks.workunit.client.1.smithi137.stdout:5/321: write d7/d16/d18/d25/f5b [1838504,126462] 0 2024-03-14T13:02:16.302 INFO:tasks.workunit.client.1.smithi137.stdout:9/493: dwrite d6/df/d19/d1f/f4a [0,4194304] 0 2024-03-14T13:02:16.302 INFO:tasks.workunit.client.1.smithi137.stdout:3/305: symlink d4/de/d16/d1c/d5d/l67 0 2024-03-14T13:02:16.303 INFO:tasks.workunit.client.0.smithi111.stdout:5/535: mknod d2/d16/d49/d7c/d8b/cb0 0 2024-03-14T13:02:16.305 INFO:tasks.workunit.client.0.smithi111.stdout:7/405: dread d0/d1/d22/f42 [0,4194304] 0 2024-03-14T13:02:16.306 INFO:tasks.workunit.client.0.smithi111.stdout:0/461: dwrite d6/d9/de/f18 [0,4194304] 0 2024-03-14T13:02:16.307 INFO:tasks.workunit.client.1.smithi137.stdout:6/395: fsync d2/d54/d55/d6d/f87 0 2024-03-14T13:02:16.307 INFO:tasks.workunit.client.0.smithi111.stdout:8/597: rmdir d3/d1b/d20/d35/d40/d48/daf 0 2024-03-14T13:02:16.310 INFO:tasks.workunit.client.1.smithi137.stdout:2/448: getdents d2/d7/d2e/d3e/d44 0 2024-03-14T13:02:16.313 INFO:tasks.workunit.client.1.smithi137.stdout:2/449: stat d2/f4 0 2024-03-14T13:02:16.313 INFO:tasks.workunit.client.1.smithi137.stdout:6/396: write d2/f25 [4943307,90704] 0 2024-03-14T13:02:16.313 INFO:tasks.workunit.client.1.smithi137.stdout:9/494: symlink d6/df/d19/d3a/la1 0 2024-03-14T13:02:16.315 INFO:tasks.workunit.client.0.smithi111.stdout:0/462: rmdir d6/d9/de/d77 39 2024-03-14T13:02:16.317 INFO:tasks.workunit.client.1.smithi137.stdout:6/397: stat d2/d54/l6f 0 2024-03-14T13:02:16.318 INFO:tasks.workunit.client.0.smithi111.stdout:5/536: dwrite d2/d16/d49/d44/d4a/f63 [0,4194304] 0 2024-03-14T13:02:16.320 INFO:tasks.workunit.client.0.smithi111.stdout:8/598: rename d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f63 to d3/d1b/d20/d35/d40/d48/d7e/fba 0 2024-03-14T13:02:16.321 INFO:tasks.workunit.client.1.smithi137.stdout:3/306: dwrite d4/de/d16/f1a [4194304,4194304] 0 2024-03-14T13:02:16.322 INFO:tasks.workunit.client.1.smithi137.stdout:9/495: chown d6/df/d19/d33/f65 287914 1 2024-03-14T13:02:16.322 INFO:tasks.workunit.client.0.smithi111.stdout:5/537: dread d2/d45/f5c [0,4194304] 0 2024-03-14T13:02:16.324 INFO:tasks.workunit.client.1.smithi137.stdout:2/450: symlink d2/d7/d2e/d3e/d44/d6b/d1d/d2d/l92 0 2024-03-14T13:02:16.324 INFO:tasks.workunit.client.1.smithi137.stdout:8/380: dwrite d1/dc/d11/d13/f3e [0,4194304] 0 2024-03-14T13:02:16.327 INFO:tasks.workunit.client.1.smithi137.stdout:6/398: creat d2/d54/d55/d6d/d76/f8f x:0 0 0 2024-03-14T13:02:16.328 INFO:tasks.workunit.client.1.smithi137.stdout:3/307: fdatasync d4/de/f3c 0 2024-03-14T13:02:16.330 INFO:tasks.workunit.client.1.smithi137.stdout:2/451: readlink d2/d7/d2e/d3e/d44/d6b/l7d 0 2024-03-14T13:02:16.333 INFO:tasks.workunit.client.1.smithi137.stdout:8/381: chown d1/dc/l57 3 1 2024-03-14T13:02:16.333 INFO:tasks.workunit.client.1.smithi137.stdout:0/341: dread d1/d5/d1b/d42/d3a/f48 [0,4194304] 0 2024-03-14T13:02:16.334 INFO:tasks.workunit.client.1.smithi137.stdout:9/496: mknod d6/df/d19/d33/ca2 0 2024-03-14T13:02:16.334 INFO:tasks.workunit.client.0.smithi111.stdout:5/538: fsync d2/d16/d49/d7c/d95/d6a/fa8 0 2024-03-14T13:02:16.335 INFO:tasks.workunit.client.1.smithi137.stdout:8/382: write d1/dc/f4b [1397489,58422] 0 2024-03-14T13:02:16.335 INFO:tasks.workunit.client.0.smithi111.stdout:6/427: read d0/d17/d20/f26 [884373,8505] 0 2024-03-14T13:02:16.336 INFO:tasks.workunit.client.0.smithi111.stdout:0/463: mkdir d6/d3d/d97 0 2024-03-14T13:02:16.336 INFO:tasks.workunit.client.1.smithi137.stdout:6/399: dread - d2/d54/d55/d6d/d76/d57/f81 zero size 2024-03-14T13:02:16.338 INFO:tasks.workunit.client.1.smithi137.stdout:2/452: unlink d2/l6 0 2024-03-14T13:02:16.339 INFO:tasks.workunit.client.0.smithi111.stdout:0/464: fdatasync d6/f90 0 2024-03-14T13:02:16.339 INFO:tasks.workunit.client.0.smithi111.stdout:8/599: dwrite d3/d22/f88 [0,4194304] 0 2024-03-14T13:02:16.341 INFO:tasks.workunit.client.1.smithi137.stdout:0/342: fsync d1/f1c 0 2024-03-14T13:02:16.342 INFO:tasks.workunit.client.1.smithi137.stdout:7/334: truncate d3/d5/f11 2377383 0 2024-03-14T13:02:16.346 INFO:tasks.workunit.client.0.smithi111.stdout:6/428: mknod d0/d50/ca3 0 2024-03-14T13:02:16.347 INFO:tasks.workunit.client.1.smithi137.stdout:0/343: write d1/d5/d1b/d1e/d25/d26/f41 [511832,69300] 0 2024-03-14T13:02:16.347 INFO:tasks.workunit.client.1.smithi137.stdout:9/497: dread d6/df/d19/d3a/d40/f4b [0,4194304] 0 2024-03-14T13:02:16.349 INFO:tasks.workunit.client.1.smithi137.stdout:8/383: mkdir d1/dc/d3c/d54/d34/d82 0 2024-03-14T13:02:16.349 INFO:tasks.workunit.client.0.smithi111.stdout:8/600: creat d3/d22/fbb x:0 0 0 2024-03-14T13:02:16.351 INFO:tasks.workunit.client.0.smithi111.stdout:0/465: creat d6/d9/de/d3a/d46/d4a/f98 x:0 0 0 2024-03-14T13:02:16.353 INFO:tasks.workunit.client.1.smithi137.stdout:6/400: creat d2/d54/d55/d6d/f90 x:0 0 0 2024-03-14T13:02:16.354 INFO:tasks.workunit.client.1.smithi137.stdout:2/453: creat d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f93 x:0 0 0 2024-03-14T13:02:16.354 INFO:tasks.workunit.client.1.smithi137.stdout:7/335: fdatasync d3/f18 0 2024-03-14T13:02:16.355 INFO:tasks.workunit.client.0.smithi111.stdout:9/471: dwrite f2 [0,4194304] 0 2024-03-14T13:02:16.355 INFO:tasks.workunit.client.0.smithi111.stdout:0/466: stat d6/d9/d15/d49/f62 0 2024-03-14T13:02:16.357 INFO:tasks.workunit.client.0.smithi111.stdout:1/461: dwrite f19 [0,4194304] 0 2024-03-14T13:02:16.357 INFO:tasks.workunit.client.0.smithi111.stdout:6/429: dread d0/f35 [0,4194304] 0 2024-03-14T13:02:16.359 INFO:tasks.workunit.client.1.smithi137.stdout:3/308: rmdir d4/df/d60 0 2024-03-14T13:02:16.359 INFO:tasks.workunit.client.0.smithi111.stdout:1/462: fdatasync d20/d25/f9f 0 2024-03-14T13:02:16.362 INFO:tasks.workunit.client.0.smithi111.stdout:1/463: write d20/d25/f64 [4845957,95282] 0 2024-03-14T13:02:16.362 INFO:tasks.workunit.client.0.smithi111.stdout:0/467: creat d6/d21/d74/f99 x:0 0 0 2024-03-14T13:02:16.363 INFO:tasks.workunit.client.0.smithi111.stdout:3/459: sync 2024-03-14T13:02:16.365 INFO:tasks.workunit.client.1.smithi137.stdout:5/322: dwrite d7/d16/d18/f66 [0,4194304] 0 2024-03-14T13:02:16.366 INFO:tasks.workunit.client.0.smithi111.stdout:6/430: creat d0/d17/d21/d69/d3d/d53/d81/fa4 x:0 0 0 2024-03-14T13:02:16.368 INFO:tasks.workunit.client.0.smithi111.stdout:5/539: link d2/l85 d2/d16/d49/d44/d72/lb1 0 2024-03-14T13:02:16.368 INFO:tasks.workunit.client.1.smithi137.stdout:9/498: dread d6/df/d19/d3a/f47 [4194304,4194304] 0 2024-03-14T13:02:16.371 INFO:tasks.workunit.client.1.smithi137.stdout:9/499: stat d6/df/d19/d1f/f24 0 2024-03-14T13:02:16.372 INFO:tasks.workunit.client.1.smithi137.stdout:7/336: write d3/d6/d14/d22/f43 [1781630,98066] 0 2024-03-14T13:02:16.374 INFO:tasks.workunit.client.0.smithi111.stdout:0/468: mknod d6/d9/d15/c9a 0 2024-03-14T13:02:16.375 INFO:tasks.workunit.client.0.smithi111.stdout:1/464: symlink d20/d25/d87/la5 0 2024-03-14T13:02:16.376 INFO:tasks.workunit.client.0.smithi111.stdout:8/601: getdents d3/dc/d31 0 2024-03-14T13:02:16.376 INFO:tasks.workunit.client.1.smithi137.stdout:5/323: rename d7/d16/d41/c54 to d7/d16/c70 0 2024-03-14T13:02:16.377 INFO:tasks.workunit.client.1.smithi137.stdout:8/384: creat d1/f83 x:0 0 0 2024-03-14T13:02:16.378 INFO:tasks.workunit.client.0.smithi111.stdout:1/465: readlink d20/d25/d87/l8d 0 2024-03-14T13:02:16.379 INFO:tasks.workunit.client.0.smithi111.stdout:6/431: mkdir d0/d17/d21/d42/da5 0 2024-03-14T13:02:16.383 INFO:tasks.workunit.client.0.smithi111.stdout:8/602: stat d3/d1b/d20/d35/d40/d48/d77/l8c 0 2024-03-14T13:02:16.383 INFO:tasks.workunit.client.0.smithi111.stdout:1/466: readlink d20/d25/d31/l8e 0 2024-03-14T13:02:16.383 INFO:tasks.workunit.client.0.smithi111.stdout:5/540: chown d2/d16/l41 146 1 2024-03-14T13:02:16.384 INFO:tasks.workunit.client.1.smithi137.stdout:3/309: creat d4/f68 x:0 0 0 2024-03-14T13:02:16.385 INFO:tasks.workunit.client.0.smithi111.stdout:8/603: write d3/d1b/d20/d35/fa0 [359754,89808] 0 2024-03-14T13:02:16.386 INFO:tasks.workunit.client.1.smithi137.stdout:5/324: dread d7/d16/d18/d25/d28/f3d [0,4194304] 0 2024-03-14T13:02:16.388 INFO:tasks.workunit.client.1.smithi137.stdout:3/310: unlink d4/f58 0 2024-03-14T13:02:16.388 INFO:tasks.workunit.client.0.smithi111.stdout:0/469: dwrite d6/d3d/f52 [0,4194304] 0 2024-03-14T13:02:16.389 INFO:tasks.workunit.client.1.smithi137.stdout:8/385: dwrite d1/dc/f5f [0,4194304] 0 2024-03-14T13:02:16.389 INFO:tasks.workunit.client.0.smithi111.stdout:1/467: mkdir d20/d40/da6 0 2024-03-14T13:02:16.390 INFO:tasks.workunit.client.0.smithi111.stdout:1/468: stat d20/d25/f39 0 2024-03-14T13:02:16.391 INFO:tasks.workunit.client.0.smithi111.stdout:8/604: write d3/d1b/d20/d35/fa0 [202351,37491] 0 2024-03-14T13:02:16.391 INFO:tasks.workunit.client.0.smithi111.stdout:0/470: chown d6/d21/d74 559088 1 2024-03-14T13:02:16.391 INFO:tasks.workunit.client.1.smithi137.stdout:9/500: dread d6/df/d19/d3a/f43 [0,4194304] 0 2024-03-14T13:02:16.393 INFO:tasks.workunit.client.0.smithi111.stdout:6/432: dwrite d0/ff [0,4194304] 0 2024-03-14T13:02:16.396 INFO:tasks.workunit.client.1.smithi137.stdout:5/325: creat d7/d36/f71 x:0 0 0 2024-03-14T13:02:16.397 INFO:tasks.workunit.client.1.smithi137.stdout:8/386: unlink d1/dc/d11/c2b 0 2024-03-14T13:02:16.397 INFO:tasks.workunit.client.0.smithi111.stdout:6/433: write d0/d17/d21/f44 [2319242,82062] 0 2024-03-14T13:02:16.399 INFO:tasks.workunit.client.0.smithi111.stdout:8/605: rename d3/dc/l42 to d3/d1b/d20/lbc 0 2024-03-14T13:02:16.402 INFO:tasks.workunit.client.1.smithi137.stdout:0/344: dread d1/d44/d56/f5a [0,4194304] 0 2024-03-14T13:02:16.403 INFO:tasks.workunit.client.1.smithi137.stdout:6/401: truncate d2/f2d 27732 0 2024-03-14T13:02:16.404 INFO:tasks.workunit.client.1.smithi137.stdout:5/326: mkdir d7/d16/d72 0 2024-03-14T13:02:16.405 INFO:tasks.workunit.client.1.smithi137.stdout:8/387: creat d1/dc/d11/d13/f84 x:0 0 0 2024-03-14T13:02:16.406 INFO:tasks.workunit.client.1.smithi137.stdout:0/345: chown d1/d5/d1b/d1e/d25/d26 85210981 1 2024-03-14T13:02:16.407 INFO:tasks.workunit.client.0.smithi111.stdout:0/471: write d6/d9/de/d3a/d46/f66 [1043868,130056] 0 2024-03-14T13:02:16.408 INFO:tasks.workunit.client.1.smithi137.stdout:9/501: creat d6/fa3 x:0 0 0 2024-03-14T13:02:16.409 INFO:tasks.workunit.client.1.smithi137.stdout:8/388: dread - d1/d5e/f63 zero size 2024-03-14T13:02:16.409 INFO:tasks.workunit.client.1.smithi137.stdout:2/454: dwrite d2/d7/db/f2c [0,4194304] 0 2024-03-14T13:02:16.412 INFO:tasks.workunit.client.0.smithi111.stdout:1/469: dwrite d20/d25/d75/f6a [0,4194304] 0 2024-03-14T13:02:16.412 INFO:tasks.workunit.client.0.smithi111.stdout:3/460: truncate d0/d2d/d35/d50/f54 323967 0 2024-03-14T13:02:16.414 INFO:tasks.workunit.client.1.smithi137.stdout:5/327: mknod d7/d16/d18/c73 0 2024-03-14T13:02:16.417 INFO:tasks.workunit.client.1.smithi137.stdout:9/502: rename d6/f1d to d6/df/d19/d1f/fa4 0 2024-03-14T13:02:16.417 INFO:tasks.workunit.client.0.smithi111.stdout:5/541: dwrite d2/f6 [0,4194304] 0 2024-03-14T13:02:16.417 INFO:tasks.workunit.client.0.smithi111.stdout:8/606: write d3/d1b/d20/d35/d40/d48/d7e/fba [4181720,68647] 0 2024-03-14T13:02:16.418 INFO:tasks.workunit.client.1.smithi137.stdout:8/389: unlink d1/dc/d3c/d54/c23 0 2024-03-14T13:02:16.419 INFO:tasks.workunit.client.0.smithi111.stdout:0/472: rename d6/d3d/f93 to d6/d9/de/d3a/f9b 0 2024-03-14T13:02:16.420 INFO:tasks.workunit.client.1.smithi137.stdout:0/346: creat d1/d5/d1b/d1e/d25/d55/d73/d76/f78 x:0 0 0 2024-03-14T13:02:16.424 INFO:tasks.workunit.client.1.smithi137.stdout:5/328: mkdir d7/d1d/d74 0 2024-03-14T13:02:16.424 INFO:tasks.workunit.client.1.smithi137.stdout:6/402: getdents d2/d54/d55/d6d/d76/d57 0 2024-03-14T13:02:16.424 INFO:tasks.workunit.client.1.smithi137.stdout:9/503: fdatasync d6/df/f13 0 2024-03-14T13:02:16.425 INFO:tasks.workunit.client.0.smithi111.stdout:1/470: mkdir d20/d21/da7 0 2024-03-14T13:02:16.426 INFO:tasks.workunit.client.1.smithi137.stdout:0/347: write d1/d44/d56/f5a [858439,37546] 0 2024-03-14T13:02:16.428 INFO:tasks.workunit.client.1.smithi137.stdout:8/390: rename d1/dc/d10/l36 to d1/dc/d11/d7f/l85 0 2024-03-14T13:02:16.428 INFO:tasks.workunit.client.0.smithi111.stdout:0/473: creat d6/d9/de/d3a/f9c x:0 0 0 2024-03-14T13:02:16.429 INFO:tasks.workunit.client.1.smithi137.stdout:0/348: mkdir d1/d44/d79 0 2024-03-14T13:02:16.429 INFO:tasks.workunit.client.1.smithi137.stdout:2/455: getdents d2/d7/db/d2f 0 2024-03-14T13:02:16.429 INFO:tasks.workunit.client.1.smithi137.stdout:5/329: write d7/d16/d18/d25/f46 [4637617,41387] 0 2024-03-14T13:02:16.431 INFO:tasks.workunit.client.1.smithi137.stdout:5/330: chown d7/d16/d18/d2d/f63 92 1 2024-03-14T13:02:16.433 INFO:tasks.workunit.client.1.smithi137.stdout:0/349: readlink d1/d1f/l33 0 2024-03-14T13:02:16.433 INFO:tasks.workunit.client.1.smithi137.stdout:2/456: chown d2/d7/db/d74 448724 1 2024-03-14T13:02:16.434 INFO:tasks.workunit.client.0.smithi111.stdout:6/434: dwrite d0/d17/d20/f26 [0,4194304] 0 2024-03-14T13:02:16.434 INFO:tasks.workunit.client.1.smithi137.stdout:6/403: symlink d2/d6/de/d45/d59/d39/d3c/l91 0 2024-03-14T13:02:16.434 INFO:tasks.workunit.client.0.smithi111.stdout:8/607: write d3/d1b/d20/d35/fb2 [204072,93753] 0 2024-03-14T13:02:16.435 INFO:tasks.workunit.client.1.smithi137.stdout:0/350: write d1/d11/f50 [4542124,8483] 0 2024-03-14T13:02:16.435 INFO:tasks.workunit.client.1.smithi137.stdout:8/391: mkdir d1/dc/d3c/d54/d34/d86 0 2024-03-14T13:02:16.435 INFO:tasks.workunit.client.1.smithi137.stdout:8/392: stat d1 0 2024-03-14T13:02:16.436 INFO:tasks.workunit.client.1.smithi137.stdout:5/331: rmdir d7/d16/d41 39 2024-03-14T13:02:16.437 INFO:tasks.workunit.client.0.smithi111.stdout:3/461: rename d0/d2d/d35/d50/da2/db3/l41 to d0/d2d/d35/d4c/dad/lb6 0 2024-03-14T13:02:16.441 INFO:tasks.workunit.client.1.smithi137.stdout:2/457: fsync d2/d7/db/d2f/d66/f73 0 2024-03-14T13:02:16.441 INFO:tasks.workunit.client.1.smithi137.stdout:6/404: creat d2/d6/de/d45/d59/d39/d3c/f92 x:0 0 0 2024-03-14T13:02:16.442 INFO:tasks.workunit.client.0.smithi111.stdout:0/474: mknod d6/d9/de/d3a/c9d 0 2024-03-14T13:02:16.442 INFO:tasks.workunit.client.1.smithi137.stdout:2/458: chown d2/d7/l15 36 1 2024-03-14T13:02:16.442 INFO:tasks.workunit.client.0.smithi111.stdout:3/462: chown d0/d30/f57 526628200 1 2024-03-14T13:02:16.443 INFO:tasks.workunit.client.1.smithi137.stdout:5/332: mkdir d7/d49/d75 0 2024-03-14T13:02:16.444 INFO:tasks.workunit.client.1.smithi137.stdout:6/405: write d2/d6/de/d45/f5b [382705,28975] 0 2024-03-14T13:02:16.444 INFO:tasks.workunit.client.0.smithi111.stdout:5/542: dwrite d2/d16/d49/d44/f50 [0,4194304] 0 2024-03-14T13:02:16.445 INFO:tasks.workunit.client.0.smithi111.stdout:8/608: readlink d3/d1b/d20/lbc 0 2024-03-14T13:02:16.445 INFO:tasks.workunit.client.1.smithi137.stdout:5/333: write d7/d33/f61 [701841,77360] 0 2024-03-14T13:02:16.447 INFO:tasks.workunit.client.0.smithi111.stdout:6/435: mknod d0/d17/d21/d69/d3d/d53/d81/da1/ca6 0 2024-03-14T13:02:16.448 INFO:tasks.workunit.client.1.smithi137.stdout:0/351: creat d1/d5/d1b/d1e/d25/d55/f7a x:0 0 0 2024-03-14T13:02:16.452 INFO:tasks.workunit.client.0.smithi111.stdout:5/543: mkdir d2/d45/d76/db2 0 2024-03-14T13:02:16.453 INFO:tasks.workunit.client.1.smithi137.stdout:2/459: getdents d2/d34 0 2024-03-14T13:02:16.454 INFO:tasks.workunit.client.0.smithi111.stdout:0/475: write d6/d9/d15/f16 [1044875,49439] 0 2024-03-14T13:02:16.454 INFO:tasks.workunit.client.0.smithi111.stdout:3/463: unlink d0/d2d/d35/d50/f56 0 2024-03-14T13:02:16.455 INFO:tasks.workunit.client.0.smithi111.stdout:6/436: chown d0/f6e 17342 1 2024-03-14T13:02:16.457 INFO:tasks.workunit.client.1.smithi137.stdout:2/460: mknod d2/d7/d2e/d3e/d44/d6b/d1d/c94 0 2024-03-14T13:02:16.461 INFO:tasks.workunit.client.0.smithi111.stdout:8/609: dwrite d3/d1b/d20/d35/d40/f90 [0,4194304] 0 2024-03-14T13:02:16.461 INFO:tasks.workunit.client.0.smithi111.stdout:0/476: unlink d6/d9/de/l17 0 2024-03-14T13:02:16.461 INFO:tasks.workunit.client.1.smithi137.stdout:0/352: rename d1/d5/d1b/d1e/f30 to d1/f7b 0 2024-03-14T13:02:16.461 INFO:tasks.workunit.client.1.smithi137.stdout:9/504: write d6/df/d19/d3a/d40/f4b [279673,12104] 0 2024-03-14T13:02:16.462 INFO:tasks.workunit.client.1.smithi137.stdout:8/393: rename d1/dc/d3c/d54 to d1/dc/d3c/d87 0 2024-03-14T13:02:16.465 INFO:tasks.workunit.client.0.smithi111.stdout:6/437: creat d0/d17/d21/d69/d3d/d53/d81/fa7 x:0 0 0 2024-03-14T13:02:16.466 INFO:tasks.workunit.client.1.smithi137.stdout:0/353: rename d1/d1f/c6e to d1/d5/d1b/d1e/d25/d55/c7c 0 2024-03-14T13:02:16.468 INFO:tasks.workunit.client.0.smithi111.stdout:0/477: mkdir d6/d9/d5b/d9e 0 2024-03-14T13:02:16.468 INFO:tasks.workunit.client.1.smithi137.stdout:6/406: write d2/d6/f3d [1755806,57720] 0 2024-03-14T13:02:16.469 INFO:tasks.workunit.client.0.smithi111.stdout:6/438: mknod d0/d17/d20/d4a/d68/ca8 0 2024-03-14T13:02:16.471 INFO:tasks.workunit.client.0.smithi111.stdout:5/544: dwrite d2/d16/d49/d7c/d95/fa3 [0,4194304] 0 2024-03-14T13:02:16.471 INFO:tasks.workunit.client.0.smithi111.stdout:0/478: dread - d6/d9/de/d3a/f61 zero size 2024-03-14T13:02:16.473 INFO:tasks.workunit.client.1.smithi137.stdout:9/505: getdents d6/df/d19/d1f 0 2024-03-14T13:02:16.475 INFO:tasks.workunit.client.1.smithi137.stdout:8/394: dwrite d1/dc/d11/f2a [0,4194304] 0 2024-03-14T13:02:16.475 INFO:tasks.workunit.client.0.smithi111.stdout:8/610: dread d3/d1b/d20/d35/d40/f90 [0,4194304] 0 2024-03-14T13:02:16.476 INFO:tasks.workunit.client.0.smithi111.stdout:0/479: symlink d6/d9/de/d3a/d46/d4a/l9f 0 2024-03-14T13:02:16.476 INFO:tasks.workunit.client.0.smithi111.stdout:1/471: dread d20/d21/d3f/f47 [0,4194304] 0 2024-03-14T13:02:16.477 INFO:tasks.workunit.client.1.smithi137.stdout:6/407: write d2/d6/de/f5d [3822900,6171] 0 2024-03-14T13:02:16.478 INFO:tasks.workunit.client.0.smithi111.stdout:0/480: fdatasync d6/d21/d74/f99 0 2024-03-14T13:02:16.478 INFO:tasks.workunit.client.1.smithi137.stdout:0/354: dread d1/d1f/f3b [0,4194304] 0 2024-03-14T13:02:16.478 INFO:tasks.workunit.client.1.smithi137.stdout:9/506: link d6/df/d19/d1f/f95 d6/df/d71/fa5 0 2024-03-14T13:02:16.479 INFO:tasks.workunit.client.0.smithi111.stdout:5/545: creat d2/d16/d49/d44/d4a/d9e/fb3 x:0 0 0 2024-03-14T13:02:16.481 INFO:tasks.workunit.client.1.smithi137.stdout:5/334: dread d7/f10 [0,4194304] 0 2024-03-14T13:02:16.481 INFO:tasks.workunit.client.1.smithi137.stdout:0/355: chown d1/d5/f2a 2578893 1 2024-03-14T13:02:16.481 INFO:tasks.workunit.client.0.smithi111.stdout:1/472: write d20/d25/d31/f3b [5152028,130315] 0 2024-03-14T13:02:16.481 INFO:tasks.workunit.client.1.smithi137.stdout:6/408: creat d2/d6/d1d/d88/f93 x:0 0 0 2024-03-14T13:02:16.483 INFO:tasks.workunit.client.1.smithi137.stdout:0/356: write d1/d5/f2a [4640347,11628] 0 2024-03-14T13:02:16.484 INFO:tasks.workunit.client.1.smithi137.stdout:8/395: mknod d1/c88 0 2024-03-14T13:02:16.486 INFO:tasks.workunit.client.0.smithi111.stdout:5/546: unlink d2/d13/f29 0 2024-03-14T13:02:16.486 INFO:tasks.workunit.client.1.smithi137.stdout:6/409: fdatasync d2/d6/de/d45/f60 0 2024-03-14T13:02:16.487 INFO:tasks.workunit.client.1.smithi137.stdout:5/335: getdents d7/d16/d72 0 2024-03-14T13:02:16.488 INFO:tasks.workunit.client.1.smithi137.stdout:0/357: dread - d1/d5/d1b/d1e/d25/d55/f60 zero size 2024-03-14T13:02:16.488 INFO:tasks.workunit.client.0.smithi111.stdout:0/481: dread d6/d9/de/f36 [0,4194304] 0 2024-03-14T13:02:16.489 INFO:tasks.workunit.client.0.smithi111.stdout:3/464: dread d0/d2d/d35/d50/d5e/d19/f5a [0,4194304] 0 2024-03-14T13:02:16.489 INFO:tasks.workunit.client.1.smithi137.stdout:9/507: dread d6/df/f10 [0,4194304] 0 2024-03-14T13:02:16.491 INFO:tasks.workunit.client.0.smithi111.stdout:5/547: mkdir d2/d16/d49/d33/db4 0 2024-03-14T13:02:16.491 INFO:tasks.workunit.client.0.smithi111.stdout:3/465: chown d0/d2d/l4d 46760 1 2024-03-14T13:02:16.493 INFO:tasks.workunit.client.0.smithi111.stdout:0/482: chown d6/d9/de/d3a/d46/f83 9717105 1 2024-03-14T13:02:16.493 INFO:tasks.workunit.client.0.smithi111.stdout:5/548: rmdir d2 39 2024-03-14T13:02:16.493 INFO:tasks.workunit.client.0.smithi111.stdout:1/473: dwrite d20/d21/d3f/f66 [0,4194304] 0 2024-03-14T13:02:16.493 INFO:tasks.workunit.client.0.smithi111.stdout:3/466: readlink d0/d2d/d35/d50/d5e/l9b 0 2024-03-14T13:02:16.494 INFO:tasks.workunit.client.1.smithi137.stdout:0/358: read d1/d11/d4a/d54/f5b [2796223,73270] 0 2024-03-14T13:02:16.498 INFO:tasks.workunit.client.0.smithi111.stdout:3/467: truncate d0/d91/f9e 555204 0 2024-03-14T13:02:16.499 INFO:tasks.workunit.client.1.smithi137.stdout:6/410: rename d2/d6/c24 to d2/d4c/c94 0 2024-03-14T13:02:16.499 INFO:tasks.workunit.client.0.smithi111.stdout:0/483: mkdir d6/d9/d15/d55/da0 0 2024-03-14T13:02:16.500 INFO:tasks.workunit.client.0.smithi111.stdout:8/611: write d3/d1b/d20/f9e [536924,7901] 0 2024-03-14T13:02:16.502 INFO:tasks.workunit.client.0.smithi111.stdout:5/549: creat d2/d45/d76/d9a/d9f/fb5 x:0 0 0 2024-03-14T13:02:16.503 INFO:tasks.workunit.client.0.smithi111.stdout:0/484: symlink d6/d9/de/d3a/la1 0 2024-03-14T13:02:16.503 INFO:tasks.workunit.client.0.smithi111.stdout:1/474: dwrite d20/d25/d87/f8f [0,4194304] 0 2024-03-14T13:02:16.505 INFO:tasks.workunit.client.1.smithi137.stdout:8/396: dwrite d1/dc/d11/d13/f45 [4194304,4194304] 0 2024-03-14T13:02:16.508 INFO:tasks.workunit.client.1.smithi137.stdout:9/508: rename d6/df/d19/d3a/l9d to d6/df/d19/la6 0 2024-03-14T13:02:16.508 INFO:tasks.workunit.client.1.smithi137.stdout:5/336: dwrite d7/f30 [0,4194304] 0 2024-03-14T13:02:16.509 INFO:tasks.workunit.client.0.smithi111.stdout:5/550: rmdir d2/d16/d49/d44 39 2024-03-14T13:02:16.509 INFO:tasks.workunit.client.0.smithi111.stdout:1/475: mkdir d20/d25/da8 0 2024-03-14T13:02:16.510 INFO:tasks.workunit.client.0.smithi111.stdout:1/476: stat d20/d40/f84 0 2024-03-14T13:02:16.511 INFO:tasks.workunit.client.0.smithi111.stdout:1/477: dread - d20/d25/d75/f77 zero size 2024-03-14T13:02:16.512 INFO:tasks.workunit.client.0.smithi111.stdout:5/551: read d2/d16/d49/d7c/d95/d6a/f71 [4188998,73607] 0 2024-03-14T13:02:16.513 INFO:tasks.workunit.client.1.smithi137.stdout:0/359: truncate d1/d5/d1b/d42/d3a/f48 4068003 0 2024-03-14T13:02:16.516 INFO:tasks.workunit.client.1.smithi137.stdout:6/411: dwrite d2/f17 [8388608,4194304] 0 2024-03-14T13:02:16.516 INFO:tasks.workunit.client.1.smithi137.stdout:8/397: chown d1/dc/d3c/d87/f6e 15040163 1 2024-03-14T13:02:16.518 INFO:tasks.workunit.client.1.smithi137.stdout:0/360: write d1/d5/d1b/d42/f4b [11398,90622] 0 2024-03-14T13:02:16.518 INFO:tasks.workunit.client.0.smithi111.stdout:8/612: rename d3/d1b/d34/c57 to d3/d22/cbd 0 2024-03-14T13:02:16.519 INFO:tasks.workunit.client.1.smithi137.stdout:8/398: write d1/dc/f52 [367194,32753] 0 2024-03-14T13:02:16.520 INFO:tasks.workunit.client.0.smithi111.stdout:1/478: creat d20/d21/da7/fa9 x:0 0 0 2024-03-14T13:02:16.521 INFO:tasks.workunit.client.0.smithi111.stdout:8/613: fsync d3/d1b/d20/d35/d40/d48/f89 0 2024-03-14T13:02:16.521 INFO:tasks.workunit.client.1.smithi137.stdout:9/509: rename d6/df/l1b to d6/df/d19/la7 0 2024-03-14T13:02:16.522 INFO:tasks.workunit.client.0.smithi111.stdout:8/614: fdatasync d3/dc/f99 0 2024-03-14T13:02:16.523 INFO:tasks.workunit.client.0.smithi111.stdout:5/552: dwrite d2/d13/f99 [0,4194304] 0 2024-03-14T13:02:16.527 INFO:tasks.workunit.client.1.smithi137.stdout:8/399: rename d1/dc/d10/l3d to d1/dc/d11/d56/l89 0 2024-03-14T13:02:16.527 INFO:tasks.workunit.client.0.smithi111.stdout:1/479: read d20/d25/f7c [60049,57152] 0 2024-03-14T13:02:16.528 INFO:tasks.workunit.client.1.smithi137.stdout:0/361: fsync d1/d44/d56/f5f 0 2024-03-14T13:02:16.528 INFO:tasks.workunit.client.1.smithi137.stdout:6/412: link d2/d6/d1d/l3e d2/d54/d8b/l95 0 2024-03-14T13:02:16.528 INFO:tasks.workunit.client.0.smithi111.stdout:8/615: creat d3/d1b/d20/d35/d40/d48/d7e/db3/db5/fbe x:0 0 0 2024-03-14T13:02:16.529 INFO:tasks.workunit.client.1.smithi137.stdout:9/510: mkdir d6/da8 0 2024-03-14T13:02:16.531 INFO:tasks.workunit.client.0.smithi111.stdout:5/553: unlink d2/l85 0 2024-03-14T13:02:16.531 INFO:tasks.workunit.client.1.smithi137.stdout:9/511: readlink d6/df/d88/l9f 0 2024-03-14T13:02:16.533 INFO:tasks.workunit.client.1.smithi137.stdout:8/400: creat d1/dc/d11/d7f/f8a x:0 0 0 2024-03-14T13:02:16.536 INFO:tasks.workunit.client.1.smithi137.stdout:6/413: mknod d2/d6/d1d/c96 0 2024-03-14T13:02:16.539 INFO:tasks.workunit.client.1.smithi137.stdout:0/362: rename d1/d11/d4a/d54/f5b to d1/d44/d56/f7d 0 2024-03-14T13:02:16.540 INFO:tasks.workunit.client.1.smithi137.stdout:8/401: creat d1/dc/d4e/d6d/f8b x:0 0 0 2024-03-14T13:02:16.540 INFO:tasks.workunit.client.1.smithi137.stdout:5/337: truncate d7/d33/f61 732963 0 2024-03-14T13:02:16.541 INFO:tasks.workunit.client.1.smithi137.stdout:6/414: unlink d2/f7 0 2024-03-14T13:02:16.543 INFO:tasks.workunit.client.1.smithi137.stdout:8/402: dread - d1/dc/d10/f64 zero size 2024-03-14T13:02:16.543 INFO:tasks.workunit.client.1.smithi137.stdout:9/512: dwrite d6/df/f13 [0,4194304] 0 2024-03-14T13:02:16.549 INFO:tasks.workunit.client.1.smithi137.stdout:5/338: rename d7/d16/d72 to d7/d16/d18/d25/d28/d3c/d76 0 2024-03-14T13:02:16.551 INFO:tasks.workunit.client.1.smithi137.stdout:0/363: dwrite d1/d11/f14 [0,4194304] 0 2024-03-14T13:02:16.552 INFO:tasks.workunit.client.1.smithi137.stdout:5/339: dread - d7/d16/d18/f34 zero size 2024-03-14T13:02:16.553 INFO:tasks.workunit.client.1.smithi137.stdout:5/340: readlink d7/d36/l4d 0 2024-03-14T13:02:16.553 INFO:tasks.workunit.client.1.smithi137.stdout:9/513: mkdir d6/df/d19/d3a/da9 0 2024-03-14T13:02:16.562 INFO:tasks.workunit.client.1.smithi137.stdout:8/403: dwrite d1/dc/d11/d13/f45 [0,4194304] 0 2024-03-14T13:02:16.563 INFO:tasks.workunit.client.1.smithi137.stdout:0/364: rename d1/d5/d1b/d1e/d25/d55/d73/d76 to d1/d44/d72/d7e 0 2024-03-14T13:02:16.565 INFO:tasks.workunit.client.1.smithi137.stdout:6/415: chown d2/d6/f65 522097 1 2024-03-14T13:02:16.566 INFO:tasks.workunit.client.1.smithi137.stdout:0/365: truncate d1/d11/f14 4337670 0 2024-03-14T13:02:16.567 INFO:tasks.workunit.client.1.smithi137.stdout:5/341: dwrite d7/d16/d18/d2d/f63 [0,4194304] 0 2024-03-14T13:02:16.570 INFO:tasks.workunit.client.1.smithi137.stdout:5/342: write d7/d16/d18/d25/f50 [2604491,23182] 0 2024-03-14T13:02:16.570 INFO:tasks.workunit.client.0.smithi111.stdout:5/554: dread d2/d16/d49/d33/f43 [0,4194304] 0 2024-03-14T13:02:16.572 INFO:tasks.workunit.client.1.smithi137.stdout:6/416: mkdir d2/d6/de/d45/d97 0 2024-03-14T13:02:16.580 INFO:tasks.workunit.client.1.smithi137.stdout:5/343: creat d7/d16/d18/d25/d28/d3c/f77 x:0 0 0 2024-03-14T13:02:16.580 INFO:tasks.workunit.client.0.smithi111.stdout:2/448: sync 2024-03-14T13:02:16.580 INFO:tasks.workunit.client.1.smithi137.stdout:6/417: truncate d2/d6/de/f4f 100413 0 2024-03-14T13:02:16.581 INFO:tasks.workunit.client.0.smithi111.stdout:2/449: readlink dd/d17/d34/l1e 0 2024-03-14T13:02:16.581 INFO:tasks.workunit.client.1.smithi137.stdout:5/344: stat d7/d33/c4b 0 2024-03-14T13:02:16.581 INFO:tasks.workunit.client.0.smithi111.stdout:2/450: write dd/f76 [256051,54921] 0 2024-03-14T13:02:16.582 INFO:tasks.workunit.client.0.smithi111.stdout:2/451: chown dd/d11/d1a/d56/f78 21231 1 2024-03-14T13:02:16.582 INFO:tasks.workunit.client.1.smithi137.stdout:5/345: write d7/d16/f1a [6051167,75923] 0 2024-03-14T13:02:16.583 INFO:tasks.workunit.client.1.smithi137.stdout:0/366: getdents d1/d5 0 2024-03-14T13:02:16.583 INFO:tasks.workunit.client.1.smithi137.stdout:6/418: dread - d2/d6/de/d45/d59/d39/d3c/f92 zero size 2024-03-14T13:02:16.584 INFO:tasks.workunit.client.1.smithi137.stdout:5/346: write d7/f15 [4129814,53050] 0 2024-03-14T13:02:16.588 INFO:tasks.workunit.client.1.smithi137.stdout:5/347: rmdir d7/d16/d41 39 2024-03-14T13:02:16.591 INFO:tasks.workunit.client.1.smithi137.stdout:5/348: creat d7/d1d/d74/f78 x:0 0 0 2024-03-14T13:02:16.592 INFO:tasks.workunit.client.1.smithi137.stdout:6/419: dwrite d2/d54/d55/d6d/d76/d57/f74 [0,4194304] 0 2024-03-14T13:02:16.595 INFO:tasks.workunit.client.1.smithi137.stdout:8/404: truncate d1/dc/d11/d13/f45 3079628 0 2024-03-14T13:02:16.595 INFO:tasks.workunit.client.1.smithi137.stdout:9/514: dread d6/df/f20 [0,4194304] 0 2024-03-14T13:02:16.597 INFO:tasks.workunit.client.1.smithi137.stdout:6/420: rmdir d2/d54 39 2024-03-14T13:02:16.601 INFO:tasks.workunit.client.1.smithi137.stdout:5/349: creat d7/d16/d18/d25/d28/d3c/d76/f79 x:0 0 0 2024-03-14T13:02:16.603 INFO:tasks.workunit.client.1.smithi137.stdout:8/405: creat d1/dc/d3c/f8c x:0 0 0 2024-03-14T13:02:16.603 INFO:tasks.workunit.client.1.smithi137.stdout:9/515: read d6/f2b [1453042,109372] 0 2024-03-14T13:02:16.605 INFO:tasks.workunit.client.1.smithi137.stdout:8/406: write d1/dc/d10/f64 [75263,18311] 0 2024-03-14T13:02:16.605 INFO:tasks.workunit.client.1.smithi137.stdout:5/350: mkdir d7/d16/d18/d25/d28/d3c/d7a 0 2024-03-14T13:02:16.607 INFO:tasks.workunit.client.1.smithi137.stdout:6/421: chown d2/d6/d1d/c96 4650 1 2024-03-14T13:02:16.610 INFO:tasks.workunit.client.1.smithi137.stdout:9/516: rename d6/d56/c8c to d6/df/d19/d3a/d40/caa 0 2024-03-14T13:02:16.611 INFO:tasks.workunit.client.1.smithi137.stdout:6/422: fdatasync d2/d6/de/d45/d59/f79 0 2024-03-14T13:02:16.615 INFO:tasks.workunit.client.1.smithi137.stdout:8/407: dwrite d1/dc/d11/d13/f3e [0,4194304] 0 2024-03-14T13:02:16.616 INFO:tasks.workunit.client.1.smithi137.stdout:8/408: chown d1/d4a 69 1 2024-03-14T13:02:16.619 INFO:tasks.workunit.client.1.smithi137.stdout:9/517: rmdir d6/d56/d9c 0 2024-03-14T13:02:16.621 INFO:tasks.workunit.client.0.smithi111.stdout:8/616: mknod d3/d1b/d20/d35/d40/d48/d7e/cbf 0 2024-03-14T13:02:16.623 INFO:tasks.workunit.client.0.smithi111.stdout:5/555: symlink d2/d45/d76/d9a/d9f/lb6 0 2024-03-14T13:02:16.624 INFO:tasks.workunit.client.1.smithi137.stdout:8/409: creat d1/dc/d3c/d87/d34/d82/f8d x:0 0 0 2024-03-14T13:02:16.624 INFO:tasks.workunit.client.1.smithi137.stdout:5/351: dwrite d7/f1b [0,4194304] 0 2024-03-14T13:02:16.624 INFO:tasks.workunit.client.1.smithi137.stdout:6/423: write d2/d6/de/d45/d59/f5e [3653124,103405] 0 2024-03-14T13:02:16.626 INFO:tasks.workunit.client.1.smithi137.stdout:9/518: mknod d6/df/d71/d94/cab 0 2024-03-14T13:02:16.626 INFO:tasks.workunit.client.0.smithi111.stdout:2/452: mknod dd/d17/d34/d4f/c94 0 2024-03-14T13:02:16.627 INFO:tasks.workunit.client.1.smithi137.stdout:8/410: mknod d1/dc/d77/c8e 0 2024-03-14T13:02:16.627 INFO:tasks.workunit.client.0.smithi111.stdout:8/617: rmdir d3/d1b/d66 39 2024-03-14T13:02:16.628 INFO:tasks.workunit.client.1.smithi137.stdout:9/519: dread - d6/df/d19/d1f/f95 zero size 2024-03-14T13:02:16.629 INFO:tasks.workunit.client.0.smithi111.stdout:5/556: chown d2/d16/d49/d44/f50 2944334 1 2024-03-14T13:02:16.630 INFO:tasks.workunit.client.1.smithi137.stdout:9/520: fsync d6/df/d71/f8b 0 2024-03-14T13:02:16.630 INFO:tasks.workunit.client.1.smithi137.stdout:5/352: creat d7/f7b x:0 0 0 2024-03-14T13:02:16.631 INFO:tasks.workunit.client.0.smithi111.stdout:2/453: unlink dd/d17/d34/d73/f3c 0 2024-03-14T13:02:16.634 INFO:tasks.workunit.client.1.smithi137.stdout:6/424: rename d2/d6/de/d45/d59/d39/d3c/f92 to d2/d54/d55/d6d/d76/d57/d58/f98 0 2024-03-14T13:02:16.637 INFO:tasks.workunit.client.1.smithi137.stdout:5/353: creat d7/d33/d65/f7c x:0 0 0 2024-03-14T13:02:16.637 INFO:tasks.workunit.client.1.smithi137.stdout:8/411: getdents d1/dc/d11 0 2024-03-14T13:02:16.638 INFO:tasks.workunit.client.1.smithi137.stdout:8/412: write d1/dc/d3c/d87/d34/f68 [4598550,104699] 0 2024-03-14T13:02:16.639 INFO:tasks.workunit.client.0.smithi111.stdout:1/480: truncate fa 919111 0 2024-03-14T13:02:16.639 INFO:tasks.workunit.client.0.smithi111.stdout:5/557: mkdir d2/d16/d49/d44/d4a/d9e/db7 0 2024-03-14T13:02:16.640 INFO:tasks.workunit.client.1.smithi137.stdout:5/354: symlink d7/d16/d18/d2d/l7d 0 2024-03-14T13:02:16.643 INFO:tasks.workunit.client.1.smithi137.stdout:8/413: rmdir d1/d5e 39 2024-03-14T13:02:16.644 INFO:tasks.workunit.client.1.smithi137.stdout:9/521: link d6/c8 d6/df/d19/d33/cac 0 2024-03-14T13:02:16.647 INFO:tasks.workunit.client.1.smithi137.stdout:8/414: mknod d1/dc/d11/c8f 0 2024-03-14T13:02:16.648 INFO:tasks.workunit.client.1.smithi137.stdout:5/355: symlink d7/d16/d18/d48/l7e 0 2024-03-14T13:02:16.649 INFO:tasks.workunit.client.0.smithi111.stdout:5/558: unlink d2/d16/d49/d7c/d95/c54 0 2024-03-14T13:02:16.653 INFO:tasks.workunit.client.0.smithi111.stdout:8/618: dread d3/d22/d30/f97 [0,4194304] 0 2024-03-14T13:02:16.655 INFO:tasks.workunit.client.0.smithi111.stdout:8/619: symlink d3/d1b/d20/d35/d40/d48/d7e/db3/lc0 0 2024-03-14T13:02:16.655 INFO:tasks.workunit.client.1.smithi137.stdout:5/356: rename d7/l31 to d7/d16/d18/d25/d28/d3c/d76/l7f 0 2024-03-14T13:02:16.658 INFO:tasks.workunit.client.1.smithi137.stdout:8/415: rename d1/d5e/c6a to d1/d4a/c90 0 2024-03-14T13:02:16.660 INFO:tasks.workunit.client.1.smithi137.stdout:9/522: truncate d6/df/d19/d1f/fa4 713913 0 2024-03-14T13:02:16.661 INFO:tasks.workunit.client.0.smithi111.stdout:1/481: truncate d20/d25/d31/f3b 4436621 0 2024-03-14T13:02:16.662 INFO:tasks.workunit.client.0.smithi111.stdout:8/620: write d3/d22/d30/f7d [69139,94082] 0 2024-03-14T13:02:16.664 INFO:tasks.workunit.client.1.smithi137.stdout:6/425: dwrite d2/d6/de/d45/d59/f40 [0,4194304] 0 2024-03-14T13:02:16.665 INFO:tasks.workunit.client.0.smithi111.stdout:8/621: chown d3/d1b/d66/fa2 0 1 2024-03-14T13:02:16.666 INFO:tasks.workunit.client.0.smithi111.stdout:2/454: dwrite f1 [0,4194304] 0 2024-03-14T13:02:16.668 INFO:tasks.workunit.client.1.smithi137.stdout:6/426: write d2/d6/f3d [2225173,13111] 0 2024-03-14T13:02:16.668 INFO:tasks.workunit.client.1.smithi137.stdout:5/357: dwrite d7/d16/d18/d2d/f63 [0,4194304] 0 2024-03-14T13:02:16.668 INFO:tasks.workunit.client.1.smithi137.stdout:8/416: unlink d1/dc/d11/d13/f84 0 2024-03-14T13:02:16.668 INFO:tasks.workunit.client.0.smithi111.stdout:8/622: fsync d3/d1b/f75 0 2024-03-14T13:02:16.668 INFO:tasks.workunit.client.0.smithi111.stdout:2/455: write dd/d11/d1a/f21 [570469,3388] 0 2024-03-14T13:02:16.672 INFO:tasks.workunit.client.1.smithi137.stdout:9/523: creat d6/d91/fad x:0 0 0 2024-03-14T13:02:16.677 INFO:tasks.workunit.client.1.smithi137.stdout:6/427: write d2/d4c/f84 [983877,24281] 0 2024-03-14T13:02:16.678 INFO:tasks.workunit.client.0.smithi111.stdout:5/559: dread d2/d16/d49/d7c/d95/d6a/f6e [0,4194304] 0 2024-03-14T13:02:16.679 INFO:tasks.workunit.client.0.smithi111.stdout:8/623: read d3/d1b/d34/f3b [86689,56016] 0 2024-03-14T13:02:16.679 INFO:tasks.workunit.client.1.smithi137.stdout:8/417: symlink d1/dc/d11/d56/l91 0 2024-03-14T13:02:16.680 INFO:tasks.workunit.client.0.smithi111.stdout:2/456: dwrite dd/d17/d34/d73/d49/f90 [0,4194304] 0 2024-03-14T13:02:16.680 INFO:tasks.workunit.client.1.smithi137.stdout:8/418: write d1/dc/f39 [4131046,124176] 0 2024-03-14T13:02:16.682 INFO:tasks.workunit.client.1.smithi137.stdout:8/419: chown d1/dc/d11/d56 0 1 2024-03-14T13:02:16.682 INFO:tasks.workunit.client.1.smithi137.stdout:9/524: dread d6/df/d19/d1f/f4a [0,4194304] 0 2024-03-14T13:02:16.683 INFO:tasks.workunit.client.1.smithi137.stdout:6/428: write d2/d54/d55/d6d/d76/d57/d58/f98 [290434,68266] 0 2024-03-14T13:02:16.686 INFO:tasks.workunit.client.1.smithi137.stdout:6/429: write f0 [4654634,128598] 0 2024-03-14T13:02:16.687 INFO:tasks.workunit.client.1.smithi137.stdout:6/430: chown d2/d6/de/f13 2 1 2024-03-14T13:02:16.688 INFO:tasks.workunit.client.0.smithi111.stdout:5/560: mknod d2/d16/d49/d7c/d95/d6a/cb8 0 2024-03-14T13:02:16.688 INFO:tasks.workunit.client.1.smithi137.stdout:8/420: unlink d1/dc/d77/c81 0 2024-03-14T13:02:16.690 INFO:tasks.workunit.client.1.smithi137.stdout:9/525: creat d6/df/d19/d3a/fae x:0 0 0 2024-03-14T13:02:16.692 INFO:tasks.workunit.client.1.smithi137.stdout:6/431: rename d2/d6/c53 to d2/d6/d1d/c99 0 2024-03-14T13:02:16.692 INFO:tasks.workunit.client.1.smithi137.stdout:8/421: symlink d1/dc/d11/l92 0 2024-03-14T13:02:16.693 INFO:tasks.workunit.client.0.smithi111.stdout:2/457: dwrite dd/d17/f3a [0,4194304] 0 2024-03-14T13:02:16.696 INFO:tasks.workunit.client.0.smithi111.stdout:2/458: creat dd/d17/d34/d73/d75/f95 x:0 0 0 2024-03-14T13:02:16.697 INFO:tasks.workunit.client.1.smithi137.stdout:9/526: unlink d6/df/d19/d33/cac 0 2024-03-14T13:02:16.697 INFO:tasks.workunit.client.1.smithi137.stdout:8/422: symlink d1/l93 0 2024-03-14T13:02:16.698 INFO:tasks.workunit.client.0.smithi111.stdout:2/459: creat dd/d8d/f96 x:0 0 0 2024-03-14T13:02:16.698 INFO:tasks.workunit.client.1.smithi137.stdout:8/423: stat d1/dc/d3c 0 2024-03-14T13:02:16.702 INFO:tasks.workunit.client.0.smithi111.stdout:1/482: dread d20/d25/f46 [0,4194304] 0 2024-03-14T13:02:16.704 INFO:tasks.workunit.client.0.smithi111.stdout:1/483: creat d20/d21/faa x:0 0 0 2024-03-14T13:02:16.706 INFO:tasks.workunit.client.1.smithi137.stdout:8/424: dwrite d1/dc/d11/f2d [0,4194304] 0 2024-03-14T13:02:16.706 INFO:tasks.workunit.client.0.smithi111.stdout:1/484: symlink d20/d21/d28/lab 0 2024-03-14T13:02:16.710 INFO:tasks.workunit.client.1.smithi137.stdout:8/425: mkdir d1/dc/d3c/d87/d34/d82/d94 0 2024-03-14T13:02:16.719 INFO:tasks.workunit.client.0.smithi111.stdout:1/485: dread d20/d25/d75/f65 [0,4194304] 0 2024-03-14T13:02:16.720 INFO:tasks.workunit.client.0.smithi111.stdout:1/486: write d20/d21/d28/d5c/f93 [965251,3692] 0 2024-03-14T13:02:16.721 INFO:tasks.workunit.client.1.smithi137.stdout:4/377: sync 2024-03-14T13:02:16.723 INFO:tasks.workunit.client.0.smithi111.stdout:2/460: read dd/d17/d53/f92 [3507431,30403] 0 2024-03-14T13:02:16.723 INFO:tasks.workunit.client.1.smithi137.stdout:4/378: write d4/d27/d5c/d6d/f72 [467540,32477] 0 2024-03-14T13:02:16.725 INFO:tasks.workunit.client.0.smithi111.stdout:2/461: rmdir dd/d17/d34/d73/d49 39 2024-03-14T13:02:16.726 INFO:tasks.workunit.client.0.smithi111.stdout:2/462: chown dd/d17/d34/d73/l8f 8249 1 2024-03-14T13:02:16.727 INFO:tasks.workunit.client.0.smithi111.stdout:2/463: write dd/d17/f37 [1559976,76492] 0 2024-03-14T13:02:16.728 INFO:tasks.workunit.client.0.smithi111.stdout:2/464: fdatasync dd/d11/d1a/d56/f85 0 2024-03-14T13:02:16.730 INFO:tasks.workunit.client.0.smithi111.stdout:2/465: chown dd/d11/c2f 2 1 2024-03-14T13:02:16.732 INFO:tasks.workunit.client.0.smithi111.stdout:2/466: unlink dd/c2a 0 2024-03-14T13:02:16.732 INFO:tasks.workunit.client.1.smithi137.stdout:4/379: dread d4/d9/d3a/d41/d38/f53 [0,4194304] 0 2024-03-14T13:02:16.733 INFO:tasks.workunit.client.0.smithi111.stdout:8/624: write d3/dc/d31/f4c [3825856,120888] 0 2024-03-14T13:02:16.734 INFO:tasks.workunit.client.1.smithi137.stdout:4/380: fsync d4/d9/f18 0 2024-03-14T13:02:16.735 INFO:tasks.workunit.client.1.smithi137.stdout:4/381: dread - d4/d27/d5c/d6d/d58/f76 zero size 2024-03-14T13:02:16.735 INFO:tasks.workunit.client.0.smithi111.stdout:2/467: rename dd/d11/d1a/f52 to dd/d17/d34/d73/d75/d59/f97 0 2024-03-14T13:02:16.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/561: truncate d2/d16/d49/d7c/d95/fa3 3921846 0 2024-03-14T13:02:16.739 INFO:tasks.workunit.client.1.smithi137.stdout:4/382: symlink d4/d9/d3a/d70/d80/l84 0 2024-03-14T13:02:16.741 INFO:tasks.workunit.client.0.smithi111.stdout:8/625: truncate d3/d1b/d20/d35/d40/d48/f89 4088633 0 2024-03-14T13:02:16.743 INFO:tasks.workunit.client.1.smithi137.stdout:4/383: dread d4/d27/d5c/d6d/d58/d65/f7b [0,4194304] 0 2024-03-14T13:02:16.743 INFO:tasks.workunit.client.0.smithi111.stdout:5/562: rename d2/d16/d49/d44/d4a/d9e/fb3 to d2/d13/fb9 0 2024-03-14T13:02:16.745 INFO:tasks.workunit.client.1.smithi137.stdout:8/426: dread d1/f6 [0,4194304] 0 2024-03-14T13:02:16.745 INFO:tasks.workunit.client.0.smithi111.stdout:2/468: link dd/d17/f83 dd/d17/d53/f98 0 2024-03-14T13:02:16.746 INFO:tasks.workunit.client.0.smithi111.stdout:2/469: chown dd/d11/d1a/d56/f5b 155834 1 2024-03-14T13:02:16.747 INFO:tasks.workunit.client.1.smithi137.stdout:8/427: symlink d1/dc/d4e/d6d/l95 0 2024-03-14T13:02:16.749 INFO:tasks.workunit.client.0.smithi111.stdout:5/563: symlink d2/d16/d49/d7c/d95/d36/lba 0 2024-03-14T13:02:16.749 INFO:tasks.workunit.client.0.smithi111.stdout:2/470: symlink dd/d11/d1a/d70/l99 0 2024-03-14T13:02:16.750 INFO:tasks.workunit.client.1.smithi137.stdout:8/428: mkdir d1/dc/d3c/d87/d34/d82/d96 0 2024-03-14T13:02:16.750 INFO:tasks.workunit.client.1.smithi137.stdout:4/384: link d4/d9/d3a/d41/d38/d55/f64 d4/d10/d5a/f85 0 2024-03-14T13:02:16.751 INFO:tasks.workunit.client.0.smithi111.stdout:8/626: mkdir d3/d1b/dc1 0 2024-03-14T13:02:16.754 INFO:tasks.workunit.client.1.smithi137.stdout:8/429: rename d1/dc/d11 to d1/dc/d3c/d87/d34/d97 0 2024-03-14T13:02:16.756 INFO:tasks.workunit.client.0.smithi111.stdout:2/471: write dd/d17/d34/d73/d49/f82 [4769835,43476] 0 2024-03-14T13:02:16.757 INFO:tasks.workunit.client.1.smithi137.stdout:4/385: getdents d4/d27/d5c/d6d/d58/d65 0 2024-03-14T13:02:16.757 INFO:tasks.workunit.client.0.smithi111.stdout:2/472: dread - dd/d11/d1a/d56/f5b zero size 2024-03-14T13:02:16.758 INFO:tasks.workunit.client.0.smithi111.stdout:8/627: mknod d3/d1b/cc2 0 2024-03-14T13:02:16.758 INFO:tasks.workunit.client.1.smithi137.stdout:8/430: mknod d1/dc/d3c/d87/d34/d97/d7f/c98 0 2024-03-14T13:02:16.761 INFO:tasks.workunit.client.0.smithi111.stdout:2/473: creat dd/d17/d34/d4f/f9a x:0 0 0 2024-03-14T13:02:16.762 INFO:tasks.workunit.client.0.smithi111.stdout:8/628: creat d3/d22/d30/fc3 x:0 0 0 2024-03-14T13:02:16.763 INFO:tasks.workunit.client.0.smithi111.stdout:2/474: fdatasync dd/d17/d34/d73/d75/f8e 0 2024-03-14T13:02:16.765 INFO:tasks.workunit.client.0.smithi111.stdout:2/475: fdatasync dd/d17/d34/d73/d49/f51 0 2024-03-14T13:02:16.766 INFO:tasks.workunit.client.0.smithi111.stdout:5/564: dread d2/d16/d49/d4c/f73 [0,4194304] 0 2024-03-14T13:02:16.766 INFO:tasks.workunit.client.0.smithi111.stdout:2/476: fsync dd/d11/d1a/f62 0 2024-03-14T13:02:16.767 INFO:tasks.workunit.client.0.smithi111.stdout:2/477: dread - dd/d17/f71 zero size 2024-03-14T13:02:16.778 INFO:tasks.workunit.client.0.smithi111.stdout:5/565: read d2/d16/d49/d7c/d95/fa3 [3844428,42153] 0 2024-03-14T13:02:16.780 INFO:tasks.workunit.client.0.smithi111.stdout:5/566: truncate d2/d16/d49/d33/fa6 608202 0 2024-03-14T13:02:16.781 INFO:tasks.workunit.client.1.smithi137.stdout:4/386: dread d4/f2e [0,4194304] 0 2024-03-14T13:02:16.781 INFO:tasks.workunit.client.0.smithi111.stdout:5/567: write d2/d16/d49/d7c/d95/d6a/f6e [3569211,79788] 0 2024-03-14T13:02:16.783 INFO:tasks.workunit.client.0.smithi111.stdout:8/629: dread d3/dc/f11 [0,4194304] 0 2024-03-14T13:02:16.784 INFO:tasks.workunit.client.0.smithi111.stdout:5/568: truncate d2/d16/f38 986648 0 2024-03-14T13:02:16.788 INFO:tasks.workunit.client.0.smithi111.stdout:8/630: link d3/d22/d30/f7d d3/d22/fc4 0 2024-03-14T13:02:16.791 INFO:tasks.workunit.client.0.smithi111.stdout:8/631: write d3/d22/fc4 [474505,51782] 0 2024-03-14T13:02:16.792 INFO:tasks.workunit.client.0.smithi111.stdout:5/569: dwrite d2/d16/d49/d7c/d95/d36/f3c [0,4194304] 0 2024-03-14T13:02:16.795 INFO:tasks.workunit.client.1.smithi137.stdout:4/387: dwrite d4/d9/f1f [0,4194304] 0 2024-03-14T13:02:16.797 INFO:tasks.workunit.client.0.smithi111.stdout:5/570: mkdir d2/d16/d49/d44/d4a/d9e/db7/dbb 0 2024-03-14T13:02:16.798 INFO:tasks.workunit.client.0.smithi111.stdout:8/632: link d3/d5/c2f d3/d1b/d20/d35/d40/d48/cc5 0 2024-03-14T13:02:16.800 INFO:tasks.workunit.client.0.smithi111.stdout:5/571: write d2/d16/d49/d44/d72/f3e [732900,35634] 0 2024-03-14T13:02:16.802 INFO:tasks.workunit.client.0.smithi111.stdout:8/633: getdents d3/d1b/d20/d35/d40 0 2024-03-14T13:02:16.803 INFO:tasks.workunit.client.0.smithi111.stdout:8/634: chown d3/d22/d30/f43 33993865 1 2024-03-14T13:02:16.804 INFO:tasks.workunit.client.0.smithi111.stdout:5/572: rename d2/d45/f34 to d2/d16/d49/d4c/fbc 0 2024-03-14T13:02:16.806 INFO:tasks.workunit.client.0.smithi111.stdout:5/573: chown d2/d16/d49/d44/d72/l81 1768 1 2024-03-14T13:02:16.809 INFO:tasks.workunit.client.0.smithi111.stdout:5/574: mkdir d2/d45/d76/d9a/dbd 0 2024-03-14T13:02:16.812 INFO:tasks.workunit.client.1.smithi137.stdout:4/388: chown d4/d9/d3a/d70/d80 102 1 2024-03-14T13:02:16.813 INFO:tasks.workunit.client.0.smithi111.stdout:5/575: getdents d2/d16/d49/d7c/d95/d36 0 2024-03-14T13:02:16.816 INFO:tasks.workunit.client.0.smithi111.stdout:8/635: dread d3/d5/f51 [0,4194304] 0 2024-03-14T13:02:16.837 INFO:tasks.workunit.client.0.smithi111.stdout:5/576: write d2/d13/f99 [4720214,113041] 0 2024-03-14T13:02:16.838 INFO:tasks.workunit.client.0.smithi111.stdout:8/636: dread f0 [4194304,4194304] 0 2024-03-14T13:02:16.841 INFO:tasks.workunit.client.0.smithi111.stdout:8/637: chown d3/d1b/d20/d35/d40/d48/d77/l8c 8762 1 2024-03-14T13:02:16.841 INFO:tasks.workunit.client.0.smithi111.stdout:5/577: creat d2/d16/d49/d44/d4a/d9e/fbe x:0 0 0 2024-03-14T13:02:16.843 INFO:tasks.workunit.client.0.smithi111.stdout:8/638: symlink d3/d22/lc6 0 2024-03-14T13:02:16.843 INFO:tasks.workunit.client.0.smithi111.stdout:5/578: write d2/d16/d49/d7c/d95/f5e [711653,85778] 0 2024-03-14T13:02:16.844 INFO:tasks.workunit.client.0.smithi111.stdout:8/639: fsync d3/d22/fc4 0 2024-03-14T13:02:16.845 INFO:tasks.workunit.client.0.smithi111.stdout:5/579: write d2/d16/d49/d33/fa6 [941985,33114] 0 2024-03-14T13:02:16.846 INFO:tasks.workunit.client.1.smithi137.stdout:4/389: write d4/d9/d3a/d41/d38/d55/f64 [2266304,7093] 0 2024-03-14T13:02:16.847 INFO:tasks.workunit.client.0.smithi111.stdout:8/640: mknod d3/d1b/dc1/cc7 0 2024-03-14T13:02:16.849 INFO:tasks.workunit.client.0.smithi111.stdout:8/641: mknod d3/d1b/dc1/cc8 0 2024-03-14T13:02:16.849 INFO:tasks.workunit.client.0.smithi111.stdout:5/580: truncate d2/d16/f38 672522 0 2024-03-14T13:02:16.851 INFO:tasks.workunit.client.1.smithi137.stdout:8/431: rmdir d1 39 2024-03-14T13:02:16.851 INFO:tasks.workunit.client.0.smithi111.stdout:8/642: write d3/dc/f4e [4978998,109342] 0 2024-03-14T13:02:16.851 INFO:tasks.workunit.client.0.smithi111.stdout:5/581: symlink d2/d45/d76/d8d/lbf 0 2024-03-14T13:02:16.854 INFO:tasks.workunit.client.1.smithi137.stdout:8/432: chown d1/dc/d4e/f7a 8340402 1 2024-03-14T13:02:16.854 INFO:tasks.workunit.client.0.smithi111.stdout:8/643: unlink d3/d5/f12 0 2024-03-14T13:02:16.855 INFO:tasks.workunit.client.0.smithi111.stdout:5/582: mknod d2/d45/d76/d9a/dbd/cc0 0 2024-03-14T13:02:16.856 INFO:tasks.workunit.client.1.smithi137.stdout:4/390: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:16.856 INFO:tasks.workunit.client.0.smithi111.stdout:5/583: dread - d2/d13/fb9 zero size 2024-03-14T13:02:16.856 INFO:tasks.workunit.client.0.smithi111.stdout:8/644: symlink d3/d1b/d20/d35/d40/d48/d7e/lc9 0 2024-03-14T13:02:16.858 INFO:tasks.workunit.client.1.smithi137.stdout:8/433: chown d1/dc/d3c/d87/d34/d97/f2a 270 1 2024-03-14T13:02:16.858 INFO:tasks.workunit.client.0.smithi111.stdout:5/584: mkdir d2/d16/d49/d7c/d95/d6a/dc1 0 2024-03-14T13:02:16.859 INFO:tasks.workunit.client.1.smithi137.stdout:1/477: sync 2024-03-14T13:02:16.859 INFO:tasks.workunit.client.1.smithi137.stdout:4/391: symlink d4/d27/d5c/d6d/l86 0 2024-03-14T13:02:16.860 INFO:tasks.workunit.client.1.smithi137.stdout:8/434: symlink d1/d4a/l99 0 2024-03-14T13:02:16.860 INFO:tasks.workunit.client.0.smithi111.stdout:8/645: creat d3/d1b/d20/d35/d40/d48/d7e/fca x:0 0 0 2024-03-14T13:02:16.861 INFO:tasks.workunit.client.1.smithi137.stdout:4/392: mknod d4/d9/d3a/d70/d80/c87 0 2024-03-14T13:02:16.863 INFO:tasks.workunit.client.0.smithi111.stdout:8/646: symlink d3/dc/d31/d9d/lcb 0 2024-03-14T13:02:16.867 INFO:tasks.workunit.client.0.smithi111.stdout:8/647: creat d3/d22/fcc x:0 0 0 2024-03-14T13:02:16.867 INFO:tasks.workunit.client.1.smithi137.stdout:8/435: dwrite d1/dc/f5f [0,4194304] 0 2024-03-14T13:02:16.869 INFO:tasks.workunit.client.1.smithi137.stdout:1/478: link d7/d9/d19/d1f/d26/c27 d7/d9/d19/d1f/c96 0 2024-03-14T13:02:16.870 INFO:tasks.workunit.client.0.smithi111.stdout:5/585: dread d2/d16/d49/f11 [0,4194304] 0 2024-03-14T13:02:16.871 INFO:tasks.workunit.client.1.smithi137.stdout:8/436: write d1/f31 [1739802,86248] 0 2024-03-14T13:02:16.871 INFO:tasks.workunit.client.1.smithi137.stdout:4/393: dwrite d4/d9/f1f [0,4194304] 0 2024-03-14T13:02:16.872 INFO:tasks.workunit.client.1.smithi137.stdout:4/394: stat d4/d9/c4b 0 2024-03-14T13:02:16.873 INFO:tasks.workunit.client.0.smithi111.stdout:5/586: symlink d2/d45/d79/lc2 0 2024-03-14T13:02:16.874 INFO:tasks.workunit.client.0.smithi111.stdout:8/648: dwrite d3/d22/fcc [0,4194304] 0 2024-03-14T13:02:16.875 INFO:tasks.workunit.client.0.smithi111.stdout:5/587: fsync d2/d16/d49/d44/d4a/f8c 0 2024-03-14T13:02:16.879 INFO:tasks.workunit.client.1.smithi137.stdout:1/479: creat d7/d9/d19/d1f/f97 x:0 0 0 2024-03-14T13:02:16.879 INFO:tasks.workunit.client.0.smithi111.stdout:8/649: mkdir d3/d1b/d20/d35/d40/d48/dcd 0 2024-03-14T13:02:16.879 INFO:tasks.workunit.client.0.smithi111.stdout:5/588: fsync d2/d16/d49/d44/f8e 0 2024-03-14T13:02:16.881 INFO:tasks.workunit.client.1.smithi137.stdout:8/437: rename d1/dc/d10/f64 to d1/dc/d3c/d87/d34/f9a 0 2024-03-14T13:02:16.882 INFO:tasks.workunit.client.0.smithi111.stdout:8/650: write d3/d22/f88 [792204,111592] 0 2024-03-14T13:02:16.883 INFO:tasks.workunit.client.1.smithi137.stdout:1/480: symlink d7/d4d/d6b/l98 0 2024-03-14T13:02:16.883 INFO:tasks.workunit.client.0.smithi111.stdout:5/589: rename d2/d16/f42 to d2/d16/d49/d33/d68/fc3 0 2024-03-14T13:02:16.884 INFO:tasks.workunit.client.0.smithi111.stdout:8/651: write d3/d1b/d20/d35/d40/f76 [3670287,64144] 0 2024-03-14T13:02:16.884 INFO:tasks.workunit.client.1.smithi137.stdout:8/438: chown d1/dc/l5d 93005684 1 2024-03-14T13:02:16.885 INFO:tasks.workunit.client.1.smithi137.stdout:4/395: dwrite d4/d9/d3a/d41/d38/d55/f59 [0,4194304] 0 2024-03-14T13:02:16.886 INFO:tasks.workunit.client.0.smithi111.stdout:8/652: rmdir d3/dc/d45/d8d 39 2024-03-14T13:02:16.887 INFO:tasks.workunit.client.1.smithi137.stdout:8/439: truncate d1/f6 5947158 0 2024-03-14T13:02:16.888 INFO:tasks.workunit.client.0.smithi111.stdout:5/590: rename d2/l1a to d2/d45/d76/lc4 0 2024-03-14T13:02:16.889 INFO:tasks.workunit.client.0.smithi111.stdout:8/653: symlink d3/dc/d31/lce 0 2024-03-14T13:02:16.889 INFO:tasks.workunit.client.1.smithi137.stdout:8/440: dread - d1/dc/d3c/d87/d34/d97/d13/f73 zero size 2024-03-14T13:02:16.890 INFO:tasks.workunit.client.1.smithi137.stdout:4/396: read d4/d9/d3a/d41/d38/d55/f64 [908932,111355] 0 2024-03-14T13:02:16.891 INFO:tasks.workunit.client.1.smithi137.stdout:1/481: mknod d7/d9/d19/d1f/d26/d7d/d91/c99 0 2024-03-14T13:02:16.893 INFO:tasks.workunit.client.0.smithi111.stdout:5/591: dread d2/d16/d49/d33/f43 [0,4194304] 0 2024-03-14T13:02:16.893 INFO:tasks.workunit.client.0.smithi111.stdout:8/654: dread d3/d22/fc4 [0,4194304] 0 2024-03-14T13:02:16.894 INFO:tasks.workunit.client.1.smithi137.stdout:1/482: dread d7/d9/d19/d40/d42/f49 [0,4194304] 0 2024-03-14T13:02:16.894 INFO:tasks.workunit.client.1.smithi137.stdout:4/397: mknod d4/d10/d3d/c88 0 2024-03-14T13:02:16.895 INFO:tasks.workunit.client.1.smithi137.stdout:1/483: dread - d7/f88 zero size 2024-03-14T13:02:16.895 INFO:tasks.workunit.client.1.smithi137.stdout:4/398: stat d4/d10/d3d/c88 0 2024-03-14T13:02:16.895 INFO:tasks.workunit.client.1.smithi137.stdout:8/441: symlink d1/dc/d3c/d87/d34/d82/d94/l9b 0 2024-03-14T13:02:16.896 INFO:tasks.workunit.client.0.smithi111.stdout:8/655: creat d3/d49/fcf x:0 0 0 2024-03-14T13:02:16.897 INFO:tasks.workunit.client.0.smithi111.stdout:5/592: rmdir d2/d16/d49/d7c/d95/d6a/dc1 0 2024-03-14T13:02:16.898 INFO:tasks.workunit.client.1.smithi137.stdout:8/442: symlink d1/l9c 0 2024-03-14T13:02:16.900 INFO:tasks.workunit.client.0.smithi111.stdout:5/593: unlink d2/d13/f15 0 2024-03-14T13:02:16.900 INFO:tasks.workunit.client.1.smithi137.stdout:8/443: fdatasync d1/dc/d3c/d87/d34/f47 0 2024-03-14T13:02:16.901 INFO:tasks.workunit.client.0.smithi111.stdout:5/594: fdatasync d2/d16/d49/d33/f7b 0 2024-03-14T13:02:16.902 INFO:tasks.workunit.client.1.smithi137.stdout:8/444: write d1/dc/d3c/d87/d34/d82/f8d [787691,82965] 0 2024-03-14T13:02:16.902 INFO:tasks.workunit.client.0.smithi111.stdout:5/595: write d2/d45/f66 [48751,87253] 0 2024-03-14T13:02:16.903 INFO:tasks.workunit.client.0.smithi111.stdout:5/596: fdatasync d2/d45/d76/d8d/fae 0 2024-03-14T13:02:16.904 INFO:tasks.workunit.client.1.smithi137.stdout:8/445: write d1/f6 [166781,2464] 0 2024-03-14T13:02:16.905 INFO:tasks.workunit.client.1.smithi137.stdout:8/446: chown d1/dc/d3c/d87/d34/d97/d56/l91 7345 1 2024-03-14T13:02:16.907 INFO:tasks.workunit.client.0.smithi111.stdout:5/597: getdents d2/d16/d49/d33 0 2024-03-14T13:02:16.907 INFO:tasks.workunit.client.1.smithi137.stdout:8/447: creat d1/d4a/f9d x:0 0 0 2024-03-14T13:02:16.909 INFO:tasks.workunit.client.0.smithi111.stdout:5/598: creat d2/d45/d79/fc5 x:0 0 0 2024-03-14T13:02:16.910 INFO:tasks.workunit.client.0.smithi111.stdout:5/599: chown d2/d16/d49/f27 3403 1 2024-03-14T13:02:16.912 INFO:tasks.workunit.client.1.smithi137.stdout:8/448: rename d1/dc/d3c/d87/d34/f47 to d1/dc/d3c/d87/d34/d97/d13/f9e 0 2024-03-14T13:02:16.915 INFO:tasks.workunit.client.1.smithi137.stdout:8/449: dread d1/dc/d3c/d87/d34/d97/d13/f26 [0,4194304] 0 2024-03-14T13:02:16.917 INFO:tasks.workunit.client.0.smithi111.stdout:5/600: dwrite d2/d16/d49/d33/f7b [0,4194304] 0 2024-03-14T13:02:16.918 INFO:tasks.workunit.client.1.smithi137.stdout:1/484: dread d7/d9/d19/f2f [4194304,4194304] 0 2024-03-14T13:02:16.921 INFO:tasks.workunit.client.1.smithi137.stdout:1/485: read d7/d9/f4f [335303,57920] 0 2024-03-14T13:02:16.925 INFO:tasks.workunit.client.1.smithi137.stdout:4/399: write d4/d9/d3a/d41/d38/f3b [277462,114931] 0 2024-03-14T13:02:16.925 INFO:tasks.workunit.client.1.smithi137.stdout:8/450: dwrite d1/f49 [0,4194304] 0 2024-03-14T13:02:16.931 INFO:tasks.workunit.client.1.smithi137.stdout:8/451: write d1/dc/d3c/d87/d34/d97/d13/f22 [1385745,116518] 0 2024-03-14T13:02:16.933 INFO:tasks.workunit.client.0.smithi111.stdout:8/656: dwrite d3/d22/d30/f97 [0,4194304] 0 2024-03-14T13:02:16.936 INFO:tasks.workunit.client.1.smithi137.stdout:8/452: rename d1/dc/d3c/d87/d34/d97/d56 to d1/dc/d77/d9f 0 2024-03-14T13:02:16.936 INFO:tasks.workunit.client.1.smithi137.stdout:4/400: getdents d4/d27/d5c 0 2024-03-14T13:02:16.937 INFO:tasks.workunit.client.1.smithi137.stdout:4/401: stat d4/d10/f3f 0 2024-03-14T13:02:16.939 INFO:tasks.workunit.client.0.smithi111.stdout:5/601: dwrite d2/d13/f4f [0,4194304] 0 2024-03-14T13:02:16.940 INFO:tasks.workunit.client.1.smithi137.stdout:4/402: creat d4/f89 x:0 0 0 2024-03-14T13:02:16.941 INFO:tasks.workunit.client.1.smithi137.stdout:1/486: dread d7/f10 [0,4194304] 0 2024-03-14T13:02:16.942 INFO:tasks.workunit.client.1.smithi137.stdout:4/403: mknod d4/d10/c8a 0 2024-03-14T13:02:16.942 INFO:tasks.workunit.client.0.smithi111.stdout:8/657: link d3/d49/c86 d3/d22/d4a/cd0 0 2024-03-14T13:02:16.943 INFO:tasks.workunit.client.1.smithi137.stdout:1/487: write d7/d9/d19/d40/d50/f80 [672320,47453] 0 2024-03-14T13:02:16.945 INFO:tasks.workunit.client.0.smithi111.stdout:5/602: read d2/d16/d49/d7c/d95/d6a/f71 [2111192,34965] 0 2024-03-14T13:02:16.946 INFO:tasks.workunit.client.0.smithi111.stdout:5/603: read d2/d16/d49/d7c/d95/fa3 [255365,52704] 0 2024-03-14T13:02:16.947 INFO:tasks.workunit.client.0.smithi111.stdout:5/604: dread - d2/d16/d49/d44/d4a/d9e/fbe zero size 2024-03-14T13:02:16.948 INFO:tasks.workunit.client.0.smithi111.stdout:5/605: chown d2/d45/d76/fab 129 1 2024-03-14T13:02:16.949 INFO:tasks.workunit.client.1.smithi137.stdout:4/404: dwrite d4/f31 [0,4194304] 0 2024-03-14T13:02:16.949 INFO:tasks.workunit.client.0.smithi111.stdout:5/606: write d2/d45/d76/d9a/d9f/fb5 [188299,47996] 0 2024-03-14T13:02:16.949 INFO:tasks.workunit.client.0.smithi111.stdout:8/658: read d3/dc/f6c [2328557,81506] 0 2024-03-14T13:02:16.952 INFO:tasks.workunit.client.1.smithi137.stdout:4/405: write d4/d9/d3a/d41/d38/f4c [476846,88643] 0 2024-03-14T13:02:16.952 INFO:tasks.workunit.client.0.smithi111.stdout:5/607: creat d2/d16/d49/d7c/fc6 x:0 0 0 2024-03-14T13:02:16.954 INFO:tasks.workunit.client.0.smithi111.stdout:5/608: readlink d2/d16/d49/d44/l56 0 2024-03-14T13:02:16.955 INFO:tasks.workunit.client.0.smithi111.stdout:5/609: write d2/d16/d49/d44/d4a/f8c [1611346,9907] 0 2024-03-14T13:02:16.955 INFO:tasks.workunit.client.1.smithi137.stdout:4/406: rename d4/d10/d3d/c88 to d4/d9/d3a/c8b 0 2024-03-14T13:02:16.961 INFO:tasks.workunit.client.0.smithi111.stdout:8/659: write d3/d1b/d66/f7b [3067674,1959] 0 2024-03-14T13:02:16.963 INFO:tasks.workunit.client.0.smithi111.stdout:8/660: write d3/dc/f11 [2639445,69853] 0 2024-03-14T13:02:16.964 INFO:tasks.workunit.client.0.smithi111.stdout:8/661: readlink d3/d1b/d20/d35/d40/l9a 0 2024-03-14T13:02:16.965 INFO:tasks.workunit.client.0.smithi111.stdout:8/662: stat d3/l54 0 2024-03-14T13:02:16.966 INFO:tasks.workunit.client.0.smithi111.stdout:8/663: symlink d3/d49/ld1 0 2024-03-14T13:02:16.967 INFO:tasks.workunit.client.1.smithi137.stdout:1/488: truncate d7/d9/d19/d1f/f81 2209300 0 2024-03-14T13:02:16.968 INFO:tasks.workunit.client.1.smithi137.stdout:4/407: dread d4/d10/d5a/f63 [0,4194304] 0 2024-03-14T13:02:16.969 INFO:tasks.workunit.client.0.smithi111.stdout:8/664: rmdir d3/d1b/d20/d35/d40/d48/d7e 39 2024-03-14T13:02:16.970 INFO:tasks.workunit.client.1.smithi137.stdout:1/489: write d7/d31/f76 [5155220,104804] 0 2024-03-14T13:02:16.973 INFO:tasks.workunit.client.0.smithi111.stdout:8/665: getdents d3/d1b/d20 0 2024-03-14T13:02:16.973 INFO:tasks.workunit.client.1.smithi137.stdout:1/490: creat d7/f9a x:0 0 0 2024-03-14T13:02:16.974 INFO:tasks.workunit.client.1.smithi137.stdout:1/491: fsync d7/d31/d54/f62 0 2024-03-14T13:02:16.975 INFO:tasks.workunit.client.1.smithi137.stdout:4/408: link d4/d9/d3a/d41/d38/d55/f59 d4/d9/d3a/d70/d80/f8c 0 2024-03-14T13:02:16.977 INFO:tasks.workunit.client.0.smithi111.stdout:8/666: getdents d3/d1b/d34 0 2024-03-14T13:02:16.978 INFO:tasks.workunit.client.0.smithi111.stdout:5/610: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:16.980 INFO:tasks.workunit.client.0.smithi111.stdout:5/611: write d2/d45/d76/d8d/fae [644075,129527] 0 2024-03-14T13:02:16.980 INFO:tasks.workunit.client.1.smithi137.stdout:8/453: dread d1/dc/d3c/d87/f38 [0,4194304] 0 2024-03-14T13:02:16.981 INFO:tasks.workunit.client.0.smithi111.stdout:5/612: mknod d2/d45/cc7 0 2024-03-14T13:02:16.981 INFO:tasks.workunit.client.0.smithi111.stdout:5/613: chown d2/d16/d49/d44/d72 3891135 1 2024-03-14T13:02:16.982 INFO:tasks.workunit.client.1.smithi137.stdout:8/454: truncate d1/dc/d77/f78 542217 0 2024-03-14T13:02:16.983 INFO:tasks.workunit.client.1.smithi137.stdout:8/455: dread - d1/dc/d3c/d87/d34/d97/d13/f67 zero size 2024-03-14T13:02:16.983 INFO:tasks.workunit.client.0.smithi111.stdout:8/667: read d3/d1b/f6f [2186216,97271] 0 2024-03-14T13:02:16.984 INFO:tasks.workunit.client.0.smithi111.stdout:8/668: chown d3/d1b/c2d 18135639 1 2024-03-14T13:02:16.985 INFO:tasks.workunit.client.1.smithi137.stdout:8/456: creat d1/dc/d3c/d87/d34/d97/d7f/fa0 x:0 0 0 2024-03-14T13:02:16.985 INFO:tasks.workunit.client.0.smithi111.stdout:5/614: creat d2/d45/d76/d9a/fc8 x:0 0 0 2024-03-14T13:02:16.986 INFO:tasks.workunit.client.0.smithi111.stdout:8/669: unlink d3/d1b/d20/d35/d40/faa 0 2024-03-14T13:02:16.987 INFO:tasks.workunit.client.1.smithi137.stdout:8/457: rename d1/dc/d3c/d87/d34/d97/d13/f67 to d1/dc/d77/d9f/fa1 0 2024-03-14T13:02:16.988 INFO:tasks.workunit.client.0.smithi111.stdout:5/615: rmdir d2/d45/d76/d8d 39 2024-03-14T13:02:16.989 INFO:tasks.workunit.client.0.smithi111.stdout:8/670: symlink d3/d22/d4a/ld2 0 2024-03-14T13:02:16.992 INFO:tasks.workunit.client.0.smithi111.stdout:9/472: sync 2024-03-14T13:02:16.992 INFO:tasks.workunit.client.0.smithi111.stdout:7/406: sync 2024-03-14T13:02:16.993 INFO:tasks.workunit.client.0.smithi111.stdout:4/726: sync 2024-03-14T13:02:16.993 INFO:tasks.workunit.client.0.smithi111.stdout:5/616: fsync d2/d16/d49/d33/d68/fc3 0 2024-03-14T13:02:16.994 INFO:tasks.workunit.client.0.smithi111.stdout:4/727: chown d1d/d22/dc6 1461652 1 2024-03-14T13:02:16.995 INFO:tasks.workunit.client.0.smithi111.stdout:7/407: write d0/d1/d4a/f6a [427421,82081] 0 2024-03-14T13:02:16.998 INFO:tasks.workunit.client.0.smithi111.stdout:8/671: dwrite d3/d22/f88 [0,4194304] 0 2024-03-14T13:02:16.998 INFO:tasks.workunit.client.0.smithi111.stdout:7/408: write d0/d1/f28 [7265915,97781] 0 2024-03-14T13:02:16.999 INFO:tasks.workunit.client.0.smithi111.stdout:5/617: rename d2/d16/d49/cf to d2/d13/cc9 0 2024-03-14T13:02:17.001 INFO:tasks.workunit.client.0.smithi111.stdout:9/473: getdents d5/d6/d2a 0 2024-03-14T13:02:17.004 INFO:tasks.workunit.client.0.smithi111.stdout:4/728: rename d1d/l1e to d1d/d22/d40/d69/d86/db1/l100 0 2024-03-14T13:02:17.005 INFO:tasks.workunit.client.0.smithi111.stdout:9/474: symlink d5/d6/d14/la1 0 2024-03-14T13:02:17.009 INFO:tasks.workunit.client.1.smithi137.stdout:3/311: sync 2024-03-14T13:02:17.009 INFO:tasks.workunit.client.1.smithi137.stdout:0/367: sync 2024-03-14T13:02:17.009 INFO:tasks.workunit.client.1.smithi137.stdout:7/337: sync 2024-03-14T13:02:17.009 INFO:tasks.workunit.client.1.smithi137.stdout:5/358: sync 2024-03-14T13:02:17.009 INFO:tasks.workunit.client.1.smithi137.stdout:2/461: sync 2024-03-14T13:02:17.012 INFO:tasks.workunit.client.1.smithi137.stdout:0/368: dread - d1/d5/d1b/d42/f32 zero size 2024-03-14T13:02:17.014 INFO:tasks.workunit.client.0.smithi111.stdout:4/729: dread d1d/d22/f55 [0,4194304] 0 2024-03-14T13:02:17.014 INFO:tasks.workunit.client.1.smithi137.stdout:3/312: fdatasync d4/df/d34/f5a 0 2024-03-14T13:02:17.016 INFO:tasks.workunit.client.0.smithi111.stdout:4/730: fdatasync d1d/d22/d40/d69/d86/db1/fbf 0 2024-03-14T13:02:17.016 INFO:tasks.workunit.client.1.smithi137.stdout:8/458: dwrite d1/d5e/f63 [0,4194304] 0 2024-03-14T13:02:17.019 INFO:tasks.workunit.client.0.smithi111.stdout:5/618: dread d2/d16/d49/d7c/d95/f24 [0,4194304] 0 2024-03-14T13:02:17.019 INFO:tasks.workunit.client.0.smithi111.stdout:5/619: chown d2/d45/c2e 4140 1 2024-03-14T13:02:17.021 INFO:tasks.workunit.client.0.smithi111.stdout:8/672: rmdir d3/d1b/d20/d35 39 2024-03-14T13:02:17.023 INFO:tasks.workunit.client.1.smithi137.stdout:3/313: symlink d4/d22/l69 0 2024-03-14T13:02:17.024 INFO:tasks.workunit.client.1.smithi137.stdout:7/338: dwrite d3/d6/f63 [0,4194304] 0 2024-03-14T13:02:17.025 INFO:tasks.workunit.client.1.smithi137.stdout:7/339: stat d3/d5/f67 0 2024-03-14T13:02:17.025 INFO:tasks.workunit.client.1.smithi137.stdout:0/369: mknod d1/d44/d79/c7f 0 2024-03-14T13:02:17.026 INFO:tasks.workunit.client.0.smithi111.stdout:5/620: unlink d2/d16/d49/d33/d68/c92 0 2024-03-14T13:02:17.028 INFO:tasks.workunit.client.1.smithi137.stdout:8/459: mknod d1/dc/d3c/d87/d34/d82/ca2 0 2024-03-14T13:02:17.031 INFO:tasks.workunit.client.0.smithi111.stdout:8/673: symlink d3/d1b/d66/ld3 0 2024-03-14T13:02:17.031 INFO:tasks.workunit.client.1.smithi137.stdout:7/340: creat d3/d5/d23/d2c/d5d/f6b x:0 0 0 2024-03-14T13:02:17.032 INFO:tasks.workunit.client.0.smithi111.stdout:4/731: link d1d/d22/lb4 d1d/d22/l101 0 2024-03-14T13:02:17.033 INFO:tasks.workunit.client.1.smithi137.stdout:0/370: mkdir d1/d1f/d80 0 2024-03-14T13:02:17.033 INFO:tasks.workunit.client.1.smithi137.stdout:7/341: stat d3/d5/d23/d3f/d5b/l69 0 2024-03-14T13:02:17.035 INFO:tasks.workunit.client.0.smithi111.stdout:5/621: unlink d2/d13/c7a 0 2024-03-14T13:02:17.035 INFO:tasks.workunit.client.1.smithi137.stdout:8/460: rmdir d1/dc/d3c/d87/d34/d82/d94 39 2024-03-14T13:02:17.036 INFO:tasks.workunit.client.0.smithi111.stdout:8/674: mknod d3/d1b/d66/cd4 0 2024-03-14T13:02:17.036 INFO:tasks.workunit.client.1.smithi137.stdout:0/371: truncate d1/f51 4969996 0 2024-03-14T13:02:17.037 INFO:tasks.workunit.client.0.smithi111.stdout:7/409: dread d0/d1/d5/f1a [4194304,4194304] 0 2024-03-14T13:02:17.038 INFO:tasks.workunit.client.0.smithi111.stdout:7/410: fdatasync d0/d1/d21/d2a/f7d 0 2024-03-14T13:02:17.038 INFO:tasks.workunit.client.0.smithi111.stdout:5/622: creat d2/d16/d49/d44/fca x:0 0 0 2024-03-14T13:02:17.039 INFO:tasks.workunit.client.0.smithi111.stdout:4/732: getdents d1d/d22/d30/d5e/db8/ddf/de3 0 2024-03-14T13:02:17.039 INFO:tasks.workunit.client.1.smithi137.stdout:7/342: read - d3/d5/d23/d2c/f53 zero size 2024-03-14T13:02:17.039 INFO:tasks.workunit.client.1.smithi137.stdout:8/461: readlink d1/dc/lf 0 2024-03-14T13:02:17.039 INFO:tasks.workunit.client.1.smithi137.stdout:3/314: creat d4/de/d16/f6a x:0 0 0 2024-03-14T13:02:17.040 INFO:tasks.workunit.client.1.smithi137.stdout:4/409: dwrite d4/d9/d3a/d41/d38/d55/f64 [0,4194304] 0 2024-03-14T13:02:17.040 INFO:tasks.workunit.client.0.smithi111.stdout:5/623: chown d2/d16/d49/d7c/d95/f24 612 1 2024-03-14T13:02:17.040 INFO:tasks.workunit.client.0.smithi111.stdout:7/411: write d0/d1/d22/f42 [2244624,111357] 0 2024-03-14T13:02:17.041 INFO:tasks.workunit.client.1.smithi137.stdout:0/372: creat d1/d44/d72/d7e/f81 x:0 0 0 2024-03-14T13:02:17.041 INFO:tasks.workunit.client.0.smithi111.stdout:8/675: rename d3/d1b/d34/f3c to d3/dc/d31/d9d/fd5 0 2024-03-14T13:02:17.042 INFO:tasks.workunit.client.0.smithi111.stdout:5/624: fsync d2/d16/d49/d44/fa7 0 2024-03-14T13:02:17.045 INFO:tasks.workunit.client.1.smithi137.stdout:8/462: mkdir d1/dc/d71/da3 0 2024-03-14T13:02:17.045 INFO:tasks.workunit.client.1.smithi137.stdout:3/315: truncate d4/d19/f66 498508 0 2024-03-14T13:02:17.045 INFO:tasks.workunit.client.0.smithi111.stdout:8/676: creat d3/d1b/d34/fd6 x:0 0 0 2024-03-14T13:02:17.046 INFO:tasks.workunit.client.1.smithi137.stdout:8/463: fsync d1/dc/d71/f72 0 2024-03-14T13:02:17.046 INFO:tasks.workunit.client.1.smithi137.stdout:3/316: chown d4/df/f30 17073403 1 2024-03-14T13:02:17.046 INFO:tasks.workunit.client.1.smithi137.stdout:7/343: fdatasync d3/d6/f45 0 2024-03-14T13:02:17.047 INFO:tasks.workunit.client.1.smithi137.stdout:0/373: mkdir d1/d1f/d82 0 2024-03-14T13:02:17.047 INFO:tasks.workunit.client.1.smithi137.stdout:4/410: mkdir d4/d9/d3a/d70/d80/d8d 0 2024-03-14T13:02:17.048 INFO:tasks.workunit.client.1.smithi137.stdout:3/317: mkdir d4/df/d34/d6b 0 2024-03-14T13:02:17.048 INFO:tasks.workunit.client.1.smithi137.stdout:0/374: chown d1/d44/d79 7 1 2024-03-14T13:02:17.049 INFO:tasks.workunit.client.1.smithi137.stdout:4/411: stat d4/d10/d5a/f71 0 2024-03-14T13:02:17.052 INFO:tasks.workunit.client.1.smithi137.stdout:7/344: write d3/d6/d14/d22/f43 [2701110,7025] 0 2024-03-14T13:02:17.055 INFO:tasks.workunit.client.1.smithi137.stdout:4/412: fsync d4/d10/d42/f75 0 2024-03-14T13:02:17.056 INFO:tasks.workunit.client.1.smithi137.stdout:8/464: dread d1/dc/d10/f27 [0,4194304] 0 2024-03-14T13:02:17.057 INFO:tasks.workunit.client.1.smithi137.stdout:4/413: read - d4/d27/d5c/d6d/d58/f7e zero size 2024-03-14T13:02:17.058 INFO:tasks.workunit.client.1.smithi137.stdout:0/375: rmdir d1/d5/d1b/d1e/d25/d26/d6c 0 2024-03-14T13:02:17.059 INFO:tasks.workunit.client.1.smithi137.stdout:8/465: creat d1/dc/d10/fa4 x:0 0 0 2024-03-14T13:02:17.061 INFO:tasks.workunit.client.1.smithi137.stdout:3/318: rename d4/df/d34/c56 to d4/de/d16/c6c 0 2024-03-14T13:02:17.061 INFO:tasks.workunit.client.1.smithi137.stdout:4/414: rmdir d4/d27/d5c/d6d/d58/d65 39 2024-03-14T13:02:17.066 INFO:tasks.workunit.client.1.smithi137.stdout:8/466: rename d1/dc/lf to d1/dc/d3c/d87/d34/d97/d7f/la5 0 2024-03-14T13:02:17.067 INFO:tasks.workunit.client.1.smithi137.stdout:0/376: dwrite d1/f3e [0,4194304] 0 2024-03-14T13:02:17.069 INFO:tasks.workunit.client.1.smithi137.stdout:8/467: chown d1/l42 36922 1 2024-03-14T13:02:17.071 INFO:tasks.workunit.client.1.smithi137.stdout:3/319: rename d4/de/d16/f5b to d4/df/d34/d6b/f6d 0 2024-03-14T13:02:17.071 INFO:tasks.workunit.client.1.smithi137.stdout:3/320: chown d4/d19 8368 1 2024-03-14T13:02:17.073 INFO:tasks.workunit.client.1.smithi137.stdout:3/321: write d4/f37 [348383,43287] 0 2024-03-14T13:02:17.073 INFO:tasks.workunit.client.1.smithi137.stdout:8/468: rmdir d1/dc/d3c/d87/d34 39 2024-03-14T13:02:17.077 INFO:tasks.workunit.client.1.smithi137.stdout:8/469: write d1/dc/d3c/d87/d34/f65 [638129,112442] 0 2024-03-14T13:02:17.081 INFO:tasks.workunit.client.1.smithi137.stdout:4/415: dread d4/d9/d3a/d41/d38/f4c [0,4194304] 0 2024-03-14T13:02:17.081 INFO:tasks.workunit.client.1.smithi137.stdout:8/470: creat d1/dc/d3c/d87/d34/fa6 x:0 0 0 2024-03-14T13:02:17.085 INFO:tasks.workunit.client.1.smithi137.stdout:0/377: dread d1/d11/f50 [4194304,4194304] 0 2024-03-14T13:02:17.087 INFO:tasks.workunit.client.1.smithi137.stdout:8/471: creat d1/dc/d3c/d87/d34/d82/fa7 x:0 0 0 2024-03-14T13:02:17.090 INFO:tasks.workunit.client.1.smithi137.stdout:0/378: fsync d1/d5/d1b/d42/d3a/f48 0 2024-03-14T13:02:17.094 INFO:tasks.workunit.client.1.smithi137.stdout:0/379: write d1/d5/d1b/d42/f32 [370747,122067] 0 2024-03-14T13:02:17.096 INFO:tasks.workunit.client.1.smithi137.stdout:3/322: dwrite d4/df/f30 [0,4194304] 0 2024-03-14T13:02:17.099 INFO:tasks.workunit.client.1.smithi137.stdout:0/380: read d1/d11/f50 [4325532,46221] 0 2024-03-14T13:02:17.100 INFO:tasks.workunit.client.1.smithi137.stdout:3/323: write d4/de/d16/f1a [2051959,29986] 0 2024-03-14T13:02:17.102 INFO:tasks.workunit.client.1.smithi137.stdout:8/472: dwrite d1/dc/d3c/d87/d34/d97/d7f/f8a [0,4194304] 0 2024-03-14T13:02:17.102 INFO:tasks.workunit.client.1.smithi137.stdout:0/381: read - d1/d5/d1b/d1e/d53/d6d/f74 zero size 2024-03-14T13:02:17.104 INFO:tasks.workunit.client.1.smithi137.stdout:4/416: dread d4/d10/f3f [0,4194304] 0 2024-03-14T13:02:17.109 INFO:tasks.workunit.client.1.smithi137.stdout:0/382: mkdir d1/d5/d1b/d1e/d25/d55/d73/d83 0 2024-03-14T13:02:17.111 INFO:tasks.workunit.client.1.smithi137.stdout:8/473: symlink d1/dc/d3c/d87/d34/d82/la8 0 2024-03-14T13:02:17.112 INFO:tasks.workunit.client.1.smithi137.stdout:0/383: mkdir d1/d44/d79/d84 0 2024-03-14T13:02:17.113 INFO:tasks.workunit.client.1.smithi137.stdout:8/474: chown d1/l6b 248140386 1 2024-03-14T13:02:17.117 INFO:tasks.workunit.client.1.smithi137.stdout:8/475: mkdir d1/dc/da9 0 2024-03-14T13:02:17.121 INFO:tasks.workunit.client.1.smithi137.stdout:3/324: dwrite d4/de/d13/f61 [0,4194304] 0 2024-03-14T13:02:17.121 INFO:tasks.workunit.client.1.smithi137.stdout:8/476: fsync d1/dc/d3c/d87/d34/d97/f2d 0 2024-03-14T13:02:17.121 INFO:tasks.workunit.client.1.smithi137.stdout:0/384: dwrite d1/d1f/f2f [0,4194304] 0 2024-03-14T13:02:17.124 INFO:tasks.workunit.client.1.smithi137.stdout:3/325: read d4/d22/f4c [2352234,7557] 0 2024-03-14T13:02:17.125 INFO:tasks.workunit.client.1.smithi137.stdout:8/477: fsync d1/dc/d3c/d87/d34/d97/d13/f73 0 2024-03-14T13:02:17.128 INFO:tasks.workunit.client.1.smithi137.stdout:0/385: unlink d1/f29 0 2024-03-14T13:02:17.130 INFO:tasks.workunit.client.1.smithi137.stdout:0/386: symlink d1/d5/d1b/d1e/l85 0 2024-03-14T13:02:17.132 INFO:tasks.workunit.client.1.smithi137.stdout:3/326: rename d4/la to d4/de/l6e 0 2024-03-14T13:02:17.137 INFO:tasks.workunit.client.1.smithi137.stdout:4/417: dread d4/d27/f2b [0,4194304] 0 2024-03-14T13:02:17.139 INFO:tasks.workunit.client.1.smithi137.stdout:8/478: dwrite d1/dc/d3c/d87/d34/d97/f2d [0,4194304] 0 2024-03-14T13:02:17.139 INFO:tasks.workunit.client.1.smithi137.stdout:0/387: truncate d1/d5/d1b/d1e/d25/d55/f7a 960145 0 2024-03-14T13:02:17.140 INFO:tasks.workunit.client.1.smithi137.stdout:4/418: write d4/d9/d3a/f56 [1147040,65622] 0 2024-03-14T13:02:17.155 INFO:tasks.workunit.client.0.smithi111.stdout:7/412: dwrite d0/d7e/f3d [4194304,4194304] 0 2024-03-14T13:02:17.155 INFO:tasks.workunit.client.0.smithi111.stdout:8/677: mknod d3/d22/da1/cd7 0 2024-03-14T13:02:17.156 INFO:tasks.workunit.client.0.smithi111.stdout:7/413: chown d0/d7e 220 1 2024-03-14T13:02:17.163 INFO:tasks.workunit.client.0.smithi111.stdout:7/414: link d0/l23 d0/d1/d21/d3a/l80 0 2024-03-14T13:02:17.164 INFO:tasks.workunit.client.0.smithi111.stdout:7/415: stat d0 0 2024-03-14T13:02:17.167 INFO:tasks.workunit.client.0.smithi111.stdout:5/625: dwrite d2/f53 [0,4194304] 0 2024-03-14T13:02:17.167 INFO:tasks.workunit.client.0.smithi111.stdout:4/733: dwrite fb [0,4194304] 0 2024-03-14T13:02:17.167 INFO:tasks.workunit.client.0.smithi111.stdout:7/416: readlink d0/d7e/l6e 0 2024-03-14T13:02:17.169 INFO:tasks.workunit.client.0.smithi111.stdout:8/678: dwrite d3/dc/f11 [4194304,4194304] 0 2024-03-14T13:02:17.175 INFO:tasks.workunit.client.0.smithi111.stdout:7/417: creat d0/d7e/f81 x:0 0 0 2024-03-14T13:02:17.178 INFO:tasks.workunit.client.0.smithi111.stdout:4/734: dread fb [0,4194304] 0 2024-03-14T13:02:17.178 INFO:tasks.workunit.client.0.smithi111.stdout:7/418: write d0/d1/d22/f42 [1737637,51745] 0 2024-03-14T13:02:17.180 INFO:tasks.workunit.client.0.smithi111.stdout:5/626: fdatasync d2/f35 0 2024-03-14T13:02:17.181 INFO:tasks.workunit.client.0.smithi111.stdout:7/419: write d0/d1/d4a/d52/f62 [579996,48835] 0 2024-03-14T13:02:17.183 INFO:tasks.workunit.client.0.smithi111.stdout:7/420: write d0/d1/d5/d56/f7c [352034,36448] 0 2024-03-14T13:02:17.184 INFO:tasks.workunit.client.0.smithi111.stdout:4/735: rename d1d/d22/d40/c60 to d1d/d22/d56/c102 0 2024-03-14T13:02:17.185 INFO:tasks.workunit.client.0.smithi111.stdout:5/627: mkdir d2/d16/d49/d4c/dcb 0 2024-03-14T13:02:17.185 INFO:tasks.workunit.client.0.smithi111.stdout:8/679: mknod d3/d1b/d20/d35/d40/d48/d77/cd8 0 2024-03-14T13:02:17.187 INFO:tasks.workunit.client.0.smithi111.stdout:7/421: getdents d0/d1/d5/d61 0 2024-03-14T13:02:17.191 INFO:tasks.workunit.client.0.smithi111.stdout:7/422: getdents d0/d1/d22/d3e 0 2024-03-14T13:02:17.192 INFO:tasks.workunit.client.0.smithi111.stdout:8/680: dwrite d3/d1b/d20/d35/d40/f76 [0,4194304] 0 2024-03-14T13:02:17.192 INFO:tasks.workunit.client.0.smithi111.stdout:7/423: write d0/d7e/f19 [1349696,30975] 0 2024-03-14T13:02:17.194 INFO:tasks.workunit.client.0.smithi111.stdout:7/424: write d0/d1/f28 [6465548,37363] 0 2024-03-14T13:02:17.195 INFO:tasks.workunit.client.0.smithi111.stdout:8/681: symlink d3/dc/d31/ld9 0 2024-03-14T13:02:17.199 INFO:tasks.workunit.client.0.smithi111.stdout:7/425: getdents d0/d1 0 2024-03-14T13:02:17.205 INFO:tasks.workunit.client.0.smithi111.stdout:7/426: unlink d0/d1/d21/d3a/l80 0 2024-03-14T13:02:17.209 INFO:tasks.workunit.client.0.smithi111.stdout:7/427: mknod d0/d1/d22/c82 0 2024-03-14T13:02:17.211 INFO:tasks.workunit.client.0.smithi111.stdout:7/428: mknod d0/d1/d5/d38/c83 0 2024-03-14T13:02:17.211 INFO:tasks.workunit.client.0.smithi111.stdout:7/429: stat d0/d1/d4a/f6a 0 2024-03-14T13:02:17.213 INFO:tasks.workunit.client.0.smithi111.stdout:7/430: readlink d0/d1/d21/d2a/l3c 0 2024-03-14T13:02:17.214 INFO:tasks.workunit.client.0.smithi111.stdout:7/431: truncate d0/d1/d21/d2a/f7d 39689 0 2024-03-14T13:02:17.216 INFO:tasks.workunit.client.0.smithi111.stdout:7/432: creat d0/d1/d21/d5d/f84 x:0 0 0 2024-03-14T13:02:17.216 INFO:tasks.workunit.client.0.smithi111.stdout:4/736: write d1d/d22/d40/d69/d86/db1/fd2 [631133,84774] 0 2024-03-14T13:02:17.217 INFO:tasks.workunit.client.0.smithi111.stdout:3/468: sync 2024-03-14T13:02:17.217 INFO:tasks.workunit.client.0.smithi111.stdout:6/439: sync 2024-03-14T13:02:17.219 INFO:tasks.workunit.client.0.smithi111.stdout:8/682: dread d3/dc/f99 [0,4194304] 0 2024-03-14T13:02:17.220 INFO:tasks.workunit.client.0.smithi111.stdout:7/433: dread d0/d1/d5/d56/f7c [0,4194304] 0 2024-03-14T13:02:17.221 INFO:tasks.workunit.client.0.smithi111.stdout:6/440: creat d0/d17/d99/fa9 x:0 0 0 2024-03-14T13:02:17.223 INFO:tasks.workunit.client.0.smithi111.stdout:3/469: creat d0/d2d/d35/d50/da2/db3/d6f/fb7 x:0 0 0 2024-03-14T13:02:17.223 INFO:tasks.workunit.client.0.smithi111.stdout:8/683: fdatasync d3/d22/d30/f47 0 2024-03-14T13:02:17.224 INFO:tasks.workunit.client.1.smithi137.stdout:3/327: rmdir d4/de/d13 39 2024-03-14T13:02:17.227 INFO:tasks.workunit.client.0.smithi111.stdout:6/441: rmdir d0/d17/d20/d4a/d59/d82 39 2024-03-14T13:02:17.228 INFO:tasks.workunit.client.1.smithi137.stdout:8/479: symlink d1/dc/d77/laa 0 2024-03-14T13:02:17.228 INFO:tasks.workunit.client.0.smithi111.stdout:4/737: dwrite d1d/d22/d30/de5/fb0 [4194304,4194304] 0 2024-03-14T13:02:17.229 INFO:tasks.workunit.client.1.smithi137.stdout:4/419: mkdir d4/d9/d8e 0 2024-03-14T13:02:17.230 INFO:tasks.workunit.client.0.smithi111.stdout:8/684: rename d3/d5/f51 to d3/d1b/dc1/fda 0 2024-03-14T13:02:17.230 INFO:tasks.workunit.client.1.smithi137.stdout:9/527: sync 2024-03-14T13:02:17.231 INFO:tasks.workunit.client.1.smithi137.stdout:6/432: sync 2024-03-14T13:02:17.231 INFO:tasks.workunit.client.1.smithi137.stdout:3/328: symlink d4/d19/l6f 0 2024-03-14T13:02:17.231 INFO:tasks.workunit.client.1.smithi137.stdout:3/329: chown d4/l8 1005618 1 2024-03-14T13:02:17.232 INFO:tasks.workunit.client.1.smithi137.stdout:8/480: creat d1/dc/d3c/d87/d34/d97/d13/fab x:0 0 0 2024-03-14T13:02:17.233 INFO:tasks.workunit.client.1.smithi137.stdout:8/481: chown d1/d4a 46 1 2024-03-14T13:02:17.236 INFO:tasks.workunit.client.0.smithi111.stdout:6/442: dread d0/d17/d20/f26 [0,4194304] 0 2024-03-14T13:02:17.237 INFO:tasks.workunit.client.1.smithi137.stdout:8/482: write d1/dc/d10/f1a [1018435,61499] 0 2024-03-14T13:02:17.237 INFO:tasks.workunit.client.1.smithi137.stdout:9/528: symlink d6/df/d88/laf 0 2024-03-14T13:02:17.238 INFO:tasks.workunit.client.0.smithi111.stdout:4/738: dwrite d1d/d22/f41 [0,4194304] 0 2024-03-14T13:02:17.239 INFO:tasks.workunit.client.0.smithi111.stdout:8/685: dwrite d3/d1b/f1d [4194304,4194304] 0 2024-03-14T13:02:17.240 INFO:tasks.workunit.client.1.smithi137.stdout:9/529: dread d6/df/d19/d3a/d40/f60 [0,4194304] 0 2024-03-14T13:02:17.241 INFO:tasks.workunit.client.1.smithi137.stdout:6/433: creat d2/d54/d55/d6d/d76/d57/f9a x:0 0 0 2024-03-14T13:02:17.242 INFO:tasks.workunit.client.1.smithi137.stdout:9/530: truncate d6/d56/f73 1668882 0 2024-03-14T13:02:17.243 INFO:tasks.workunit.client.0.smithi111.stdout:6/443: mkdir d0/d17/d20/d94/daa 0 2024-03-14T13:02:17.243 INFO:tasks.workunit.client.0.smithi111.stdout:8/686: write d3/d1b/d20/d35/fb2 [413301,51084] 0 2024-03-14T13:02:17.245 INFO:tasks.workunit.client.1.smithi137.stdout:6/434: chown d2/d6/de/d45/d59/d39/d3c/f47 15 1 2024-03-14T13:02:17.245 INFO:tasks.workunit.client.1.smithi137.stdout:9/531: truncate d6/df/d19/d33/f36 643486 0 2024-03-14T13:02:17.247 INFO:tasks.workunit.client.0.smithi111.stdout:6/444: symlink d0/d17/lab 0 2024-03-14T13:02:17.249 INFO:tasks.workunit.client.1.smithi137.stdout:6/435: stat d2/f17 0 2024-03-14T13:02:17.249 INFO:tasks.workunit.client.0.smithi111.stdout:8/687: mkdir d3/dc/ddb 0 2024-03-14T13:02:17.251 INFO:tasks.workunit.client.0.smithi111.stdout:4/739: dwrite d1d/d22/d23/fc7 [0,4194304] 0 2024-03-14T13:02:17.252 INFO:tasks.workunit.client.1.smithi137.stdout:9/532: rename d6/df/d19/d1f/f4a to d6/df/d19/d3a/d40/fb0 0 2024-03-14T13:02:17.253 INFO:tasks.workunit.client.1.smithi137.stdout:8/483: rename d1/c3b to d1/cac 0 2024-03-14T13:02:17.254 INFO:tasks.workunit.client.1.smithi137.stdout:8/484: rename d1/dc/d4e to d1/dc/d4e/dad 22 2024-03-14T13:02:17.254 INFO:tasks.workunit.client.0.smithi111.stdout:7/434: dread d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:17.255 INFO:tasks.workunit.client.0.smithi111.stdout:6/445: symlink d0/d17/d20/d94/daa/lac 0 2024-03-14T13:02:17.255 INFO:tasks.workunit.client.0.smithi111.stdout:8/688: creat d3/d1b/d20/d35/d40/fdc x:0 0 0 2024-03-14T13:02:17.256 INFO:tasks.workunit.client.1.smithi137.stdout:8/485: write d1/dc/d3c/d87/d34/d82/f8d [436726,31739] 0 2024-03-14T13:02:17.257 INFO:tasks.workunit.client.0.smithi111.stdout:7/435: stat d0/d1/d4a/f6a 0 2024-03-14T13:02:17.257 INFO:tasks.workunit.client.1.smithi137.stdout:8/486: fsync d1/dc/d3c/d87/d34/fa6 0 2024-03-14T13:02:17.257 INFO:tasks.workunit.client.1.smithi137.stdout:9/533: link d6/df/d19/d3a/d40/l79 d6/df/d19/d3a/d40/lb1 0 2024-03-14T13:02:17.258 INFO:tasks.workunit.client.0.smithi111.stdout:8/689: chown c2 29285461 1 2024-03-14T13:02:17.261 INFO:tasks.workunit.client.0.smithi111.stdout:7/436: mknod d0/d1/d21/d2a/c85 0 2024-03-14T13:02:17.262 INFO:tasks.workunit.client.1.smithi137.stdout:9/534: creat d6/df/d19/d33/d8f/fb2 x:0 0 0 2024-03-14T13:02:17.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/446: creat d0/d17/d21/d42/da5/fad x:0 0 0 2024-03-14T13:02:17.263 INFO:tasks.workunit.client.0.smithi111.stdout:7/437: read - d0/d1/d22/f7b zero size 2024-03-14T13:02:17.264 INFO:tasks.workunit.client.1.smithi137.stdout:4/420: mknod d4/d9/d3a/d41/d38/d79/c8f 0 2024-03-14T13:02:17.264 INFO:tasks.workunit.client.1.smithi137.stdout:9/535: fsync d6/df/fa0 0 2024-03-14T13:02:17.265 INFO:tasks.workunit.client.1.smithi137.stdout:8/487: dwrite d1/dc/f4b [0,4194304] 0 2024-03-14T13:02:17.267 INFO:tasks.workunit.client.1.smithi137.stdout:9/536: fdatasync d6/df/d19/d1f/f66 0 2024-03-14T13:02:17.268 INFO:tasks.workunit.client.0.smithi111.stdout:8/690: dwrite d3/d1b/d20/d35/d40/fdc [0,4194304] 0 2024-03-14T13:02:17.268 INFO:tasks.workunit.client.0.smithi111.stdout:4/740: dread d1d/d22/d30/d7c/ff9 [0,4194304] 0 2024-03-14T13:02:17.270 INFO:tasks.workunit.client.0.smithi111.stdout:7/438: rename d0/l26 to d0/d1/d5/d38/d5c/l86 0 2024-03-14T13:02:17.272 INFO:tasks.workunit.client.1.smithi137.stdout:8/488: dread - d1/dc/d3c/d87/d34/d82/fa7 zero size 2024-03-14T13:02:17.274 INFO:tasks.workunit.client.1.smithi137.stdout:8/489: dread - d1/dc/d10/fa4 zero size 2024-03-14T13:02:17.274 INFO:tasks.workunit.client.0.smithi111.stdout:7/439: write d0/d11/f29 [806676,3528] 0 2024-03-14T13:02:17.276 INFO:tasks.workunit.client.0.smithi111.stdout:8/691: getdents d3/d49 0 2024-03-14T13:02:17.277 INFO:tasks.workunit.client.1.smithi137.stdout:3/330: dwrite d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:17.277 INFO:tasks.workunit.client.1.smithi137.stdout:8/490: chown d1/dc/d10/c80 30754090 1 2024-03-14T13:02:17.277 INFO:tasks.workunit.client.0.smithi111.stdout:8/692: read d3/d22/fcc [3783434,66828] 0 2024-03-14T13:02:17.281 INFO:tasks.workunit.client.1.smithi137.stdout:9/537: dwrite d6/df/d19/d3a/d40/f41 [0,4194304] 0 2024-03-14T13:02:17.282 INFO:tasks.workunit.client.0.smithi111.stdout:6/447: truncate d0/f14 4038625 0 2024-03-14T13:02:17.282 INFO:tasks.workunit.client.1.smithi137.stdout:3/331: truncate d4/de/d13/f61 4673931 0 2024-03-14T13:02:17.284 INFO:tasks.workunit.client.1.smithi137.stdout:8/491: creat d1/dc/d3c/d87/d34/d82/d96/fae x:0 0 0 2024-03-14T13:02:17.285 INFO:tasks.workunit.client.0.smithi111.stdout:7/440: dwrite d0/d1/d5/d38/f68 [4194304,4194304] 0 2024-03-14T13:02:17.286 INFO:tasks.workunit.client.1.smithi137.stdout:3/332: creat d4/d19/f70 x:0 0 0 2024-03-14T13:02:17.287 INFO:tasks.workunit.client.0.smithi111.stdout:4/741: link d1d/d22/d23/ca6 d1d/d22/d30/d5e/d6f/d7e/d66/dba/c103 0 2024-03-14T13:02:17.288 INFO:tasks.workunit.client.1.smithi137.stdout:8/492: mkdir d1/d5e/daf 0 2024-03-14T13:02:17.288 INFO:tasks.workunit.client.1.smithi137.stdout:3/333: chown d4/df/d34/l3b 177 1 2024-03-14T13:02:17.290 INFO:tasks.workunit.client.0.smithi111.stdout:4/742: creat d1d/d22/d30/d87/f104 x:0 0 0 2024-03-14T13:02:17.291 INFO:tasks.workunit.client.1.smithi137.stdout:8/493: fdatasync d1/dc/d77/d9f/fa1 0 2024-03-14T13:02:17.292 INFO:tasks.workunit.client.0.smithi111.stdout:6/448: rename d0/d17/d21/c33 to d0/d17/d20/d4a/cae 0 2024-03-14T13:02:17.294 INFO:tasks.workunit.client.0.smithi111.stdout:6/449: chown d0/d17/d20/d65/f92 13920803 1 2024-03-14T13:02:17.295 INFO:tasks.workunit.client.1.smithi137.stdout:4/421: dwrite d4/f32 [0,4194304] 0 2024-03-14T13:02:17.296 INFO:tasks.workunit.client.0.smithi111.stdout:7/441: dwrite d0/d1/d21/f3f [0,4194304] 0 2024-03-14T13:02:17.296 INFO:tasks.workunit.client.1.smithi137.stdout:4/422: stat d4/f11 0 2024-03-14T13:02:17.305 INFO:tasks.workunit.client.1.smithi137.stdout:4/423: dread d4/d9/f1f [0,4194304] 0 2024-03-14T13:02:17.311 INFO:tasks.workunit.client.1.smithi137.stdout:4/424: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:17.319 INFO:tasks.workunit.client.1.smithi137.stdout:4/425: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:17.320 INFO:tasks.workunit.client.1.smithi137.stdout:3/334: dread f1 [0,4194304] 0 2024-03-14T13:02:17.322 INFO:tasks.workunit.client.0.smithi111.stdout:7/442: fdatasync d0/d1/d21/f3f 0 2024-03-14T13:02:17.325 INFO:tasks.workunit.client.1.smithi137.stdout:4/426: link d4/c44 d4/d9/d3a/d41/c90 0 2024-03-14T13:02:17.326 INFO:tasks.workunit.client.1.smithi137.stdout:4/427: write d4/d27/f28 [851695,13868] 0 2024-03-14T13:02:17.329 INFO:tasks.workunit.client.1.smithi137.stdout:3/335: rename d4/d19/l6f to d4/de/d16/l71 0 2024-03-14T13:02:17.336 INFO:tasks.workunit.client.1.smithi137.stdout:3/336: dwrite d4/de/d16/d1c/d5d/f43 [0,4194304] 0 2024-03-14T13:02:17.338 INFO:tasks.workunit.client.1.smithi137.stdout:3/337: fdatasync d4/de/d16/d1c/f41 0 2024-03-14T13:02:17.343 INFO:tasks.workunit.client.1.smithi137.stdout:3/338: rmdir d4/de/d16 39 2024-03-14T13:02:17.344 INFO:tasks.workunit.client.1.smithi137.stdout:4/428: write d4/d9/d3a/d41/f49 [1493795,9603] 0 2024-03-14T13:02:17.345 INFO:tasks.workunit.client.1.smithi137.stdout:3/339: symlink d4/df/l72 0 2024-03-14T13:02:17.345 INFO:tasks.workunit.client.1.smithi137.stdout:4/429: readlink d4/d9/d3a/d41/l35 0 2024-03-14T13:02:17.346 INFO:tasks.workunit.client.1.smithi137.stdout:4/430: write d4/f13 [4534732,29796] 0 2024-03-14T13:02:17.349 INFO:tasks.workunit.client.1.smithi137.stdout:3/340: creat d4/de/d16/d27/f73 x:0 0 0 2024-03-14T13:02:17.350 INFO:tasks.workunit.client.1.smithi137.stdout:4/431: mknod d4/d9/d3a/d41/d38/d55/c91 0 2024-03-14T13:02:17.353 INFO:tasks.workunit.client.1.smithi137.stdout:4/432: mknod d4/d10/d5a/c92 0 2024-03-14T13:02:17.354 INFO:tasks.workunit.client.1.smithi137.stdout:4/433: symlink d4/d10/l93 0 2024-03-14T13:02:17.359 INFO:tasks.workunit.client.1.smithi137.stdout:1/492: sync 2024-03-14T13:02:17.359 INFO:tasks.workunit.client.1.smithi137.stdout:4/434: dwrite d4/d10/d26/f6f [0,4194304] 0 2024-03-14T13:02:17.367 INFO:tasks.workunit.client.0.smithi111.stdout:4/743: mkdir d1d/d105 0 2024-03-14T13:02:17.368 INFO:tasks.workunit.client.1.smithi137.stdout:1/493: dwrite d7/d9/d19/d1f/f95 [0,4194304] 0 2024-03-14T13:02:17.371 INFO:tasks.workunit.client.1.smithi137.stdout:1/494: truncate d7/f88 674914 0 2024-03-14T13:02:17.372 INFO:tasks.workunit.client.0.smithi111.stdout:4/744: symlink d1d/d22/l106 0 2024-03-14T13:02:17.373 INFO:tasks.workunit.client.1.smithi137.stdout:3/341: dwrite d4/de/d16/d1c/f41 [0,4194304] 0 2024-03-14T13:02:17.374 INFO:tasks.workunit.client.0.smithi111.stdout:4/745: dread - d1d/d22/fd8 zero size 2024-03-14T13:02:17.375 INFO:tasks.workunit.client.0.smithi111.stdout:6/450: dwrite d0/d17/d20/d4a/d59/d82/f8f [0,4194304] 0 2024-03-14T13:02:17.377 INFO:tasks.workunit.client.0.smithi111.stdout:4/746: read d1d/d22/d30/de5/fb0 [5945430,99821] 0 2024-03-14T13:02:17.381 INFO:tasks.workunit.client.0.smithi111.stdout:6/451: getdents d0/d17/d20/d94 0 2024-03-14T13:02:17.381 INFO:tasks.workunit.client.0.smithi111.stdout:4/747: readlink d1d/l73 0 2024-03-14T13:02:17.381 INFO:tasks.workunit.client.1.smithi137.stdout:3/342: mkdir d4/de/d16/d74 0 2024-03-14T13:02:17.382 INFO:tasks.workunit.client.0.smithi111.stdout:4/748: write d1d/d22/d30/d87/ffc [618340,104328] 0 2024-03-14T13:02:17.383 INFO:tasks.workunit.client.0.smithi111.stdout:6/452: write d0/d17/d21/d42/da5/fad [661242,60180] 0 2024-03-14T13:02:17.385 INFO:tasks.workunit.client.0.smithi111.stdout:4/749: mkdir d1d/d22/dc6/de6/d107 0 2024-03-14T13:02:17.386 INFO:tasks.workunit.client.0.smithi111.stdout:4/750: dread - d1d/d22/d56/f71 zero size 2024-03-14T13:02:17.387 INFO:tasks.workunit.client.0.smithi111.stdout:6/453: getdents d0/d17/d20/d4a/d59 0 2024-03-14T13:02:17.391 INFO:tasks.workunit.client.0.smithi111.stdout:6/454: link d0/d17/d20/d65/l78 d0/d17/d21/d69/d3d/d53/laf 0 2024-03-14T13:02:17.391 INFO:tasks.workunit.client.0.smithi111.stdout:6/455: chown d0/d17/d21/d42/c62 0 1 2024-03-14T13:02:17.397 INFO:tasks.workunit.client.0.smithi111.stdout:6/456: dwrite d0/d17/f1e [0,4194304] 0 2024-03-14T13:02:17.398 INFO:tasks.workunit.client.0.smithi111.stdout:6/457: chown d0/c11 21665 1 2024-03-14T13:02:17.399 INFO:tasks.workunit.client.0.smithi111.stdout:6/458: mknod d0/d17/d20/d5a/cb0 0 2024-03-14T13:02:17.403 INFO:tasks.workunit.client.0.smithi111.stdout:6/459: link d0/d17/d24/f38 d0/d17/d20/fb1 0 2024-03-14T13:02:17.407 INFO:tasks.workunit.client.1.smithi137.stdout:3/343: dread d4/de/f2a [0,4194304] 0 2024-03-14T13:02:17.408 INFO:tasks.workunit.client.0.smithi111.stdout:4/751: truncate f11 2116584 0 2024-03-14T13:02:17.412 INFO:tasks.workunit.client.0.smithi111.stdout:6/460: truncate d0/d50/f67 100046 0 2024-03-14T13:02:17.413 INFO:tasks.workunit.client.0.smithi111.stdout:6/461: chown d0/d17/d24/l8d 111306285 1 2024-03-14T13:02:17.413 INFO:tasks.workunit.client.0.smithi111.stdout:6/462: chown d0/d17/d20/f4b 50 1 2024-03-14T13:02:17.414 INFO:tasks.workunit.client.0.smithi111.stdout:6/463: dread - d0/d17/d21/d69/f45 zero size 2024-03-14T13:02:17.416 INFO:tasks.workunit.client.0.smithi111.stdout:1/487: sync 2024-03-14T13:02:17.416 INFO:tasks.workunit.client.0.smithi111.stdout:0/485: sync 2024-03-14T13:02:17.417 INFO:tasks.workunit.client.0.smithi111.stdout:6/464: symlink d0/d17/d20/d94/lb2 0 2024-03-14T13:02:17.417 INFO:tasks.workunit.client.0.smithi111.stdout:2/478: sync 2024-03-14T13:02:17.417 INFO:tasks.workunit.client.0.smithi111.stdout:4/752: write d1d/d22/d30/d7c/ff9 [1018929,91973] 0 2024-03-14T13:02:17.419 INFO:tasks.workunit.client.0.smithi111.stdout:6/465: chown d0/d17/d20/l7c 71561475 1 2024-03-14T13:02:17.420 INFO:tasks.workunit.client.1.smithi137.stdout:3/344: truncate d4/de/d16/d1c/f2d 881493 0 2024-03-14T13:02:17.421 INFO:tasks.workunit.client.0.smithi111.stdout:4/753: write d1d/d22/d30/d5e/d6f/fc3 [1663978,116211] 0 2024-03-14T13:02:17.422 INFO:tasks.workunit.client.0.smithi111.stdout:1/488: rename l7 to d20/d21/d3f/d59/lac 0 2024-03-14T13:02:17.424 INFO:tasks.workunit.client.1.smithi137.stdout:3/345: rename d4/de/f3f to d4/d19/f75 0 2024-03-14T13:02:17.424 INFO:tasks.workunit.client.0.smithi111.stdout:1/489: read d20/d21/f42 [4159336,111089] 0 2024-03-14T13:02:17.424 INFO:tasks.workunit.client.1.smithi137.stdout:1/495: dwrite d7/d9/f4b [0,4194304] 0 2024-03-14T13:02:17.425 INFO:tasks.workunit.client.1.smithi137.stdout:3/346: write d4/f37 [103343,127842] 0 2024-03-14T13:02:17.426 INFO:tasks.workunit.client.0.smithi111.stdout:6/466: mknod d0/d17/d20/d65/cb3 0 2024-03-14T13:02:17.426 INFO:tasks.workunit.client.1.smithi137.stdout:1/496: stat d7/d9/d19/d1f/c21 0 2024-03-14T13:02:17.430 INFO:tasks.workunit.client.0.smithi111.stdout:0/486: creat d6/d21/fa2 x:0 0 0 2024-03-14T13:02:17.430 INFO:tasks.workunit.client.0.smithi111.stdout:4/754: getdents d1d/d22/d30/d5e/d6f/d7e/d66/df4/df5 0 2024-03-14T13:02:17.431 INFO:tasks.workunit.client.1.smithi137.stdout:3/347: unlink d4/de/d16/d1c/d5d/d4b/f55 0 2024-03-14T13:02:17.431 INFO:tasks.workunit.client.0.smithi111.stdout:1/490: truncate fa 276355 0 2024-03-14T13:02:17.432 INFO:tasks.workunit.client.1.smithi137.stdout:3/348: dread - d4/de/f3c zero size 2024-03-14T13:02:17.433 INFO:tasks.workunit.client.0.smithi111.stdout:4/755: dread d1d/d22/d30/d5e/f7a [0,4194304] 0 2024-03-14T13:02:17.435 INFO:tasks.workunit.client.0.smithi111.stdout:0/487: getdents d6/d3d/d97 0 2024-03-14T13:02:17.437 INFO:tasks.workunit.client.0.smithi111.stdout:1/491: dread d20/d25/d75/f6a [0,4194304] 0 2024-03-14T13:02:17.439 INFO:tasks.workunit.client.0.smithi111.stdout:6/467: getdents d0 0 2024-03-14T13:02:17.439 INFO:tasks.workunit.client.0.smithi111.stdout:2/479: truncate dd/d17/f3a 339005 0 2024-03-14T13:02:17.439 INFO:tasks.workunit.client.1.smithi137.stdout:3/349: dwrite d4/df/d34/f42 [4194304,4194304] 0 2024-03-14T13:02:17.440 INFO:tasks.workunit.client.0.smithi111.stdout:1/492: write d20/d21/d3f/f6f [195789,30532] 0 2024-03-14T13:02:17.441 INFO:tasks.workunit.client.1.smithi137.stdout:3/350: truncate d4/df/d34/d6b/f6d 880305 0 2024-03-14T13:02:17.442 INFO:tasks.workunit.client.0.smithi111.stdout:6/468: read d0/f98 [100903,58887] 0 2024-03-14T13:02:17.442 INFO:tasks.workunit.client.0.smithi111.stdout:0/488: symlink d6/d9/d15/d55/d89/la3 0 2024-03-14T13:02:17.442 INFO:tasks.workunit.client.1.smithi137.stdout:3/351: read - d4/de/d16/f6a zero size 2024-03-14T13:02:17.443 INFO:tasks.workunit.client.0.smithi111.stdout:6/469: readlink d0/l15 0 2024-03-14T13:02:17.444 INFO:tasks.workunit.client.1.smithi137.stdout:3/352: write d4/de/f3c [684578,10778] 0 2024-03-14T13:02:17.444 INFO:tasks.workunit.client.0.smithi111.stdout:1/493: write d20/d25/d75/f77 [370491,21471] 0 2024-03-14T13:02:17.445 INFO:tasks.workunit.client.1.smithi137.stdout:3/353: fdatasync f1 0 2024-03-14T13:02:17.446 INFO:tasks.workunit.client.0.smithi111.stdout:2/480: creat dd/d8d/f9b x:0 0 0 2024-03-14T13:02:17.447 INFO:tasks.workunit.client.1.smithi137.stdout:3/354: truncate d4/de/d13/f21 2220129 0 2024-03-14T13:02:17.449 INFO:tasks.workunit.client.1.smithi137.stdout:3/355: rmdir d4/d19 39 2024-03-14T13:02:17.451 INFO:tasks.workunit.client.1.smithi137.stdout:3/356: mknod d4/df/c76 0 2024-03-14T13:02:17.451 INFO:tasks.workunit.client.0.smithi111.stdout:0/489: unlink d6/d9/d15/d49/d56/l67 0 2024-03-14T13:02:17.452 INFO:tasks.workunit.client.0.smithi111.stdout:1/494: rmdir d20/d40 39 2024-03-14T13:02:17.453 INFO:tasks.workunit.client.1.smithi137.stdout:3/357: truncate d4/df/d34/f44 343619 0 2024-03-14T13:02:17.453 INFO:tasks.workunit.client.1.smithi137.stdout:3/358: readlink d4/de/d16/d1c/d5d/d4b/l5f 0 2024-03-14T13:02:17.454 INFO:tasks.workunit.client.0.smithi111.stdout:2/481: rename dd/d8d/f96 to dd/d8d/f9c 0 2024-03-14T13:02:17.454 INFO:tasks.workunit.client.0.smithi111.stdout:6/470: rmdir d0/d17/d20/d4a/d68 39 2024-03-14T13:02:17.456 INFO:tasks.workunit.client.0.smithi111.stdout:0/490: readlink d6/d9/l65 0 2024-03-14T13:02:17.456 INFO:tasks.workunit.client.1.smithi137.stdout:3/359: creat d4/de/f77 x:0 0 0 2024-03-14T13:02:17.459 INFO:tasks.workunit.client.1.smithi137.stdout:3/360: dread d4/f37 [0,4194304] 0 2024-03-14T13:02:17.459 INFO:tasks.workunit.client.0.smithi111.stdout:2/482: fdatasync dd/d17/f63 0 2024-03-14T13:02:17.461 INFO:tasks.workunit.client.0.smithi111.stdout:6/471: creat d0/d17/d21/d42/da5/fb4 x:0 0 0 2024-03-14T13:02:17.461 INFO:tasks.workunit.client.1.smithi137.stdout:3/361: unlink d4/d22/c48 0 2024-03-14T13:02:17.463 INFO:tasks.workunit.client.1.smithi137.stdout:3/362: mknod d4/de/c78 0 2024-03-14T13:02:17.463 INFO:tasks.workunit.client.0.smithi111.stdout:1/495: mknod d20/d21/d28/cad 0 2024-03-14T13:02:17.463 INFO:tasks.workunit.client.1.smithi137.stdout:3/363: readlink d4/de/d13/l59 0 2024-03-14T13:02:17.464 INFO:tasks.workunit.client.0.smithi111.stdout:1/496: write d20/d25/f64 [2222361,59367] 0 2024-03-14T13:02:17.464 INFO:tasks.workunit.client.0.smithi111.stdout:4/756: dread f12 [0,4194304] 0 2024-03-14T13:02:17.465 INFO:tasks.workunit.client.0.smithi111.stdout:6/472: creat d0/d17/d21/d69/d3d/fb5 x:0 0 0 2024-03-14T13:02:17.466 INFO:tasks.workunit.client.1.smithi137.stdout:3/364: rename d4/de/d16/d1c/d5d/d4b/l5f to d4/de/d16/d1c/d5d/d4b/l79 0 2024-03-14T13:02:17.467 INFO:tasks.workunit.client.0.smithi111.stdout:1/497: dread d20/d25/f38 [0,4194304] 0 2024-03-14T13:02:17.468 INFO:tasks.workunit.client.0.smithi111.stdout:1/498: dread - d20/d25/f3d zero size 2024-03-14T13:02:17.472 INFO:tasks.workunit.client.1.smithi137.stdout:3/365: truncate d4/de/f2a 28773 0 2024-03-14T13:02:17.473 INFO:tasks.workunit.client.1.smithi137.stdout:3/366: chown d4/df 975379 1 2024-03-14T13:02:17.475 INFO:tasks.workunit.client.1.smithi137.stdout:3/367: symlink d4/d19/l7a 0 2024-03-14T13:02:17.475 INFO:tasks.workunit.client.1.smithi137.stdout:3/368: chown d4/de/d16/d1c/d5d/f43 2026 1 2024-03-14T13:02:17.481 INFO:tasks.workunit.client.1.smithi137.stdout:3/369: write d4/df/d34/f49 [270382,16965] 0 2024-03-14T13:02:17.483 INFO:tasks.workunit.client.1.smithi137.stdout:3/370: mknod d4/de/d16/c7b 0 2024-03-14T13:02:17.485 INFO:tasks.workunit.client.1.smithi137.stdout:3/371: creat d4/de/d16/d1c/d5d/f7c x:0 0 0 2024-03-14T13:02:17.490 INFO:tasks.workunit.client.1.smithi137.stdout:3/372: dwrite d4/df/d34/d6b/f6d [0,4194304] 0 2024-03-14T13:02:17.495 INFO:tasks.workunit.client.1.smithi137.stdout:5/359: sync 2024-03-14T13:02:17.495 INFO:tasks.workunit.client.1.smithi137.stdout:2/462: sync 2024-03-14T13:02:17.496 INFO:tasks.workunit.client.1.smithi137.stdout:3/373: write d4/de/f2a [850331,77302] 0 2024-03-14T13:02:17.498 INFO:tasks.workunit.client.1.smithi137.stdout:5/360: creat d7/d49/d75/f80 x:0 0 0 2024-03-14T13:02:17.499 INFO:tasks.workunit.client.1.smithi137.stdout:5/361: read d7/f10 [785957,23887] 0 2024-03-14T13:02:17.499 INFO:tasks.workunit.client.1.smithi137.stdout:2/463: rmdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64 39 2024-03-14T13:02:17.502 INFO:tasks.workunit.client.1.smithi137.stdout:5/362: mkdir d7/d1d/d74/d81 0 2024-03-14T13:02:17.504 INFO:tasks.workunit.client.1.smithi137.stdout:3/374: getdents d4/d19/d5c 0 2024-03-14T13:02:17.505 INFO:tasks.workunit.client.1.smithi137.stdout:5/363: rmdir d7/d16/d18/d25/d28/d3c/d76 39 2024-03-14T13:02:17.507 INFO:tasks.workunit.client.1.smithi137.stdout:3/375: rmdir d4/de 39 2024-03-14T13:02:17.511 INFO:tasks.workunit.client.1.smithi137.stdout:2/464: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f75 to d2/d7/f95 0 2024-03-14T13:02:17.512 INFO:tasks.workunit.client.1.smithi137.stdout:5/364: dread d7/fb [0,4194304] 0 2024-03-14T13:02:17.514 INFO:tasks.workunit.client.1.smithi137.stdout:3/376: getdents d4/de 0 2024-03-14T13:02:17.514 INFO:tasks.workunit.client.1.smithi137.stdout:5/365: fsync d7/f10 0 2024-03-14T13:02:17.515 INFO:tasks.workunit.client.1.smithi137.stdout:3/377: truncate d4/d19/f23 13031213 0 2024-03-14T13:02:17.515 INFO:tasks.workunit.client.1.smithi137.stdout:2/465: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f96 x:0 0 0 2024-03-14T13:02:17.516 INFO:tasks.workunit.client.1.smithi137.stdout:2/466: readlink d2/d7/db/d2f/l6c 0 2024-03-14T13:02:17.517 INFO:tasks.workunit.client.1.smithi137.stdout:2/467: chown d2/d34/d3c/l42 98543423 1 2024-03-14T13:02:17.517 INFO:tasks.workunit.client.1.smithi137.stdout:5/366: link d7/f10 d7/d36/f82 0 2024-03-14T13:02:17.519 INFO:tasks.workunit.client.1.smithi137.stdout:2/468: write d2/d7/d2e/f47 [364640,13307] 0 2024-03-14T13:02:17.520 INFO:tasks.workunit.client.1.smithi137.stdout:5/367: mkdir d7/d1d/d74/d83 0 2024-03-14T13:02:17.523 INFO:tasks.workunit.client.1.smithi137.stdout:5/368: creat d7/d33/d65/f84 x:0 0 0 2024-03-14T13:02:17.523 INFO:tasks.workunit.client.1.smithi137.stdout:3/378: dwrite d4/d19/f66 [0,4194304] 0 2024-03-14T13:02:17.526 INFO:tasks.workunit.client.1.smithi137.stdout:5/369: creat d7/d49/d75/f85 x:0 0 0 2024-03-14T13:02:17.527 INFO:tasks.workunit.client.1.smithi137.stdout:5/370: fdatasync d7/f1b 0 2024-03-14T13:02:17.528 INFO:tasks.workunit.client.1.smithi137.stdout:2/469: rename d2/d7/d2e/d3e/d44/d6b/d1d/d8e to d2/d7/d2e/d3e/d44/d6b/d22/d97 0 2024-03-14T13:02:17.532 INFO:tasks.workunit.client.1.smithi137.stdout:3/379: rename d4/d22/f4c to d4/de/f7d 0 2024-03-14T13:02:17.535 INFO:tasks.workunit.client.0.smithi111.stdout:0/491: mkdir d6/d9/de/d77/da4 0 2024-03-14T13:02:17.537 INFO:tasks.workunit.client.1.smithi137.stdout:2/470: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f98 x:0 0 0 2024-03-14T13:02:17.538 INFO:tasks.workunit.client.0.smithi111.stdout:1/499: creat d20/d21/d28/d5c/fae x:0 0 0 2024-03-14T13:02:17.538 INFO:tasks.workunit.client.0.smithi111.stdout:4/757: getdents d1d/d48/dbe 0 2024-03-14T13:02:17.539 INFO:tasks.workunit.client.0.smithi111.stdout:0/492: symlink d6/d9/d5b/la5 0 2024-03-14T13:02:17.540 INFO:tasks.workunit.client.0.smithi111.stdout:6/473: symlink d0/d17/d21/d42/lb6 0 2024-03-14T13:02:17.541 INFO:tasks.workunit.client.0.smithi111.stdout:1/500: mkdir d20/d21/d3f/d59/daf 0 2024-03-14T13:02:17.541 INFO:tasks.workunit.client.0.smithi111.stdout:0/493: fdatasync d6/d9/de/f36 0 2024-03-14T13:02:17.542 INFO:tasks.workunit.client.1.smithi137.stdout:2/471: getdents d2/d7/d2e/d3e/d44/d6b/d22/d97 0 2024-03-14T13:02:17.542 INFO:tasks.workunit.client.0.smithi111.stdout:1/501: readlink d20/d25/l89 0 2024-03-14T13:02:17.542 INFO:tasks.workunit.client.0.smithi111.stdout:4/758: mknod d1d/d22/d30/c108 0 2024-03-14T13:02:17.543 INFO:tasks.workunit.client.0.smithi111.stdout:6/474: mknod d0/d17/d24/cb7 0 2024-03-14T13:02:17.543 INFO:tasks.workunit.client.0.smithi111.stdout:0/494: chown d6/c71 912041 1 2024-03-14T13:02:17.544 INFO:tasks.workunit.client.1.smithi137.stdout:2/472: unlink d2/d7/d2e/c3b 0 2024-03-14T13:02:17.545 INFO:tasks.workunit.client.1.smithi137.stdout:3/380: getdents d4/de/d16 0 2024-03-14T13:02:17.545 INFO:tasks.workunit.client.0.smithi111.stdout:6/475: mkdir d0/d17/d24/db8 0 2024-03-14T13:02:17.546 INFO:tasks.workunit.client.0.smithi111.stdout:4/759: symlink d1d/d22/d40/d69/l109 0 2024-03-14T13:02:17.546 INFO:tasks.workunit.client.0.smithi111.stdout:0/495: mkdir d6/d3d/da6 0 2024-03-14T13:02:17.548 INFO:tasks.workunit.client.0.smithi111.stdout:6/476: mknod d0/d17/d21/d69/d3d/cb9 0 2024-03-14T13:02:17.549 INFO:tasks.workunit.client.0.smithi111.stdout:0/496: creat d6/d3d/fa7 x:0 0 0 2024-03-14T13:02:17.549 INFO:tasks.workunit.client.1.smithi137.stdout:2/473: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99 0 2024-03-14T13:02:17.549 INFO:tasks.workunit.client.0.smithi111.stdout:6/477: mknod d0/d50/cba 0 2024-03-14T13:02:17.550 INFO:tasks.workunit.client.0.smithi111.stdout:0/497: write d6/d9/de/f3b [3174570,65218] 0 2024-03-14T13:02:17.551 INFO:tasks.workunit.client.1.smithi137.stdout:2/474: mknod d2/d34/d3c/c9a 0 2024-03-14T13:02:17.553 INFO:tasks.workunit.client.1.smithi137.stdout:3/381: dwrite d4/de/d13/f36 [0,4194304] 0 2024-03-14T13:02:17.555 INFO:tasks.workunit.client.1.smithi137.stdout:3/382: write d4/de/d16/f20 [124157,50660] 0 2024-03-14T13:02:17.558 INFO:tasks.workunit.client.0.smithi111.stdout:0/498: rename d6/d21/f42 to d6/d9/d5b/d9e/fa8 0 2024-03-14T13:02:17.558 INFO:tasks.workunit.client.0.smithi111.stdout:1/502: getdents d20/d21/d28/d5c 0 2024-03-14T13:02:17.560 INFO:tasks.workunit.client.1.smithi137.stdout:3/383: rename d4/de/c78 to d4/df/d34/d6b/c7e 0 2024-03-14T13:02:17.560 INFO:tasks.workunit.client.0.smithi111.stdout:4/760: dread d1d/d22/d30/d5e/f8d [0,4194304] 0 2024-03-14T13:02:17.561 INFO:tasks.workunit.client.1.smithi137.stdout:2/475: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f98 [0,4194304] 0 2024-03-14T13:02:17.563 INFO:tasks.workunit.client.0.smithi111.stdout:0/499: mkdir d6/d9/d15/da9 0 2024-03-14T13:02:17.566 INFO:tasks.workunit.client.0.smithi111.stdout:4/761: rename d1d/d22/d30/d5e/db8/ldd to d1d/d22/d30/d87/dd1/l10a 0 2024-03-14T13:02:17.567 INFO:tasks.workunit.client.1.smithi137.stdout:2/476: creat d2/d7/db/d74/f9b x:0 0 0 2024-03-14T13:02:17.571 INFO:tasks.workunit.client.0.smithi111.stdout:1/503: link d20/d21/d28/lab d20/d25/d87/lb0 0 2024-03-14T13:02:17.575 INFO:tasks.workunit.client.0.smithi111.stdout:1/504: dread d20/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:17.575 INFO:tasks.workunit.client.0.smithi111.stdout:3/470: sync 2024-03-14T13:02:17.575 INFO:tasks.workunit.client.0.smithi111.stdout:9/475: sync 2024-03-14T13:02:17.575 INFO:tasks.workunit.client.0.smithi111.stdout:5/628: sync 2024-03-14T13:02:17.575 INFO:tasks.workunit.client.1.smithi137.stdout:2/477: dwrite d2/d34/f7f [0,4194304] 0 2024-03-14T13:02:17.576 INFO:tasks.workunit.client.0.smithi111.stdout:4/762: dread d1d/d22/d30/d5e/d6f/fc3 [0,4194304] 0 2024-03-14T13:02:17.578 INFO:tasks.workunit.client.0.smithi111.stdout:9/476: write d5/d6/d2a/d2d/d8b/f8e [64848,78531] 0 2024-03-14T13:02:17.578 INFO:tasks.workunit.client.0.smithi111.stdout:1/505: write d20/d40/f4f [2072458,56843] 0 2024-03-14T13:02:17.581 INFO:tasks.workunit.client.1.smithi137.stdout:3/384: dwrite d4/de/f2a [0,4194304] 0 2024-03-14T13:02:17.581 INFO:tasks.workunit.client.0.smithi111.stdout:1/506: dread - d20/d21/d3f/d59/f74 zero size 2024-03-14T13:02:17.582 INFO:tasks.workunit.client.1.smithi137.stdout:3/385: stat d4/df/l72 0 2024-03-14T13:02:17.585 INFO:tasks.workunit.client.1.smithi137.stdout:2/478: truncate d2/f12 1230539 0 2024-03-14T13:02:17.589 INFO:tasks.workunit.client.1.smithi137.stdout:3/386: link d4/de/d16/l33 d4/de/l7f 0 2024-03-14T13:02:17.590 INFO:tasks.workunit.client.1.smithi137.stdout:2/479: truncate d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 9077501 0 2024-03-14T13:02:17.590 INFO:tasks.workunit.client.0.smithi111.stdout:4/763: truncate d1d/d22/d40/f63 2882783 0 2024-03-14T13:02:17.591 INFO:tasks.workunit.client.0.smithi111.stdout:5/629: link d2/d16/d49/d7c/d95/d36/lba d2/d16/d49/d7c/d95/d6a/lcc 0 2024-03-14T13:02:17.592 INFO:tasks.workunit.client.1.smithi137.stdout:2/480: write d2/d7/db/d2f/f89 [385700,58049] 0 2024-03-14T13:02:17.592 INFO:tasks.workunit.client.1.smithi137.stdout:3/387: stat d4/de/d16/d27/f65 0 2024-03-14T13:02:17.594 INFO:tasks.workunit.client.1.smithi137.stdout:2/481: symlink d2/d7/l9c 0 2024-03-14T13:02:17.594 INFO:tasks.workunit.client.1.smithi137.stdout:2/482: fdatasync d2/d7/f87 0 2024-03-14T13:02:17.598 INFO:tasks.workunit.client.0.smithi111.stdout:0/500: dwrite d6/d9/d15/d55/f82 [4194304,4194304] 0 2024-03-14T13:02:17.599 INFO:tasks.workunit.client.1.smithi137.stdout:2/483: read d2/d7/db/f38 [181530,105743] 0 2024-03-14T13:02:17.599 INFO:tasks.workunit.client.0.smithi111.stdout:3/471: dwrite d0/d2d/d35/d69/f9d [0,4194304] 0 2024-03-14T13:02:17.600 INFO:tasks.workunit.client.0.smithi111.stdout:5/630: truncate d2/d16/d49/d7c/d95/d6a/f6e 4850114 0 2024-03-14T13:02:17.601 INFO:tasks.workunit.client.1.smithi137.stdout:2/484: write d2/d7/d2e/d3e/d44/d6b/f20 [775378,42585] 0 2024-03-14T13:02:17.602 INFO:tasks.workunit.client.0.smithi111.stdout:0/501: chown d6/d9/de/d32/f31 0 1 2024-03-14T13:02:17.603 INFO:tasks.workunit.client.1.smithi137.stdout:2/485: chown d2/d7/d2e/f6a 7889 1 2024-03-14T13:02:17.607 INFO:tasks.workunit.client.1.smithi137.stdout:3/388: dwrite d4/f18 [0,4194304] 0 2024-03-14T13:02:17.608 INFO:tasks.workunit.client.0.smithi111.stdout:4/764: truncate d1d/d22/d30/d5e/d6f/d7e/d66/f85 1051405 0 2024-03-14T13:02:17.608 INFO:tasks.workunit.client.0.smithi111.stdout:0/502: mknod d6/d3d/caa 0 2024-03-14T13:02:17.609 INFO:tasks.workunit.client.0.smithi111.stdout:0/503: write d6/d9/de/d32/f5e [492504,71276] 0 2024-03-14T13:02:17.612 INFO:tasks.workunit.client.0.smithi111.stdout:0/504: write d6/d9/de/d3a/f61 [772068,76638] 0 2024-03-14T13:02:17.612 INFO:tasks.workunit.client.0.smithi111.stdout:3/472: rename d0/l5d to d0/d2d/d35/d69/lb8 0 2024-03-14T13:02:17.615 INFO:tasks.workunit.client.0.smithi111.stdout:5/631: getdents d2 0 2024-03-14T13:02:17.616 INFO:tasks.workunit.client.1.smithi137.stdout:3/389: creat d4/f80 x:0 0 0 2024-03-14T13:02:17.617 INFO:tasks.workunit.client.1.smithi137.stdout:3/390: stat d4/d19/l1e 0 2024-03-14T13:02:17.617 INFO:tasks.workunit.client.0.smithi111.stdout:4/765: rename d1d/d22/d23/fc7 to d1d/d22/d40/d69/d86/f10b 0 2024-03-14T13:02:17.619 INFO:tasks.workunit.client.0.smithi111.stdout:0/505: creat d6/d9/d15/d49/d56/fab x:0 0 0 2024-03-14T13:02:17.620 INFO:tasks.workunit.client.0.smithi111.stdout:5/632: symlink d2/d45/d76/d9a/d9f/lcd 0 2024-03-14T13:02:17.621 INFO:tasks.workunit.client.0.smithi111.stdout:0/506: fdatasync d6/d9/de/d3a/d46/d4a/f81 0 2024-03-14T13:02:17.622 INFO:tasks.workunit.client.0.smithi111.stdout:4/766: creat d1d/f10c x:0 0 0 2024-03-14T13:02:17.623 INFO:tasks.workunit.client.0.smithi111.stdout:5/633: readlink d2/d13/l46 0 2024-03-14T13:02:17.623 INFO:tasks.workunit.client.0.smithi111.stdout:1/507: dread d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:17.624 INFO:tasks.workunit.client.1.smithi137.stdout:3/391: dread d4/df/f30 [0,4194304] 0 2024-03-14T13:02:17.624 INFO:tasks.workunit.client.1.smithi137.stdout:2/486: dwrite d2/d7/db/d2f/d66/f7a [0,4194304] 0 2024-03-14T13:02:17.625 INFO:tasks.workunit.client.0.smithi111.stdout:0/507: mkdir d6/d9/de/d3a/dac 0 2024-03-14T13:02:17.627 INFO:tasks.workunit.client.0.smithi111.stdout:4/767: truncate d1d/d22/d30/d7c/ff6 23762 0 2024-03-14T13:02:17.633 INFO:tasks.workunit.client.0.smithi111.stdout:0/508: creat d6/fad x:0 0 0 2024-03-14T13:02:17.634 INFO:tasks.workunit.client.0.smithi111.stdout:1/508: dwrite d20/d21/d28/d5c/f93 [0,4194304] 0 2024-03-14T13:02:17.637 INFO:tasks.workunit.client.1.smithi137.stdout:3/392: dwrite d4/de/d16/f20 [4194304,4194304] 0 2024-03-14T13:02:17.637 INFO:tasks.workunit.client.1.smithi137.stdout:2/487: dwrite d2/d7/d2e/d3e/d44/d6b/f39 [0,4194304] 0 2024-03-14T13:02:17.638 INFO:tasks.workunit.client.0.smithi111.stdout:0/509: write d6/d9/de/d77/f7c [817761,36875] 0 2024-03-14T13:02:17.639 INFO:tasks.workunit.client.0.smithi111.stdout:5/634: dwrite d2/d45/d76/fab [0,4194304] 0 2024-03-14T13:02:17.642 INFO:tasks.workunit.client.1.smithi137.stdout:3/393: unlink d4/f37 0 2024-03-14T13:02:17.643 INFO:tasks.workunit.client.1.smithi137.stdout:2/488: mkdir d2/d34/d3c/d9d 0 2024-03-14T13:02:17.645 INFO:tasks.workunit.client.1.smithi137.stdout:7/345: sync 2024-03-14T13:02:17.645 INFO:tasks.workunit.client.1.smithi137.stdout:0/388: sync 2024-03-14T13:02:17.645 INFO:tasks.workunit.client.1.smithi137.stdout:0/389: stat d1/d1f 0 2024-03-14T13:02:17.648 INFO:tasks.workunit.client.0.smithi111.stdout:3/473: dread d0/d91/f9e [0,4194304] 0 2024-03-14T13:02:17.650 INFO:tasks.workunit.client.1.smithi137.stdout:2/489: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d9e 0 2024-03-14T13:02:17.650 INFO:tasks.workunit.client.1.smithi137.stdout:0/390: creat d1/d1f/d80/f86 x:0 0 0 2024-03-14T13:02:17.654 INFO:tasks.workunit.client.1.smithi137.stdout:0/391: unlink d1/d11/f50 0 2024-03-14T13:02:17.657 INFO:tasks.workunit.client.1.smithi137.stdout:2/490: unlink d2/d34/d3c/d53/f5e 0 2024-03-14T13:02:17.658 INFO:tasks.workunit.client.1.smithi137.stdout:7/346: dwrite d3/d6/d14/f2f [0,4194304] 0 2024-03-14T13:02:17.659 INFO:tasks.workunit.client.0.smithi111.stdout:4/768: getdents d1d/d22/d30/d5e/db8/ddf 0 2024-03-14T13:02:17.661 INFO:tasks.workunit.client.0.smithi111.stdout:0/510: chown d6/d9/de/d3a/f95 269 1 2024-03-14T13:02:17.662 INFO:tasks.workunit.client.0.smithi111.stdout:0/511: chown d6/d9/d15/f16 0 1 2024-03-14T13:02:17.663 INFO:tasks.workunit.client.0.smithi111.stdout:4/769: fsync d1d/d22/d56/f71 0 2024-03-14T13:02:17.665 INFO:tasks.workunit.client.1.smithi137.stdout:3/394: fdatasync d4/de/d16/f20 0 2024-03-14T13:02:17.665 INFO:tasks.workunit.client.0.smithi111.stdout:0/512: truncate d6/d9/de/f53 5207453 0 2024-03-14T13:02:17.665 INFO:tasks.workunit.client.0.smithi111.stdout:1/509: link d20/f58 d20/d21/da7/fb1 0 2024-03-14T13:02:17.665 INFO:tasks.workunit.client.0.smithi111.stdout:0/513: fsync f1 0 2024-03-14T13:02:17.667 INFO:tasks.workunit.client.0.smithi111.stdout:4/770: creat d1d/d22/d30/d5e/d6f/f10d x:0 0 0 2024-03-14T13:02:17.668 INFO:tasks.workunit.client.1.smithi137.stdout:3/395: rmdir d4/df/d34/d6b 39 2024-03-14T13:02:17.669 INFO:tasks.workunit.client.0.smithi111.stdout:5/635: link d2/d16/c18 d2/d16/d49/cce 0 2024-03-14T13:02:17.670 INFO:tasks.workunit.client.0.smithi111.stdout:0/514: symlink d6/d9/d15/da9/lae 0 2024-03-14T13:02:17.671 INFO:tasks.workunit.client.1.smithi137.stdout:7/347: dwrite d3/d6/d14/f1a [0,4194304] 0 2024-03-14T13:02:17.671 INFO:tasks.workunit.client.0.smithi111.stdout:0/515: fsync d6/d9/de/f6a 0 2024-03-14T13:02:17.672 INFO:tasks.workunit.client.0.smithi111.stdout:1/510: rename d20/d21/d3f/d59/lac to d20/d40/da6/lb2 0 2024-03-14T13:02:17.672 INFO:tasks.workunit.client.1.smithi137.stdout:7/348: fsync d3/d5/f37 0 2024-03-14T13:02:17.673 INFO:tasks.workunit.client.1.smithi137.stdout:3/396: rename d4/de/d16/d1c/c3e to d4/de/d16/c81 0 2024-03-14T13:02:17.673 INFO:tasks.workunit.client.0.smithi111.stdout:4/771: fsync d1d/d22/d40/fab 0 2024-03-14T13:02:17.674 INFO:tasks.workunit.client.0.smithi111.stdout:1/511: truncate d20/d25/d31/d80/f92 491471 0 2024-03-14T13:02:17.676 INFO:tasks.workunit.client.0.smithi111.stdout:0/516: creat d6/d9/d5b/d9e/faf x:0 0 0 2024-03-14T13:02:17.677 INFO:tasks.workunit.client.1.smithi137.stdout:3/397: mkdir d4/df/d82 0 2024-03-14T13:02:17.677 INFO:tasks.workunit.client.1.smithi137.stdout:7/349: creat d3/d6/d14/d27/f6c x:0 0 0 2024-03-14T13:02:17.678 INFO:tasks.workunit.client.0.smithi111.stdout:0/517: write d6/d9/de/d3a/d46/d4a/f81 [462053,20806] 0 2024-03-14T13:02:17.678 INFO:tasks.workunit.client.0.smithi111.stdout:1/512: symlink d20/d25/d31/d80/lb3 0 2024-03-14T13:02:17.679 INFO:tasks.workunit.client.0.smithi111.stdout:1/513: dread - d20/d40/f84 zero size 2024-03-14T13:02:17.680 INFO:tasks.workunit.client.0.smithi111.stdout:4/772: symlink d1d/d22/d30/d5e/l10e 0 2024-03-14T13:02:17.681 INFO:tasks.workunit.client.0.smithi111.stdout:5/636: getdents d2/d16/d49/d7c/d8b 0 2024-03-14T13:02:17.682 INFO:tasks.workunit.client.0.smithi111.stdout:1/514: mkdir d20/d40/db4 0 2024-03-14T13:02:17.683 INFO:tasks.workunit.client.0.smithi111.stdout:1/515: chown d20/d25/f46 83644 1 2024-03-14T13:02:17.684 INFO:tasks.workunit.client.0.smithi111.stdout:5/637: write d2/d16/d49/f78 [3607599,111938] 0 2024-03-14T13:02:17.685 INFO:tasks.workunit.client.0.smithi111.stdout:4/773: mkdir d1d/d22/d30/d5e/db8/ddf/d10f 0 2024-03-14T13:02:17.687 INFO:tasks.workunit.client.0.smithi111.stdout:1/516: rmdir d20/d21/d28 39 2024-03-14T13:02:17.689 INFO:tasks.workunit.client.0.smithi111.stdout:5/638: write d2/d16/d49/d44/d4a/d9e/fbe [628719,41082] 0 2024-03-14T13:02:17.689 INFO:tasks.workunit.client.0.smithi111.stdout:0/518: dwrite d6/d9/de/d3a/f9b [0,4194304] 0 2024-03-14T13:02:17.691 INFO:tasks.workunit.client.0.smithi111.stdout:1/517: chown d20/f57 2 1 2024-03-14T13:02:17.693 INFO:tasks.workunit.client.0.smithi111.stdout:0/519: write d6/d9/d5b/f6d [1006853,90735] 0 2024-03-14T13:02:17.693 INFO:tasks.workunit.client.1.smithi137.stdout:0/392: dread d1/d5/d1b/d1e/d25/d26/f2d [0,4194304] 0 2024-03-14T13:02:17.693 INFO:tasks.workunit.client.0.smithi111.stdout:1/518: chown d20/d25/f9f 48605 1 2024-03-14T13:02:17.697 INFO:tasks.workunit.client.1.smithi137.stdout:0/393: symlink d1/d5/d1b/d1e/d25/l87 0 2024-03-14T13:02:17.697 INFO:tasks.workunit.client.0.smithi111.stdout:1/519: chown d20/d21/d28/d5c/fae 40085 1 2024-03-14T13:02:17.698 INFO:tasks.workunit.client.0.smithi111.stdout:0/520: creat d6/d45/d87/fb0 x:0 0 0 2024-03-14T13:02:17.700 INFO:tasks.workunit.client.0.smithi111.stdout:1/520: mknod d20/d40/cb5 0 2024-03-14T13:02:17.701 INFO:tasks.workunit.client.1.smithi137.stdout:0/394: stat d1/d1f/c66 0 2024-03-14T13:02:17.701 INFO:tasks.workunit.client.1.smithi137.stdout:2/491: truncate d2/d7/d2e/d3e/d44/d6b/d1d/f48 4301565 0 2024-03-14T13:02:17.701 INFO:tasks.workunit.client.1.smithi137.stdout:7/350: dread d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:17.702 INFO:tasks.workunit.client.0.smithi111.stdout:5/639: link d2/d16/d49/f78 d2/d16/d49/d44/d4a/d9e/db7/dbb/fcf 0 2024-03-14T13:02:17.704 INFO:tasks.workunit.client.0.smithi111.stdout:1/521: mkdir d20/d21/d28/da0/db6 0 2024-03-14T13:02:17.704 INFO:tasks.workunit.client.0.smithi111.stdout:4/774: dread d1d/d22/d56/f7f [0,4194304] 0 2024-03-14T13:02:17.706 INFO:tasks.workunit.client.0.smithi111.stdout:1/522: creat d20/d25/d87/fb7 x:0 0 0 2024-03-14T13:02:17.707 INFO:tasks.workunit.client.0.smithi111.stdout:1/523: stat d20/d21/l55 0 2024-03-14T13:02:17.711 INFO:tasks.workunit.client.0.smithi111.stdout:5/640: getdents d2/d16 0 2024-03-14T13:02:17.712 INFO:tasks.workunit.client.1.smithi137.stdout:3/398: rename d4/de/d16/d1c/f2d to d4/df/f83 0 2024-03-14T13:02:17.713 INFO:tasks.workunit.client.0.smithi111.stdout:4/775: write d1d/d22/d56/f71 [858598,102221] 0 2024-03-14T13:02:17.713 INFO:tasks.workunit.client.1.smithi137.stdout:0/395: getdents d1/d5/d1b/d42 0 2024-03-14T13:02:17.714 INFO:tasks.workunit.client.0.smithi111.stdout:5/641: mknod d2/d16/d49/d7c/d95/d36/cd0 0 2024-03-14T13:02:17.715 INFO:tasks.workunit.client.0.smithi111.stdout:5/642: chown d2/d16/d49/d7c/d95/d6a 903483303 1 2024-03-14T13:02:17.716 INFO:tasks.workunit.client.1.smithi137.stdout:0/396: read d1/f1c [391786,13897] 0 2024-03-14T13:02:17.717 INFO:tasks.workunit.client.1.smithi137.stdout:2/492: mkdir d2/d7/db/d74/d9f 0 2024-03-14T13:02:17.718 INFO:tasks.workunit.client.1.smithi137.stdout:3/399: symlink d4/df/d34/l84 0 2024-03-14T13:02:17.719 INFO:tasks.workunit.client.1.smithi137.stdout:7/351: dwrite d3/d5/f19 [0,4194304] 0 2024-03-14T13:02:17.720 INFO:tasks.workunit.client.1.smithi137.stdout:0/397: write d1/d44/d72/d7e/f78 [256388,43331] 0 2024-03-14T13:02:17.720 INFO:tasks.workunit.client.0.smithi111.stdout:5/643: creat d2/d45/d76/fd1 x:0 0 0 2024-03-14T13:02:17.723 INFO:tasks.workunit.client.1.smithi137.stdout:3/400: readlink d4/de/d16/d27/l4d 0 2024-03-14T13:02:17.724 INFO:tasks.workunit.client.0.smithi111.stdout:1/524: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:17.725 INFO:tasks.workunit.client.1.smithi137.stdout:0/398: dread d1/d5/d1b/d1e/d25/d26/f41 [0,4194304] 0 2024-03-14T13:02:17.726 INFO:tasks.workunit.client.1.smithi137.stdout:0/399: fsync d1/d5/d1b/d42/f4f 0 2024-03-14T13:02:17.726 INFO:tasks.workunit.client.1.smithi137.stdout:7/352: unlink d3/d6/c56 0 2024-03-14T13:02:17.727 INFO:tasks.workunit.client.1.smithi137.stdout:0/400: fdatasync d1/f13 0 2024-03-14T13:02:17.727 INFO:tasks.workunit.client.0.smithi111.stdout:1/525: dread - d20/d21/d28/d5c/fae zero size 2024-03-14T13:02:17.728 INFO:tasks.workunit.client.0.smithi111.stdout:1/526: readlink l1 0 2024-03-14T13:02:17.728 INFO:tasks.workunit.client.0.smithi111.stdout:5/644: rename d2/d16/d49/d7c/d95/d6a/f6e to d2/d16/d49/d44/d4a/d9e/db7/dbb/fd2 0 2024-03-14T13:02:17.729 INFO:tasks.workunit.client.1.smithi137.stdout:0/401: read d1/d1f/f3b [7469,117388] 0 2024-03-14T13:02:17.732 INFO:tasks.workunit.client.1.smithi137.stdout:2/493: rmdir d2/d7/d2e/d3e/d44/d6b/d22/d9e 0 2024-03-14T13:02:17.732 INFO:tasks.workunit.client.1.smithi137.stdout:2/494: stat d2/d7/d2e/d3e 0 2024-03-14T13:02:17.733 INFO:tasks.workunit.client.0.smithi111.stdout:5/645: creat d2/d13/fd3 x:0 0 0 2024-03-14T13:02:17.733 INFO:tasks.workunit.client.1.smithi137.stdout:0/402: mknod d1/d5/d1b/d1e/d25/d55/c88 0 2024-03-14T13:02:17.733 INFO:tasks.workunit.client.1.smithi137.stdout:7/353: symlink d3/d5/d68/l6d 0 2024-03-14T13:02:17.733 INFO:tasks.workunit.client.0.smithi111.stdout:5/646: fdatasync d2/d16/d49/d44/fa7 0 2024-03-14T13:02:17.736 INFO:tasks.workunit.client.1.smithi137.stdout:3/401: rename d4/df/d34 to d4/de/d16/d1c/d5d/d85 0 2024-03-14T13:02:17.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/647: symlink d2/d16/d49/d7c/d95/d6a/ld4 0 2024-03-14T13:02:17.738 INFO:tasks.workunit.client.0.smithi111.stdout:4/776: write d1d/d22/d23/f34 [4418696,87948] 0 2024-03-14T13:02:17.739 INFO:tasks.workunit.client.1.smithi137.stdout:7/354: mknod d3/d5/d23/d3f/d5b/c6e 0 2024-03-14T13:02:17.739 INFO:tasks.workunit.client.0.smithi111.stdout:1/527: dwrite d20/d21/d3f/f47 [0,4194304] 0 2024-03-14T13:02:17.741 INFO:tasks.workunit.client.1.smithi137.stdout:3/402: symlink d4/df/l86 0 2024-03-14T13:02:17.741 INFO:tasks.workunit.client.0.smithi111.stdout:5/648: write d2/d16/d49/d4c/f73 [1248789,104844] 0 2024-03-14T13:02:17.741 INFO:tasks.workunit.client.0.smithi111.stdout:4/777: dread - d1d/d22/d40/d69/fef zero size 2024-03-14T13:02:17.742 INFO:tasks.workunit.client.1.smithi137.stdout:2/495: truncate d2/d7/d2e/d3e/d44/d6b/d1d/f48 1604120 0 2024-03-14T13:02:17.742 INFO:tasks.workunit.client.0.smithi111.stdout:1/528: write f9 [2485337,48849] 0 2024-03-14T13:02:17.742 INFO:tasks.workunit.client.1.smithi137.stdout:3/403: write d4/d19/f66 [1393164,67016] 0 2024-03-14T13:02:17.744 INFO:tasks.workunit.client.1.smithi137.stdout:7/355: stat d3/c25 0 2024-03-14T13:02:17.744 INFO:tasks.workunit.client.0.smithi111.stdout:5/649: fdatasync d2/d16/d49/f27 0 2024-03-14T13:02:17.747 INFO:tasks.workunit.client.0.smithi111.stdout:5/650: truncate d2/d45/d76/d9a/fc8 425413 0 2024-03-14T13:02:17.747 INFO:tasks.workunit.client.0.smithi111.stdout:5/651: chown d2/d16/f70 14874699 1 2024-03-14T13:02:17.748 INFO:tasks.workunit.client.1.smithi137.stdout:3/404: read d4/de/f2a [741595,60502] 0 2024-03-14T13:02:17.750 INFO:tasks.workunit.client.0.smithi111.stdout:5/652: symlink d2/d16/d49/d44/d72/ld5 0 2024-03-14T13:02:17.751 INFO:tasks.workunit.client.1.smithi137.stdout:3/405: write d4/de/d16/d1c/d5d/d4b/f5e [3018528,40104] 0 2024-03-14T13:02:17.753 INFO:tasks.workunit.client.0.smithi111.stdout:5/653: mkdir d2/d16/d49/d44/d4a/d9e/db7/dd6 0 2024-03-14T13:02:17.753 INFO:tasks.workunit.client.1.smithi137.stdout:2/496: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/fa0 x:0 0 0 2024-03-14T13:02:17.755 INFO:tasks.workunit.client.1.smithi137.stdout:8/494: sync 2024-03-14T13:02:17.755 INFO:tasks.workunit.client.1.smithi137.stdout:6/436: sync 2024-03-14T13:02:17.755 INFO:tasks.workunit.client.1.smithi137.stdout:4/435: sync 2024-03-14T13:02:17.755 INFO:tasks.workunit.client.1.smithi137.stdout:1/497: sync 2024-03-14T13:02:17.756 INFO:tasks.workunit.client.0.smithi111.stdout:5/654: mknod d2/d16/d49/d7c/d95/d36/cd7 0 2024-03-14T13:02:17.756 INFO:tasks.workunit.client.1.smithi137.stdout:9/538: sync 2024-03-14T13:02:17.756 INFO:tasks.workunit.client.1.smithi137.stdout:0/403: dwrite d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:17.756 INFO:tasks.workunit.client.1.smithi137.stdout:7/356: link d3/d5/f11 d3/f6f 0 2024-03-14T13:02:17.757 INFO:tasks.workunit.client.0.smithi111.stdout:5/655: fsync d2/d16/d49/d44/d4a/f8c 0 2024-03-14T13:02:17.757 INFO:tasks.workunit.client.0.smithi111.stdout:4/778: dread d1d/d22/d30/de5/fb6 [0,4194304] 0 2024-03-14T13:02:17.757 INFO:tasks.workunit.client.0.smithi111.stdout:5/656: chown d2/c9 816411 1 2024-03-14T13:02:17.758 INFO:tasks.workunit.client.1.smithi137.stdout:2/497: chown d2/f1e 790641 1 2024-03-14T13:02:17.761 INFO:tasks.workunit.client.0.smithi111.stdout:4/779: rename d1d/d22/d30/d87/f104 to d1d/d22/dc6/de6/d107/f110 0 2024-03-14T13:02:17.761 INFO:tasks.workunit.client.1.smithi137.stdout:8/495: read - d1/d4a/f9d zero size 2024-03-14T13:02:17.762 INFO:tasks.workunit.client.0.smithi111.stdout:5/657: symlink d2/d45/d76/db2/ld8 0 2024-03-14T13:02:17.766 INFO:tasks.workunit.client.1.smithi137.stdout:1/498: stat d7/d9/d19/d1f/d26 0 2024-03-14T13:02:17.767 INFO:tasks.workunit.client.1.smithi137.stdout:2/498: chown d2/d34/l62 529414041 1 2024-03-14T13:02:17.767 INFO:tasks.workunit.client.0.smithi111.stdout:5/658: rename d2/d16/d49/ce to d2/cd9 0 2024-03-14T13:02:17.767 INFO:tasks.workunit.client.1.smithi137.stdout:4/436: rename d4/f31 to d4/d9/d3a/d41/d38/d79/f94 0 2024-03-14T13:02:17.768 INFO:tasks.workunit.client.1.smithi137.stdout:0/404: mknod d1/d5/d1b/c89 0 2024-03-14T13:02:17.774 INFO:tasks.workunit.client.0.smithi111.stdout:5/659: dread d2/d16/d49/d7c/d95/f5e [0,4194304] 0 2024-03-14T13:02:17.775 INFO:tasks.workunit.client.1.smithi137.stdout:7/357: symlink d3/d5/d23/l70 0 2024-03-14T13:02:17.779 INFO:tasks.workunit.client.1.smithi137.stdout:3/406: dwrite d4/d19/f54 [0,4194304] 0 2024-03-14T13:02:17.780 INFO:tasks.workunit.client.1.smithi137.stdout:8/496: write d1/d4a/f66 [566991,97001] 0 2024-03-14T13:02:17.780 INFO:tasks.workunit.client.0.smithi111.stdout:5/660: truncate d2/f6 1033904 0 2024-03-14T13:02:17.780 INFO:tasks.workunit.client.1.smithi137.stdout:2/499: chown d2/d34/d3c/f6d 175621990 1 2024-03-14T13:02:17.781 INFO:tasks.workunit.client.0.smithi111.stdout:4/780: dwrite d1d/d22/d40/fab [0,4194304] 0 2024-03-14T13:02:17.782 INFO:tasks.workunit.client.0.smithi111.stdout:4/781: stat d1d/d22/l106 0 2024-03-14T13:02:17.782 INFO:tasks.workunit.client.1.smithi137.stdout:5/371: sync 2024-03-14T13:02:17.783 INFO:tasks.workunit.client.1.smithi137.stdout:4/437: mkdir d4/d9/d8e/d95 0 2024-03-14T13:02:17.784 INFO:tasks.workunit.client.0.smithi111.stdout:5/661: unlink d2/d16/d49/d44/d72/l7f 0 2024-03-14T13:02:17.785 INFO:tasks.workunit.client.0.smithi111.stdout:4/782: write d1d/d22/d30/d7c/d59/dc4/fdb [288709,111451] 0 2024-03-14T13:02:17.785 INFO:tasks.workunit.client.1.smithi137.stdout:9/539: dread d6/df/d19/d3a/d40/f41 [0,4194304] 0 2024-03-14T13:02:17.785 INFO:tasks.workunit.client.1.smithi137.stdout:7/358: mknod d3/d6/d59/c71 0 2024-03-14T13:02:17.789 INFO:tasks.workunit.client.0.smithi111.stdout:4/783: mkdir d1d/d22/d30/d87/dd1/de1/d111 0 2024-03-14T13:02:17.789 INFO:tasks.workunit.client.0.smithi111.stdout:5/662: truncate d2/d16/d49/d7c/d8b/f91 4946441 0 2024-03-14T13:02:17.789 INFO:tasks.workunit.client.1.smithi137.stdout:8/497: truncate d1/dc/f39 5547483 0 2024-03-14T13:02:17.790 INFO:tasks.workunit.client.0.smithi111.stdout:5/663: truncate d2/d13/fd3 110793 0 2024-03-14T13:02:17.791 INFO:tasks.workunit.client.1.smithi137.stdout:9/540: chown d6/d91/fad 612 1 2024-03-14T13:02:17.795 INFO:tasks.workunit.client.1.smithi137.stdout:1/499: dwrite d7/d9/d19/d1f/d26/d84/f8c [4194304,4194304] 0 2024-03-14T13:02:17.796 INFO:tasks.workunit.client.1.smithi137.stdout:8/498: chown d1/dc/d77/d9f/l91 33342 1 2024-03-14T13:02:17.796 INFO:tasks.workunit.client.1.smithi137.stdout:0/405: dwrite d1/d44/d56/f70 [0,4194304] 0 2024-03-14T13:02:17.797 INFO:tasks.workunit.client.1.smithi137.stdout:2/500: creat d2/d34/d3c/d9d/fa1 x:0 0 0 2024-03-14T13:02:17.798 INFO:tasks.workunit.client.1.smithi137.stdout:4/438: mknod d4/d9/d3a/d70/d80/d8d/c96 0 2024-03-14T13:02:17.800 INFO:tasks.workunit.client.1.smithi137.stdout:9/541: fsync d6/df/d19/d3a/d40/f9e 0 2024-03-14T13:02:17.804 INFO:tasks.workunit.client.1.smithi137.stdout:9/542: write d6/df/d19/d3a/d40/f5c [248630,50729] 0 2024-03-14T13:02:17.804 INFO:tasks.workunit.client.1.smithi137.stdout:5/372: creat d7/d1d/d74/d83/f86 x:0 0 0 2024-03-14T13:02:17.807 INFO:tasks.workunit.client.1.smithi137.stdout:8/499: dread - d1/dc/d3c/d87/d34/d82/fa7 zero size 2024-03-14T13:02:17.811 INFO:tasks.workunit.client.1.smithi137.stdout:7/359: dwrite d3/d5/d23/d2c/d5d/f6b [0,4194304] 0 2024-03-14T13:02:17.812 INFO:tasks.workunit.client.1.smithi137.stdout:2/501: read - d2/d7/db/d2f/f5f zero size 2024-03-14T13:02:17.813 INFO:tasks.workunit.client.0.smithi111.stdout:7/443: sync 2024-03-14T13:02:17.813 INFO:tasks.workunit.client.0.smithi111.stdout:8/693: sync 2024-03-14T13:02:17.814 INFO:tasks.workunit.client.0.smithi111.stdout:7/444: chown d0/d11/l35 2 1 2024-03-14T13:02:17.814 INFO:tasks.workunit.client.1.smithi137.stdout:9/543: write d6/df/d19/d3a/fae [482519,121408] 0 2024-03-14T13:02:17.814 INFO:tasks.workunit.client.1.smithi137.stdout:4/439: rename d4/f12 to d4/d9/d3a/d41/f97 0 2024-03-14T13:02:17.817 INFO:tasks.workunit.client.1.smithi137.stdout:1/500: creat d7/d9/d19/d1f/d2e/f9b x:0 0 0 2024-03-14T13:02:17.818 INFO:tasks.workunit.client.1.smithi137.stdout:5/373: truncate d7/d16/d18/f34 682679 0 2024-03-14T13:02:17.821 INFO:tasks.workunit.client.1.smithi137.stdout:4/440: chown d4/d4d/l60 58716214 1 2024-03-14T13:02:17.823 INFO:tasks.workunit.client.1.smithi137.stdout:0/406: dwrite d1/f51 [0,4194304] 0 2024-03-14T13:02:17.823 INFO:tasks.workunit.client.1.smithi137.stdout:5/374: stat d7/d16/d18/c73 0 2024-03-14T13:02:17.829 INFO:tasks.workunit.client.1.smithi137.stdout:2/502: mknod d2/ca2 0 2024-03-14T13:02:17.829 INFO:tasks.workunit.client.1.smithi137.stdout:6/437: dread d2/d6/de/f5d [0,4194304] 0 2024-03-14T13:02:17.829 INFO:tasks.workunit.client.1.smithi137.stdout:8/500: dwrite d1/d4a/f9d [0,4194304] 0 2024-03-14T13:02:17.831 INFO:tasks.workunit.client.1.smithi137.stdout:0/407: read d1/d5/d1b/d1e/d25/d26/f2d [4140861,82832] 0 2024-03-14T13:02:17.836 INFO:tasks.workunit.client.1.smithi137.stdout:9/544: dwrite d6/df/f13 [0,4194304] 0 2024-03-14T13:02:17.836 INFO:tasks.workunit.client.1.smithi137.stdout:0/408: chown d1/d5/d1b/d1e/d25/d26/f39 17199047 1 2024-03-14T13:02:17.844 INFO:tasks.workunit.client.1.smithi137.stdout:0/409: read d1/d1f/f28 [5303079,103322] 0 2024-03-14T13:02:17.845 INFO:tasks.workunit.client.1.smithi137.stdout:4/441: creat d4/d9/d3a/d41/d38/d79/f98 x:0 0 0 2024-03-14T13:02:17.846 INFO:tasks.workunit.client.1.smithi137.stdout:9/545: truncate d6/df/d19/d3a/f67 785575 0 2024-03-14T13:02:17.846 INFO:tasks.workunit.client.1.smithi137.stdout:8/501: dwrite d1/dc/d3c/d87/d34/d97/f2d [0,4194304] 0 2024-03-14T13:02:17.848 INFO:tasks.workunit.client.1.smithi137.stdout:2/503: symlink d2/d7/db/la3 0 2024-03-14T13:02:17.849 INFO:tasks.workunit.client.1.smithi137.stdout:5/375: rename d7/d16/f3e to d7/d16/d18/d25/f87 0 2024-03-14T13:02:17.850 INFO:tasks.workunit.client.1.smithi137.stdout:5/376: read d7/fb [1259863,14880] 0 2024-03-14T13:02:17.850 INFO:tasks.workunit.client.1.smithi137.stdout:4/442: unlink d4/d10/d26/f40 0 2024-03-14T13:02:17.852 INFO:tasks.workunit.client.1.smithi137.stdout:9/546: creat d6/d56/fb3 x:0 0 0 2024-03-14T13:02:17.852 INFO:tasks.workunit.client.1.smithi137.stdout:5/377: dread - d7/d1d/d74/d83/f86 zero size 2024-03-14T13:02:17.852 INFO:tasks.workunit.client.1.smithi137.stdout:4/443: write d4/d27/d5c/d77/f7c [650778,99686] 0 2024-03-14T13:02:17.854 INFO:tasks.workunit.client.1.smithi137.stdout:5/378: write d7/d16/d18/f66 [4454953,45454] 0 2024-03-14T13:02:17.855 INFO:tasks.workunit.client.1.smithi137.stdout:6/438: rename d2/d54/d8b to d2/d6/de/d45/d9b 0 2024-03-14T13:02:17.857 INFO:tasks.workunit.client.1.smithi137.stdout:6/439: chown d2/d6/d1d/c96 158 1 2024-03-14T13:02:17.858 INFO:tasks.workunit.client.0.smithi111.stdout:5/664: symlink d2/d16/d49/d44/d4a/d9e/lda 0 2024-03-14T13:02:17.859 INFO:tasks.workunit.client.1.smithi137.stdout:5/379: symlink d7/d36/l88 0 2024-03-14T13:02:17.861 INFO:tasks.workunit.client.0.smithi111.stdout:7/445: mkdir d0/d1/d4a/d87 0 2024-03-14T13:02:17.862 INFO:tasks.workunit.client.1.smithi137.stdout:0/410: rename d1/d11 to d1/d5/d1b/d1e/d25/d8a 0 2024-03-14T13:02:17.862 INFO:tasks.workunit.client.0.smithi111.stdout:4/784: rmdir d1d/d22 39 2024-03-14T13:02:17.863 INFO:tasks.workunit.client.1.smithi137.stdout:2/504: getdents d2/d7/d2e 0 2024-03-14T13:02:17.863 INFO:tasks.workunit.client.0.smithi111.stdout:5/665: symlink d2/d16/d49/d7c/ldb 0 2024-03-14T13:02:17.865 INFO:tasks.workunit.client.1.smithi137.stdout:2/505: fsync d2/d7/d2e/d3e/d44/d6b/d1d/d2d/f61 0 2024-03-14T13:02:17.865 INFO:tasks.workunit.client.1.smithi137.stdout:9/547: rename d6/df/d19/d3a/f5b to d6/df/d19/d1f/fb4 0 2024-03-14T13:02:17.866 INFO:tasks.workunit.client.1.smithi137.stdout:0/411: dread d1/d5/d1b/d42/f32 [0,4194304] 0 2024-03-14T13:02:17.867 INFO:tasks.workunit.client.1.smithi137.stdout:6/440: mkdir d2/d54/d55/d9c 0 2024-03-14T13:02:17.867 INFO:tasks.workunit.client.1.smithi137.stdout:9/548: write d6/f49 [2836413,77098] 0 2024-03-14T13:02:17.868 INFO:tasks.workunit.client.0.smithi111.stdout:7/446: rmdir d0/d1/d5/d61 0 2024-03-14T13:02:17.869 INFO:tasks.workunit.client.0.smithi111.stdout:4/785: dwrite d1d/f43 [0,4194304] 0 2024-03-14T13:02:17.870 INFO:tasks.workunit.client.1.smithi137.stdout:5/380: link d7/d16/d18/d25/f6b d7/d1d/d74/d81/f89 0 2024-03-14T13:02:17.871 INFO:tasks.workunit.client.1.smithi137.stdout:9/549: truncate d6/d56/f73 2002331 0 2024-03-14T13:02:17.872 INFO:tasks.workunit.client.1.smithi137.stdout:2/506: dread d2/f28 [0,4194304] 0 2024-03-14T13:02:17.872 INFO:tasks.workunit.client.0.smithi111.stdout:7/447: mknod d0/d1/d22/c88 0 2024-03-14T13:02:17.876 INFO:tasks.workunit.client.0.smithi111.stdout:7/448: rmdir d0/d1/d4a/d52 39 2024-03-14T13:02:17.876 INFO:tasks.workunit.client.1.smithi137.stdout:9/550: chown d6/df/d19/d3a/la1 23 1 2024-03-14T13:02:17.880 INFO:tasks.workunit.client.0.smithi111.stdout:7/449: write d0/d1/d21/f15 [704276,45312] 0 2024-03-14T13:02:17.881 INFO:tasks.workunit.client.0.smithi111.stdout:4/786: dwrite d1d/d22/d40/d69/d86/f9f [0,4194304] 0 2024-03-14T13:02:17.881 INFO:tasks.workunit.client.1.smithi137.stdout:2/507: mknod d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/ca4 0 2024-03-14T13:02:17.882 INFO:tasks.workunit.client.1.smithi137.stdout:9/551: write d6/df/d19/d1f/f35 [4429865,21925] 0 2024-03-14T13:02:17.883 INFO:tasks.workunit.client.1.smithi137.stdout:6/441: dwrite d2/d54/d55/d6d/f90 [0,4194304] 0 2024-03-14T13:02:17.887 INFO:tasks.workunit.client.0.smithi111.stdout:4/787: mkdir d1d/d48/d112 0 2024-03-14T13:02:17.887 INFO:tasks.workunit.client.0.smithi111.stdout:7/450: mknod d0/d1/d4a/d87/c89 0 2024-03-14T13:02:17.887 INFO:tasks.workunit.client.0.smithi111.stdout:8/694: dwrite d3/dc/f6c [4194304,4194304] 0 2024-03-14T13:02:17.893 INFO:tasks.workunit.client.1.smithi137.stdout:6/442: creat d2/d6/d1d/d88/f9d x:0 0 0 2024-03-14T13:02:17.893 INFO:tasks.workunit.client.0.smithi111.stdout:8/695: write d3/d1b/f92 [2065980,45116] 0 2024-03-14T13:02:17.894 INFO:tasks.workunit.client.1.smithi137.stdout:5/381: dwrite d7/f7b [0,4194304] 0 2024-03-14T13:02:17.897 INFO:tasks.workunit.client.0.smithi111.stdout:4/788: dread d1d/d22/d30/d5e/f8d [0,4194304] 0 2024-03-14T13:02:17.897 INFO:tasks.workunit.client.1.smithi137.stdout:7/360: dwrite d3/f32 [0,4194304] 0 2024-03-14T13:02:17.898 INFO:tasks.workunit.client.1.smithi137.stdout:2/508: mkdir d2/d7/d2e/d3e/d44/d6b/d22/da5 0 2024-03-14T13:02:17.900 INFO:tasks.workunit.client.0.smithi111.stdout:7/451: dwrite d0/d1/d21/f3f [8388608,4194304] 0 2024-03-14T13:02:17.902 INFO:tasks.workunit.client.0.smithi111.stdout:5/666: dread d2/d45/f66 [0,4194304] 0 2024-03-14T13:02:17.904 INFO:tasks.workunit.client.1.smithi137.stdout:7/361: read d3/d6/d14/f2f [4097778,98783] 0 2024-03-14T13:02:17.904 INFO:tasks.workunit.client.0.smithi111.stdout:4/789: mknod d1d/d48/dbe/c113 0 2024-03-14T13:02:17.906 INFO:tasks.workunit.client.1.smithi137.stdout:9/552: dwrite d6/df/f92 [0,4194304] 0 2024-03-14T13:02:17.906 INFO:tasks.workunit.client.0.smithi111.stdout:7/452: creat d0/d1/d21/d2a/f8a x:0 0 0 2024-03-14T13:02:17.907 INFO:tasks.workunit.client.1.smithi137.stdout:2/509: rmdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57 39 2024-03-14T13:02:17.910 INFO:tasks.workunit.client.0.smithi111.stdout:7/453: truncate d0/d1/d5/d38/f7f 67869 0 2024-03-14T13:02:17.913 INFO:tasks.workunit.client.0.smithi111.stdout:5/667: unlink d2/d13/f93 0 2024-03-14T13:02:17.913 INFO:tasks.workunit.client.0.smithi111.stdout:4/790: creat d1d/d22/d40/d69/d86/db1/f114 x:0 0 0 2024-03-14T13:02:17.913 INFO:tasks.workunit.client.1.smithi137.stdout:6/443: link d2/d6/de/lf d2/d6/de/d45/d59/d39/d3c/d83/l9e 0 2024-03-14T13:02:17.914 INFO:tasks.workunit.client.0.smithi111.stdout:4/791: chown fe 5755 1 2024-03-14T13:02:17.914 INFO:tasks.workunit.client.0.smithi111.stdout:8/696: dread d3/d1b/d20/f9e [0,4194304] 0 2024-03-14T13:02:17.918 INFO:tasks.workunit.client.1.smithi137.stdout:2/510: rename d2/d7/f95 to d2/d7/d2e/d3e/d44/d6b/fa6 0 2024-03-14T13:02:17.919 INFO:tasks.workunit.client.1.smithi137.stdout:4/444: truncate d4/f8 919297 0 2024-03-14T13:02:17.919 INFO:tasks.workunit.client.0.smithi111.stdout:8/697: unlink d3/d1b/dc1/cc7 0 2024-03-14T13:02:17.921 INFO:tasks.workunit.client.0.smithi111.stdout:5/668: unlink d2/d16/d49/d44/d72/ld5 0 2024-03-14T13:02:17.922 INFO:tasks.workunit.client.1.smithi137.stdout:6/444: rename d2/d6/de/d45/d9b to d2/d6/de/d45/d97/d9f 0 2024-03-14T13:02:17.923 INFO:tasks.workunit.client.0.smithi111.stdout:8/698: dread d3/dc/d45/f7c [0,4194304] 0 2024-03-14T13:02:17.923 INFO:tasks.workunit.client.0.smithi111.stdout:4/792: getdents d1d/d22/d30/d5e/d6f/d7e 0 2024-03-14T13:02:17.925 INFO:tasks.workunit.client.1.smithi137.stdout:6/445: creat d2/d54/d55/d33/fa0 x:0 0 0 2024-03-14T13:02:17.927 INFO:tasks.workunit.client.0.smithi111.stdout:5/669: rename d2/d16/d49/d7c/d8b/cad to d2/d16/d49/d33/d68/cdc 0 2024-03-14T13:02:17.927 INFO:tasks.workunit.client.1.smithi137.stdout:4/445: rename d4/d9/l47 to d4/d27/l99 0 2024-03-14T13:02:17.928 INFO:tasks.workunit.client.1.smithi137.stdout:2/511: getdents d2/d7/db 0 2024-03-14T13:02:17.929 INFO:tasks.workunit.client.0.smithi111.stdout:4/793: rename d1d/d22/d56 to d1d/d22/dc6/d115 0 2024-03-14T13:02:17.930 INFO:tasks.workunit.client.0.smithi111.stdout:5/670: mkdir d2/d45/d76/d9a/ddd 0 2024-03-14T13:02:17.930 INFO:tasks.workunit.client.1.smithi137.stdout:2/512: write d2/d7/d2e/d3e/d44/d6b/d22/f91 [1122540,34839] 0 2024-03-14T13:02:17.934 INFO:tasks.workunit.client.0.smithi111.stdout:8/699: dread d3/d22/d30/f47 [0,4194304] 0 2024-03-14T13:02:17.935 INFO:tasks.workunit.client.0.smithi111.stdout:4/794: rename d1d/d22/d30/d5e/d6f/c90 to d1d/d22/d40/d69/d86/db1/c116 0 2024-03-14T13:02:17.936 INFO:tasks.workunit.client.0.smithi111.stdout:4/795: stat d1d/d22/d30/d87/cea 0 2024-03-14T13:02:17.937 INFO:tasks.workunit.client.0.smithi111.stdout:8/700: read d3/d1b/d20/d35/d40/fac [3810171,23190] 0 2024-03-14T13:02:17.938 INFO:tasks.workunit.client.0.smithi111.stdout:4/796: creat d1d/d48/dbe/f117 x:0 0 0 2024-03-14T13:02:17.938 INFO:tasks.workunit.client.0.smithi111.stdout:8/701: stat d3/d5 0 2024-03-14T13:02:17.938 INFO:tasks.workunit.client.0.smithi111.stdout:4/797: dread - d1d/d22/d23/fe7 zero size 2024-03-14T13:02:17.940 INFO:tasks.workunit.client.0.smithi111.stdout:8/702: mknod d3/dc/d31/d9d/cdd 0 2024-03-14T13:02:17.941 INFO:tasks.workunit.client.0.smithi111.stdout:4/798: mknod d1d/d105/c118 0 2024-03-14T13:02:17.942 INFO:tasks.workunit.client.0.smithi111.stdout:4/799: rmdir d1d/d22/d30/d5e/d6f 39 2024-03-14T13:02:17.944 INFO:tasks.workunit.client.0.smithi111.stdout:4/800: chown d1d/d22/d30/d7c/c9b 41918 1 2024-03-14T13:02:17.945 INFO:tasks.workunit.client.1.smithi137.stdout:2/513: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/f91 0 2024-03-14T13:02:17.946 INFO:tasks.workunit.client.0.smithi111.stdout:8/703: dwrite d3/d5/fd [4194304,4194304] 0 2024-03-14T13:02:17.948 INFO:tasks.workunit.client.1.smithi137.stdout:2/514: creat d2/d34/d3c/d53/fa7 x:0 0 0 2024-03-14T13:02:17.950 INFO:tasks.workunit.client.1.smithi137.stdout:0/412: write d1/d5/d1b/d1e/f3d [10970,124026] 0 2024-03-14T13:02:17.953 INFO:tasks.workunit.client.0.smithi111.stdout:4/801: creat d1d/d22/f119 x:0 0 0 2024-03-14T13:02:17.954 INFO:tasks.workunit.client.0.smithi111.stdout:4/802: fdatasync d1d/d22/d30/d7c/ff6 0 2024-03-14T13:02:17.957 INFO:tasks.workunit.client.1.smithi137.stdout:6/446: dread d2/d6/de/d45/d59/d39/f73 [0,4194304] 0 2024-03-14T13:02:17.958 INFO:tasks.workunit.client.1.smithi137.stdout:4/446: dread d4/d10/d5a/f63 [0,4194304] 0 2024-03-14T13:02:17.959 INFO:tasks.workunit.client.1.smithi137.stdout:2/515: rename d2/d7/db/d2f/d66/f7a to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/fa8 0 2024-03-14T13:02:17.960 INFO:tasks.workunit.client.0.smithi111.stdout:8/704: getdents d3/d22 0 2024-03-14T13:02:17.961 INFO:tasks.workunit.client.1.smithi137.stdout:2/516: chown d2/ca2 12857573 1 2024-03-14T13:02:17.962 INFO:tasks.workunit.client.0.smithi111.stdout:8/705: stat d3/dc/d31/lce 0 2024-03-14T13:02:17.962 INFO:tasks.workunit.client.1.smithi137.stdout:5/382: truncate d7/f7b 2752282 0 2024-03-14T13:02:17.963 INFO:tasks.workunit.client.0.smithi111.stdout:5/671: dwrite d2/d45/f20 [0,4194304] 0 2024-03-14T13:02:17.964 INFO:tasks.workunit.client.1.smithi137.stdout:6/447: rename l1 to d2/d6/d86/d8a/la1 0 2024-03-14T13:02:17.964 INFO:tasks.workunit.client.1.smithi137.stdout:2/517: read - d2/d7/d2e/d3e/d44/d6b/d1d/f7b zero size 2024-03-14T13:02:17.965 INFO:tasks.workunit.client.0.smithi111.stdout:8/706: rmdir d3/d1b/dc1 39 2024-03-14T13:02:17.966 INFO:tasks.workunit.client.1.smithi137.stdout:7/362: truncate d3/d4a/f5e 89993 0 2024-03-14T13:02:17.967 INFO:tasks.workunit.client.0.smithi111.stdout:4/803: link d1d/d22/d30/d7c/d59/c95 d1d/d48/c11a 0 2024-03-14T13:02:17.968 INFO:tasks.workunit.client.0.smithi111.stdout:4/804: chown d1d/d22/d30/d87/ffc 31853773 1 2024-03-14T13:02:17.969 INFO:tasks.workunit.client.0.smithi111.stdout:4/805: write d1d/d22/d30/d7c/ff6 [214271,65734] 0 2024-03-14T13:02:17.969 INFO:tasks.workunit.client.1.smithi137.stdout:6/448: fsync d2/d6/de/d45/d59/d39/f7a 0 2024-03-14T13:02:17.969 INFO:tasks.workunit.client.0.smithi111.stdout:8/707: rename d3/d22/d4a/c82 to d3/d5/cde 0 2024-03-14T13:02:17.970 INFO:tasks.workunit.client.0.smithi111.stdout:8/708: fsync d3/d22/d30/fc3 0 2024-03-14T13:02:17.971 INFO:tasks.workunit.client.1.smithi137.stdout:2/518: dread - d2/d34/d3c/d53/fa7 zero size 2024-03-14T13:02:17.972 INFO:tasks.workunit.client.0.smithi111.stdout:4/806: creat d1d/d22/d30/d7c/f11b x:0 0 0 2024-03-14T13:02:17.972 INFO:tasks.workunit.client.1.smithi137.stdout:4/447: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:17.972 INFO:tasks.workunit.client.0.smithi111.stdout:4/807: dread - d1d/f10c zero size 2024-03-14T13:02:17.975 INFO:tasks.workunit.client.0.smithi111.stdout:4/808: mknod d1d/d22/d30/d7c/df8/c11c 0 2024-03-14T13:02:17.977 INFO:tasks.workunit.client.0.smithi111.stdout:4/809: write d1d/d22/fd7 [715611,105473] 0 2024-03-14T13:02:17.979 INFO:tasks.workunit.client.0.smithi111.stdout:8/709: rename d3/d1b/d20/d35/d40/d48/f68 to d3/d1b/d20/d35/d40/d48/d77/fdf 0 2024-03-14T13:02:17.980 INFO:tasks.workunit.client.0.smithi111.stdout:4/810: fsync d1d/fbb 0 2024-03-14T13:02:17.980 INFO:tasks.workunit.client.1.smithi137.stdout:5/383: dwrite d7/d16/d18/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:17.980 INFO:tasks.workunit.client.1.smithi137.stdout:7/363: dread d3/d5/f11 [0,4194304] 0 2024-03-14T13:02:17.981 INFO:tasks.workunit.client.1.smithi137.stdout:2/519: dread d2/d34/d3c/f6d [0,4194304] 0 2024-03-14T13:02:17.982 INFO:tasks.workunit.client.1.smithi137.stdout:5/384: fsync d7/d16/d18/d25/f5b 0 2024-03-14T13:02:17.984 INFO:tasks.workunit.client.1.smithi137.stdout:4/448: creat d4/d27/d5c/d6d/d58/d65/f9a x:0 0 0 2024-03-14T13:02:17.985 INFO:tasks.workunit.client.1.smithi137.stdout:6/449: unlink d2/d6/de/c7f 0 2024-03-14T13:02:17.990 INFO:tasks.workunit.client.1.smithi137.stdout:7/364: symlink d3/d5/l72 0 2024-03-14T13:02:17.991 INFO:tasks.workunit.client.1.smithi137.stdout:7/365: chown d3/d5/l72 349340 1 2024-03-14T13:02:17.991 INFO:tasks.workunit.client.1.smithi137.stdout:2/520: unlink d2/d7/db/f38 0 2024-03-14T13:02:18.002 INFO:tasks.workunit.client.0.smithi111.stdout:8/710: dwrite d3/d22/d30/f74 [0,4194304] 0 2024-03-14T13:02:18.004 INFO:tasks.workunit.client.1.smithi137.stdout:4/449: write d4/d10/d42/f4e [5630566,32278] 0 2024-03-14T13:02:18.004 INFO:tasks.workunit.client.0.smithi111.stdout:3/474: sync 2024-03-14T13:02:18.005 INFO:tasks.workunit.client.0.smithi111.stdout:2/483: sync 2024-03-14T13:02:18.005 INFO:tasks.workunit.client.0.smithi111.stdout:6/478: sync 2024-03-14T13:02:18.005 INFO:tasks.workunit.client.0.smithi111.stdout:9/477: sync 2024-03-14T13:02:18.005 INFO:tasks.workunit.client.1.smithi137.stdout:6/450: creat d2/d6/d1d/d88/fa2 x:0 0 0 2024-03-14T13:02:18.006 INFO:tasks.workunit.client.0.smithi111.stdout:8/711: mkdir d3/d1b/dc1/de0 0 2024-03-14T13:02:18.006 INFO:tasks.workunit.client.0.smithi111.stdout:6/479: chown d0/f1a 1791 1 2024-03-14T13:02:18.007 INFO:tasks.workunit.client.1.smithi137.stdout:7/366: dwrite d3/d6/d59/f5f [0,4194304] 0 2024-03-14T13:02:18.010 INFO:tasks.workunit.client.0.smithi111.stdout:2/484: chown dd/d8d/f9b 3663 1 2024-03-14T13:02:18.012 INFO:tasks.workunit.client.0.smithi111.stdout:4/811: dread d1d/d22/fad [0,4194304] 0 2024-03-14T13:02:18.016 INFO:tasks.workunit.client.1.smithi137.stdout:2/521: getdents d2/d7/d2e/d3e/d44/d6b/d1d/d3a 0 2024-03-14T13:02:18.016 INFO:tasks.workunit.client.1.smithi137.stdout:4/450: write d4/d9/d3a/f56 [1267958,60728] 0 2024-03-14T13:02:18.017 INFO:tasks.workunit.client.0.smithi111.stdout:3/475: dread - d0/d2d/d35/d50/d5e/d19/f44 zero size 2024-03-14T13:02:18.017 INFO:tasks.workunit.client.0.smithi111.stdout:9/478: creat d5/d6/d8/d5b/d84/fa2 x:0 0 0 2024-03-14T13:02:18.017 INFO:tasks.workunit.client.0.smithi111.stdout:6/480: fdatasync d0/d50/f67 0 2024-03-14T13:02:18.017 INFO:tasks.workunit.client.0.smithi111.stdout:2/485: unlink dd/d17/l35 0 2024-03-14T13:02:18.017 INFO:tasks.workunit.client.1.smithi137.stdout:5/385: dread d7/f15 [0,4194304] 0 2024-03-14T13:02:18.018 INFO:tasks.workunit.client.0.smithi111.stdout:4/812: creat d1d/d22/dc6/de6/d107/f11d x:0 0 0 2024-03-14T13:02:18.018 INFO:tasks.workunit.client.0.smithi111.stdout:4/813: stat d1d/f2e 0 2024-03-14T13:02:18.018 INFO:tasks.workunit.client.1.smithi137.stdout:7/367: link d3/d6/d2e/f3e d3/d5/d5a/f73 0 2024-03-14T13:02:18.019 INFO:tasks.workunit.client.1.smithi137.stdout:6/451: creat d2/d54/d55/fa3 x:0 0 0 2024-03-14T13:02:18.020 INFO:tasks.workunit.client.1.smithi137.stdout:6/452: chown d2/d6/de/d45/d59/f75 46847653 1 2024-03-14T13:02:18.021 INFO:tasks.workunit.client.1.smithi137.stdout:2/522: symlink d2/d34/d3c/la9 0 2024-03-14T13:02:18.021 INFO:tasks.workunit.client.1.smithi137.stdout:6/453: fsync d2/d4c/f84 0 2024-03-14T13:02:18.022 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:17 smithi111 ceph-mon[39688]: pgmap v23: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 21 MiB/s rd, 169 MiB/s wr, 261 op/s 2024-03-14T13:02:18.023 INFO:tasks.workunit.client.1.smithi137.stdout:4/451: truncate d4/d27/d5c/d6d/d58/f76 131501 0 2024-03-14T13:02:18.025 INFO:tasks.workunit.client.0.smithi111.stdout:2/486: symlink dd/d11/d1a/d70/l9d 0 2024-03-14T13:02:18.026 INFO:tasks.workunit.client.0.smithi111.stdout:4/814: dwrite d1d/d22/f41 [0,4194304] 0 2024-03-14T13:02:18.027 INFO:tasks.workunit.client.0.smithi111.stdout:9/479: link d5/d6/d14/la1 d5/d6/d2a/la3 0 2024-03-14T13:02:18.028 INFO:tasks.workunit.client.1.smithi137.stdout:1/501: sync 2024-03-14T13:02:18.028 INFO:tasks.workunit.client.1.smithi137.stdout:3/407: sync 2024-03-14T13:02:18.034 INFO:tasks.workunit.client.1.smithi137.stdout:4/452: unlink d4/d27/d5c/d6d/d58/d65/f9a 0 2024-03-14T13:02:18.034 INFO:tasks.workunit.client.0.smithi111.stdout:4/815: creat d1d/f11e x:0 0 0 2024-03-14T13:02:18.034 INFO:tasks.workunit.client.1.smithi137.stdout:3/408: readlink d4/de/l6e 0 2024-03-14T13:02:18.037 INFO:tasks.workunit.client.1.smithi137.stdout:5/386: rename d7/d16/f68 to d7/d16/d18/f8a 0 2024-03-14T13:02:18.039 INFO:tasks.workunit.client.0.smithi111.stdout:3/476: write d0/f20 [110156,130538] 0 2024-03-14T13:02:18.041 INFO:tasks.workunit.client.0.smithi111.stdout:4/816: dread - d1d/d22/d40/d69/d86/db1/f114 zero size 2024-03-14T13:02:18.043 INFO:tasks.workunit.client.1.smithi137.stdout:5/387: dread d7/f7b [0,4194304] 0 2024-03-14T13:02:18.043 INFO:tasks.workunit.client.1.smithi137.stdout:7/368: dread d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:18.045 INFO:tasks.workunit.client.1.smithi137.stdout:5/388: fsync d7/d16/d18/d25/f55 0 2024-03-14T13:02:18.046 INFO:tasks.workunit.client.0.smithi111.stdout:2/487: dwrite dd/d17/d34/d73/d75/f95 [0,4194304] 0 2024-03-14T13:02:18.046 INFO:tasks.workunit.client.0.smithi111.stdout:9/480: write d5/d6/d2a/d2d/d8b/f8e [323478,61755] 0 2024-03-14T13:02:18.047 INFO:tasks.workunit.client.1.smithi137.stdout:7/369: truncate d3/d54/f5c 262893 0 2024-03-14T13:02:18.048 INFO:tasks.workunit.client.0.smithi111.stdout:3/477: rmdir d0/d2d/d35/d69/d84 39 2024-03-14T13:02:18.049 INFO:tasks.workunit.client.0.smithi111.stdout:9/481: fsync f2 0 2024-03-14T13:02:18.049 INFO:tasks.workunit.client.0.smithi111.stdout:4/817: creat d1d/d22/d30/d7c/f11f x:0 0 0 2024-03-14T13:02:18.051 INFO:tasks.workunit.client.1.smithi137.stdout:2/523: dread d2/d7/d2e/d3e/d44/d6b/f20 [0,4194304] 0 2024-03-14T13:02:18.051 INFO:tasks.workunit.client.1.smithi137.stdout:3/409: mknod d4/de/d16/d1c/d5d/d85/c87 0 2024-03-14T13:02:18.052 INFO:tasks.workunit.client.1.smithi137.stdout:5/389: symlink d7/d1d/d74/l8b 0 2024-03-14T13:02:18.053 INFO:tasks.workunit.client.1.smithi137.stdout:6/454: mkdir d2/d6/de/d45/d97/d9f/da4 0 2024-03-14T13:02:18.054 INFO:tasks.workunit.client.1.smithi137.stdout:7/370: mkdir d3/d6/d64/d74 0 2024-03-14T13:02:18.055 INFO:tasks.workunit.client.0.smithi111.stdout:6/481: dread d0/d17/d20/d4a/d59/f7e [0,4194304] 0 2024-03-14T13:02:18.056 INFO:tasks.workunit.client.1.smithi137.stdout:6/455: write d2/d6/de/d45/f5b [1011736,131010] 0 2024-03-14T13:02:18.056 INFO:tasks.workunit.client.1.smithi137.stdout:7/371: stat d3/d54/f61 0 2024-03-14T13:02:18.057 INFO:tasks.workunit.client.1.smithi137.stdout:2/524: write d2/d7/d2e/d3e/d44/d6b/d1d/f7b [797979,57044] 0 2024-03-14T13:02:18.058 INFO:tasks.workunit.client.0.smithi111.stdout:9/482: rename d5/d6/d2a/d2d/d9b to d5/d42/da4 0 2024-03-14T13:02:18.059 INFO:tasks.workunit.client.1.smithi137.stdout:1/502: truncate d7/d9/d19/f2f 5278769 0 2024-03-14T13:02:18.060 INFO:tasks.workunit.client.0.smithi111.stdout:3/478: dread - d0/d2d/d35/d50/da2/db3/d6f/fb7 zero size 2024-03-14T13:02:18.060 INFO:tasks.workunit.client.1.smithi137.stdout:4/453: truncate d4/d9/d3a/f56 351816 0 2024-03-14T13:02:18.060 INFO:tasks.workunit.client.1.smithi137.stdout:3/410: mknod d4/de/d16/d27/c88 0 2024-03-14T13:02:18.062 INFO:tasks.workunit.client.1.smithi137.stdout:6/456: creat d2/d54/d55/d6d/d76/d57/d58/fa5 x:0 0 0 2024-03-14T13:02:18.065 INFO:tasks.workunit.client.1.smithi137.stdout:7/372: mkdir d3/d5/d23/d2c/d5d/d6a/d75 0 2024-03-14T13:02:18.067 INFO:tasks.workunit.client.0.smithi111.stdout:2/488: getdents dd/d17/d34/d73/d75 0 2024-03-14T13:02:18.067 INFO:tasks.workunit.client.1.smithi137.stdout:1/503: mkdir d7/d9/d19/d40/d9c 0 2024-03-14T13:02:18.068 INFO:tasks.workunit.client.1.smithi137.stdout:3/411: symlink d4/de/l89 0 2024-03-14T13:02:18.069 INFO:tasks.workunit.client.0.smithi111.stdout:4/818: write d1d/d22/d30/d5e/f84 [3733698,112990] 0 2024-03-14T13:02:18.069 INFO:tasks.workunit.client.1.smithi137.stdout:4/454: creat d4/d4d/d6e/f9b x:0 0 0 2024-03-14T13:02:18.070 INFO:tasks.workunit.client.0.smithi111.stdout:6/482: dwrite d0/d17/d99/fa9 [0,4194304] 0 2024-03-14T13:02:18.071 INFO:tasks.workunit.client.1.smithi137.stdout:2/525: getdents d2/d7/db/d74/d84 0 2024-03-14T13:02:18.071 INFO:tasks.workunit.client.1.smithi137.stdout:2/526: readlink d2/l3 0 2024-03-14T13:02:18.073 INFO:tasks.workunit.client.0.smithi111.stdout:4/819: creat d1d/d22/d30/d87/dd1/f120 x:0 0 0 2024-03-14T13:02:18.073 INFO:tasks.workunit.client.1.smithi137.stdout:7/373: creat d3/d5/d23/d3f/f76 x:0 0 0 2024-03-14T13:02:18.073 INFO:tasks.workunit.client.1.smithi137.stdout:2/527: write d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f83 [1031163,11156] 0 2024-03-14T13:02:18.078 INFO:tasks.workunit.client.0.smithi111.stdout:4/820: write d1d/d22/d23/fe7 [792291,64994] 0 2024-03-14T13:02:18.079 INFO:tasks.workunit.client.0.smithi111.stdout:9/483: link d5/d6/d8/d5b/l69 d5/d6/d8/d5b/d7c/la5 0 2024-03-14T13:02:18.079 INFO:tasks.workunit.client.0.smithi111.stdout:3/479: write d0/f7 [4279288,101239] 0 2024-03-14T13:02:18.079 INFO:tasks.workunit.client.1.smithi137.stdout:5/390: truncate d7/d16/d18/f2a 715206 0 2024-03-14T13:02:18.082 INFO:tasks.workunit.client.0.smithi111.stdout:2/489: dwrite dd/d17/d34/d73/d75/f6e [0,4194304] 0 2024-03-14T13:02:18.082 INFO:tasks.workunit.client.1.smithi137.stdout:2/528: mkdir d2/d7/d2e/d3e/daa 0 2024-03-14T13:02:18.082 INFO:tasks.workunit.client.1.smithi137.stdout:5/391: readlink d7/d1d/d74/l8b 0 2024-03-14T13:02:18.083 INFO:tasks.workunit.client.0.smithi111.stdout:6/483: rename d0/d17/d20/d4a/d59/d82/f8f to d0/d17/d20/d4a/d59/fbb 0 2024-03-14T13:02:18.083 INFO:tasks.workunit.client.0.smithi111.stdout:9/484: write d5/d6/d8/d5b/d84/fa2 [679370,121895] 0 2024-03-14T13:02:18.085 INFO:tasks.workunit.client.1.smithi137.stdout:6/457: rmdir d2/d6/de/d45/d97/d9f/da4 0 2024-03-14T13:02:18.087 INFO:tasks.workunit.client.1.smithi137.stdout:4/455: symlink d4/d9/d8e/d95/l9c 0 2024-03-14T13:02:18.088 INFO:tasks.workunit.client.1.smithi137.stdout:1/504: dwrite d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:18.089 INFO:tasks.workunit.client.1.smithi137.stdout:7/374: mkdir d3/d52/d77 0 2024-03-14T13:02:18.092 INFO:tasks.workunit.client.0.smithi111.stdout:3/480: write d0/f8 [1287758,27066] 0 2024-03-14T13:02:18.092 INFO:tasks.workunit.client.0.smithi111.stdout:2/490: creat dd/d17/d34/d74/f9e x:0 0 0 2024-03-14T13:02:18.092 INFO:tasks.workunit.client.0.smithi111.stdout:4/821: creat d1d/d22/d30/d5e/d6f/d7e/d66/df4/df5/f121 x:0 0 0 2024-03-14T13:02:18.094 INFO:tasks.workunit.client.1.smithi137.stdout:2/529: creat d2/d7/d2e/d3e/d44/fab x:0 0 0 2024-03-14T13:02:18.094 INFO:tasks.workunit.client.1.smithi137.stdout:7/375: chown d3/d6/d59/c71 0 1 2024-03-14T13:02:18.094 INFO:tasks.workunit.client.1.smithi137.stdout:6/458: creat d2/d6/de/fa6 x:0 0 0 2024-03-14T13:02:18.095 INFO:tasks.workunit.client.0.smithi111.stdout:2/491: truncate dd/d17/f83 401718 0 2024-03-14T13:02:18.096 INFO:tasks.workunit.client.0.smithi111.stdout:6/484: link d0/f74 d0/d17/d21/d69/d3d/d53/d81/da1/fbc 0 2024-03-14T13:02:18.099 INFO:tasks.workunit.client.1.smithi137.stdout:3/412: rename d4/de/d16/f20 to d4/de/d16/d1c/d5d/d85/d6b/f8a 0 2024-03-14T13:02:18.099 INFO:tasks.workunit.client.0.smithi111.stdout:4/822: rename d1d/d22/dc6 to d1d/d22/d30/d7c/df8/d122 0 2024-03-14T13:02:18.100 INFO:tasks.workunit.client.1.smithi137.stdout:5/392: creat d7/d49/d75/f8c x:0 0 0 2024-03-14T13:02:18.101 INFO:tasks.workunit.client.0.smithi111.stdout:2/492: unlink dd/d17/d34/d73/d75/f6e 0 2024-03-14T13:02:18.102 INFO:tasks.workunit.client.0.smithi111.stdout:2/493: write dd/d11/d1a/f3d [489350,89306] 0 2024-03-14T13:02:18.103 INFO:tasks.workunit.client.1.smithi137.stdout:6/459: truncate d2/d6/de/f13 5707053 0 2024-03-14T13:02:18.103 INFO:tasks.workunit.client.0.smithi111.stdout:2/494: stat dd/d17 0 2024-03-14T13:02:18.103 INFO:tasks.workunit.client.1.smithi137.stdout:1/505: dwrite d7/d9/d19/d1f/d26/d84/f8c [0,4194304] 0 2024-03-14T13:02:18.104 INFO:tasks.workunit.client.0.smithi111.stdout:6/485: mkdir d0/d17/d20/d94/dbd 0 2024-03-14T13:02:18.105 INFO:tasks.workunit.client.1.smithi137.stdout:2/530: mkdir d2/d7/d2e/d3e/d44/d6b/d22/dac 0 2024-03-14T13:02:18.106 INFO:tasks.workunit.client.1.smithi137.stdout:2/531: stat d2/d7/db/d74 0 2024-03-14T13:02:18.107 INFO:tasks.workunit.client.0.smithi111.stdout:3/481: creat d0/d2d/d35/d50/d5e/d19/fb9 x:0 0 0 2024-03-14T13:02:18.107 INFO:tasks.workunit.client.1.smithi137.stdout:7/376: truncate d3/d6/d2e/f3e 574910 0 2024-03-14T13:02:18.107 INFO:tasks.workunit.client.1.smithi137.stdout:4/456: mkdir d4/d10/d42/d73/d9d 0 2024-03-14T13:02:18.109 INFO:tasks.workunit.client.1.smithi137.stdout:6/460: write d2/d6/de/fa6 [831271,50019] 0 2024-03-14T13:02:18.110 INFO:tasks.workunit.client.1.smithi137.stdout:4/457: chown d4/d9/d3a/d41/d38/d55/c91 214381 1 2024-03-14T13:02:18.111 INFO:tasks.workunit.client.1.smithi137.stdout:5/393: chown d7/d16/d41 126836 1 2024-03-14T13:02:18.114 INFO:tasks.workunit.client.0.smithi111.stdout:2/495: mkdir dd/d17/d53/d9f 0 2024-03-14T13:02:18.114 INFO:tasks.workunit.client.0.smithi111.stdout:9/485: write d5/d6/d8/d1a/f64 [405666,86264] 0 2024-03-14T13:02:18.114 INFO:tasks.workunit.client.0.smithi111.stdout:6/486: chown d0/d17/d21/d42/ca0 292515977 1 2024-03-14T13:02:18.116 INFO:tasks.workunit.client.1.smithi137.stdout:1/506: mkdir d7/d9/d19/d40/d42/d9d 0 2024-03-14T13:02:18.118 INFO:tasks.workunit.client.1.smithi137.stdout:4/458: symlink d4/d10/d42/d73/l9e 0 2024-03-14T13:02:18.120 INFO:tasks.workunit.client.1.smithi137.stdout:5/394: rmdir d7/d1d 39 2024-03-14T13:02:18.120 INFO:tasks.workunit.client.0.smithi111.stdout:4/823: write d1d/f20 [1496956,47690] 0 2024-03-14T13:02:18.122 INFO:tasks.workunit.client.1.smithi137.stdout:2/532: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d41/dad 0 2024-03-14T13:02:18.124 INFO:tasks.workunit.client.1.smithi137.stdout:6/461: mkdir d2/d6/da7 0 2024-03-14T13:02:18.125 INFO:tasks.workunit.client.0.smithi111.stdout:3/482: write d0/d2d/d35/d50/f54 [53344,85930] 0 2024-03-14T13:02:18.125 INFO:tasks.workunit.client.1.smithi137.stdout:6/462: chown d2/d6/de/d45/d97/d9f 0 1 2024-03-14T13:02:18.126 INFO:tasks.workunit.client.1.smithi137.stdout:6/463: chown d2/d4c/c6a 917456435 1 2024-03-14T13:02:18.126 INFO:tasks.workunit.client.1.smithi137.stdout:1/507: creat d7/d4d/d6b/f9e x:0 0 0 2024-03-14T13:02:18.127 INFO:tasks.workunit.client.0.smithi111.stdout:6/487: dwrite d0/d17/d99/fa9 [0,4194304] 0 2024-03-14T13:02:18.127 INFO:tasks.workunit.client.1.smithi137.stdout:4/459: mknod d4/d10/d42/c9f 0 2024-03-14T13:02:18.128 INFO:tasks.workunit.client.0.smithi111.stdout:4/824: rename d1d/d22/d30/d7c/df8/d122 to d1d/d22/d30/d7c/d59/dc4/d123 0 2024-03-14T13:02:18.129 INFO:tasks.workunit.client.1.smithi137.stdout:2/533: fdatasync d2/d34/d3c/f6d 0 2024-03-14T13:02:18.131 INFO:tasks.workunit.client.0.smithi111.stdout:9/486: dwrite d5/d6/d2a/d5a/f9f [0,4194304] 0 2024-03-14T13:02:18.132 INFO:tasks.workunit.client.0.smithi111.stdout:2/496: truncate dd/d11/d1a/f1b 1843730 0 2024-03-14T13:02:18.133 INFO:tasks.workunit.client.0.smithi111.stdout:2/497: dread - dd/d17/f71 zero size 2024-03-14T13:02:18.134 INFO:tasks.workunit.client.0.smithi111.stdout:6/488: creat d0/d17/d20/d94/fbe x:0 0 0 2024-03-14T13:02:18.136 INFO:tasks.workunit.client.0.smithi111.stdout:6/489: chown d0/d17/d20/d5a 113573 1 2024-03-14T13:02:18.140 INFO:tasks.workunit.client.0.smithi111.stdout:2/498: creat dd/d17/d34/fa0 x:0 0 0 2024-03-14T13:02:18.140 INFO:tasks.workunit.client.1.smithi137.stdout:2/534: fdatasync d2/d7/db/d2f/d66/f80 0 2024-03-14T13:02:18.146 INFO:tasks.workunit.client.0.smithi111.stdout:6/490: symlink d0/d17/d20/d4a/d59/lbf 0 2024-03-14T13:02:18.146 INFO:tasks.workunit.client.1.smithi137.stdout:2/535: dread - d2/d7/d2e/d3e/d44/d6b/fa6 zero size 2024-03-14T13:02:18.146 INFO:tasks.workunit.client.0.smithi111.stdout:9/487: write d5/d6/d8/f3a [4864751,26754] 0 2024-03-14T13:02:18.147 INFO:tasks.workunit.client.0.smithi111.stdout:4/825: truncate d1d/d22/d40/d69/d86/faa 65046 0 2024-03-14T13:02:18.147 INFO:tasks.workunit.client.1.smithi137.stdout:3/413: creat d4/de/d16/d1c/d5d/d85/d6b/f8b x:0 0 0 2024-03-14T13:02:18.148 INFO:tasks.workunit.client.0.smithi111.stdout:6/491: truncate d0/d17/f2b 5704986 0 2024-03-14T13:02:18.151 INFO:tasks.workunit.client.1.smithi137.stdout:1/508: creat d7/f9f x:0 0 0 2024-03-14T13:02:18.151 INFO:tasks.workunit.client.1.smithi137.stdout:4/460: dwrite d4/d9/d3a/d41/d38/f4c [0,4194304] 0 2024-03-14T13:02:18.151 INFO:tasks.workunit.client.0.smithi111.stdout:4/826: truncate d1d/d22/d30/d7c/ff6 363933 0 2024-03-14T13:02:18.152 INFO:tasks.workunit.client.1.smithi137.stdout:1/509: write f3 [4097278,65140] 0 2024-03-14T13:02:18.154 INFO:tasks.workunit.client.1.smithi137.stdout:2/536: mknod d2/d7/db/d2f/cae 0 2024-03-14T13:02:18.154 INFO:tasks.workunit.client.0.smithi111.stdout:9/488: rename d5/d6/d8/c1d to d5/d6/d2a/d86/ca6 0 2024-03-14T13:02:18.156 INFO:tasks.workunit.client.0.smithi111.stdout:2/499: dwrite dd/d17/d53/f92 [0,4194304] 0 2024-03-14T13:02:18.158 INFO:tasks.workunit.client.0.smithi111.stdout:4/827: creat d1d/d22/d30/d5e/db8/ddf/f124 x:0 0 0 2024-03-14T13:02:18.159 INFO:tasks.workunit.client.0.smithi111.stdout:4/828: chown d1d/f43 17 1 2024-03-14T13:02:18.159 INFO:tasks.workunit.client.1.smithi137.stdout:3/414: symlink d4/de/d16/l8c 0 2024-03-14T13:02:18.160 INFO:tasks.workunit.client.0.smithi111.stdout:9/489: symlink d5/d42/d52/la7 0 2024-03-14T13:02:18.162 INFO:tasks.workunit.client.0.smithi111.stdout:6/492: unlink d0/d17/d20/d4a/d68/l9d 0 2024-03-14T13:02:18.162 INFO:tasks.workunit.client.0.smithi111.stdout:9/490: chown d5/d6/c18 98 1 2024-03-14T13:02:18.163 INFO:tasks.workunit.client.1.smithi137.stdout:4/461: stat d4/d9/d3a/f56 0 2024-03-14T13:02:18.163 INFO:tasks.workunit.client.0.smithi111.stdout:2/500: dread dd/f87 [0,4194304] 0 2024-03-14T13:02:18.165 INFO:tasks.workunit.client.1.smithi137.stdout:1/510: mkdir d7/d9/d19/d1f/da0 0 2024-03-14T13:02:18.166 INFO:tasks.workunit.client.0.smithi111.stdout:4/829: mkdir d1d/d22/d30/d87/dd1/de1/d125 0 2024-03-14T13:02:18.167 INFO:tasks.workunit.client.1.smithi137.stdout:4/462: creat d4/d4d/fa0 x:0 0 0 2024-03-14T13:02:18.168 INFO:tasks.workunit.client.0.smithi111.stdout:2/501: write dd/d17/d34/d73/d75/d59/f65 [381618,23777] 0 2024-03-14T13:02:18.169 INFO:tasks.workunit.client.1.smithi137.stdout:7/377: dwrite d3/d6/d2e/f49 [0,4194304] 0 2024-03-14T13:02:18.169 INFO:tasks.workunit.client.0.smithi111.stdout:4/830: symlink d1d/d22/d40/d69/d86/db1/l126 0 2024-03-14T13:02:18.169 INFO:tasks.workunit.client.0.smithi111.stdout:2/502: write dd/d11/d1a/f21 [7909538,126455] 0 2024-03-14T13:02:18.170 INFO:tasks.workunit.client.0.smithi111.stdout:2/503: fdatasync dd/d17/d34/f6f 0 2024-03-14T13:02:18.171 INFO:tasks.workunit.client.0.smithi111.stdout:6/493: link d0/l90 d0/d17/d20/d5a/lc0 0 2024-03-14T13:02:18.172 INFO:tasks.workunit.client.0.smithi111.stdout:4/831: fdatasync d1d/d22/d30/d5e/d6f/d7e/f65 0 2024-03-14T13:02:18.174 INFO:tasks.workunit.client.0.smithi111.stdout:6/494: fsync d0/f4 0 2024-03-14T13:02:18.174 INFO:tasks.workunit.client.0.smithi111.stdout:0/521: sync 2024-03-14T13:02:18.177 INFO:tasks.workunit.client.0.smithi111.stdout:6/495: truncate d0/d17/d21/d69/f76 929106 0 2024-03-14T13:02:18.178 INFO:tasks.workunit.client.1.smithi137.stdout:6/464: dread d2/d54/d55/d38/f51 [0,4194304] 0 2024-03-14T13:02:18.179 INFO:tasks.workunit.client.0.smithi111.stdout:0/522: mknod d6/d9/d15/cb1 0 2024-03-14T13:02:18.179 INFO:tasks.workunit.client.0.smithi111.stdout:6/496: read d0/d17/f1e [2380540,55911] 0 2024-03-14T13:02:18.179 INFO:tasks.workunit.client.1.smithi137.stdout:2/537: getdents d2/d7/d2e/d3e/d44/d6b/d1d 0 2024-03-14T13:02:18.179 INFO:tasks.workunit.client.1.smithi137.stdout:1/511: mknod d7/d9/d19/d1f/d26/d84/ca1 0 2024-03-14T13:02:18.180 INFO:tasks.workunit.client.1.smithi137.stdout:3/415: getdents d4/d19/d5c 0 2024-03-14T13:02:18.180 INFO:tasks.workunit.client.0.smithi111.stdout:3/483: dread d0/d30/f95 [4194304,4194304] 0 2024-03-14T13:02:18.182 INFO:tasks.workunit.client.1.smithi137.stdout:4/463: mkdir d4/d9/d3a/d41/d38/d79/da1 0 2024-03-14T13:02:18.182 INFO:tasks.workunit.client.1.smithi137.stdout:1/512: chown d7/d9/d19/d1f/d2e 703 1 2024-03-14T13:02:18.183 INFO:tasks.workunit.client.1.smithi137.stdout:6/465: creat d2/d6/de/d45/fa8 x:0 0 0 2024-03-14T13:02:18.184 INFO:tasks.workunit.client.0.smithi111.stdout:2/504: link dd/d11/d1a/d70/l99 dd/d17/d34/la1 0 2024-03-14T13:02:18.184 INFO:tasks.workunit.client.0.smithi111.stdout:3/484: chown d0/d2d/d35/d50/d5e/d19/lb5 1173910 1 2024-03-14T13:02:18.184 INFO:tasks.workunit.client.0.smithi111.stdout:4/832: dwrite d1d/d22/d30/f81 [0,4194304] 0 2024-03-14T13:02:18.186 INFO:tasks.workunit.client.1.smithi137.stdout:7/378: mkdir d3/d5/d23/d2c/d5d/d6a/d75/d78 0 2024-03-14T13:02:18.186 INFO:tasks.workunit.client.1.smithi137.stdout:4/464: symlink d4/d4d/d6e/la2 0 2024-03-14T13:02:18.187 INFO:tasks.workunit.client.1.smithi137.stdout:6/466: stat d2/d6/de/d45/d97 0 2024-03-14T13:02:18.188 INFO:tasks.workunit.client.0.smithi111.stdout:6/497: truncate d0/d17/d24/f38 3382718 0 2024-03-14T13:02:18.188 INFO:tasks.workunit.client.1.smithi137.stdout:3/416: write d4/de/d13/f21 [2712354,33103] 0 2024-03-14T13:02:18.189 INFO:tasks.workunit.client.0.smithi111.stdout:0/523: creat d6/d9/fb2 x:0 0 0 2024-03-14T13:02:18.190 INFO:tasks.workunit.client.0.smithi111.stdout:3/485: truncate d0/d2d/d35/d50/d5e/f12 1438164 0 2024-03-14T13:02:18.192 INFO:tasks.workunit.client.0.smithi111.stdout:6/498: unlink d0/d17/d20/l7c 0 2024-03-14T13:02:18.198 INFO:tasks.workunit.client.0.smithi111.stdout:2/505: rename dd/d11/d1a/d70/c7c to dd/d17/ca2 0 2024-03-14T13:02:18.198 INFO:tasks.workunit.client.1.smithi137.stdout:7/379: creat d3/d5/d23/d3f/d5b/f79 x:0 0 0 2024-03-14T13:02:18.199 INFO:tasks.workunit.client.0.smithi111.stdout:4/833: dwrite d1d/f2e [0,4194304] 0 2024-03-14T13:02:18.199 INFO:tasks.workunit.client.1.smithi137.stdout:4/465: mknod d4/d27/d5c/d6d/d58/d65/ca3 0 2024-03-14T13:02:18.199 INFO:tasks.workunit.client.1.smithi137.stdout:2/538: link d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f93 d2/d7/d2e/d3e/d44/d6b/d22/dac/faf 0 2024-03-14T13:02:18.203 INFO:tasks.workunit.client.0.smithi111.stdout:6/499: read d0/d17/d20/d4a/d59/f7e [3633156,41518] 0 2024-03-14T13:02:18.204 INFO:tasks.workunit.client.1.smithi137.stdout:6/467: rename d2/d6/f3d to d2/d6/d86/fa9 0 2024-03-14T13:02:18.205 INFO:tasks.workunit.client.0.smithi111.stdout:3/486: dwrite d0/d2d/d35/d50/da2/db3/d71/fa6 [0,4194304] 0 2024-03-14T13:02:18.205 INFO:tasks.workunit.client.0.smithi111.stdout:4/834: chown d1d/d22/d40/l99 327 1 2024-03-14T13:02:18.207 INFO:tasks.workunit.client.0.smithi111.stdout:6/500: write d0/f1a [1542215,82930] 0 2024-03-14T13:02:18.208 INFO:tasks.workunit.client.1.smithi137.stdout:7/380: rename d3/d5/c40 to d3/d5/d23/d2c/c7a 0 2024-03-14T13:02:18.208 INFO:tasks.workunit.client.0.smithi111.stdout:2/506: mknod dd/d17/d34/d73/d49/ca3 0 2024-03-14T13:02:18.208 INFO:tasks.workunit.client.0.smithi111.stdout:3/487: dread - d0/d2d/d35/d50/d6c/da3/daa/fb1 zero size 2024-03-14T13:02:18.209 INFO:tasks.workunit.client.0.smithi111.stdout:2/507: readlink lc 0 2024-03-14T13:02:18.210 INFO:tasks.workunit.client.1.smithi137.stdout:6/468: mkdir d2/d6/de/d45/d59/daa 0 2024-03-14T13:02:18.212 INFO:tasks.workunit.client.0.smithi111.stdout:0/524: rename d6/d3d/f94 to d6/fb3 0 2024-03-14T13:02:18.213 INFO:tasks.workunit.client.0.smithi111.stdout:2/508: chown dd/l1f 141104 1 2024-03-14T13:02:18.213 INFO:tasks.workunit.client.0.smithi111.stdout:3/488: write d0/d2d/d35/d4c/d82/f77 [256559,117760] 0 2024-03-14T13:02:18.214 INFO:tasks.workunit.client.0.smithi111.stdout:4/835: symlink d1d/d22/d30/d5e/d6f/d7e/d66/l127 0 2024-03-14T13:02:18.216 INFO:tasks.workunit.client.0.smithi111.stdout:4/836: fdatasync d1d/f20 0 2024-03-14T13:02:18.217 INFO:tasks.workunit.client.1.smithi137.stdout:6/469: dread d2/d54/d55/d38/f51 [0,4194304] 0 2024-03-14T13:02:18.217 INFO:tasks.workunit.client.1.smithi137.stdout:7/381: dread d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:18.219 INFO:tasks.workunit.client.0.smithi111.stdout:9/491: dread d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:18.220 INFO:tasks.workunit.client.0.smithi111.stdout:6/501: rename d0/d17/d20/c36 to d0/d17/d20/d94/daa/cc1 0 2024-03-14T13:02:18.221 INFO:tasks.workunit.client.0.smithi111.stdout:0/525: creat d6/d9/de/d3a/d46/d4a/fb4 x:0 0 0 2024-03-14T13:02:18.222 INFO:tasks.workunit.client.1.smithi137.stdout:6/470: write d2/d6/fc [12268543,60023] 0 2024-03-14T13:02:18.223 INFO:tasks.workunit.client.0.smithi111.stdout:6/502: dread - d0/d17/d21/d69/d3d/d53/d81/fa4 zero size 2024-03-14T13:02:18.224 INFO:tasks.workunit.client.0.smithi111.stdout:3/489: creat d0/d2d/d35/d50/d6c/fba x:0 0 0 2024-03-14T13:02:18.226 INFO:tasks.workunit.client.1.smithi137.stdout:6/471: write d2/d6/de/d45/d59/f5e [4540067,53539] 0 2024-03-14T13:02:18.226 INFO:tasks.workunit.client.0.smithi111.stdout:9/492: chown d5/d6/d2a/f75 101 1 2024-03-14T13:02:18.228 INFO:tasks.workunit.client.1.smithi137.stdout:1/513: dread d7/d9/d19/d1f/d2e/f34 [0,4194304] 0 2024-03-14T13:02:18.228 INFO:tasks.workunit.client.0.smithi111.stdout:2/509: rename dd/d11/d1a/d70 to dd/d17/d53/da4 0 2024-03-14T13:02:18.229 INFO:tasks.workunit.client.1.smithi137.stdout:7/382: dwrite d3/f18 [0,4194304] 0 2024-03-14T13:02:18.230 INFO:tasks.workunit.client.0.smithi111.stdout:0/526: creat d6/d45/d87/fb5 x:0 0 0 2024-03-14T13:02:18.230 INFO:tasks.workunit.client.1.smithi137.stdout:3/417: dread d4/de/f7d [0,4194304] 0 2024-03-14T13:02:18.232 INFO:tasks.workunit.client.0.smithi111.stdout:6/503: mkdir d0/d17/d20/d4a/d68/dc2 0 2024-03-14T13:02:18.233 INFO:tasks.workunit.client.1.smithi137.stdout:1/514: creat d7/d9/d19/d1f/d22/fa2 x:0 0 0 2024-03-14T13:02:18.233 INFO:tasks.workunit.client.1.smithi137.stdout:6/472: getdents d2/d6/de/d45/d59/daa 0 2024-03-14T13:02:18.234 INFO:tasks.workunit.client.0.smithi111.stdout:6/504: write d0/d17/d21/d42/da5/fad [1612606,28092] 0 2024-03-14T13:02:18.234 INFO:tasks.workunit.client.0.smithi111.stdout:9/493: creat d5/d6/d2a/d2d/fa8 x:0 0 0 2024-03-14T13:02:18.236 INFO:tasks.workunit.client.0.smithi111.stdout:9/494: write d5/d6/d8/d5b/d84/fa2 [1071351,27306] 0 2024-03-14T13:02:18.237 INFO:tasks.workunit.client.1.smithi137.stdout:1/515: rename d7/d9/f4f to d7/d4d/d6b/fa3 0 2024-03-14T13:02:18.237 INFO:tasks.workunit.client.0.smithi111.stdout:4/837: creat d1d/d22/d30/d7c/d59/f128 x:0 0 0 2024-03-14T13:02:18.238 INFO:tasks.workunit.client.0.smithi111.stdout:9/495: stat d5/d6/d8/d5b/d84 0 2024-03-14T13:02:18.239 INFO:tasks.workunit.client.1.smithi137.stdout:7/383: link d3/d6/d2e/f3e d3/d6/d14/d27/f7b 0 2024-03-14T13:02:18.241 INFO:tasks.workunit.client.0.smithi111.stdout:6/505: fdatasync d0/d17/d21/d69/f4d 0 2024-03-14T13:02:18.242 INFO:tasks.workunit.client.0.smithi111.stdout:6/506: chown d0/d17/d21/d42/lb6 351445 1 2024-03-14T13:02:18.243 INFO:tasks.workunit.client.1.smithi137.stdout:3/418: dread d4/de/d16/d1c/d5d/d85/d6b/f6d [0,4194304] 0 2024-03-14T13:02:18.243 INFO:tasks.workunit.client.0.smithi111.stdout:6/507: chown d0/d17/d24/f46 928894545 1 2024-03-14T13:02:18.243 INFO:tasks.workunit.client.0.smithi111.stdout:0/527: mknod d6/d9/de/d3a/dac/cb6 0 2024-03-14T13:02:18.243 INFO:tasks.workunit.client.0.smithi111.stdout:6/508: stat d0/l9 0 2024-03-14T13:02:18.244 INFO:tasks.workunit.client.0.smithi111.stdout:9/496: symlink d5/d42/da4/la9 0 2024-03-14T13:02:18.245 INFO:tasks.workunit.client.0.smithi111.stdout:0/528: write d6/d9/de/d3a/d46/f78 [1726086,21240] 0 2024-03-14T13:02:18.246 INFO:tasks.workunit.client.0.smithi111.stdout:9/497: fsync d5/d6/d8/d5b/d87/f8c 0 2024-03-14T13:02:18.247 INFO:tasks.workunit.client.0.smithi111.stdout:4/838: symlink d1d/d22/d40/d69/l129 0 2024-03-14T13:02:18.248 INFO:tasks.workunit.client.1.smithi137.stdout:1/516: link d7/d9/d19/d1f/d22/l75 d7/d9/d19/d1f/d26/d7d/la4 0 2024-03-14T13:02:18.248 INFO:tasks.workunit.client.0.smithi111.stdout:2/510: getdents dd/d17/d34/d4f 0 2024-03-14T13:02:18.249 INFO:tasks.workunit.client.1.smithi137.stdout:1/517: write d7/d9/d19/d1f/f23 [1217192,117436] 0 2024-03-14T13:02:18.250 INFO:tasks.workunit.client.0.smithi111.stdout:6/509: mkdir d0/d17/d20/d4a/d59/dc3 0 2024-03-14T13:02:18.251 INFO:tasks.workunit.client.0.smithi111.stdout:0/529: creat d6/d3d/d97/fb7 x:0 0 0 2024-03-14T13:02:18.252 INFO:tasks.workunit.client.1.smithi137.stdout:1/518: truncate d7/d9/d19/d1f/d2e/f9b 177198 0 2024-03-14T13:02:18.253 INFO:tasks.workunit.client.0.smithi111.stdout:2/511: mknod dd/d11/d1a/d88/ca5 0 2024-03-14T13:02:18.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:17 smithi137 ceph-mon[50336]: pgmap v23: 65 pgs: 65 active+clean; 1.4 GiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 21 MiB/s rd, 169 MiB/s wr, 261 op/s 2024-03-14T13:02:18.256 INFO:tasks.workunit.client.0.smithi111.stdout:0/530: creat d6/d9/d5b/fb8 x:0 0 0 2024-03-14T13:02:18.256 INFO:tasks.workunit.client.1.smithi137.stdout:1/519: creat d7/d9/d19/d40/d9c/fa5 x:0 0 0 2024-03-14T13:02:18.256 INFO:tasks.workunit.client.1.smithi137.stdout:6/473: truncate d2/d6/fc 3786269 0 2024-03-14T13:02:18.257 INFO:tasks.workunit.client.1.smithi137.stdout:3/419: write d4/de/d16/f31 [586473,106788] 0 2024-03-14T13:02:18.257 INFO:tasks.workunit.client.0.smithi111.stdout:9/498: creat d5/d6/faa x:0 0 0 2024-03-14T13:02:18.257 INFO:tasks.workunit.client.0.smithi111.stdout:0/531: stat d6/d9/d15/d49/d56/fab 0 2024-03-14T13:02:18.258 INFO:tasks.workunit.client.0.smithi111.stdout:6/510: dwrite d0/d17/d20/f26 [0,4194304] 0 2024-03-14T13:02:18.259 INFO:tasks.workunit.client.1.smithi137.stdout:1/520: mkdir d7/d9/d19/d1f/d2e/da6 0 2024-03-14T13:02:18.259 INFO:tasks.workunit.client.1.smithi137.stdout:7/384: dread d3/d6/d14/d22/f43 [0,4194304] 0 2024-03-14T13:02:18.260 INFO:tasks.workunit.client.0.smithi111.stdout:2/512: mkdir dd/d17/d53/da4/da6 0 2024-03-14T13:02:18.262 INFO:tasks.workunit.client.0.smithi111.stdout:9/499: mknod d5/d6/d8/d74/cab 0 2024-03-14T13:02:18.263 INFO:tasks.workunit.client.0.smithi111.stdout:2/513: truncate dd/d17/d53/f68 932411 0 2024-03-14T13:02:18.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/511: rmdir d0/d17/d20/d4a/d68 39 2024-03-14T13:02:18.263 INFO:tasks.workunit.client.0.smithi111.stdout:9/500: dread - d5/d6/d14/f91 zero size 2024-03-14T13:02:18.266 INFO:tasks.workunit.client.1.smithi137.stdout:6/474: dwrite d2/d4c/f69 [0,4194304] 0 2024-03-14T13:02:18.266 INFO:tasks.workunit.client.0.smithi111.stdout:2/514: mknod dd/d8d/ca7 0 2024-03-14T13:02:18.267 INFO:tasks.workunit.client.0.smithi111.stdout:6/512: read d0/d17/d21/d69/d3d/d53/d81/da1/fbc [110885,76687] 0 2024-03-14T13:02:18.267 INFO:tasks.workunit.client.0.smithi111.stdout:2/515: fdatasync dd/d11/d1a/f21 0 2024-03-14T13:02:18.268 INFO:tasks.workunit.client.1.smithi137.stdout:1/521: rmdir d7/d9/d19/d40/d7a 0 2024-03-14T13:02:18.268 INFO:tasks.workunit.client.0.smithi111.stdout:0/532: creat d6/d9/d15/d55/da0/fb9 x:0 0 0 2024-03-14T13:02:18.269 INFO:tasks.workunit.client.1.smithi137.stdout:3/420: dwrite d4/f80 [0,4194304] 0 2024-03-14T13:02:18.272 INFO:tasks.workunit.client.1.smithi137.stdout:1/522: fdatasync d7/d9/d19/d5f/f7f 0 2024-03-14T13:02:18.273 INFO:tasks.workunit.client.0.smithi111.stdout:9/501: creat d5/d42/d71/d96/fac x:0 0 0 2024-03-14T13:02:18.275 INFO:tasks.workunit.client.1.smithi137.stdout:1/523: dread - d7/f9f zero size 2024-03-14T13:02:18.276 INFO:tasks.workunit.client.1.smithi137.stdout:6/475: symlink d2/d6/lab 0 2024-03-14T13:02:18.276 INFO:tasks.workunit.client.0.smithi111.stdout:6/513: symlink d0/d17/d24/lc4 0 2024-03-14T13:02:18.278 INFO:tasks.workunit.client.0.smithi111.stdout:0/533: mkdir d6/d3d/dba 0 2024-03-14T13:02:18.278 INFO:tasks.workunit.client.1.smithi137.stdout:3/421: mkdir d4/de/d16/d1c/d8d 0 2024-03-14T13:02:18.278 INFO:tasks.workunit.client.1.smithi137.stdout:1/524: symlink d7/d29/la7 0 2024-03-14T13:02:18.279 INFO:tasks.workunit.client.1.smithi137.stdout:3/422: chown d4/df/l86 5 1 2024-03-14T13:02:18.282 INFO:tasks.workunit.client.0.smithi111.stdout:2/516: dwrite dd/d17/d34/f6f [0,4194304] 0 2024-03-14T13:02:18.283 INFO:tasks.workunit.client.0.smithi111.stdout:4/839: dread d1d/d22/d30/d5e/d6f/d7e/fc8 [0,4194304] 0 2024-03-14T13:02:18.285 INFO:tasks.workunit.client.1.smithi137.stdout:1/525: creat d7/d9/d19/d40/fa8 x:0 0 0 2024-03-14T13:02:18.286 INFO:tasks.workunit.client.1.smithi137.stdout:3/423: getdents d4/de/d16/d1c/d5d/d4b 0 2024-03-14T13:02:18.288 INFO:tasks.workunit.client.0.smithi111.stdout:9/502: dwrite d5/d6/d8/d5b/d7c/f90 [0,4194304] 0 2024-03-14T13:02:18.289 INFO:tasks.workunit.client.1.smithi137.stdout:6/476: dwrite d2/d6/f65 [0,4194304] 0 2024-03-14T13:02:18.291 INFO:tasks.workunit.client.0.smithi111.stdout:6/514: write d0/d17/f1f [4712233,76970] 0 2024-03-14T13:02:18.294 INFO:tasks.workunit.client.0.smithi111.stdout:4/840: rename d1d/d22/d30/d7c/df8/c11c to d1d/d48/d112/c12a 0 2024-03-14T13:02:18.294 INFO:tasks.workunit.client.0.smithi111.stdout:9/503: creat d5/d6/d2a/d5a/fad x:0 0 0 2024-03-14T13:02:18.295 INFO:tasks.workunit.client.0.smithi111.stdout:0/534: dwrite d6/d9/de/d32/f31 [4194304,4194304] 0 2024-03-14T13:02:18.295 INFO:tasks.workunit.client.1.smithi137.stdout:6/477: creat d2/d54/d55/d33/fac x:0 0 0 2024-03-14T13:02:18.298 INFO:tasks.workunit.client.1.smithi137.stdout:6/478: readlink d2/d54/l4b 0 2024-03-14T13:02:18.299 INFO:tasks.workunit.client.1.smithi137.stdout:3/424: dwrite d4/de/d16/f17 [4194304,4194304] 0 2024-03-14T13:02:18.299 INFO:tasks.workunit.client.1.smithi137.stdout:8/502: sync 2024-03-14T13:02:18.301 INFO:tasks.workunit.client.0.smithi111.stdout:0/535: dread - d6/d9/d5b/fb8 zero size 2024-03-14T13:02:18.301 INFO:tasks.workunit.client.1.smithi137.stdout:3/425: creat d4/d22/f8e x:0 0 0 2024-03-14T13:02:18.304 INFO:tasks.workunit.client.0.smithi111.stdout:4/841: symlink d1d/d22/d30/d5e/d6f/d7e/d66/l12b 0 2024-03-14T13:02:18.305 INFO:tasks.workunit.client.1.smithi137.stdout:3/426: chown d4/de/d16/d1c/d5d/d85/d6b/f6d 123 1 2024-03-14T13:02:18.305 INFO:tasks.workunit.client.1.smithi137.stdout:8/503: mkdir d1/dc/d71/db0 0 2024-03-14T13:02:18.305 INFO:tasks.workunit.client.0.smithi111.stdout:6/515: creat d0/d17/d20/fc5 x:0 0 0 2024-03-14T13:02:18.307 INFO:tasks.workunit.client.1.smithi137.stdout:3/427: getdents d4/df/d82 0 2024-03-14T13:02:18.307 INFO:tasks.workunit.client.1.smithi137.stdout:8/504: truncate d1/dc/d3c/d87/d34/d82/d96/fae 117834 0 2024-03-14T13:02:18.308 INFO:tasks.workunit.client.0.smithi111.stdout:0/536: creat d6/d45/fbb x:0 0 0 2024-03-14T13:02:18.309 INFO:tasks.workunit.client.0.smithi111.stdout:0/537: chown d6/d5a/l8e 33727334 1 2024-03-14T13:02:18.310 INFO:tasks.workunit.client.0.smithi111.stdout:4/842: chown d1d/d22/d40/d69/d86/db1/l126 19 1 2024-03-14T13:02:18.310 INFO:tasks.workunit.client.0.smithi111.stdout:6/516: mkdir d0/d17/d20/d5a/d5e/dc6 0 2024-03-14T13:02:18.311 INFO:tasks.workunit.client.0.smithi111.stdout:0/538: dread - d6/d45/d87/fb5 zero size 2024-03-14T13:02:18.314 INFO:tasks.workunit.client.0.smithi111.stdout:9/504: creat d5/d6/d8/d5b/fae x:0 0 0 2024-03-14T13:02:18.314 INFO:tasks.workunit.client.0.smithi111.stdout:0/539: write d6/d9/de/f18 [5012309,115292] 0 2024-03-14T13:02:18.315 INFO:tasks.workunit.client.0.smithi111.stdout:0/540: chown d6/d9/d5b/d9e 0 1 2024-03-14T13:02:18.316 INFO:tasks.workunit.client.1.smithi137.stdout:6/479: dread d2/d6/de/f4f [0,4194304] 0 2024-03-14T13:02:18.316 INFO:tasks.workunit.client.1.smithi137.stdout:8/505: dwrite d1/f6 [0,4194304] 0 2024-03-14T13:02:18.316 INFO:tasks.workunit.client.1.smithi137.stdout:6/480: fsync d2/d6/de/d45/d59/d39/d3c/f47 0 2024-03-14T13:02:18.317 INFO:tasks.workunit.client.0.smithi111.stdout:6/517: creat d0/d17/d20/d65/fc7 x:0 0 0 2024-03-14T13:02:18.318 INFO:tasks.workunit.client.0.smithi111.stdout:4/843: creat d1d/d22/d30/d5e/db8/ddf/d10f/f12c x:0 0 0 2024-03-14T13:02:18.319 INFO:tasks.workunit.client.0.smithi111.stdout:0/541: truncate d6/d9/de/d3a/f95 3003127 0 2024-03-14T13:02:18.320 INFO:tasks.workunit.client.0.smithi111.stdout:6/518: mkdir d0/d17/d21/d69/d3d/d53/d81/da1/dc8 0 2024-03-14T13:02:18.321 INFO:tasks.workunit.client.0.smithi111.stdout:4/844: chown d1d/d22/d23/ca6 605 1 2024-03-14T13:02:18.322 INFO:tasks.workunit.client.0.smithi111.stdout:0/542: creat d6/d9/de/d77/da4/fbc x:0 0 0 2024-03-14T13:02:18.324 INFO:tasks.workunit.client.1.smithi137.stdout:8/506: dwrite d1/dc/d3c/d87/d34/fa6 [0,4194304] 0 2024-03-14T13:02:18.325 INFO:tasks.workunit.client.0.smithi111.stdout:6/519: creat d0/d17/d21/d69/d3d/d53/d81/da1/dc8/fc9 x:0 0 0 2024-03-14T13:02:18.325 INFO:tasks.workunit.client.0.smithi111.stdout:0/543: read d6/d9/d15/d49/f62 [703740,26863] 0 2024-03-14T13:02:18.325 INFO:tasks.workunit.client.0.smithi111.stdout:4/845: mkdir d1d/d22/d30/d87/dd1/de1/d12d 0 2024-03-14T13:02:18.326 INFO:tasks.workunit.client.0.smithi111.stdout:6/520: chown d0/d17/c31 10861130 1 2024-03-14T13:02:18.326 INFO:tasks.workunit.client.1.smithi137.stdout:8/507: truncate d1/f44 5265779 0 2024-03-14T13:02:18.329 INFO:tasks.workunit.client.0.smithi111.stdout:0/544: mknod d6/d9/de/d3a/dac/cbd 0 2024-03-14T13:02:18.331 INFO:tasks.workunit.client.0.smithi111.stdout:4/846: creat d1d/d22/d30/d5e/db8/ddf/f12e x:0 0 0 2024-03-14T13:02:18.333 INFO:tasks.workunit.client.0.smithi111.stdout:4/847: write d1d/d22/d30/de5/fe4 [977688,60023] 0 2024-03-14T13:02:18.333 INFO:tasks.workunit.client.0.smithi111.stdout:0/545: unlink d6/d9/d15/c50 0 2024-03-14T13:02:18.339 INFO:tasks.workunit.client.0.smithi111.stdout:0/546: getdents d6/d9/de/d3a 0 2024-03-14T13:02:18.341 INFO:tasks.workunit.client.0.smithi111.stdout:0/547: rename d6/d9/d15/d55/d89/la3 to d6/d3d/d97/lbe 0 2024-03-14T13:02:18.341 INFO:tasks.workunit.client.1.smithi137.stdout:1/526: truncate d7/d9/f4b 1096430 0 2024-03-14T13:02:18.343 INFO:tasks.workunit.client.0.smithi111.stdout:0/548: unlink d6/d9/d15/d49/f62 0 2024-03-14T13:02:18.345 INFO:tasks.workunit.client.0.smithi111.stdout:0/549: symlink d6/d9/d15/da9/lbf 0 2024-03-14T13:02:18.346 INFO:tasks.workunit.client.1.smithi137.stdout:8/508: rmdir d1/dc/d71 39 2024-03-14T13:02:18.347 INFO:tasks.workunit.client.0.smithi111.stdout:0/550: truncate d6/d45/d87/fb5 963583 0 2024-03-14T13:02:18.347 INFO:tasks.workunit.client.1.smithi137.stdout:3/428: dread d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:18.348 INFO:tasks.workunit.client.0.smithi111.stdout:6/521: fsync d0/d17/d20/d65/fc7 0 2024-03-14T13:02:18.349 INFO:tasks.workunit.client.1.smithi137.stdout:1/527: write d7/d9/d19/d40/d42/f49 [567515,110802] 0 2024-03-14T13:02:18.350 INFO:tasks.workunit.client.0.smithi111.stdout:0/551: rename d6/d9/de/f53 to d6/d3d/fc0 0 2024-03-14T13:02:18.351 INFO:tasks.workunit.client.1.smithi137.stdout:6/481: dread d2/d6/de/f68 [0,4194304] 0 2024-03-14T13:02:18.352 INFO:tasks.workunit.client.1.smithi137.stdout:3/429: creat d4/de/d16/d1c/f8f x:0 0 0 2024-03-14T13:02:18.353 INFO:tasks.workunit.client.1.smithi137.stdout:6/482: truncate d2/d54/d55/f8d 15852 0 2024-03-14T13:02:18.354 INFO:tasks.workunit.client.1.smithi137.stdout:3/430: write d4/d19/f54 [4747848,79521] 0 2024-03-14T13:02:18.354 INFO:tasks.workunit.client.0.smithi111.stdout:6/522: mkdir d0/d17/d21/d69/d3d/dca 0 2024-03-14T13:02:18.355 INFO:tasks.workunit.client.0.smithi111.stdout:6/523: chown d0/d17/d21/d69/d3d/c7b 92687243 1 2024-03-14T13:02:18.356 INFO:tasks.workunit.client.1.smithi137.stdout:6/483: write d2/d54/d55/d6d/d76/d57/f70 [1404176,60955] 0 2024-03-14T13:02:18.358 INFO:tasks.workunit.client.1.smithi137.stdout:8/509: link d1/dc/d3c/d87/d34/d97/l92 d1/d5e/daf/lb1 0 2024-03-14T13:02:18.359 INFO:tasks.workunit.client.1.smithi137.stdout:6/484: write d2/d54/d55/fa3 [146044,59196] 0 2024-03-14T13:02:18.362 INFO:tasks.workunit.client.1.smithi137.stdout:3/431: dread d4/f80 [0,4194304] 0 2024-03-14T13:02:18.363 INFO:tasks.workunit.client.0.smithi111.stdout:4/848: truncate d1d/d22/d30/de5/fe4 814601 0 2024-03-14T13:02:18.363 INFO:tasks.workunit.client.1.smithi137.stdout:3/432: fdatasync d4/de/d16/f31 0 2024-03-14T13:02:18.366 INFO:tasks.workunit.client.1.smithi137.stdout:6/485: rename d2/d6/de/d45/d59/d39/l49 to d2/d54/d55/d6d/d76/d57/lad 0 2024-03-14T13:02:18.367 INFO:tasks.workunit.client.1.smithi137.stdout:1/528: truncate f5 5114140 0 2024-03-14T13:02:18.368 INFO:tasks.workunit.client.0.smithi111.stdout:0/552: dwrite d6/d21/f73 [0,4194304] 0 2024-03-14T13:02:18.369 INFO:tasks.workunit.client.1.smithi137.stdout:3/433: creat d4/d19/f90 x:0 0 0 2024-03-14T13:02:18.369 INFO:tasks.workunit.client.1.smithi137.stdout:1/529: chown d7/d4d/d6b/f9e 105728 1 2024-03-14T13:02:18.371 INFO:tasks.workunit.client.1.smithi137.stdout:8/510: truncate d1/dc/f5f 3466550 0 2024-03-14T13:02:18.372 INFO:tasks.workunit.client.1.smithi137.stdout:3/434: write d4/de/d13/f36 [10561173,57472] 0 2024-03-14T13:02:18.372 INFO:tasks.workunit.client.0.smithi111.stdout:0/553: mkdir d6/d3d/dc1 0 2024-03-14T13:02:18.374 INFO:tasks.workunit.client.0.smithi111.stdout:4/849: rename d1d/d22/d30/cd5 to d1d/d22/d30/d7c/df8/c12f 0 2024-03-14T13:02:18.375 INFO:tasks.workunit.client.0.smithi111.stdout:4/850: write d1d/f26 [2909222,4125] 0 2024-03-14T13:02:18.375 INFO:tasks.workunit.client.0.smithi111.stdout:0/554: creat d6/d9/d15/d49/fc2 x:0 0 0 2024-03-14T13:02:18.377 INFO:tasks.workunit.client.1.smithi137.stdout:8/511: creat d1/dc/d3c/d87/d34/fb2 x:0 0 0 2024-03-14T13:02:18.379 INFO:tasks.workunit.client.1.smithi137.stdout:6/486: dwrite d2/d54/d55/d6d/d76/d57/d58/f98 [0,4194304] 0 2024-03-14T13:02:18.379 INFO:tasks.workunit.client.0.smithi111.stdout:4/851: mknod d1d/d22/d30/d7c/d59/dc4/d123/c130 0 2024-03-14T13:02:18.384 INFO:tasks.workunit.client.1.smithi137.stdout:3/435: dwrite d4/de/d16/f17 [4194304,4194304] 0 2024-03-14T13:02:18.384 INFO:tasks.workunit.client.1.smithi137.stdout:8/512: creat d1/dc/d3c/d87/d34/d97/fb3 x:0 0 0 2024-03-14T13:02:18.385 INFO:tasks.workunit.client.0.smithi111.stdout:6/524: dwrite d0/d17/d21/d69/f61 [0,4194304] 0 2024-03-14T13:02:18.385 INFO:tasks.workunit.client.0.smithi111.stdout:4/852: write d1d/d22/d30/d7c/d59/dc4/d123/d115/fca [88165,29795] 0 2024-03-14T13:02:18.386 INFO:tasks.workunit.client.0.smithi111.stdout:6/525: stat d0/d17/d24/cb7 0 2024-03-14T13:02:18.386 INFO:tasks.workunit.client.1.smithi137.stdout:8/513: dread - d1/dc/d4e/d6d/f74 zero size 2024-03-14T13:02:18.390 INFO:tasks.workunit.client.1.smithi137.stdout:3/436: link d4/de/d16/d1c/d5d/d4b/c51 d4/d19/d5c/c91 0 2024-03-14T13:02:18.390 INFO:tasks.workunit.client.0.smithi111.stdout:6/526: read d0/ff [1413404,12186] 0 2024-03-14T13:02:18.392 INFO:tasks.workunit.client.1.smithi137.stdout:3/437: unlink d4/df/f30 0 2024-03-14T13:02:18.394 INFO:tasks.workunit.client.0.smithi111.stdout:4/853: mkdir d1d/d22/d30/d5e/d131 0 2024-03-14T13:02:18.395 INFO:tasks.workunit.client.1.smithi137.stdout:3/438: mknod d4/de/d13/c92 0 2024-03-14T13:02:18.395 INFO:tasks.workunit.client.0.smithi111.stdout:4/854: chown d1d/d22/d40/d69 35 1 2024-03-14T13:02:18.400 INFO:tasks.workunit.client.0.smithi111.stdout:4/855: rename d1d/d22/d30/d5e/fb3 to d1d/d22/d30/d7c/d59/dc4/f132 0 2024-03-14T13:02:18.402 INFO:tasks.workunit.client.0.smithi111.stdout:4/856: fdatasync d1d/d22/d40/d69/d86/f9f 0 2024-03-14T13:02:18.405 INFO:tasks.workunit.client.0.smithi111.stdout:4/857: rmdir d1d/d22/d30/d7c/d59/dc4/d123/de6 39 2024-03-14T13:02:18.405 INFO:tasks.workunit.client.0.smithi111.stdout:4/858: write f11 [658178,30407] 0 2024-03-14T13:02:18.407 INFO:tasks.workunit.client.1.smithi137.stdout:8/514: truncate d1/d5e/f63 1173229 0 2024-03-14T13:02:18.409 INFO:tasks.workunit.client.1.smithi137.stdout:8/515: dread - d1/dc/d10/f32 zero size 2024-03-14T13:02:18.412 INFO:tasks.workunit.client.0.smithi111.stdout:0/555: dread d6/d9/d5b/d9e/fa8 [0,4194304] 0 2024-03-14T13:02:18.415 INFO:tasks.workunit.client.1.smithi137.stdout:8/516: dwrite d1/f44 [0,4194304] 0 2024-03-14T13:02:18.415 INFO:tasks.workunit.client.0.smithi111.stdout:0/556: dread d6/d9/de/f41 [0,4194304] 0 2024-03-14T13:02:18.417 INFO:tasks.workunit.client.1.smithi137.stdout:8/517: chown d1/dc/d3c/d87/d34/d82/f8d 21170716 1 2024-03-14T13:02:18.417 INFO:tasks.workunit.client.1.smithi137.stdout:6/487: dread d2/f25 [0,4194304] 0 2024-03-14T13:02:18.419 INFO:tasks.workunit.client.0.smithi111.stdout:0/557: creat d6/d9/de/d3a/fc3 x:0 0 0 2024-03-14T13:02:18.424 INFO:tasks.workunit.client.0.smithi111.stdout:0/558: mknod d6/d9/d15/d55/cc4 0 2024-03-14T13:02:18.424 INFO:tasks.workunit.client.0.smithi111.stdout:6/527: dwrite d0/d50/f5f [0,4194304] 0 2024-03-14T13:02:18.425 INFO:tasks.workunit.client.1.smithi137.stdout:6/488: dwrite d2/d54/d55/d38/f51 [0,4194304] 0 2024-03-14T13:02:18.428 INFO:tasks.workunit.client.1.smithi137.stdout:6/489: chown d2/d6/d1d 2 1 2024-03-14T13:02:18.429 INFO:tasks.workunit.client.0.smithi111.stdout:6/528: creat d0/d17/d20/d5a/fcb x:0 0 0 2024-03-14T13:02:18.429 INFO:tasks.workunit.client.1.smithi137.stdout:6/490: write d2/d6/d86/fa9 [3296117,58089] 0 2024-03-14T13:02:18.429 INFO:tasks.workunit.client.0.smithi111.stdout:0/559: creat d6/d9/de/d77/da4/fc5 x:0 0 0 2024-03-14T13:02:18.431 INFO:tasks.workunit.client.1.smithi137.stdout:8/518: dread d1/dc/d3c/d87/d34/d97/d13/f4f [0,4194304] 0 2024-03-14T13:02:18.432 INFO:tasks.workunit.client.0.smithi111.stdout:6/529: dread - d0/d17/d21/d69/d3d/fb5 zero size 2024-03-14T13:02:18.432 INFO:tasks.workunit.client.0.smithi111.stdout:0/560: symlink d6/d9/d15/d55/lc6 0 2024-03-14T13:02:18.434 INFO:tasks.workunit.client.0.smithi111.stdout:6/530: truncate d0/d50/f66 524093 0 2024-03-14T13:02:18.435 INFO:tasks.workunit.client.1.smithi137.stdout:8/519: write d1/dc/d3c/d87/d34/d97/d13/f45 [1825458,78749] 0 2024-03-14T13:02:18.435 INFO:tasks.workunit.client.1.smithi137.stdout:6/491: link d2/d6/de/d45/d59/d39/d3c/l91 d2/d54/d55/d6d/lae 0 2024-03-14T13:02:18.435 INFO:tasks.workunit.client.0.smithi111.stdout:0/561: dread d6/d45/d87/fb5 [0,4194304] 0 2024-03-14T13:02:18.437 INFO:tasks.workunit.client.0.smithi111.stdout:0/562: mknod d6/cc7 0 2024-03-14T13:02:18.440 INFO:tasks.workunit.client.0.smithi111.stdout:6/531: read d0/f6e [2006596,74906] 0 2024-03-14T13:02:18.442 INFO:tasks.workunit.client.1.smithi137.stdout:6/492: dwrite d2/d54/d55/d33/fa0 [0,4194304] 0 2024-03-14T13:02:18.443 INFO:tasks.workunit.client.0.smithi111.stdout:6/532: link d0/ce d0/d17/d20/d65/d93/ccc 0 2024-03-14T13:02:18.444 INFO:tasks.workunit.client.1.smithi137.stdout:6/493: write d2/d4c/f69 [4437989,920] 0 2024-03-14T13:02:18.444 INFO:tasks.workunit.client.0.smithi111.stdout:6/533: chown d0/d17/d21/d69/f4d 1804 1 2024-03-14T13:02:18.445 INFO:tasks.workunit.client.1.smithi137.stdout:6/494: symlink d2/d6/de/laf 0 2024-03-14T13:02:18.446 INFO:tasks.workunit.client.0.smithi111.stdout:6/534: write d0/f1a [5454983,95374] 0 2024-03-14T13:02:18.448 INFO:tasks.workunit.client.1.smithi137.stdout:8/520: dread d1/dc/d10/f27 [0,4194304] 0 2024-03-14T13:02:18.450 INFO:tasks.workunit.client.0.smithi111.stdout:6/535: dread - d0/d17/d20/d4a/d59/d80/f9c zero size 2024-03-14T13:02:18.454 INFO:tasks.workunit.client.0.smithi111.stdout:0/563: truncate d6/d9/d5b/d9e/fa8 3571348 0 2024-03-14T13:02:18.455 INFO:tasks.workunit.client.0.smithi111.stdout:0/564: chown d6/d9/de/d3a/d46/d4a/f8f 398 1 2024-03-14T13:02:18.457 INFO:tasks.workunit.client.0.smithi111.stdout:0/565: read d6/d9/d15/f16 [974002,50350] 0 2024-03-14T13:02:18.459 INFO:tasks.workunit.client.0.smithi111.stdout:0/566: symlink d6/d9/de/d32/lc8 0 2024-03-14T13:02:18.461 INFO:tasks.workunit.client.0.smithi111.stdout:0/567: creat d6/d9/d15/d49/d56/fc9 x:0 0 0 2024-03-14T13:02:18.466 INFO:tasks.workunit.client.0.smithi111.stdout:0/568: dwrite d6/d9/de/f6a [4194304,4194304] 0 2024-03-14T13:02:18.469 INFO:tasks.workunit.client.0.smithi111.stdout:0/569: mkdir d6/d9/de/d77/da4/dca 0 2024-03-14T13:02:18.471 INFO:tasks.workunit.client.0.smithi111.stdout:0/570: symlink d6/d3d/lcb 0 2024-03-14T13:02:18.476 INFO:tasks.workunit.client.0.smithi111.stdout:0/571: dread d6/d9/de/d32/f31 [4194304,4194304] 0 2024-03-14T13:02:18.480 INFO:tasks.workunit.client.0.smithi111.stdout:0/572: getdents d6/d3d/d97 0 2024-03-14T13:02:18.482 INFO:tasks.workunit.client.0.smithi111.stdout:0/573: symlink d6/d9/d5b/d9e/lcc 0 2024-03-14T13:02:18.488 INFO:tasks.workunit.client.0.smithi111.stdout:0/574: dwrite d6/d3d/fa7 [0,4194304] 0 2024-03-14T13:02:18.489 INFO:tasks.workunit.client.0.smithi111.stdout:0/575: fdatasync d6/fd 0 2024-03-14T13:02:18.490 INFO:tasks.workunit.client.0.smithi111.stdout:0/576: dread - d6/d9/de/d3a/d46/d4a/f98 zero size 2024-03-14T13:02:18.491 INFO:tasks.workunit.client.0.smithi111.stdout:0/577: fsync d6/d9/de/d32/f5e 0 2024-03-14T13:02:18.492 INFO:tasks.workunit.client.0.smithi111.stdout:0/578: dread - d6/d3d/d97/fb7 zero size 2024-03-14T13:02:18.493 INFO:tasks.workunit.client.0.smithi111.stdout:0/579: mknod d6/d9/d15/d55/da0/ccd 0 2024-03-14T13:02:18.495 INFO:tasks.workunit.client.1.smithi137.stdout:0/413: sync 2024-03-14T13:02:18.496 INFO:tasks.workunit.client.1.smithi137.stdout:9/553: sync 2024-03-14T13:02:18.498 INFO:tasks.workunit.client.0.smithi111.stdout:0/580: dwrite d6/f1f [0,4194304] 0 2024-03-14T13:02:18.506 INFO:tasks.workunit.client.0.smithi111.stdout:0/581: dread d6/d9/de/d3a/d46/f66 [0,4194304] 0 2024-03-14T13:02:18.508 INFO:tasks.workunit.client.0.smithi111.stdout:0/582: read f5 [106230,24798] 0 2024-03-14T13:02:18.509 INFO:tasks.workunit.client.0.smithi111.stdout:0/583: read - d6/d9/de/d3a/f9c zero size 2024-03-14T13:02:18.515 INFO:tasks.workunit.client.0.smithi111.stdout:0/584: dwrite d6/d9/de/d3a/fc3 [0,4194304] 0 2024-03-14T13:02:18.517 INFO:tasks.workunit.client.0.smithi111.stdout:0/585: creat d6/d9/de/d3a/dac/fce x:0 0 0 2024-03-14T13:02:18.518 INFO:tasks.workunit.client.0.smithi111.stdout:0/586: chown d6/d9/de/d3a/d46/f66 2369921 1 2024-03-14T13:02:18.519 INFO:tasks.workunit.client.0.smithi111.stdout:0/587: mknod d6/d9/de/d32/d64/ccf 0 2024-03-14T13:02:18.519 INFO:tasks.workunit.client.0.smithi111.stdout:0/588: write d6/f90 [864303,60444] 0 2024-03-14T13:02:18.531 INFO:tasks.workunit.client.1.smithi137.stdout:6/495: dread d2/d6/f65 [0,4194304] 0 2024-03-14T13:02:18.531 INFO:tasks.workunit.client.1.smithi137.stdout:8/521: creat d1/dc/d3c/d87/d34/fb4 x:0 0 0 2024-03-14T13:02:18.532 INFO:tasks.workunit.client.1.smithi137.stdout:0/414: read d1/d5/d1b/d42/f58 [55164,68732] 0 2024-03-14T13:02:18.535 INFO:tasks.workunit.client.1.smithi137.stdout:8/522: mkdir d1/dc/d4e/d6d/db5 0 2024-03-14T13:02:18.536 INFO:tasks.workunit.client.1.smithi137.stdout:6/496: rename d2/d6/de/d45/d59/d39/d3c/f47 to d2/d54/d55/d6d/fb0 0 2024-03-14T13:02:18.537 INFO:tasks.workunit.client.1.smithi137.stdout:6/497: dread - d2/d6/d1d/d88/fa2 zero size 2024-03-14T13:02:18.538 INFO:tasks.workunit.client.1.smithi137.stdout:0/415: rename d1/d5/d1b/d1e/d25/d26/f5d to d1/d1f/d82/f8b 0 2024-03-14T13:02:18.540 INFO:tasks.workunit.client.1.smithi137.stdout:8/523: creat d1/dc/d71/fb6 x:0 0 0 2024-03-14T13:02:18.542 INFO:tasks.workunit.client.1.smithi137.stdout:0/416: creat d1/d44/d79/d84/f8c x:0 0 0 2024-03-14T13:02:18.545 INFO:tasks.workunit.client.1.smithi137.stdout:0/417: rename d1/l5c to d1/d5/d1b/d42/l8d 0 2024-03-14T13:02:18.550 INFO:tasks.workunit.client.0.smithi111.stdout:1/529: sync 2024-03-14T13:02:18.555 INFO:tasks.workunit.client.0.smithi111.stdout:1/530: rename d20/d21/f41 to d20/d25/d87/fb8 0 2024-03-14T13:02:18.556 INFO:tasks.workunit.client.0.smithi111.stdout:0/589: dread d6/d9/de/f1d [0,4194304] 0 2024-03-14T13:02:18.558 INFO:tasks.workunit.client.0.smithi111.stdout:0/590: chown d6/d9/d5b/d9e/lcc 1611272336 1 2024-03-14T13:02:18.559 INFO:tasks.workunit.client.0.smithi111.stdout:1/531: creat d20/d21/d28/da0/db6/fb9 x:0 0 0 2024-03-14T13:02:18.559 INFO:tasks.workunit.client.1.smithi137.stdout:0/418: dread d1/d5/d1b/d1e/d25/d55/f67 [0,4194304] 0 2024-03-14T13:02:18.560 INFO:tasks.workunit.client.0.smithi111.stdout:0/591: mkdir d6/d9/de/d77/da4/dca/dd0 0 2024-03-14T13:02:18.561 INFO:tasks.workunit.client.0.smithi111.stdout:1/532: chown d20/d25/d75/c94 242 1 2024-03-14T13:02:18.561 INFO:tasks.workunit.client.1.smithi137.stdout:8/524: dread d1/dc/d3c/d87/d34/d97/d13/f9e [0,4194304] 0 2024-03-14T13:02:18.563 INFO:tasks.workunit.client.1.smithi137.stdout:8/525: fdatasync d1/dc/d4e/d6d/f74 0 2024-03-14T13:02:18.565 INFO:tasks.workunit.client.1.smithi137.stdout:9/554: dread d6/df/d19/f45 [0,4194304] 0 2024-03-14T13:02:18.566 INFO:tasks.workunit.client.1.smithi137.stdout:0/419: dwrite d1/d1f/f24 [0,4194304] 0 2024-03-14T13:02:18.568 INFO:tasks.workunit.client.1.smithi137.stdout:9/555: mknod d6/d26/cb5 0 2024-03-14T13:02:18.568 INFO:tasks.workunit.client.0.smithi111.stdout:0/592: rename d6/d9/de/d3a/dac to d6/d9/de/d3a/dd1 0 2024-03-14T13:02:18.568 INFO:tasks.workunit.client.0.smithi111.stdout:0/593: stat d6/d9/de/f3c 0 2024-03-14T13:02:18.569 INFO:tasks.workunit.client.1.smithi137.stdout:0/420: write d1/d5/d1b/d42/f4b [1030083,79618] 0 2024-03-14T13:02:18.569 INFO:tasks.workunit.client.0.smithi111.stdout:0/594: chown d6/d9/de/d32/d64/l6c 80185 1 2024-03-14T13:02:18.570 INFO:tasks.workunit.client.1.smithi137.stdout:9/556: dread - d6/fa3 zero size 2024-03-14T13:02:18.575 INFO:tasks.workunit.client.1.smithi137.stdout:0/421: creat d1/d5/d1b/d42/f8e x:0 0 0 2024-03-14T13:02:18.575 INFO:tasks.workunit.client.1.smithi137.stdout:8/526: dwrite d1/dc/d3c/d87/d34/d97/d13/f22 [0,4194304] 0 2024-03-14T13:02:18.578 INFO:tasks.workunit.client.1.smithi137.stdout:9/557: unlink d6/df/d19/d1f/f2a 0 2024-03-14T13:02:18.579 INFO:tasks.workunit.client.1.smithi137.stdout:9/558: read - d6/d56/fb3 zero size 2024-03-14T13:02:18.580 INFO:tasks.workunit.client.1.smithi137.stdout:6/498: write d2/d54/d55/d6d/d76/d57/d58/f98 [4541763,7098] 0 2024-03-14T13:02:18.583 INFO:tasks.workunit.client.1.smithi137.stdout:8/527: mknod d1/dc/d3c/d87/cb7 0 2024-03-14T13:02:18.586 INFO:tasks.workunit.client.0.smithi111.stdout:1/533: dread d20/d25/d31/f3b [0,4194304] 0 2024-03-14T13:02:18.586 INFO:tasks.workunit.client.1.smithi137.stdout:6/499: creat d2/d6/de/d45/d97/d9f/fb1 x:0 0 0 2024-03-14T13:02:18.587 INFO:tasks.workunit.client.0.smithi111.stdout:0/595: dread d6/d9/f58 [0,4194304] 0 2024-03-14T13:02:18.587 INFO:tasks.workunit.client.1.smithi137.stdout:6/500: write f0 [8815717,11934] 0 2024-03-14T13:02:18.588 INFO:tasks.workunit.client.0.smithi111.stdout:1/534: truncate d20/d25/fa1 951547 0 2024-03-14T13:02:18.588 INFO:tasks.workunit.client.0.smithi111.stdout:0/596: write d6/d9/de/d3a/dd1/fce [751189,103136] 0 2024-03-14T13:02:18.591 INFO:tasks.workunit.client.0.smithi111.stdout:0/597: dread d6/d45/d87/fb5 [0,4194304] 0 2024-03-14T13:02:18.593 INFO:tasks.workunit.client.1.smithi137.stdout:6/501: dread d2/d6/f65 [0,4194304] 0 2024-03-14T13:02:18.593 INFO:tasks.workunit.client.1.smithi137.stdout:8/528: link d1/dc/l5b d1/d5e/lb8 0 2024-03-14T13:02:18.593 INFO:tasks.workunit.client.0.smithi111.stdout:0/598: symlink d6/d45/d87/ld2 0 2024-03-14T13:02:18.594 INFO:tasks.workunit.client.1.smithi137.stdout:0/422: write d1/d5/d1b/d1e/d25/d55/f60 [191291,24975] 0 2024-03-14T13:02:18.595 INFO:tasks.workunit.client.1.smithi137.stdout:9/559: getdents d6/df/d19/d33 0 2024-03-14T13:02:18.595 INFO:tasks.workunit.client.0.smithi111.stdout:0/599: symlink d6/d9/d15/d55/d89/ld3 0 2024-03-14T13:02:18.597 INFO:tasks.workunit.client.0.smithi111.stdout:0/600: mknod d6/d9/de/d77/da4/cd4 0 2024-03-14T13:02:18.598 INFO:tasks.workunit.client.1.smithi137.stdout:8/529: rename d1/dc/l57 to d1/dc/d3c/lb9 0 2024-03-14T13:02:18.598 INFO:tasks.workunit.client.0.smithi111.stdout:0/601: rmdir d6/d9/de/d77/da4/dca 39 2024-03-14T13:02:18.599 INFO:tasks.workunit.client.1.smithi137.stdout:0/423: rmdir d1/d44/d72 39 2024-03-14T13:02:18.599 INFO:tasks.workunit.client.1.smithi137.stdout:8/530: write d1/dc/d10/f32 [1027298,121848] 0 2024-03-14T13:02:18.600 INFO:tasks.workunit.client.0.smithi111.stdout:0/602: write d6/d3d/f5f [751255,80771] 0 2024-03-14T13:02:18.601 INFO:tasks.workunit.client.1.smithi137.stdout:9/560: creat d6/df/d19/d3a/d40/fb6 x:0 0 0 2024-03-14T13:02:18.603 INFO:tasks.workunit.client.1.smithi137.stdout:0/424: unlink d1/d5/d1b/l20 0 2024-03-14T13:02:18.603 INFO:tasks.workunit.client.1.smithi137.stdout:6/502: write d2/f2d [547728,92782] 0 2024-03-14T13:02:18.604 INFO:tasks.workunit.client.1.smithi137.stdout:0/425: chown d1/f3e 8983 1 2024-03-14T13:02:18.604 INFO:tasks.workunit.client.1.smithi137.stdout:8/531: creat d1/dc/d71/fba x:0 0 0 2024-03-14T13:02:18.604 INFO:tasks.workunit.client.0.smithi111.stdout:0/603: rmdir d6/d9/de 39 2024-03-14T13:02:18.605 INFO:tasks.workunit.client.0.smithi111.stdout:0/604: chown d6/d45/d87/fb5 43845 1 2024-03-14T13:02:18.605 INFO:tasks.workunit.client.1.smithi137.stdout:9/561: write d6/df/f20 [4101643,88157] 0 2024-03-14T13:02:18.606 INFO:tasks.workunit.client.1.smithi137.stdout:8/532: dread - d1/dc/d3c/d87/d34/d97/fb3 zero size 2024-03-14T13:02:18.607 INFO:tasks.workunit.client.0.smithi111.stdout:0/605: truncate d6/d9/d15/d49/d56/fab 651149 0 2024-03-14T13:02:18.607 INFO:tasks.workunit.client.1.smithi137.stdout:9/562: chown d6/df/d19/f45 29 1 2024-03-14T13:02:18.608 INFO:tasks.workunit.client.0.smithi111.stdout:1/535: dread d20/d25/d87/f8f [0,4194304] 0 2024-03-14T13:02:18.609 INFO:tasks.workunit.client.1.smithi137.stdout:0/426: creat d1/d5/d1b/d1e/d25/d55/d73/f8f x:0 0 0 2024-03-14T13:02:18.609 INFO:tasks.workunit.client.0.smithi111.stdout:0/606: stat d6/d9/de/d3a/d46/d4a/f5c 0 2024-03-14T13:02:18.609 INFO:tasks.workunit.client.1.smithi137.stdout:9/563: chown d6/df/d19/d3a/f67 4019 1 2024-03-14T13:02:18.610 INFO:tasks.workunit.client.0.smithi111.stdout:1/536: symlink d20/d21/lba 0 2024-03-14T13:02:18.611 INFO:tasks.workunit.client.0.smithi111.stdout:0/607: write d6/d9/de/f1d [726098,52640] 0 2024-03-14T13:02:18.612 INFO:tasks.workunit.client.1.smithi137.stdout:6/503: rename d2/d6/de/d45/d59/d39/d3c/l61 to d2/d6/lb2 0 2024-03-14T13:02:18.612 INFO:tasks.workunit.client.1.smithi137.stdout:9/564: stat d6/df/d19/l6b 0 2024-03-14T13:02:18.613 INFO:tasks.workunit.client.0.smithi111.stdout:1/537: mknod d20/d21/cbb 0 2024-03-14T13:02:18.613 INFO:tasks.workunit.client.0.smithi111.stdout:0/608: read d6/f1f [5592815,90457] 0 2024-03-14T13:02:18.613 INFO:tasks.workunit.client.1.smithi137.stdout:0/427: mknod d1/d5/d1b/d1e/d53/c90 0 2024-03-14T13:02:18.614 INFO:tasks.workunit.client.0.smithi111.stdout:1/538: fsync d20/d25/d87/f8f 0 2024-03-14T13:02:18.614 INFO:tasks.workunit.client.1.smithi137.stdout:6/504: fdatasync d2/d54/d55/d6d/d76/d57/f81 0 2024-03-14T13:02:18.615 INFO:tasks.workunit.client.1.smithi137.stdout:0/428: dread d1/d5/d1b/d1e/f3d [0,4194304] 0 2024-03-14T13:02:18.616 INFO:tasks.workunit.client.0.smithi111.stdout:0/609: read d6/d9/de/f23 [3788961,53088] 0 2024-03-14T13:02:18.616 INFO:tasks.workunit.client.1.smithi137.stdout:6/505: truncate d2/d6/de/fa6 1193425 0 2024-03-14T13:02:18.617 INFO:tasks.workunit.client.1.smithi137.stdout:6/506: read - d2/d6/de/f8c zero size 2024-03-14T13:02:18.618 INFO:tasks.workunit.client.1.smithi137.stdout:8/533: rename d1/dc/d71/f72 to d1/dc/d3c/d87/d34/d97/fbb 0 2024-03-14T13:02:18.619 INFO:tasks.workunit.client.0.smithi111.stdout:0/610: rename d6/f90 to d6/d9/d5b/d9e/fd5 0 2024-03-14T13:02:18.619 INFO:tasks.workunit.client.1.smithi137.stdout:6/507: chown d2/d54/d55/d38/f85 499047338 1 2024-03-14T13:02:18.620 INFO:tasks.workunit.client.0.smithi111.stdout:0/611: truncate d6/d9/de/d3a/d46/d4a/fb4 418671 0 2024-03-14T13:02:18.620 INFO:tasks.workunit.client.1.smithi137.stdout:0/429: fsync d1/d1f/d82/f8b 0 2024-03-14T13:02:18.622 INFO:tasks.workunit.client.0.smithi111.stdout:0/612: mkdir d6/d3d/dc1/dd6 0 2024-03-14T13:02:18.624 INFO:tasks.workunit.client.0.smithi111.stdout:0/613: creat d6/d9/d15/d55/da0/fd7 x:0 0 0 2024-03-14T13:02:18.624 INFO:tasks.workunit.client.1.smithi137.stdout:2/539: sync 2024-03-14T13:02:18.624 INFO:tasks.workunit.client.1.smithi137.stdout:5/395: sync 2024-03-14T13:02:18.624 INFO:tasks.workunit.client.1.smithi137.stdout:4/466: sync 2024-03-14T13:02:18.625 INFO:tasks.workunit.client.0.smithi111.stdout:0/614: read d6/d3d/fa7 [4190629,24695] 0 2024-03-14T13:02:18.625 INFO:tasks.workunit.client.1.smithi137.stdout:6/508: fdatasync d2/d6/de/f4f 0 2024-03-14T13:02:18.625 INFO:tasks.workunit.client.1.smithi137.stdout:0/430: dread d1/d5/d1b/d42/f58 [0,4194304] 0 2024-03-14T13:02:18.626 INFO:tasks.workunit.client.0.smithi111.stdout:0/615: fdatasync d6/d9/d15/d55/da0/fd7 0 2024-03-14T13:02:18.627 INFO:tasks.workunit.client.1.smithi137.stdout:9/565: getdents d6/df/d19 0 2024-03-14T13:02:18.628 INFO:tasks.workunit.client.1.smithi137.stdout:8/534: symlink d1/dc/da9/lbc 0 2024-03-14T13:02:18.629 INFO:tasks.workunit.client.0.smithi111.stdout:0/616: creat d6/d21/d74/fd8 x:0 0 0 2024-03-14T13:02:18.629 INFO:tasks.workunit.client.1.smithi137.stdout:9/566: chown d6/df/f13 7179374 1 2024-03-14T13:02:18.630 INFO:tasks.workunit.client.0.smithi111.stdout:1/539: dread d20/d25/f46 [0,4194304] 0 2024-03-14T13:02:18.631 INFO:tasks.workunit.client.1.smithi137.stdout:8/535: read - d1/dc/d3c/d87/d34/fb4 zero size 2024-03-14T13:02:18.633 INFO:tasks.workunit.client.1.smithi137.stdout:4/467: fdatasync d4/d9/d3a/d70/d80/f8c 0 2024-03-14T13:02:18.634 INFO:tasks.workunit.client.0.smithi111.stdout:1/540: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:18.635 INFO:tasks.workunit.client.0.smithi111.stdout:0/617: write d6/d9/d15/f2e [881929,123179] 0 2024-03-14T13:02:18.636 INFO:tasks.workunit.client.0.smithi111.stdout:1/541: symlink d20/d40/lbc 0 2024-03-14T13:02:18.637 INFO:tasks.workunit.client.1.smithi137.stdout:5/396: rename d7/d16/d18/d25/d28/c24 to d7/d1d/d74/d83/c8d 0 2024-03-14T13:02:18.637 INFO:tasks.workunit.client.0.smithi111.stdout:1/542: write d20/d25/d75/f5e [696928,8719] 0 2024-03-14T13:02:18.637 INFO:tasks.workunit.client.0.smithi111.stdout:0/618: truncate d6/d9/de/d3a/dd1/fce 1264697 0 2024-03-14T13:02:18.638 INFO:tasks.workunit.client.0.smithi111.stdout:1/543: fdatasync d20/d40/f9b 0 2024-03-14T13:02:18.638 INFO:tasks.workunit.client.1.smithi137.stdout:9/567: dwrite d6/df/f13 [0,4194304] 0 2024-03-14T13:02:18.638 INFO:tasks.workunit.client.1.smithi137.stdout:8/536: mknod d1/dc/d77/d9f/cbd 0 2024-03-14T13:02:18.639 INFO:tasks.workunit.client.0.smithi111.stdout:0/619: read d6/d9/de/f3c [2833275,91816] 0 2024-03-14T13:02:18.642 INFO:tasks.workunit.client.0.smithi111.stdout:0/620: dread d6/d9/de/f1d [0,4194304] 0 2024-03-14T13:02:18.642 INFO:tasks.workunit.client.1.smithi137.stdout:9/568: dread - d6/df/d19/d1f/fb4 zero size 2024-03-14T13:02:18.642 INFO:tasks.workunit.client.1.smithi137.stdout:2/540: creat d2/d7/d2e/d3e/daa/fb0 x:0 0 0 2024-03-14T13:02:18.642 INFO:tasks.workunit.client.1.smithi137.stdout:0/431: getdents d1/d5/d1b/d1e/d25/d55/d73/d83 0 2024-03-14T13:02:18.642 INFO:tasks.workunit.client.1.smithi137.stdout:0/432: fdatasync d1/d1f/f2f 0 2024-03-14T13:02:18.644 INFO:tasks.workunit.client.0.smithi111.stdout:0/621: creat d6/d9/de/d3a/fd9 x:0 0 0 2024-03-14T13:02:18.644 INFO:tasks.workunit.client.0.smithi111.stdout:0/622: chown d6/d3d/l7e 92060 1 2024-03-14T13:02:18.644 INFO:tasks.workunit.client.1.smithi137.stdout:6/509: link d2/d6/de/d45/d59/f75 d2/d6/de/fb3 0 2024-03-14T13:02:18.645 INFO:tasks.workunit.client.1.smithi137.stdout:4/468: mknod d4/d9/d3a/d41/d38/d55/ca4 0 2024-03-14T13:02:18.645 INFO:tasks.workunit.client.1.smithi137.stdout:5/397: creat d7/d16/d18/d2d/f8e x:0 0 0 2024-03-14T13:02:18.645 INFO:tasks.workunit.client.1.smithi137.stdout:8/537: dread d1/dc/d10/f1a [0,4194304] 0 2024-03-14T13:02:18.646 INFO:tasks.workunit.client.0.smithi111.stdout:0/623: creat d6/d9/d15/d55/fda x:0 0 0 2024-03-14T13:02:18.647 INFO:tasks.workunit.client.1.smithi137.stdout:9/569: mkdir d6/d53/db7 0 2024-03-14T13:02:18.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/469: truncate d4/f89 223478 0 2024-03-14T13:02:18.650 INFO:tasks.workunit.client.1.smithi137.stdout:8/538: write d1/dc/d3c/d87/d34/d97/f2d [5072730,29162] 0 2024-03-14T13:02:18.651 INFO:tasks.workunit.client.1.smithi137.stdout:6/510: write d2/f25 [586705,122192] 0 2024-03-14T13:02:18.651 INFO:tasks.workunit.client.1.smithi137.stdout:9/570: creat d6/d53/fb8 x:0 0 0 2024-03-14T13:02:18.651 INFO:tasks.workunit.client.1.smithi137.stdout:4/470: truncate d4/d4d/fa0 184923 0 2024-03-14T13:02:18.655 INFO:tasks.workunit.client.1.smithi137.stdout:4/471: chown d4/d9/l4a 428895 1 2024-03-14T13:02:18.656 INFO:tasks.workunit.client.0.smithi111.stdout:1/544: truncate d20/d25/fa1 217007 0 2024-03-14T13:02:18.658 INFO:tasks.workunit.client.1.smithi137.stdout:8/539: creat d1/dc/da9/fbe x:0 0 0 2024-03-14T13:02:18.659 INFO:tasks.workunit.client.1.smithi137.stdout:0/433: dwrite d1/d5/d1b/d1e/d25/d26/f2d [0,4194304] 0 2024-03-14T13:02:18.659 INFO:tasks.workunit.client.1.smithi137.stdout:4/472: mknod d4/d10/d42/ca5 0 2024-03-14T13:02:18.661 INFO:tasks.workunit.client.1.smithi137.stdout:9/571: symlink d6/df/d19/d3a/lb9 0 2024-03-14T13:02:18.662 INFO:tasks.workunit.client.1.smithi137.stdout:2/541: write d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f93 [511766,119415] 0 2024-03-14T13:02:18.662 INFO:tasks.workunit.client.1.smithi137.stdout:6/511: mknod d2/d6/da7/cb4 0 2024-03-14T13:02:18.663 INFO:tasks.workunit.client.0.smithi111.stdout:0/624: rename f1 to d6/d9/de/d3a/dd1/fdb 0 2024-03-14T13:02:18.663 INFO:tasks.workunit.client.0.smithi111.stdout:1/545: readlink d20/d25/d75/l78 0 2024-03-14T13:02:18.664 INFO:tasks.workunit.client.1.smithi137.stdout:8/540: mknod d1/dc/d77/cbf 0 2024-03-14T13:02:18.665 INFO:tasks.workunit.client.0.smithi111.stdout:1/546: write d20/d25/d31/f7f [72058,62869] 0 2024-03-14T13:02:18.667 INFO:tasks.workunit.client.1.smithi137.stdout:5/398: write d7/d16/d18/f20 [3217621,119236] 0 2024-03-14T13:02:18.667 INFO:tasks.workunit.client.1.smithi137.stdout:1/530: sync 2024-03-14T13:02:18.667 INFO:tasks.workunit.client.1.smithi137.stdout:7/385: sync 2024-03-14T13:02:18.668 INFO:tasks.workunit.client.1.smithi137.stdout:4/473: creat d4/d9/d3a/d70/d80/fa6 x:0 0 0 2024-03-14T13:02:18.669 INFO:tasks.workunit.client.1.smithi137.stdout:9/572: creat d6/d56/fba x:0 0 0 2024-03-14T13:02:18.670 INFO:tasks.workunit.client.0.smithi111.stdout:0/625: dwrite d6/d9/d15/d55/f86 [0,4194304] 0 2024-03-14T13:02:18.670 INFO:tasks.workunit.client.0.smithi111.stdout:1/547: creat d20/d21/d3f/d59/daf/fbd x:0 0 0 2024-03-14T13:02:18.673 INFO:tasks.workunit.client.0.smithi111.stdout:0/626: creat d6/d45/d87/fdc x:0 0 0 2024-03-14T13:02:18.674 INFO:tasks.workunit.client.0.smithi111.stdout:1/548: truncate d20/f58 2923866 0 2024-03-14T13:02:18.675 INFO:tasks.workunit.client.1.smithi137.stdout:8/541: creat d1/dc/d3c/d87/d34/d97/fc0 x:0 0 0 2024-03-14T13:02:18.675 INFO:tasks.workunit.client.0.smithi111.stdout:0/627: fdatasync d6/d9/de/d3a/d46/d4a/f81 0 2024-03-14T13:02:18.676 INFO:tasks.workunit.client.1.smithi137.stdout:2/542: getdents d2/d7/d2e/d3e/d4e 0 2024-03-14T13:02:18.677 INFO:tasks.workunit.client.1.smithi137.stdout:9/573: symlink d6/df/d19/d3a/da9/lbb 0 2024-03-14T13:02:18.679 INFO:tasks.workunit.client.1.smithi137.stdout:5/399: rename d7/d16/d18/d25/f3f to d7/d16/d18/d2d/f8f 0 2024-03-14T13:02:18.683 INFO:tasks.workunit.client.0.smithi111.stdout:0/628: rmdir d6/d9/d15/d49 39 2024-03-14T13:02:18.684 INFO:tasks.workunit.client.1.smithi137.stdout:6/512: dread d2/d54/d55/d38/f85 [0,4194304] 0 2024-03-14T13:02:18.684 INFO:tasks.workunit.client.0.smithi111.stdout:0/629: chown d6/d9/de/d3a/f95 0 1 2024-03-14T13:02:18.689 INFO:tasks.workunit.client.0.smithi111.stdout:7/454: sync 2024-03-14T13:02:18.689 INFO:tasks.workunit.client.0.smithi111.stdout:5/672: sync 2024-03-14T13:02:18.690 INFO:tasks.workunit.client.0.smithi111.stdout:2/517: sync 2024-03-14T13:02:18.690 INFO:tasks.workunit.client.0.smithi111.stdout:3/490: sync 2024-03-14T13:02:18.690 INFO:tasks.workunit.client.0.smithi111.stdout:8/712: sync 2024-03-14T13:02:18.692 INFO:tasks.workunit.client.1.smithi137.stdout:0/434: dwrite d1/d5/d1b/d1e/d25/d55/f60 [0,4194304] 0 2024-03-14T13:02:18.694 INFO:tasks.workunit.client.0.smithi111.stdout:8/713: read - d3/d22/f72 zero size 2024-03-14T13:02:18.695 INFO:tasks.workunit.client.0.smithi111.stdout:0/630: dwrite d6/d3d/fc0 [4194304,4194304] 0 2024-03-14T13:02:18.695 INFO:tasks.workunit.client.0.smithi111.stdout:5/673: mknod d2/d13/cde 0 2024-03-14T13:02:18.697 INFO:tasks.workunit.client.1.smithi137.stdout:7/386: dwrite d3/f32 [0,4194304] 0 2024-03-14T13:02:18.697 INFO:tasks.workunit.client.1.smithi137.stdout:9/574: symlink d6/df/d19/d33/d8f/lbc 0 2024-03-14T13:02:18.697 INFO:tasks.workunit.client.1.smithi137.stdout:8/542: dread d1/dc/d3c/d87/d34/f9a [0,4194304] 0 2024-03-14T13:02:18.701 INFO:tasks.workunit.client.1.smithi137.stdout:1/531: dwrite d7/d9/d19/d1f/f23 [0,4194304] 0 2024-03-14T13:02:18.701 INFO:tasks.workunit.client.0.smithi111.stdout:2/518: truncate dd/d17/f63 4871395 0 2024-03-14T13:02:18.702 INFO:tasks.workunit.client.1.smithi137.stdout:6/513: mknod d2/d54/d55/d6d/d76/d57/cb5 0 2024-03-14T13:02:18.703 INFO:tasks.workunit.client.0.smithi111.stdout:3/491: rmdir d0/d2d/d35/d50/d6c/da3/daa 39 2024-03-14T13:02:18.704 INFO:tasks.workunit.client.0.smithi111.stdout:1/549: dread d20/d21/d28/d5c/f93 [0,4194304] 0 2024-03-14T13:02:18.708 INFO:tasks.workunit.client.0.smithi111.stdout:0/631: creat d6/d3d/dba/fdd x:0 0 0 2024-03-14T13:02:18.708 INFO:tasks.workunit.client.0.smithi111.stdout:1/550: chown d20/d25/d31/f69 248940106 1 2024-03-14T13:02:18.708 INFO:tasks.workunit.client.0.smithi111.stdout:7/455: dwrite d0/d1/d5/d56/f5e [0,4194304] 0 2024-03-14T13:02:18.709 INFO:tasks.workunit.client.0.smithi111.stdout:8/714: mkdir d3/dc/d45/de1 0 2024-03-14T13:02:18.712 INFO:tasks.workunit.client.0.smithi111.stdout:2/519: read dd/d17/d34/d73/d75/d59/f65 [78662,120186] 0 2024-03-14T13:02:18.713 INFO:tasks.workunit.client.0.smithi111.stdout:3/492: creat d0/d2d/d35/d50/da2/db3/da7/fbb x:0 0 0 2024-03-14T13:02:18.714 INFO:tasks.workunit.client.0.smithi111.stdout:0/632: mkdir d6/d9/de/d3a/d46/dde 0 2024-03-14T13:02:18.714 INFO:tasks.workunit.client.0.smithi111.stdout:7/456: write d0/d1/d22/d3e/f64 [319216,100424] 0 2024-03-14T13:02:18.715 INFO:tasks.workunit.client.1.smithi137.stdout:3/439: sync 2024-03-14T13:02:18.716 INFO:tasks.workunit.client.0.smithi111.stdout:2/520: truncate dd/f76 589238 0 2024-03-14T13:02:18.716 INFO:tasks.workunit.client.1.smithi137.stdout:2/543: dwrite d2/d34/d3c/f5d [0,4194304] 0 2024-03-14T13:02:18.717 INFO:tasks.workunit.client.1.smithi137.stdout:0/435: mknod d1/d5/d1b/d42/d3a/c91 0 2024-03-14T13:02:18.718 INFO:tasks.workunit.client.1.smithi137.stdout:5/400: dread d7/d36/f82 [0,4194304] 0 2024-03-14T13:02:18.718 INFO:tasks.workunit.client.0.smithi111.stdout:3/493: write d0/d30/f95 [2242600,109373] 0 2024-03-14T13:02:18.718 INFO:tasks.workunit.client.0.smithi111.stdout:0/633: creat d6/d9/d15/d55/da0/fdf x:0 0 0 2024-03-14T13:02:18.721 INFO:tasks.workunit.client.0.smithi111.stdout:2/521: symlink dd/d17/d34/d73/d75/la8 0 2024-03-14T13:02:18.721 INFO:tasks.workunit.client.0.smithi111.stdout:3/494: write d0/d2d/f86 [495711,57687] 0 2024-03-14T13:02:18.724 INFO:tasks.workunit.client.0.smithi111.stdout:1/551: dwrite f9 [4194304,4194304] 0 2024-03-14T13:02:18.727 INFO:tasks.workunit.client.1.smithi137.stdout:0/436: readlink d1/d5/l22 0 2024-03-14T13:02:18.729 INFO:tasks.workunit.client.1.smithi137.stdout:4/474: dread d4/d9/d3a/d41/f97 [0,4194304] 0 2024-03-14T13:02:18.731 INFO:tasks.workunit.client.0.smithi111.stdout:0/634: rename d6/d9/d15/d55/d89/ld3 to d6/d3d/le0 0 2024-03-14T13:02:18.731 INFO:tasks.workunit.client.0.smithi111.stdout:3/495: mkdir d0/d2d/d35/d50/da2/db3/d71/dbc 0 2024-03-14T13:02:18.732 INFO:tasks.workunit.client.1.smithi137.stdout:3/440: mkdir d4/de/d93 0 2024-03-14T13:02:18.734 INFO:tasks.workunit.client.0.smithi111.stdout:0/635: mknod d6/d9/d15/d55/d89/ce1 0 2024-03-14T13:02:18.736 INFO:tasks.workunit.client.0.smithi111.stdout:1/552: link d20/d25/d87/fb8 d20/d21/da7/fbe 0 2024-03-14T13:02:18.739 INFO:tasks.workunit.client.0.smithi111.stdout:1/553: unlink d20/d21/d3f/f96 0 2024-03-14T13:02:18.740 INFO:tasks.workunit.client.0.smithi111.stdout:0/636: dread d6/d9/de/d3a/f61 [0,4194304] 0 2024-03-14T13:02:18.741 INFO:tasks.workunit.client.1.smithi137.stdout:8/543: dwrite d1/dc/d10/fa4 [0,4194304] 0 2024-03-14T13:02:18.742 INFO:tasks.workunit.client.0.smithi111.stdout:1/554: mknod d20/d21/d28/da0/db6/cbf 0 2024-03-14T13:02:18.742 INFO:tasks.workunit.client.1.smithi137.stdout:5/401: chown d7/d33/d65/l6d 10740833 1 2024-03-14T13:02:18.742 INFO:tasks.workunit.client.0.smithi111.stdout:1/555: read d20/d40/f54 [489783,58630] 0 2024-03-14T13:02:18.743 INFO:tasks.workunit.client.1.smithi137.stdout:8/544: stat d1/dc/d77 0 2024-03-14T13:02:18.744 INFO:tasks.workunit.client.1.smithi137.stdout:9/575: dwrite d6/df/d19/d3a/d40/f9e [0,4194304] 0 2024-03-14T13:02:18.745 INFO:tasks.workunit.client.0.smithi111.stdout:5/674: dread d2/d16/d49/d33/f43 [0,4194304] 0 2024-03-14T13:02:18.745 INFO:tasks.workunit.client.1.smithi137.stdout:4/475: symlink d4/d9/d3a/d70/d80/d8d/la7 0 2024-03-14T13:02:18.745 INFO:tasks.workunit.client.0.smithi111.stdout:1/556: creat d20/d25/fc0 x:0 0 0 2024-03-14T13:02:18.746 INFO:tasks.workunit.client.0.smithi111.stdout:1/557: write d20/f9d [989324,94382] 0 2024-03-14T13:02:18.748 INFO:tasks.workunit.client.1.smithi137.stdout:1/532: dwrite d7/f9f [0,4194304] 0 2024-03-14T13:02:18.748 INFO:tasks.workunit.client.1.smithi137.stdout:6/514: rmdir d2/d6/de/d45/d59/d39/d67 0 2024-03-14T13:02:18.748 INFO:tasks.workunit.client.1.smithi137.stdout:3/441: symlink d4/de/d16/d1c/d5d/d4b/l94 0 2024-03-14T13:02:18.748 INFO:tasks.workunit.client.1.smithi137.stdout:5/402: fsync d7/d16/d18/d25/d28/f60 0 2024-03-14T13:02:18.749 INFO:tasks.workunit.client.0.smithi111.stdout:5/675: rmdir d2/d45/d79/dac 0 2024-03-14T13:02:18.750 INFO:tasks.workunit.client.0.smithi111.stdout:5/676: chown d2/d13/f6c 0 1 2024-03-14T13:02:18.753 INFO:tasks.workunit.client.1.smithi137.stdout:4/476: mknod d4/d10/ca8 0 2024-03-14T13:02:18.753 INFO:tasks.workunit.client.1.smithi137.stdout:8/545: read d1/dc/d3c/d87/d34/d97/d13/f9e [2140507,27613] 0 2024-03-14T13:02:18.754 INFO:tasks.workunit.client.1.smithi137.stdout:3/442: creat d4/d22/f95 x:0 0 0 2024-03-14T13:02:18.755 INFO:tasks.workunit.client.0.smithi111.stdout:1/558: dwrite d20/d40/f9b [0,4194304] 0 2024-03-14T13:02:18.758 INFO:tasks.workunit.client.1.smithi137.stdout:9/576: fsync d6/df/d19/d3a/d40/fb0 0 2024-03-14T13:02:18.759 INFO:tasks.workunit.client.1.smithi137.stdout:1/533: write d7/d9/d19/d40/d42/f49 [405143,26402] 0 2024-03-14T13:02:18.767 INFO:tasks.workunit.client.1.smithi137.stdout:3/443: rename d4/de/f77 to d4/df/f96 0 2024-03-14T13:02:18.768 INFO:tasks.workunit.client.0.smithi111.stdout:8/715: write d3/d22/d30/fa4 [280682,2746] 0 2024-03-14T13:02:18.769 INFO:tasks.workunit.client.0.smithi111.stdout:8/716: mknod d3/d1b/d34/ce2 0 2024-03-14T13:02:18.772 INFO:tasks.workunit.client.0.smithi111.stdout:8/717: creat d3/d1b/d20/d35/d40/d48/dcd/fe3 x:0 0 0 2024-03-14T13:02:18.773 INFO:tasks.workunit.client.1.smithi137.stdout:6/515: rmdir d2/d54/d55/d6d/d76/d57 39 2024-03-14T13:02:18.773 INFO:tasks.workunit.client.1.smithi137.stdout:8/546: dwrite d1/dc/d3c/d87/d34/d97/d13/f9e [0,4194304] 0 2024-03-14T13:02:18.773 INFO:tasks.workunit.client.1.smithi137.stdout:3/444: symlink d4/d19/l97 0 2024-03-14T13:02:18.774 INFO:tasks.workunit.client.1.smithi137.stdout:9/577: creat d6/df/d19/d33/d8f/fbd x:0 0 0 2024-03-14T13:02:18.774 INFO:tasks.workunit.client.0.smithi111.stdout:8/718: mkdir d3/d1b/dc1/de4 0 2024-03-14T13:02:18.777 INFO:tasks.workunit.client.1.smithi137.stdout:2/544: dread d2/d34/f7f [0,4194304] 0 2024-03-14T13:02:18.778 INFO:tasks.workunit.client.1.smithi137.stdout:6/516: stat d2/d54/d55/d6d/lae 0 2024-03-14T13:02:18.778 INFO:tasks.workunit.client.0.smithi111.stdout:8/719: rename d3/dc/d31/f4c to d3/d1b/dc1/de4/fe5 0 2024-03-14T13:02:18.783 INFO:tasks.workunit.client.0.smithi111.stdout:8/720: mkdir d3/d22/da1/de6 0 2024-03-14T13:02:18.785 INFO:tasks.workunit.client.1.smithi137.stdout:9/578: mknod d6/df/d88/cbe 0 2024-03-14T13:02:18.786 INFO:tasks.workunit.client.0.smithi111.stdout:0/637: dread d6/d9/de/d32/f31 [0,4194304] 0 2024-03-14T13:02:18.789 INFO:tasks.workunit.client.1.smithi137.stdout:5/403: dread d7/d16/d18/d25/f5b [0,4194304] 0 2024-03-14T13:02:18.789 INFO:tasks.workunit.client.1.smithi137.stdout:7/387: truncate d3/d6/d59/f5f 3909421 0 2024-03-14T13:02:18.789 INFO:tasks.workunit.client.1.smithi137.stdout:7/388: stat d3 0 2024-03-14T13:02:18.789 INFO:tasks.workunit.client.1.smithi137.stdout:7/389: fsync d3/d6/d14/d27/f6c 0 2024-03-14T13:02:18.789 INFO:tasks.workunit.client.0.smithi111.stdout:8/721: dread - d3/d1b/d20/d35/d40/d48/d7e/fca zero size 2024-03-14T13:02:18.790 INFO:tasks.workunit.client.1.smithi137.stdout:2/545: write d2/d7/d2e/d3e/d44/d6b/d22/f8f [81883,96726] 0 2024-03-14T13:02:18.790 INFO:tasks.workunit.client.0.smithi111.stdout:2/522: dwrite dd/d17/d53/f98 [0,4194304] 0 2024-03-14T13:02:18.793 INFO:tasks.workunit.client.1.smithi137.stdout:0/437: truncate d1/d5/d1b/d1e/d25/d55/f7a 174973 0 2024-03-14T13:02:18.795 INFO:tasks.workunit.client.0.smithi111.stdout:8/722: mknod d3/d1b/d20/d35/d40/ce7 0 2024-03-14T13:02:18.795 INFO:tasks.workunit.client.0.smithi111.stdout:1/559: truncate d20/d21/f42 1754557 0 2024-03-14T13:02:18.796 INFO:tasks.workunit.client.0.smithi111.stdout:5/677: write d2/d16/d49/d33/f43 [2751517,25220] 0 2024-03-14T13:02:18.798 INFO:tasks.workunit.client.0.smithi111.stdout:0/638: dread d6/d3d/fa7 [0,4194304] 0 2024-03-14T13:02:18.799 INFO:tasks.workunit.client.1.smithi137.stdout:1/534: dread d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:18.800 INFO:tasks.workunit.client.0.smithi111.stdout:1/560: write d20/d21/d3f/d59/f74 [283054,73230] 0 2024-03-14T13:02:18.800 INFO:tasks.workunit.client.1.smithi137.stdout:1/535: write d7/f9f [1969179,38870] 0 2024-03-14T13:02:18.802 INFO:tasks.workunit.client.0.smithi111.stdout:8/723: unlink d3/d5/c2f 0 2024-03-14T13:02:18.802 INFO:tasks.workunit.client.0.smithi111.stdout:1/561: readlink d20/d21/d28/d5c/l67 0 2024-03-14T13:02:18.803 INFO:tasks.workunit.client.0.smithi111.stdout:2/523: truncate dd/d11/f6d 2364466 0 2024-03-14T13:02:18.804 INFO:tasks.workunit.client.0.smithi111.stdout:0/639: creat d6/d45/d87/fe2 x:0 0 0 2024-03-14T13:02:18.805 INFO:tasks.workunit.client.0.smithi111.stdout:8/724: unlink c2 0 2024-03-14T13:02:18.807 INFO:tasks.workunit.client.0.smithi111.stdout:2/524: mkdir dd/da9 0 2024-03-14T13:02:18.808 INFO:tasks.workunit.client.0.smithi111.stdout:1/562: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:18.810 INFO:tasks.workunit.client.0.smithi111.stdout:8/725: creat d3/d1b/d20/d35/d40/d48/d77/fe8 x:0 0 0 2024-03-14T13:02:18.810 INFO:tasks.workunit.client.0.smithi111.stdout:5/678: link d2/d16/d49/d33/d68/cdc d2/d16/d49/d7c/d95/d6a/cdf 0 2024-03-14T13:02:18.811 INFO:tasks.workunit.client.0.smithi111.stdout:0/640: dread d6/d9/de/d32/f33 [4194304,4194304] 0 2024-03-14T13:02:18.812 INFO:tasks.workunit.client.0.smithi111.stdout:5/679: write d2/d16/d49/d33/fa6 [656804,10259] 0 2024-03-14T13:02:18.812 INFO:tasks.workunit.client.0.smithi111.stdout:0/641: dread - d6/d9/d5b/fb8 zero size 2024-03-14T13:02:18.812 INFO:tasks.workunit.client.0.smithi111.stdout:2/525: mknod dd/da9/caa 0 2024-03-14T13:02:18.814 INFO:tasks.workunit.client.0.smithi111.stdout:8/726: symlink d3/d5/le9 0 2024-03-14T13:02:18.815 INFO:tasks.workunit.client.0.smithi111.stdout:1/563: mkdir d20/d21/dc1 0 2024-03-14T13:02:18.817 INFO:tasks.workunit.client.0.smithi111.stdout:0/642: fsync d6/d9/d15/d49/d56/fab 0 2024-03-14T13:02:18.818 INFO:tasks.workunit.client.0.smithi111.stdout:5/680: dread d2/d16/d49/d7c/d95/f5e [0,4194304] 0 2024-03-14T13:02:18.821 INFO:tasks.workunit.client.0.smithi111.stdout:0/643: write d6/d9/de/d3a/d46/f83 [153199,27011] 0 2024-03-14T13:02:18.822 INFO:tasks.workunit.client.0.smithi111.stdout:8/727: dwrite d3/d22/fbb [0,4194304] 0 2024-03-14T13:02:18.824 INFO:tasks.workunit.client.0.smithi111.stdout:0/644: fsync d6/d9/de/f36 0 2024-03-14T13:02:18.824 INFO:tasks.workunit.client.0.smithi111.stdout:1/564: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:18.829 INFO:tasks.workunit.client.0.smithi111.stdout:5/681: mkdir d2/d16/d49/d44/d72/de0 0 2024-03-14T13:02:18.832 INFO:tasks.workunit.client.0.smithi111.stdout:1/565: stat d20/d21/da7/fb1 0 2024-03-14T13:02:18.833 INFO:tasks.workunit.client.0.smithi111.stdout:8/728: unlink d3/d1b/d20/l5e 0 2024-03-14T13:02:18.838 INFO:tasks.workunit.client.0.smithi111.stdout:5/682: mknod d2/d16/d49/d4c/d9c/ce1 0 2024-03-14T13:02:18.839 INFO:tasks.workunit.client.0.smithi111.stdout:1/566: dwrite d20/f9d [0,4194304] 0 2024-03-14T13:02:18.843 INFO:tasks.workunit.client.0.smithi111.stdout:0/645: dread d6/fb3 [0,4194304] 0 2024-03-14T13:02:18.843 INFO:tasks.workunit.client.0.smithi111.stdout:0/646: chown d6/d3d/d97/fb7 54817192 1 2024-03-14T13:02:18.845 INFO:tasks.workunit.client.0.smithi111.stdout:0/647: truncate d6/d9/de/d32/f5e 1009507 0 2024-03-14T13:02:18.845 INFO:tasks.workunit.client.0.smithi111.stdout:8/729: creat d3/dc/d31/d9d/fea x:0 0 0 2024-03-14T13:02:18.847 INFO:tasks.workunit.client.0.smithi111.stdout:0/648: write d6/d9/de/d3a/d46/d4a/f81 [833398,66077] 0 2024-03-14T13:02:18.849 INFO:tasks.workunit.client.0.smithi111.stdout:0/649: creat d6/d9/de/d77/fe3 x:0 0 0 2024-03-14T13:02:18.850 INFO:tasks.workunit.client.0.smithi111.stdout:0/650: rmdir d6/d21 39 2024-03-14T13:02:18.852 INFO:tasks.workunit.client.0.smithi111.stdout:0/651: write d6/d9/d15/d55/da0/fb9 [778374,5927] 0 2024-03-14T13:02:18.853 INFO:tasks.workunit.client.0.smithi111.stdout:0/652: mknod d6/d9/d15/d49/ce4 0 2024-03-14T13:02:18.855 INFO:tasks.workunit.client.0.smithi111.stdout:0/653: getdents d6/d9/de/d3a/d46/dde 0 2024-03-14T13:02:18.860 INFO:tasks.workunit.client.0.smithi111.stdout:0/654: dwrite d6/d9/d15/d55/da0/fb9 [0,4194304] 0 2024-03-14T13:02:18.865 INFO:tasks.workunit.client.0.smithi111.stdout:0/655: rename d6/l38 to d6/d3d/da6/le5 0 2024-03-14T13:02:18.867 INFO:tasks.workunit.client.0.smithi111.stdout:0/656: write d6/d9/d5b/d9e/faf [304691,122549] 0 2024-03-14T13:02:18.867 INFO:tasks.workunit.client.1.smithi137.stdout:9/579: unlink d6/df/f7d 0 2024-03-14T13:02:18.871 INFO:tasks.workunit.client.1.smithi137.stdout:4/477: truncate d4/d9/d3a/f4f 3577017 0 2024-03-14T13:02:18.872 INFO:tasks.workunit.client.1.smithi137.stdout:6/517: rename d2/c10 to d2/d6/d1d/d88/cb6 0 2024-03-14T13:02:18.873 INFO:tasks.workunit.client.1.smithi137.stdout:3/445: rmdir d4/d19 39 2024-03-14T13:02:18.874 INFO:tasks.workunit.client.0.smithi111.stdout:1/567: write d20/d25/f7c [1279731,75901] 0 2024-03-14T13:02:18.876 INFO:tasks.workunit.client.1.smithi137.stdout:0/438: mknod d1/d44/d79/c92 0 2024-03-14T13:02:18.876 INFO:tasks.workunit.client.0.smithi111.stdout:5/683: truncate d2/d16/d49/d4c/f73 1592123 0 2024-03-14T13:02:18.878 INFO:tasks.workunit.client.1.smithi137.stdout:2/546: dwrite d2/d7/d2e/d3e/d44/d6b/f20 [4194304,4194304] 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.1.smithi137.stdout:4/478: dread d4/d27/f28 [0,4194304] 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.1.smithi137.stdout:0/439: read d1/d44/d56/f7d [2473056,22068] 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.0.smithi111.stdout:8/730: symlink d3/d1b/d20/d35/d40/d48/dcd/leb 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.0.smithi111.stdout:1/568: mknod d20/d21/d28/da0/cc2 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.0.smithi111.stdout:5/684: mknod d2/d16/d49/d7c/d8b/ce2 0 2024-03-14T13:02:18.882 INFO:tasks.workunit.client.0.smithi111.stdout:0/657: write d6/d9/de/d3a/f95 [333178,66809] 0 2024-03-14T13:02:18.883 INFO:tasks.workunit.client.1.smithi137.stdout:7/390: fdatasync d3/d6/d59/f5f 0 2024-03-14T13:02:18.885 INFO:tasks.workunit.client.1.smithi137.stdout:0/440: write d1/d5/d1b/d1e/d25/d55/d73/f8f [802231,72887] 0 2024-03-14T13:02:18.886 INFO:tasks.workunit.client.0.smithi111.stdout:0/658: truncate d6/d9/de/d3a/fd9 377205 0 2024-03-14T13:02:18.887 INFO:tasks.workunit.client.0.smithi111.stdout:0/659: fsync f5 0 2024-03-14T13:02:18.888 INFO:tasks.workunit.client.0.smithi111.stdout:0/660: read d6/d9/f58 [377680,8413] 0 2024-03-14T13:02:18.888 INFO:tasks.workunit.client.0.smithi111.stdout:1/569: unlink d20/d21/cbb 0 2024-03-14T13:02:18.889 INFO:tasks.workunit.client.1.smithi137.stdout:6/518: read - d2/d54/d55/d6d/d76/d57/d58/fa5 zero size 2024-03-14T13:02:18.891 INFO:tasks.workunit.client.0.smithi111.stdout:5/685: unlink d2/d16/d49/d33/c97 0 2024-03-14T13:02:18.892 INFO:tasks.workunit.client.1.smithi137.stdout:4/479: dwrite d4/d4d/f6b [0,4194304] 0 2024-03-14T13:02:18.893 INFO:tasks.workunit.client.1.smithi137.stdout:4/480: chown d4/d10/d42/c82 295691946 1 2024-03-14T13:02:18.893 INFO:tasks.workunit.client.0.smithi111.stdout:8/731: dread - d3/d1b/d20/d35/d40/d48/d7e/fca zero size 2024-03-14T13:02:18.895 INFO:tasks.workunit.client.1.smithi137.stdout:8/547: write d1/dc/d3c/d87/f1e [332163,57939] 0 2024-03-14T13:02:18.896 INFO:tasks.workunit.client.0.smithi111.stdout:1/570: creat d20/d21/d28/fc3 x:0 0 0 2024-03-14T13:02:18.897 INFO:tasks.workunit.client.0.smithi111.stdout:0/661: dwrite d6/d9/d15/d49/d56/fc9 [0,4194304] 0 2024-03-14T13:02:18.898 INFO:tasks.workunit.client.1.smithi137.stdout:8/548: stat d1/dc/d3c/d87/d34/d82/ca2 0 2024-03-14T13:02:18.898 INFO:tasks.workunit.client.1.smithi137.stdout:8/549: read - d1/f83 zero size 2024-03-14T13:02:18.901 INFO:tasks.workunit.client.0.smithi111.stdout:1/571: symlink d20/d40/lc4 0 2024-03-14T13:02:18.901 INFO:tasks.workunit.client.1.smithi137.stdout:9/580: rmdir d6/df 39 2024-03-14T13:02:18.901 INFO:tasks.workunit.client.1.smithi137.stdout:1/536: link d7/l55 d7/d4d/la9 0 2024-03-14T13:02:18.902 INFO:tasks.workunit.client.1.smithi137.stdout:2/547: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/cb1 0 2024-03-14T13:02:18.903 INFO:tasks.workunit.client.1.smithi137.stdout:1/537: fdatasync d7/f9f 0 2024-03-14T13:02:18.903 INFO:tasks.workunit.client.0.smithi111.stdout:8/732: creat d3/fec x:0 0 0 2024-03-14T13:02:18.904 INFO:tasks.workunit.client.0.smithi111.stdout:8/733: chown d3/dc/f11 18970 1 2024-03-14T13:02:18.905 INFO:tasks.workunit.client.0.smithi111.stdout:8/734: stat d3/d1b/d34/f3b 0 2024-03-14T13:02:18.906 INFO:tasks.workunit.client.1.smithi137.stdout:2/548: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 3 1 2024-03-14T13:02:18.906 INFO:tasks.workunit.client.0.smithi111.stdout:0/662: creat d6/d9/de/d77/da4/fe6 x:0 0 0 2024-03-14T13:02:18.906 INFO:tasks.workunit.client.0.smithi111.stdout:8/735: dread - d3/f78 zero size 2024-03-14T13:02:18.907 INFO:tasks.workunit.client.0.smithi111.stdout:0/663: stat d6/d45 0 2024-03-14T13:02:18.908 INFO:tasks.workunit.client.0.smithi111.stdout:0/664: write d6/d9/de/f1d [361109,48576] 0 2024-03-14T13:02:18.909 INFO:tasks.workunit.client.1.smithi137.stdout:8/550: readlink d1/dc/d3c/d87/d34/d82/d94/l9b 0 2024-03-14T13:02:18.910 INFO:tasks.workunit.client.0.smithi111.stdout:8/736: rename d3/dc/f4e to d3/d1b/d20/d35/d40/fed 0 2024-03-14T13:02:18.913 INFO:tasks.workunit.client.0.smithi111.stdout:0/665: mkdir d6/d9/de/d32/d64/de7 0 2024-03-14T13:02:18.913 INFO:tasks.workunit.client.1.smithi137.stdout:1/538: creat d7/d4d/d6b/faa x:0 0 0 2024-03-14T13:02:18.914 INFO:tasks.workunit.client.0.smithi111.stdout:5/686: write d2/d16/d49/f78 [1368192,120888] 0 2024-03-14T13:02:18.914 INFO:tasks.workunit.client.1.smithi137.stdout:0/441: creat d1/d44/f93 x:0 0 0 2024-03-14T13:02:18.915 INFO:tasks.workunit.client.0.smithi111.stdout:8/737: chown d3/d5/cde 247 1 2024-03-14T13:02:18.917 INFO:tasks.workunit.client.1.smithi137.stdout:3/446: dwrite d4/de/f7d [0,4194304] 0 2024-03-14T13:02:18.917 INFO:tasks.workunit.client.1.smithi137.stdout:4/481: link d4/d9/d3a/d41/l35 d4/d10/d26/la9 0 2024-03-14T13:02:18.918 INFO:tasks.workunit.client.1.smithi137.stdout:0/442: chown d1/d5/d1b/d1e/d25/d26/f2d 855941 1 2024-03-14T13:02:18.919 INFO:tasks.workunit.client.0.smithi111.stdout:8/738: creat d3/d22/d30/fee x:0 0 0 2024-03-14T13:02:18.919 INFO:tasks.workunit.client.1.smithi137.stdout:3/447: stat d4/de/f7d 0 2024-03-14T13:02:18.919 INFO:tasks.workunit.client.0.smithi111.stdout:5/687: truncate d2/d13/f2f 1363318 0 2024-03-14T13:02:18.920 INFO:tasks.workunit.client.0.smithi111.stdout:0/666: mkdir d6/d9/de/d77/da4/dca/dd0/de8 0 2024-03-14T13:02:18.921 INFO:tasks.workunit.client.1.smithi137.stdout:3/448: readlink d4/de/d16/d1c/d5d/d4b/l94 0 2024-03-14T13:02:18.921 INFO:tasks.workunit.client.0.smithi111.stdout:8/739: creat d3/d22/d30/fef x:0 0 0 2024-03-14T13:02:18.923 INFO:tasks.workunit.client.1.smithi137.stdout:9/581: link d6/df/d19/l23 d6/d53/lbf 0 2024-03-14T13:02:18.925 INFO:tasks.workunit.client.1.smithi137.stdout:0/443: dread d1/d5/d1b/d1e/d25/d55/f67 [0,4194304] 0 2024-03-14T13:02:18.927 INFO:tasks.workunit.client.1.smithi137.stdout:1/539: link d7/d9/d19/d1f/d26/l4e d7/d9/d19/d1f/da0/lab 0 2024-03-14T13:02:18.928 INFO:tasks.workunit.client.0.smithi111.stdout:8/740: rename d3/d1b/d20/d35/d40/f91 to d3/d1b/d20/d35/d40/d48/d7e/db3/db5/ff0 0 2024-03-14T13:02:18.929 INFO:tasks.workunit.client.1.smithi137.stdout:4/482: fsync d4/d4d/f6b 0 2024-03-14T13:02:18.930 INFO:tasks.workunit.client.0.smithi111.stdout:8/741: truncate d3/d22/d30/f7d 627409 0 2024-03-14T13:02:18.931 INFO:tasks.workunit.client.1.smithi137.stdout:7/391: truncate d3/d5/d23/f2b 4055672 0 2024-03-14T13:02:18.932 INFO:tasks.workunit.client.0.smithi111.stdout:8/742: creat d3/d22/ff1 x:0 0 0 2024-03-14T13:02:18.934 INFO:tasks.workunit.client.1.smithi137.stdout:0/444: fdatasync d1/d5/d1b/d1e/d25/d8a/f14 0 2024-03-14T13:02:18.935 INFO:tasks.workunit.client.1.smithi137.stdout:8/551: dwrite d1/dc/d3c/d87/d34/d97/d13/f4f [0,4194304] 0 2024-03-14T13:02:18.936 INFO:tasks.workunit.client.0.smithi111.stdout:1/572: dwrite d20/d40/f54 [0,4194304] 0 2024-03-14T13:02:18.937 INFO:tasks.workunit.client.0.smithi111.stdout:1/573: chown d20/d25/d31/f3b 70 1 2024-03-14T13:02:18.938 INFO:tasks.workunit.client.0.smithi111.stdout:1/574: stat d20/d40/l50 0 2024-03-14T13:02:18.938 INFO:tasks.workunit.client.1.smithi137.stdout:7/392: creat d3/d5/f7c x:0 0 0 2024-03-14T13:02:18.941 INFO:tasks.workunit.client.0.smithi111.stdout:9/505: sync 2024-03-14T13:02:18.942 INFO:tasks.workunit.client.1.smithi137.stdout:8/552: read d1/dc/f52 [50877,20244] 0 2024-03-14T13:02:18.943 INFO:tasks.workunit.client.0.smithi111.stdout:0/667: truncate d6/d9/de/d3a/f61 183222 0 2024-03-14T13:02:18.944 INFO:tasks.workunit.client.1.smithi137.stdout:3/449: dwrite d4/d22/f95 [0,4194304] 0 2024-03-14T13:02:18.949 INFO:tasks.workunit.client.0.smithi111.stdout:0/668: fsync d6/d9/de/d3a/d46/d4a/fb4 0 2024-03-14T13:02:18.949 INFO:tasks.workunit.client.1.smithi137.stdout:9/582: dwrite d6/df/d19/d1f/f35 [0,4194304] 0 2024-03-14T13:02:18.953 INFO:tasks.workunit.client.0.smithi111.stdout:5/688: dwrite d2/d16/f38 [0,4194304] 0 2024-03-14T13:02:18.956 INFO:tasks.workunit.client.0.smithi111.stdout:1/575: dwrite d20/d21/d28/d5c/f9a [0,4194304] 0 2024-03-14T13:02:18.957 INFO:tasks.workunit.client.1.smithi137.stdout:8/553: chown d1/dc/d4e/d6d/l95 86871 1 2024-03-14T13:02:18.957 INFO:tasks.workunit.client.1.smithi137.stdout:9/583: chown d6/fe 0 1 2024-03-14T13:02:18.957 INFO:tasks.workunit.client.0.smithi111.stdout:0/669: rename d6/d9/de/d32/l8b to d6/d9/de/d3a/le9 0 2024-03-14T13:02:18.959 INFO:tasks.workunit.client.1.smithi137.stdout:2/549: dread d2/d7/db/d2f/f45 [0,4194304] 0 2024-03-14T13:02:18.959 INFO:tasks.workunit.client.1.smithi137.stdout:3/450: write d4/de/d16/d1c/f8f [177966,117247] 0 2024-03-14T13:02:18.962 INFO:tasks.workunit.client.1.smithi137.stdout:0/445: dwrite d1/d5/d1b/d1e/d25/d8a/d4a/d54/f6f [0,4194304] 0 2024-03-14T13:02:18.963 INFO:tasks.workunit.client.1.smithi137.stdout:7/393: dwrite d3/f32 [0,4194304] 0 2024-03-14T13:02:18.964 INFO:tasks.workunit.client.0.smithi111.stdout:9/506: getdents d5/d6/d2a/d86 0 2024-03-14T13:02:18.970 INFO:tasks.workunit.client.1.smithi137.stdout:7/394: write d3/f18 [4471929,26647] 0 2024-03-14T13:02:18.970 INFO:tasks.workunit.client.1.smithi137.stdout:9/584: creat d6/df/d71/fc0 x:0 0 0 2024-03-14T13:02:18.973 INFO:tasks.workunit.client.0.smithi111.stdout:5/689: rename d2/d16/f38 to d2/d45/d76/fe3 0 2024-03-14T13:02:18.974 INFO:tasks.workunit.client.1.smithi137.stdout:2/550: fdatasync d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f8a 0 2024-03-14T13:02:18.974 INFO:tasks.workunit.client.1.smithi137.stdout:3/451: write d4/de/d16/d1c/d5d/d85/d6b/f8a [453465,118476] 0 2024-03-14T13:02:18.977 INFO:tasks.workunit.client.0.smithi111.stdout:5/690: fdatasync d2/d16/d49/f83 0 2024-03-14T13:02:18.979 INFO:tasks.workunit.client.0.smithi111.stdout:8/743: dread d3/f6b [0,4194304] 0 2024-03-14T13:02:18.981 INFO:tasks.workunit.client.1.smithi137.stdout:8/554: dwrite d1/dc/d10/f1a [0,4194304] 0 2024-03-14T13:02:18.982 INFO:tasks.workunit.client.0.smithi111.stdout:1/576: dwrite d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:18.982 INFO:tasks.workunit.client.0.smithi111.stdout:9/507: unlink d5/d6/d8/d5b/fae 0 2024-03-14T13:02:18.983 INFO:tasks.workunit.client.1.smithi137.stdout:2/551: write d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 [7573442,86104] 0 2024-03-14T13:02:18.985 INFO:tasks.workunit.client.1.smithi137.stdout:0/446: dread d1/d5/d1b/d1e/d25/d55/f60 [0,4194304] 0 2024-03-14T13:02:18.986 INFO:tasks.workunit.client.0.smithi111.stdout:9/508: truncate d5/d6/d8/d5b/d84/fa2 1697727 0 2024-03-14T13:02:18.990 INFO:tasks.workunit.client.1.smithi137.stdout:0/447: chown d1/d5/d1b/d1e/c2e 31741899 1 2024-03-14T13:02:18.990 INFO:tasks.workunit.client.1.smithi137.stdout:1/540: truncate d7/d4d/f85 42913 0 2024-03-14T13:02:18.991 INFO:tasks.workunit.client.0.smithi111.stdout:1/577: symlink d20/d21/lc5 0 2024-03-14T13:02:18.991 INFO:tasks.workunit.client.1.smithi137.stdout:7/395: dwrite d3/d5/f7c [0,4194304] 0 2024-03-14T13:02:18.992 INFO:tasks.workunit.client.0.smithi111.stdout:0/670: dread d6/d21/f73 [0,4194304] 0 2024-03-14T13:02:18.996 INFO:tasks.workunit.client.0.smithi111.stdout:9/509: rename d5/d6/d8/d1a/c23 to d5/d6/d8/d5b/d84/caf 0 2024-03-14T13:02:18.996 INFO:tasks.workunit.client.0.smithi111.stdout:8/744: getdents d3/d1b/dc1/de4 0 2024-03-14T13:02:19.000 INFO:tasks.workunit.client.0.smithi111.stdout:9/510: fsync d5/d6/d8/f93 0 2024-03-14T13:02:19.000 INFO:tasks.workunit.client.0.smithi111.stdout:8/745: creat d3/d1b/d66/ff2 x:0 0 0 2024-03-14T13:02:19.000 INFO:tasks.workunit.client.1.smithi137.stdout:0/448: write d1/d5/d1b/d42/f4b [857892,56687] 0 2024-03-14T13:02:19.001 INFO:tasks.workunit.client.1.smithi137.stdout:0/449: truncate d1/d44/d56/f70 4295860 0 2024-03-14T13:02:19.001 INFO:tasks.workunit.client.1.smithi137.stdout:8/555: fdatasync d1/dc/d3c/d87/d34/d97/d13/f16 0 2024-03-14T13:02:19.001 INFO:tasks.workunit.client.1.smithi137.stdout:2/552: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/fa0 to d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 0 2024-03-14T13:02:19.005 INFO:tasks.workunit.client.0.smithi111.stdout:9/511: rename d5/d6/d8/c38 to d5/d6/d8/d5b/d87/cb0 0 2024-03-14T13:02:19.005 INFO:tasks.workunit.client.1.smithi137.stdout:0/450: creat d1/d44/d72/d7e/f94 x:0 0 0 2024-03-14T13:02:19.006 INFO:tasks.workunit.client.1.smithi137.stdout:8/556: unlink d1/l1d 0 2024-03-14T13:02:19.007 INFO:tasks.workunit.client.0.smithi111.stdout:8/746: mknod d3/d1b/d20/d35/d40/d48/d7e/db3/db5/cf3 0 2024-03-14T13:02:19.007 INFO:tasks.workunit.client.0.smithi111.stdout:9/512: readlink d5/d6/d14/l3b 0 2024-03-14T13:02:19.007 INFO:tasks.workunit.client.0.smithi111.stdout:0/671: dwrite d6/d9/d5b/d9e/fd5 [0,4194304] 0 2024-03-14T13:02:19.007 INFO:tasks.workunit.client.0.smithi111.stdout:9/513: stat d5/c28 0 2024-03-14T13:02:19.007 INFO:tasks.workunit.client.1.smithi137.stdout:8/557: fsync d1/dc/f5f 0 2024-03-14T13:02:19.008 INFO:tasks.workunit.client.1.smithi137.stdout:0/451: symlink d1/d5/d1b/d1e/d25/d8a/d4a/d54/l95 0 2024-03-14T13:02:19.008 INFO:tasks.workunit.client.1.smithi137.stdout:8/558: chown d1/dc/d3c/d87/d34/d97/d13 240653 1 2024-03-14T13:02:19.009 INFO:tasks.workunit.client.0.smithi111.stdout:0/672: chown d6/d9/c22 1112 1 2024-03-14T13:02:19.010 INFO:tasks.workunit.client.0.smithi111.stdout:8/747: unlink d3/dc/f11 0 2024-03-14T13:02:19.011 INFO:tasks.workunit.client.0.smithi111.stdout:9/514: creat d5/d42/da4/fb1 x:0 0 0 2024-03-14T13:02:19.012 INFO:tasks.workunit.client.0.smithi111.stdout:0/673: mkdir d6/d9/d15/d49/d56/dea 0 2024-03-14T13:02:19.013 INFO:tasks.workunit.client.0.smithi111.stdout:9/515: mknod d5/d6/d8/d5b/d7c/cb2 0 2024-03-14T13:02:19.015 INFO:tasks.workunit.client.1.smithi137.stdout:2/553: dwrite d2/d7/d2e/d3e/d44/d6b/f1b [0,4194304] 0 2024-03-14T13:02:19.016 INFO:tasks.workunit.client.0.smithi111.stdout:9/516: truncate d5/d6/d2a/d5a/f9f 4530635 0 2024-03-14T13:02:19.016 INFO:tasks.workunit.client.1.smithi137.stdout:0/452: link d1/d44/d72/d7e/f78 d1/d1f/d80/f96 0 2024-03-14T13:02:19.017 INFO:tasks.workunit.client.0.smithi111.stdout:9/517: truncate d5/d6/f56 1231112 0 2024-03-14T13:02:19.017 INFO:tasks.workunit.client.1.smithi137.stdout:8/559: creat d1/dc/d3c/d87/fc1 x:0 0 0 2024-03-14T13:02:19.019 INFO:tasks.workunit.client.0.smithi111.stdout:0/674: rename d6/d9/de/d3a/d46/d4a/f5c to d6/d9/de/d77/da4/feb 0 2024-03-14T13:02:19.020 INFO:tasks.workunit.client.1.smithi137.stdout:0/453: symlink d1/d44/d79/d84/l97 0 2024-03-14T13:02:19.021 INFO:tasks.workunit.client.1.smithi137.stdout:2/554: write d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 [6181261,49392] 0 2024-03-14T13:02:19.021 INFO:tasks.workunit.client.0.smithi111.stdout:0/675: mknod d6/d9/de/d77/da4/dca/cec 0 2024-03-14T13:02:19.022 INFO:tasks.workunit.client.0.smithi111.stdout:0/676: chown d6/d9/d15/d55/lc6 7940 1 2024-03-14T13:02:19.023 INFO:tasks.workunit.client.1.smithi137.stdout:0/454: readlink d1/d1f/l3c 0 2024-03-14T13:02:19.023 INFO:tasks.workunit.client.0.smithi111.stdout:8/748: dread d3/d1b/d20/d35/fa0 [0,4194304] 0 2024-03-14T13:02:19.023 INFO:tasks.workunit.client.1.smithi137.stdout:3/452: dread d4/de/d16/d1c/f8f [0,4194304] 0 2024-03-14T13:02:19.028 INFO:tasks.workunit.client.0.smithi111.stdout:0/677: rename d6/d3d/c63 to d6/d9/de/d3a/d46/ced 0 2024-03-14T13:02:19.028 INFO:tasks.workunit.client.1.smithi137.stdout:3/453: rename d4/d19/f2f to d4/d22/f98 0 2024-03-14T13:02:19.029 INFO:tasks.workunit.client.0.smithi111.stdout:0/678: chown d6/f1f 123372175 1 2024-03-14T13:02:19.030 INFO:tasks.workunit.client.1.smithi137.stdout:3/454: mknod d4/d19/c99 0 2024-03-14T13:02:19.030 INFO:tasks.workunit.client.0.smithi111.stdout:8/749: fdatasync d3/d22/d30/f43 0 2024-03-14T13:02:19.037 INFO:tasks.workunit.client.1.smithi137.stdout:3/455: dwrite d4/d19/f70 [0,4194304] 0 2024-03-14T13:02:19.044 INFO:tasks.workunit.client.1.smithi137.stdout:9/585: truncate d6/df/d19/d3a/d40/f41 3269326 0 2024-03-14T13:02:19.045 INFO:tasks.workunit.client.1.smithi137.stdout:1/541: write d7/d9/d19/f2f [2062218,10545] 0 2024-03-14T13:02:19.049 INFO:tasks.workunit.client.1.smithi137.stdout:1/542: creat d7/d9/d19/d1f/d89/fac x:0 0 0 2024-03-14T13:02:19.052 INFO:tasks.workunit.client.1.smithi137.stdout:9/586: link d6/d26/l4d d6/df/lc1 0 2024-03-14T13:02:19.052 INFO:tasks.workunit.client.1.smithi137.stdout:1/543: chown d7/d9/d19/d1f/d22/l75 0 1 2024-03-14T13:02:19.055 INFO:tasks.workunit.client.1.smithi137.stdout:9/587: read d6/f69 [715320,115065] 0 2024-03-14T13:02:19.056 INFO:tasks.workunit.client.1.smithi137.stdout:1/544: creat d7/d9/fad x:0 0 0 2024-03-14T13:02:19.057 INFO:tasks.workunit.client.1.smithi137.stdout:9/588: stat d6/df/d19/d3a/da9/lbb 0 2024-03-14T13:02:19.059 INFO:tasks.workunit.client.1.smithi137.stdout:8/560: truncate d1/dc/d3c/d87/f1e 1238806 0 2024-03-14T13:02:19.063 INFO:tasks.workunit.client.1.smithi137.stdout:7/396: dwrite d3/d6/d2e/f3e [0,4194304] 0 2024-03-14T13:02:19.063 INFO:tasks.workunit.client.0.smithi111.stdout:0/679: rename d6/d9/de/d3a/dd1/fdb to d6/d9/d15/d55/fee 0 2024-03-14T13:02:19.064 INFO:tasks.workunit.client.0.smithi111.stdout:5/691: write d2/d16/d49/d4c/f73 [546200,100104] 0 2024-03-14T13:02:19.065 INFO:tasks.workunit.client.1.smithi137.stdout:7/397: write d3/d54/f5c [583910,104328] 0 2024-03-14T13:02:19.066 INFO:tasks.workunit.client.0.smithi111.stdout:0/680: write d6/d9/de/d3a/d46/f83 [224596,4763] 0 2024-03-14T13:02:19.069 INFO:tasks.workunit.client.0.smithi111.stdout:8/750: mknod d3/dc/d45/d8d/cf4 0 2024-03-14T13:02:19.069 INFO:tasks.workunit.client.0.smithi111.stdout:0/681: write d6/d9/d15/d49/fc2 [301226,104752] 0 2024-03-14T13:02:19.070 INFO:tasks.workunit.client.0.smithi111.stdout:1/578: dwrite d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:19.072 INFO:tasks.workunit.client.1.smithi137.stdout:9/589: dwrite d6/df/d19/d3a/f47 [0,4194304] 0 2024-03-14T13:02:19.073 INFO:tasks.workunit.client.0.smithi111.stdout:8/751: chown d3/d49/c60 23 1 2024-03-14T13:02:19.074 INFO:tasks.workunit.client.1.smithi137.stdout:0/455: truncate d1/d1f/f24 2857943 0 2024-03-14T13:02:19.080 INFO:tasks.workunit.client.0.smithi111.stdout:0/682: dwrite d6/d21/fa2 [0,4194304] 0 2024-03-14T13:02:19.081 INFO:tasks.workunit.client.0.smithi111.stdout:1/579: dwrite f1a [4194304,4194304] 0 2024-03-14T13:02:19.081 INFO:tasks.workunit.client.1.smithi137.stdout:8/561: rmdir d1/dc/d4e/d6d 39 2024-03-14T13:02:19.086 INFO:tasks.workunit.client.1.smithi137.stdout:9/590: creat d6/df/d19/d33/d8f/fc2 x:0 0 0 2024-03-14T13:02:19.087 INFO:tasks.workunit.client.1.smithi137.stdout:9/591: chown d6/df/l42 251260032 1 2024-03-14T13:02:19.088 INFO:tasks.workunit.client.0.smithi111.stdout:0/683: symlink d6/d9/d15/lef 0 2024-03-14T13:02:19.089 INFO:tasks.workunit.client.1.smithi137.stdout:0/456: creat d1/d5/d1b/d1e/d25/d55/d73/f98 x:0 0 0 2024-03-14T13:02:19.089 INFO:tasks.workunit.client.0.smithi111.stdout:1/580: dwrite d20/d25/d75/f5e [0,4194304] 0 2024-03-14T13:02:19.090 INFO:tasks.workunit.client.1.smithi137.stdout:0/457: chown d1/d44/d72/d7e 1 1 2024-03-14T13:02:19.091 INFO:tasks.workunit.client.1.smithi137.stdout:1/545: rename d7/d9/d19/d40/d42/f49 to d7/d9/d19/fae 0 2024-03-14T13:02:19.092 INFO:tasks.workunit.client.0.smithi111.stdout:1/581: stat d20 0 2024-03-14T13:02:19.092 INFO:tasks.workunit.client.0.smithi111.stdout:1/582: write d20/d25/f39 [1650938,59758] 0 2024-03-14T13:02:19.093 INFO:tasks.workunit.client.1.smithi137.stdout:9/592: symlink d6/df/d19/d33/d8f/lc3 0 2024-03-14T13:02:19.094 INFO:tasks.workunit.client.0.smithi111.stdout:0/684: unlink d6/f1e 0 2024-03-14T13:02:19.096 INFO:tasks.workunit.client.0.smithi111.stdout:0/685: creat d6/d9/d15/d55/da0/ff0 x:0 0 0 2024-03-14T13:02:19.097 INFO:tasks.workunit.client.0.smithi111.stdout:1/583: creat d20/d25/d31/fc6 x:0 0 0 2024-03-14T13:02:19.098 INFO:tasks.workunit.client.1.smithi137.stdout:9/593: creat d6/df/d71/fc4 x:0 0 0 2024-03-14T13:02:19.106 INFO:tasks.workunit.client.1.smithi137.stdout:8/562: getdents d1/dc/d3c/d87 0 2024-03-14T13:02:19.107 INFO:tasks.workunit.client.1.smithi137.stdout:1/546: link d7/d31/d54/f62 d7/d31/faf 0 2024-03-14T13:02:19.107 INFO:tasks.workunit.client.1.smithi137.stdout:1/547: stat d7/d29/l2c 0 2024-03-14T13:02:19.108 INFO:tasks.workunit.client.1.smithi137.stdout:8/563: chown d1/dc/d3c/d87/d34/d97/d13/c4c 61519 1 2024-03-14T13:02:19.110 INFO:tasks.workunit.client.1.smithi137.stdout:1/548: mknod d7/d31/d82/cb0 0 2024-03-14T13:02:19.111 INFO:tasks.workunit.client.1.smithi137.stdout:8/564: chown d1/dc/d3c/d87/d34/d97/d13/f3e 1246 1 2024-03-14T13:02:19.112 INFO:tasks.workunit.client.1.smithi137.stdout:8/565: write d1/dc/f1f [2932643,32081] 0 2024-03-14T13:02:19.114 INFO:tasks.workunit.client.1.smithi137.stdout:1/549: mkdir d7/d9/d19/d1f/d89/d90/db1 0 2024-03-14T13:02:19.116 INFO:tasks.workunit.client.1.smithi137.stdout:1/550: creat d7/d9/d19/d1f/d89/fb2 x:0 0 0 2024-03-14T13:02:19.118 INFO:tasks.workunit.client.1.smithi137.stdout:1/551: truncate d7/d4d/d6b/faa 654950 0 2024-03-14T13:02:19.118 INFO:tasks.workunit.client.1.smithi137.stdout:7/398: dread d3/d6/d14/f1a [0,4194304] 0 2024-03-14T13:02:19.118 INFO:tasks.workunit.client.1.smithi137.stdout:1/552: readlink d7/d4d/l73 0 2024-03-14T13:02:19.122 INFO:tasks.workunit.client.1.smithi137.stdout:9/594: dread d6/fe [0,4194304] 0 2024-03-14T13:02:19.123 INFO:tasks.workunit.client.1.smithi137.stdout:7/399: dread d3/f18 [0,4194304] 0 2024-03-14T13:02:19.125 INFO:tasks.workunit.client.1.smithi137.stdout:9/595: mknod d6/df/cc5 0 2024-03-14T13:02:19.127 INFO:tasks.workunit.client.1.smithi137.stdout:0/458: write d1/d44/f4c [13677,106977] 0 2024-03-14T13:02:19.132 INFO:tasks.workunit.client.1.smithi137.stdout:9/596: dwrite d6/df/f13 [0,4194304] 0 2024-03-14T13:02:19.132 INFO:tasks.workunit.client.1.smithi137.stdout:0/459: rename d1/d5/d1b/d1e/d25/d8a/c69 to d1/d5/d1b/c99 0 2024-03-14T13:02:19.135 INFO:tasks.workunit.client.1.smithi137.stdout:9/597: creat d6/d91/fc6 x:0 0 0 2024-03-14T13:02:19.136 INFO:tasks.workunit.client.0.smithi111.stdout:6/536: sync 2024-03-14T13:02:19.136 INFO:tasks.workunit.client.0.smithi111.stdout:4/859: sync 2024-03-14T13:02:19.136 INFO:tasks.workunit.client.0.smithi111.stdout:3/496: sync 2024-03-14T13:02:19.136 INFO:tasks.workunit.client.0.smithi111.stdout:7/457: sync 2024-03-14T13:02:19.139 INFO:tasks.workunit.client.0.smithi111.stdout:4/860: dread - d1d/d22/fd8 zero size 2024-03-14T13:02:19.139 INFO:tasks.workunit.client.0.smithi111.stdout:3/497: readlink d0/d2d/d35/d50/d5e/l2f 0 2024-03-14T13:02:19.142 INFO:tasks.workunit.client.1.smithi137.stdout:9/598: dwrite d6/d56/fb3 [0,4194304] 0 2024-03-14T13:02:19.144 INFO:tasks.workunit.client.0.smithi111.stdout:4/861: dread - d1d/d22/f119 zero size 2024-03-14T13:02:19.144 INFO:tasks.workunit.client.1.smithi137.stdout:9/599: unlink d6/f2b 0 2024-03-14T13:02:19.144 INFO:tasks.workunit.client.0.smithi111.stdout:4/862: readlink d1d/d22/l106 0 2024-03-14T13:02:19.145 INFO:tasks.workunit.client.1.smithi137.stdout:9/600: write d6/f69 [638738,31929] 0 2024-03-14T13:02:19.147 INFO:tasks.workunit.client.0.smithi111.stdout:6/537: fsync d0/d17/d21/d69/f39 0 2024-03-14T13:02:19.147 INFO:tasks.workunit.client.1.smithi137.stdout:9/601: creat d6/df/d19/d3a/da9/fc7 x:0 0 0 2024-03-14T13:02:19.149 INFO:tasks.workunit.client.0.smithi111.stdout:7/458: rename d0/d1/d5/l37 to d0/d1/d21/d2a/l8b 0 2024-03-14T13:02:19.151 INFO:tasks.workunit.client.0.smithi111.stdout:7/459: fdatasync d0/f2e 0 2024-03-14T13:02:19.152 INFO:tasks.workunit.client.1.smithi137.stdout:9/602: dwrite d6/f49 [0,4194304] 0 2024-03-14T13:02:19.154 INFO:tasks.workunit.client.1.smithi137.stdout:9/603: mkdir d6/d91/dc8 0 2024-03-14T13:02:19.157 INFO:tasks.workunit.client.1.smithi137.stdout:5/404: sync 2024-03-14T13:02:19.158 INFO:tasks.workunit.client.0.smithi111.stdout:7/460: write d0/d1/d21/d2a/f8a [1016579,9512] 0 2024-03-14T13:02:19.162 INFO:tasks.workunit.client.0.smithi111.stdout:3/498: dwrite d0/f23 [4194304,4194304] 0 2024-03-14T13:02:19.164 INFO:tasks.workunit.client.1.smithi137.stdout:5/405: truncate d7/d16/d18/d25/f5b 1199631 0 2024-03-14T13:02:19.165 INFO:tasks.workunit.client.0.smithi111.stdout:3/499: stat d0/d2d/d35/d50/d6c/da3 0 2024-03-14T13:02:19.166 INFO:tasks.workunit.client.0.smithi111.stdout:4/863: rename d1d/d22/d30/d5e/d6f/d7e/d66/l8f to d1d/d22/d30/d7c/d59/dc4/d123/l133 0 2024-03-14T13:02:19.166 INFO:tasks.workunit.client.0.smithi111.stdout:3/500: chown d0/d2d/d35/d50/d5e/d19/fb9 13875625 1 2024-03-14T13:02:19.167 INFO:tasks.workunit.client.0.smithi111.stdout:6/538: getdents d0/d17/d21/d69/d3d/d53/d81/da1/dc8 0 2024-03-14T13:02:19.170 INFO:tasks.workunit.client.0.smithi111.stdout:6/539: dread - d0/d17/d20/f9a zero size 2024-03-14T13:02:19.173 INFO:tasks.workunit.client.0.smithi111.stdout:0/686: dwrite d6/d21/fa2 [4194304,4194304] 0 2024-03-14T13:02:19.173 INFO:tasks.workunit.client.0.smithi111.stdout:4/864: unlink d1d/d22/d30/d87/f97 0 2024-03-14T13:02:19.175 INFO:tasks.workunit.client.0.smithi111.stdout:7/461: mkdir d0/d1/d21/d2a/d8c 0 2024-03-14T13:02:19.177 INFO:tasks.workunit.client.0.smithi111.stdout:3/501: chown d0/d2d/d35/d4c/la5 771029 1 2024-03-14T13:02:19.178 INFO:tasks.workunit.client.0.smithi111.stdout:3/502: chown d0/d2d/d35/d69 6145132 1 2024-03-14T13:02:19.178 INFO:tasks.workunit.client.1.smithi137.stdout:1/553: dread d7/d9/d19/d1f/f2a [0,4194304] 0 2024-03-14T13:02:19.181 INFO:tasks.workunit.client.1.smithi137.stdout:1/554: symlink d7/d9/d19/d1f/d26/d7d/lb3 0 2024-03-14T13:02:19.182 INFO:tasks.workunit.client.0.smithi111.stdout:4/865: truncate fe 3187452 0 2024-03-14T13:02:19.183 INFO:tasks.workunit.client.1.smithi137.stdout:1/555: rmdir d7/d9/d19/d1f/da0 39 2024-03-14T13:02:19.188 INFO:tasks.workunit.client.0.smithi111.stdout:3/503: fsync d0/d2d/d35/d69/d84/fb2 0 2024-03-14T13:02:19.188 INFO:tasks.workunit.client.0.smithi111.stdout:3/504: chown d0/d30/c52 5844 1 2024-03-14T13:02:19.189 INFO:tasks.workunit.client.0.smithi111.stdout:3/505: chown d0/d30/l8a 43345 1 2024-03-14T13:02:19.190 INFO:tasks.workunit.client.0.smithi111.stdout:4/866: rename d1d/d22/d40/d69/d86/db1 to d1d/d22/d30/d87/d134 0 2024-03-14T13:02:19.195 INFO:tasks.workunit.client.0.smithi111.stdout:3/506: rename d0/d2d/d35/d50/da2/db3/d71/la4 to d0/d2d/d35/d50/da2/db3/da7/lbd 0 2024-03-14T13:02:19.196 INFO:tasks.workunit.client.0.smithi111.stdout:4/867: truncate d1d/d22/d30/d5e/d6f/d7e/d66/f85 1205373 0 2024-03-14T13:02:19.198 INFO:tasks.workunit.client.0.smithi111.stdout:3/507: mknod d0/d2d/d35/d50/da2/db3/da7/cbe 0 2024-03-14T13:02:19.200 INFO:tasks.workunit.client.0.smithi111.stdout:4/868: mknod d1d/dfe/c135 0 2024-03-14T13:02:19.203 INFO:tasks.workunit.client.0.smithi111.stdout:6/540: dread d0/d17/d21/f22 [0,4194304] 0 2024-03-14T13:02:19.204 INFO:tasks.workunit.client.1.smithi137.stdout:0/460: truncate d1/d5/d1b/d1e/d25/d26/f2d 152839 0 2024-03-14T13:02:19.205 INFO:tasks.workunit.client.1.smithi137.stdout:0/461: fsync d1/d44/f4c 0 2024-03-14T13:02:19.208 INFO:tasks.workunit.client.1.smithi137.stdout:9/604: truncate d6/df/f92 4167887 0 2024-03-14T13:02:19.209 INFO:tasks.workunit.client.0.smithi111.stdout:4/869: rename d1d/d22/d30/d5e/d6f/d7e/d66/df4/df5 to d1d/d22/d30/d5e/d6f/d7e/d66/df4/d136 0 2024-03-14T13:02:19.211 INFO:tasks.workunit.client.1.smithi137.stdout:5/406: truncate d7/d16/d18/f39 2449872 0 2024-03-14T13:02:19.211 INFO:tasks.workunit.client.0.smithi111.stdout:4/870: write d1d/d22/d30/d5e/db8/ddf/f12e [528941,114198] 0 2024-03-14T13:02:19.213 INFO:tasks.workunit.client.1.smithi137.stdout:1/556: write d7/d9/f4b [1838955,18162] 0 2024-03-14T13:02:19.215 INFO:tasks.workunit.client.1.smithi137.stdout:0/462: dwrite d1/d5/d1b/d42/f4b [0,4194304] 0 2024-03-14T13:02:19.215 INFO:tasks.workunit.client.0.smithi111.stdout:3/508: dread d0/f1c [0,4194304] 0 2024-03-14T13:02:19.216 INFO:tasks.workunit.client.1.smithi137.stdout:1/557: stat d7/d9/d19/c52 0 2024-03-14T13:02:19.217 INFO:tasks.workunit.client.1.smithi137.stdout:5/407: creat d7/d16/d18/d25/d28/d3c/d76/f90 x:0 0 0 2024-03-14T13:02:19.219 INFO:tasks.workunit.client.0.smithi111.stdout:3/509: creat d0/d2d/d35/d50/d6c/fbf x:0 0 0 2024-03-14T13:02:19.225 INFO:tasks.workunit.client.1.smithi137.stdout:9/605: write d6/df/d19/d3a/f76 [318209,63285] 0 2024-03-14T13:02:19.226 INFO:tasks.workunit.client.1.smithi137.stdout:0/463: creat d1/d5/d1b/d1e/d53/d6d/f9a x:0 0 0 2024-03-14T13:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout:0/687: write d6/d9/de/f41 [1640185,57214] 0 2024-03-14T13:02:19.227 INFO:tasks.workunit.client.0.smithi111.stdout:7/462: rmdir d0/d1/d21/d5d 39 2024-03-14T13:02:19.228 INFO:tasks.workunit.client.0.smithi111.stdout:0/688: chown d6/d9/de/d77/f7c 0 1 2024-03-14T13:02:19.229 INFO:tasks.workunit.client.1.smithi137.stdout:5/408: fsync d7/d16/d18/f2a 0 2024-03-14T13:02:19.230 INFO:tasks.workunit.client.1.smithi137.stdout:0/464: dread d1/d1f/f3b [0,4194304] 0 2024-03-14T13:02:19.232 INFO:tasks.workunit.client.0.smithi111.stdout:7/463: creat d0/d1/d5/d38/d5c/f8d x:0 0 0 2024-03-14T13:02:19.233 INFO:tasks.workunit.client.0.smithi111.stdout:3/510: symlink d0/d2d/d35/d50/lc0 0 2024-03-14T13:02:19.234 INFO:tasks.workunit.client.0.smithi111.stdout:0/689: symlink d6/d9/de/d77/da4/dca/lf1 0 2024-03-14T13:02:19.235 INFO:tasks.workunit.client.1.smithi137.stdout:5/409: rmdir d7/d36 39 2024-03-14T13:02:19.235 INFO:tasks.workunit.client.0.smithi111.stdout:6/541: write d0/d17/d20/f30 [165451,114694] 0 2024-03-14T13:02:19.236 INFO:tasks.workunit.client.0.smithi111.stdout:7/464: rmdir d0/d1 39 2024-03-14T13:02:19.238 INFO:tasks.workunit.client.0.smithi111.stdout:3/511: symlink d0/d30/lc1 0 2024-03-14T13:02:19.239 INFO:tasks.workunit.client.1.smithi137.stdout:5/410: mknod d7/d16/d18/d48/c91 0 2024-03-14T13:02:19.239 INFO:tasks.workunit.client.0.smithi111.stdout:4/871: dread d1d/d22/d30/d7c/d59/fb7 [4194304,4194304] 0 2024-03-14T13:02:19.241 INFO:tasks.workunit.client.0.smithi111.stdout:0/690: mknod d6/d9/d15/d49/cf2 0 2024-03-14T13:02:19.242 INFO:tasks.workunit.client.1.smithi137.stdout:0/465: creat d1/f9b x:0 0 0 2024-03-14T13:02:19.242 INFO:tasks.workunit.client.1.smithi137.stdout:5/411: read - d7/d49/d75/f8c zero size 2024-03-14T13:02:19.243 INFO:tasks.workunit.client.0.smithi111.stdout:0/691: readlink d6/d9/l65 0 2024-03-14T13:02:19.245 INFO:tasks.workunit.client.1.smithi137.stdout:9/606: dwrite d6/df/d19/d1f/f95 [0,4194304] 0 2024-03-14T13:02:19.246 INFO:tasks.workunit.client.1.smithi137.stdout:5/412: write d7/d16/d18/f66 [2542579,45256] 0 2024-03-14T13:02:19.246 INFO:tasks.workunit.client.1.smithi137.stdout:1/558: dread d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:19.247 INFO:tasks.workunit.client.0.smithi111.stdout:6/542: dwrite d0/d17/d21/d69/d3d/f71 [4194304,4194304] 0 2024-03-14T13:02:19.249 INFO:tasks.workunit.client.0.smithi111.stdout:3/512: creat d0/d2d/d35/d50/fc2 x:0 0 0 2024-03-14T13:02:19.249 INFO:tasks.workunit.client.1.smithi137.stdout:0/466: unlink d1/d5/c2b 0 2024-03-14T13:02:19.253 INFO:tasks.workunit.client.0.smithi111.stdout:0/692: rename d6/d9/de/d3a/d46/d4a/f98 to d6/d9/de/d3a/d46/dde/ff3 0 2024-03-14T13:02:19.254 INFO:tasks.workunit.client.1.smithi137.stdout:1/559: creat d7/d9/d19/d1f/d26/d6c/fb4 x:0 0 0 2024-03-14T13:02:19.254 INFO:tasks.workunit.client.1.smithi137.stdout:5/413: mknod d7/d16/d18/d25/d28/d3c/c92 0 2024-03-14T13:02:19.255 INFO:tasks.workunit.client.1.smithi137.stdout:1/560: fsync d7/f9a 0 2024-03-14T13:02:19.255 INFO:tasks.workunit.client.0.smithi111.stdout:6/543: mknod d0/d17/d21/d69/ccd 0 2024-03-14T13:02:19.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/465: getdents d0/d1/d4a/d87 0 2024-03-14T13:02:19.256 INFO:tasks.workunit.client.0.smithi111.stdout:6/544: readlink d0/d17/d20/d4a/d59/lbf 0 2024-03-14T13:02:19.256 INFO:tasks.workunit.client.0.smithi111.stdout:4/872: write d1d/d22/d30/d87/fcb [1038545,73290] 0 2024-03-14T13:02:19.257 INFO:tasks.workunit.client.1.smithi137.stdout:9/607: dwrite d6/df/d19/d1f/f66 [0,4194304] 0 2024-03-14T13:02:19.259 INFO:tasks.workunit.client.0.smithi111.stdout:3/513: unlink d0/d2d/d35/d50/da2/db3/d71/f7c 0 2024-03-14T13:02:19.262 INFO:tasks.workunit.client.0.smithi111.stdout:0/693: creat d6/d9/d15/d49/ff4 x:0 0 0 2024-03-14T13:02:19.264 INFO:tasks.workunit.client.0.smithi111.stdout:4/873: unlink d1d/d22/d30/f3d 0 2024-03-14T13:02:19.265 INFO:tasks.workunit.client.1.smithi137.stdout:9/608: link d6/df/d19/d3a/lb9 d6/d53/lc9 0 2024-03-14T13:02:19.267 INFO:tasks.workunit.client.1.smithi137.stdout:1/561: rename d7/d9/d19/d1f/d26/d84/f8c to d7/d9/d19/d40/fb5 0 2024-03-14T13:02:19.268 INFO:tasks.workunit.client.0.smithi111.stdout:6/545: dwrite d0/f4 [0,4194304] 0 2024-03-14T13:02:19.268 INFO:tasks.workunit.client.1.smithi137.stdout:1/562: chown d7/d9/d19/d1f/d89/d90 92661884 1 2024-03-14T13:02:19.268 INFO:tasks.workunit.client.1.smithi137.stdout:9/609: mknod d6/df/d19/d3a/da9/cca 0 2024-03-14T13:02:19.270 INFO:tasks.workunit.client.0.smithi111.stdout:4/874: mknod d1d/d22/d23/c137 0 2024-03-14T13:02:19.270 INFO:tasks.workunit.client.0.smithi111.stdout:7/466: dwrite d0/d1/d5/f4c [0,4194304] 0 2024-03-14T13:02:19.270 INFO:tasks.workunit.client.1.smithi137.stdout:9/610: read d6/df/d71/f72 [40407,124644] 0 2024-03-14T13:02:19.272 INFO:tasks.workunit.client.1.smithi137.stdout:9/611: stat d6/c9 0 2024-03-14T13:02:19.273 INFO:tasks.workunit.client.0.smithi111.stdout:7/467: chown d0/d1/d5/d56 45 1 2024-03-14T13:02:19.273 INFO:tasks.workunit.client.0.smithi111.stdout:6/546: mknod d0/d17/d24/cce 0 2024-03-14T13:02:19.274 INFO:tasks.workunit.client.0.smithi111.stdout:0/694: truncate d6/d9/de/f3c 4935052 0 2024-03-14T13:02:19.276 INFO:tasks.workunit.client.0.smithi111.stdout:7/468: symlink d0/d1/d5/d38/d5c/d6d/l8e 0 2024-03-14T13:02:19.277 INFO:tasks.workunit.client.0.smithi111.stdout:6/547: dread - d0/d17/d20/d4a/d59/d80/f9c zero size 2024-03-14T13:02:19.278 INFO:tasks.workunit.client.1.smithi137.stdout:0/467: getdents d1/d5/d1b/d1e/d25 0 2024-03-14T13:02:19.278 INFO:tasks.workunit.client.1.smithi137.stdout:9/612: fsync d6/df/d19/d3a/da9/fc7 0 2024-03-14T13:02:19.278 INFO:tasks.workunit.client.0.smithi111.stdout:6/548: chown d0/d17/d99/fa9 17284 1 2024-03-14T13:02:19.279 INFO:tasks.workunit.client.0.smithi111.stdout:4/875: creat d1d/d22/d40/d69/f138 x:0 0 0 2024-03-14T13:02:19.279 INFO:tasks.workunit.client.1.smithi137.stdout:0/468: stat d1/f46 0 2024-03-14T13:02:19.280 INFO:tasks.workunit.client.0.smithi111.stdout:7/469: symlink d0/d11/l8f 0 2024-03-14T13:02:19.281 INFO:tasks.workunit.client.0.smithi111.stdout:6/549: truncate d0/d17/d20/d65/f92 1157864 0 2024-03-14T13:02:19.282 INFO:tasks.workunit.client.0.smithi111.stdout:0/695: write d6/d9/de/d77/da4/feb [1622402,18848] 0 2024-03-14T13:02:19.284 INFO:tasks.workunit.client.0.smithi111.stdout:4/876: symlink d1d/d22/d30/d5e/l139 0 2024-03-14T13:02:19.285 INFO:tasks.workunit.client.0.smithi111.stdout:0/696: chown d6/d9/de/d3a/fd9 377664169 1 2024-03-14T13:02:19.286 INFO:tasks.workunit.client.1.smithi137.stdout:1/563: dwrite d7/d9/d19/d40/fa8 [0,4194304] 0 2024-03-14T13:02:19.286 INFO:tasks.workunit.client.0.smithi111.stdout:4/877: fsync d1d/d22/d30/d87/fcb 0 2024-03-14T13:02:19.289 INFO:tasks.workunit.client.1.smithi137.stdout:5/414: dwrite d7/d16/d18/d25/f5b [0,4194304] 0 2024-03-14T13:02:19.289 INFO:tasks.workunit.client.0.smithi111.stdout:0/697: creat d6/d9/d5b/d9e/ff5 x:0 0 0 2024-03-14T13:02:19.290 INFO:tasks.workunit.client.0.smithi111.stdout:0/698: creat d6/d9/d15/d55/ff6 x:0 0 0 2024-03-14T13:02:19.291 INFO:tasks.workunit.client.0.smithi111.stdout:4/878: getdents d1d/d48 0 2024-03-14T13:02:19.293 INFO:tasks.workunit.client.0.smithi111.stdout:0/699: creat d6/d21/ff7 x:0 0 0 2024-03-14T13:02:19.294 INFO:tasks.workunit.client.1.smithi137.stdout:1/564: fsync d7/d9/d19/d1f/d22/f25 0 2024-03-14T13:02:19.298 INFO:tasks.workunit.client.0.smithi111.stdout:0/700: rename d6/d9/d15/d49/c4e to d6/d3d/cf8 0 2024-03-14T13:02:19.299 INFO:tasks.workunit.client.0.smithi111.stdout:2/526: sync 2024-03-14T13:02:19.299 INFO:tasks.workunit.client.1.smithi137.stdout:5/415: creat d7/d33/f93 x:0 0 0 2024-03-14T13:02:19.300 INFO:tasks.workunit.client.0.smithi111.stdout:4/879: dwrite d1d/d22/d30/d87/dd1/f120 [0,4194304] 0 2024-03-14T13:02:19.301 INFO:tasks.workunit.client.1.smithi137.stdout:1/565: mknod d7/d9/d19/d1f/d26/d6c/cb6 0 2024-03-14T13:02:19.301 INFO:tasks.workunit.client.0.smithi111.stdout:3/514: dread d0/fd [0,4194304] 0 2024-03-14T13:02:19.301 INFO:tasks.workunit.client.1.smithi137.stdout:5/416: write d7/d33/f93 [37866,23309] 0 2024-03-14T13:02:19.302 INFO:tasks.workunit.client.1.smithi137.stdout:9/613: creat d6/df/d19/d3a/fcb x:0 0 0 2024-03-14T13:02:19.302 INFO:tasks.workunit.client.0.smithi111.stdout:0/701: write d6/d9/de/d77/da4/fc5 [836153,99139] 0 2024-03-14T13:02:19.303 INFO:tasks.workunit.client.0.smithi111.stdout:4/880: dread - d1d/d48/dbe/f117 zero size 2024-03-14T13:02:19.303 INFO:tasks.workunit.client.1.smithi137.stdout:1/566: mkdir d7/d4d/db7 0 2024-03-14T13:02:19.303 INFO:tasks.workunit.client.0.smithi111.stdout:2/527: write dd/d17/d34/d73/f5e [1098705,84554] 0 2024-03-14T13:02:19.304 INFO:tasks.workunit.client.0.smithi111.stdout:0/702: readlink d6/d9/d15/d55/lc6 0 2024-03-14T13:02:19.305 INFO:tasks.workunit.client.1.smithi137.stdout:5/417: mknod d7/d1d/d74/d83/c94 0 2024-03-14T13:02:19.306 INFO:tasks.workunit.client.0.smithi111.stdout:3/515: symlink d0/d2d/d35/lc3 0 2024-03-14T13:02:19.307 INFO:tasks.workunit.client.0.smithi111.stdout:4/881: creat d1d/d22/d30/d5e/d6f/d7e/f13a x:0 0 0 2024-03-14T13:02:19.308 INFO:tasks.workunit.client.1.smithi137.stdout:1/567: read d7/d4d/f85 [10677,86830] 0 2024-03-14T13:02:19.309 INFO:tasks.workunit.client.1.smithi137.stdout:9/614: link d6/d53/f99 d6/df/d19/fcc 0 2024-03-14T13:02:19.309 INFO:tasks.workunit.client.0.smithi111.stdout:3/516: unlink d0/fd 0 2024-03-14T13:02:19.310 INFO:tasks.workunit.client.1.smithi137.stdout:5/418: creat d7/d36/f95 x:0 0 0 2024-03-14T13:02:19.310 INFO:tasks.workunit.client.1.smithi137.stdout:5/419: stat d7/d36/f95 0 2024-03-14T13:02:19.311 INFO:tasks.workunit.client.0.smithi111.stdout:4/882: creat d1d/d22/d30/d7c/f13b x:0 0 0 2024-03-14T13:02:19.313 INFO:tasks.workunit.client.0.smithi111.stdout:0/703: creat d6/d9/de/ff9 x:0 0 0 2024-03-14T13:02:19.315 INFO:tasks.workunit.client.0.smithi111.stdout:4/883: chown d1d/d22/d30/d87/dd1/l10a 732 1 2024-03-14T13:02:19.317 INFO:tasks.workunit.client.0.smithi111.stdout:0/704: rename d6/d9/de/d32/f33 to d6/d3d/ffa 0 2024-03-14T13:02:19.317 INFO:tasks.workunit.client.0.smithi111.stdout:4/884: mknod d1d/d48/c13c 0 2024-03-14T13:02:19.318 INFO:tasks.workunit.client.0.smithi111.stdout:3/517: dread d0/fc [0,4194304] 0 2024-03-14T13:02:19.320 INFO:tasks.workunit.client.1.smithi137.stdout:9/615: dwrite d6/d53/f99 [0,4194304] 0 2024-03-14T13:02:19.321 INFO:tasks.workunit.client.0.smithi111.stdout:0/705: rename d6/d9/d15/da9/lae to d6/lfb 0 2024-03-14T13:02:19.322 INFO:tasks.workunit.client.1.smithi137.stdout:5/420: dwrite d7/d16/d18/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:19.322 INFO:tasks.workunit.client.0.smithi111.stdout:4/885: mknod d1d/d22/d30/d87/dd1/de1/d12d/c13d 0 2024-03-14T13:02:19.323 INFO:tasks.workunit.client.0.smithi111.stdout:0/706: chown d6/d9/d15/d55/fda 2732 1 2024-03-14T13:02:19.324 INFO:tasks.workunit.client.0.smithi111.stdout:4/886: rmdir d1d/dfe 39 2024-03-14T13:02:19.329 INFO:tasks.workunit.client.0.smithi111.stdout:3/518: creat d0/d2d/d35/d69/fc4 x:0 0 0 2024-03-14T13:02:19.330 INFO:tasks.workunit.client.0.smithi111.stdout:4/887: write d1d/d22/d30/de5/fb6 [4768716,50902] 0 2024-03-14T13:02:19.330 INFO:tasks.workunit.client.1.smithi137.stdout:0/469: write d1/d5/d1b/d1e/d25/d55/f7a [104025,18787] 0 2024-03-14T13:02:19.333 INFO:tasks.workunit.client.1.smithi137.stdout:9/616: dread - d6/df/d19/d33/f65 zero size 2024-03-14T13:02:19.334 INFO:tasks.workunit.client.0.smithi111.stdout:0/707: creat d6/d9/ffc x:0 0 0 2024-03-14T13:02:19.334 INFO:tasks.workunit.client.0.smithi111.stdout:4/888: chown d1d/d22/d40/ld3 204663 1 2024-03-14T13:02:19.335 INFO:tasks.workunit.client.1.smithi137.stdout:0/470: link d1/d44/d56/f5f d1/d44/d56/f9c 0 2024-03-14T13:02:19.336 INFO:tasks.workunit.client.0.smithi111.stdout:3/519: chown d0/d30/f57 1343 1 2024-03-14T13:02:19.336 INFO:tasks.workunit.client.1.smithi137.stdout:5/421: dwrite d7/d16/d18/d25/d28/d3c/f77 [0,4194304] 0 2024-03-14T13:02:19.338 INFO:tasks.workunit.client.1.smithi137.stdout:0/471: chown d1/d5/d1b/d1e/d25/d55/d73/d83 2562 1 2024-03-14T13:02:19.338 INFO:tasks.workunit.client.1.smithi137.stdout:5/422: dread - d7/d49/d75/f8c zero size 2024-03-14T13:02:19.339 INFO:tasks.workunit.client.0.smithi111.stdout:0/708: creat d6/d9/de/d32/d64/de7/ffd x:0 0 0 2024-03-14T13:02:19.340 INFO:tasks.workunit.client.1.smithi137.stdout:9/617: write d6/df/d19/d3a/d40/f9e [1633153,122505] 0 2024-03-14T13:02:19.340 INFO:tasks.workunit.client.0.smithi111.stdout:3/520: creat d0/d2d/d35/d50/da2/db3/d6f/fc5 x:0 0 0 2024-03-14T13:02:19.341 INFO:tasks.workunit.client.0.smithi111.stdout:3/521: fsync d0/d2d/d35/d50/da2/db3/d6f/fb7 0 2024-03-14T13:02:19.342 INFO:tasks.workunit.client.0.smithi111.stdout:4/889: mkdir d1d/d22/d30/d13e 0 2024-03-14T13:02:19.342 INFO:tasks.workunit.client.1.smithi137.stdout:0/472: mkdir d1/d5/d1b/d1e/d25/d8a/d9d 0 2024-03-14T13:02:19.343 INFO:tasks.workunit.client.0.smithi111.stdout:2/528: truncate dd/d11/d1a/d56/f78 2301984 0 2024-03-14T13:02:19.344 INFO:tasks.workunit.client.0.smithi111.stdout:0/709: mkdir d6/d3d/da6/dfe 0 2024-03-14T13:02:19.345 INFO:tasks.workunit.client.1.smithi137.stdout:5/423: creat d7/f96 x:0 0 0 2024-03-14T13:02:19.345 INFO:tasks.workunit.client.0.smithi111.stdout:2/529: write dd/f4b [2456220,42870] 0 2024-03-14T13:02:19.346 INFO:tasks.workunit.client.1.smithi137.stdout:9/618: symlink d6/d53/db7/lcd 0 2024-03-14T13:02:19.347 INFO:tasks.workunit.client.1.smithi137.stdout:5/424: chown d7/d16/d18/d2d/f8f 2 1 2024-03-14T13:02:19.348 INFO:tasks.workunit.client.1.smithi137.stdout:5/425: fsync d7/d49/d75/f80 0 2024-03-14T13:02:19.348 INFO:tasks.workunit.client.1.smithi137.stdout:9/619: chown d6/df/d19/d1f/f35 236 1 2024-03-14T13:02:19.349 INFO:tasks.workunit.client.1.smithi137.stdout:5/426: chown d7/d1d/l32 1861071531 1 2024-03-14T13:02:19.349 INFO:tasks.workunit.client.1.smithi137.stdout:9/620: dread - d6/df/d19/d3a/fcb zero size 2024-03-14T13:02:19.349 INFO:tasks.workunit.client.0.smithi111.stdout:2/530: write dd/d17/d34/f42 [4607046,42700] 0 2024-03-14T13:02:19.351 INFO:tasks.workunit.client.1.smithi137.stdout:5/427: unlink d7/d16/d18/d25/f87 0 2024-03-14T13:02:19.351 INFO:tasks.workunit.client.0.smithi111.stdout:3/522: getdents d0/d2d/d35/d50/d6c 0 2024-03-14T13:02:19.352 INFO:tasks.workunit.client.1.smithi137.stdout:9/621: creat d6/df/d19/d1f/fce x:0 0 0 2024-03-14T13:02:19.352 INFO:tasks.workunit.client.1.smithi137.stdout:5/428: read - d7/d16/d18/d25/d28/d3c/f4f zero size 2024-03-14T13:02:19.353 INFO:tasks.workunit.client.0.smithi111.stdout:0/710: creat d6/d9/de/d77/da4/fff x:0 0 0 2024-03-14T13:02:19.353 INFO:tasks.workunit.client.0.smithi111.stdout:4/890: dwrite d1d/d22/f119 [0,4194304] 0 2024-03-14T13:02:19.355 INFO:tasks.workunit.client.1.smithi137.stdout:9/622: read d6/df/d19/d33/f61 [389575,61968] 0 2024-03-14T13:02:19.356 INFO:tasks.workunit.client.1.smithi137.stdout:6/519: sync 2024-03-14T13:02:19.356 INFO:tasks.workunit.client.0.smithi111.stdout:4/891: chown d1d/d48/dbe/f117 13 1 2024-03-14T13:02:19.357 INFO:tasks.workunit.client.0.smithi111.stdout:3/523: fsync d0/d2d/d35/d50/d5e/f97 0 2024-03-14T13:02:19.357 INFO:tasks.workunit.client.1.smithi137.stdout:5/429: symlink d7/d16/d18/d25/l97 0 2024-03-14T13:02:19.357 INFO:tasks.workunit.client.1.smithi137.stdout:9/623: fdatasync d6/df/d19/d3a/d40/fb6 0 2024-03-14T13:02:19.359 INFO:tasks.workunit.client.1.smithi137.stdout:6/520: symlink d2/d6/de/d45/lb7 0 2024-03-14T13:02:19.359 INFO:tasks.workunit.client.1.smithi137.stdout:9/624: dread - d6/df/d19/d33/d8f/fc2 zero size 2024-03-14T13:02:19.360 INFO:tasks.workunit.client.0.smithi111.stdout:4/892: rename d1d/d48/dbe to d1d/d22/d30/d87/d134/d13f 0 2024-03-14T13:02:19.363 INFO:tasks.workunit.client.0.smithi111.stdout:3/524: dread d0/d91/f9e [0,4194304] 0 2024-03-14T13:02:19.364 INFO:tasks.workunit.client.1.smithi137.stdout:9/625: dread d6/d53/f99 [0,4194304] 0 2024-03-14T13:02:19.364 INFO:tasks.workunit.client.0.smithi111.stdout:0/711: rename d6/d9/d15/d55/da0/ccd to d6/d9/de/d77/da4/dca/c100 0 2024-03-14T13:02:19.365 INFO:tasks.workunit.client.0.smithi111.stdout:4/893: symlink d1d/d22/d30/d5e/d6f/d7e/d66/l140 0 2024-03-14T13:02:19.366 INFO:tasks.workunit.client.0.smithi111.stdout:3/525: write d0/d2d/d35/d4c/d82/fab [892558,87065] 0 2024-03-14T13:02:19.367 INFO:tasks.workunit.client.1.smithi137.stdout:5/430: creat d7/d16/f98 x:0 0 0 2024-03-14T13:02:19.368 INFO:tasks.workunit.client.0.smithi111.stdout:0/712: unlink d6/d9/d15/d49/ff4 0 2024-03-14T13:02:19.369 INFO:tasks.workunit.client.1.smithi137.stdout:6/521: creat d2/d6/de/d45/d97/d9f/fb8 x:0 0 0 2024-03-14T13:02:19.370 INFO:tasks.workunit.client.0.smithi111.stdout:4/894: symlink d1d/d48/l141 0 2024-03-14T13:02:19.370 INFO:tasks.workunit.client.1.smithi137.stdout:0/473: dread d1/d5/f2a [0,4194304] 0 2024-03-14T13:02:19.370 INFO:tasks.workunit.client.0.smithi111.stdout:4/895: dread - d1d/d22/d30/d7c/f11f zero size 2024-03-14T13:02:19.372 INFO:tasks.workunit.client.1.smithi137.stdout:5/431: symlink d7/d16/d18/l99 0 2024-03-14T13:02:19.373 INFO:tasks.workunit.client.0.smithi111.stdout:0/713: chown d6/cc 11749 1 2024-03-14T13:02:19.374 INFO:tasks.workunit.client.0.smithi111.stdout:3/526: symlink d0/d2d/d35/d4c/db0/lc6 0 2024-03-14T13:02:19.376 INFO:tasks.workunit.client.1.smithi137.stdout:9/626: dwrite d6/df/d19/d1f/f35 [4194304,4194304] 0 2024-03-14T13:02:19.377 INFO:tasks.workunit.client.1.smithi137.stdout:5/432: creat d7/d1d/d74/d83/f9a x:0 0 0 2024-03-14T13:02:19.377 INFO:tasks.workunit.client.0.smithi111.stdout:0/714: symlink d6/d9/de/d3a/dd1/l101 0 2024-03-14T13:02:19.378 INFO:tasks.workunit.client.0.smithi111.stdout:4/896: dwrite d1d/d22/d30/d7c/d59/dc4/fdb [0,4194304] 0 2024-03-14T13:02:19.379 INFO:tasks.workunit.client.0.smithi111.stdout:0/715: stat d6/d3d/d97 0 2024-03-14T13:02:19.380 INFO:tasks.workunit.client.0.smithi111.stdout:3/527: truncate d0/d30/f48 684075 0 2024-03-14T13:02:19.380 INFO:tasks.workunit.client.0.smithi111.stdout:0/716: dread - d6/d3d/dba/fdd zero size 2024-03-14T13:02:19.381 INFO:tasks.workunit.client.1.smithi137.stdout:6/522: dwrite d2/d6/de/d45/d59/f40 [0,4194304] 0 2024-03-14T13:02:19.382 INFO:tasks.workunit.client.0.smithi111.stdout:3/528: rmdir d0/d2d/d35/d50/d5e 39 2024-03-14T13:02:19.388 INFO:tasks.workunit.client.0.smithi111.stdout:0/717: rename d6/d9/de/d3a/la1 to d6/d9/de/d3a/d46/d4a/l102 0 2024-03-14T13:02:19.389 INFO:tasks.workunit.client.1.smithi137.stdout:0/474: dread d1/d1f/f2f [0,4194304] 0 2024-03-14T13:02:19.389 INFO:tasks.workunit.client.0.smithi111.stdout:0/718: fdatasync d6/d9/de/d77/da4/fc5 0 2024-03-14T13:02:19.390 INFO:tasks.workunit.client.1.smithi137.stdout:5/433: rename d7/d16/d18/d25/d28/d3c/d76/l7f to d7/d16/d18/d25/d28/l9b 0 2024-03-14T13:02:19.392 INFO:tasks.workunit.client.1.smithi137.stdout:5/434: truncate d7/d33/d65/f84 29239 0 2024-03-14T13:02:19.392 INFO:tasks.workunit.client.0.smithi111.stdout:0/719: mknod d6/d9/de/d32/d64/de7/c103 0 2024-03-14T13:02:19.392 INFO:tasks.workunit.client.1.smithi137.stdout:0/475: creat d1/d5/d1b/d42/d3a/f9e x:0 0 0 2024-03-14T13:02:19.393 INFO:tasks.workunit.client.1.smithi137.stdout:5/435: fsync d7/d16/d18/f66 0 2024-03-14T13:02:19.395 INFO:tasks.workunit.client.0.smithi111.stdout:0/720: truncate d6/d9/de/f23 2939020 0 2024-03-14T13:02:19.397 INFO:tasks.workunit.client.1.smithi137.stdout:0/476: unlink d1/d5/d1b/d1e/l3f 0 2024-03-14T13:02:19.398 INFO:tasks.workunit.client.0.smithi111.stdout:0/721: rmdir d6/d3d/dba 39 2024-03-14T13:02:19.400 INFO:tasks.workunit.client.1.smithi137.stdout:0/477: read d1/d5/d1b/d1e/d25/d26/f41 [432912,55287] 0 2024-03-14T13:02:19.400 INFO:tasks.workunit.client.0.smithi111.stdout:0/722: rmdir d6/d9/d15/d49/d56 39 2024-03-14T13:02:19.401 INFO:tasks.workunit.client.0.smithi111.stdout:0/723: stat d6/d3d/fa7 0 2024-03-14T13:02:19.401 INFO:tasks.workunit.client.0.smithi111.stdout:3/529: write d0/d2d/d35/f61 [1249408,117117] 0 2024-03-14T13:02:19.403 INFO:tasks.workunit.client.0.smithi111.stdout:4/897: dread d1d/d22/d30/de5/f3e [0,4194304] 0 2024-03-14T13:02:19.405 INFO:tasks.workunit.client.0.smithi111.stdout:3/530: fsync d0/d2d/d35/d50/f74 0 2024-03-14T13:02:19.405 INFO:tasks.workunit.client.1.smithi137.stdout:0/478: dread d1/d44/d72/d7e/f78 [0,4194304] 0 2024-03-14T13:02:19.407 INFO:tasks.workunit.client.0.smithi111.stdout:3/531: mkdir d0/d2d/dc7 0 2024-03-14T13:02:19.408 INFO:tasks.workunit.client.1.smithi137.stdout:0/479: unlink d1/d5/d1b/d1e/d53/d6d/l71 0 2024-03-14T13:02:19.412 INFO:tasks.workunit.client.0.smithi111.stdout:3/532: rename d0/d30/l64 to d0/d2d/d35/d50/da2/db3/d6f/lc8 0 2024-03-14T13:02:19.413 INFO:tasks.workunit.client.1.smithi137.stdout:9/627: dread d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:19.415 INFO:tasks.workunit.client.0.smithi111.stdout:3/533: mknod d0/d2d/d35/d50/da2/db3/d71/cc9 0 2024-03-14T13:02:19.415 INFO:tasks.workunit.client.0.smithi111.stdout:4/898: dread d1d/d22/d30/d5e/d6f/fc3 [0,4194304] 0 2024-03-14T13:02:19.417 INFO:tasks.workunit.client.1.smithi137.stdout:0/480: dwrite d1/d44/f65 [0,4194304] 0 2024-03-14T13:02:19.417 INFO:tasks.workunit.client.1.smithi137.stdout:9/628: rmdir d6/df/d19/d33/d8f 39 2024-03-14T13:02:19.417 INFO:tasks.workunit.client.0.smithi111.stdout:0/724: dread d6/d3d/ffa [0,4194304] 0 2024-03-14T13:02:19.418 INFO:tasks.workunit.client.1.smithi137.stdout:5/436: dread d7/f15 [4194304,4194304] 0 2024-03-14T13:02:19.420 INFO:tasks.workunit.client.0.smithi111.stdout:3/534: rmdir d0/d2d/d35/d4c/d82 39 2024-03-14T13:02:19.421 INFO:tasks.workunit.client.1.smithi137.stdout:5/437: fdatasync d7/d16/d18/d25/d28/d3c/f4f 0 2024-03-14T13:02:19.422 INFO:tasks.workunit.client.1.smithi137.stdout:9/629: mkdir d6/df/d19/dcf 0 2024-03-14T13:02:19.423 INFO:tasks.workunit.client.0.smithi111.stdout:0/725: rmdir d6/d3d/dba 39 2024-03-14T13:02:19.423 INFO:tasks.workunit.client.1.smithi137.stdout:6/523: dread d2/d6/f34 [0,4194304] 0 2024-03-14T13:02:19.424 INFO:tasks.workunit.client.1.smithi137.stdout:9/630: write d6/df/f13 [3635165,130238] 0 2024-03-14T13:02:19.424 INFO:tasks.workunit.client.1.smithi137.stdout:5/438: truncate d7/d16/d18/f34 1222285 0 2024-03-14T13:02:19.425 INFO:tasks.workunit.client.0.smithi111.stdout:0/726: dread d6/d45/d87/fb5 [0,4194304] 0 2024-03-14T13:02:19.427 INFO:tasks.workunit.client.1.smithi137.stdout:5/439: write d7/f7b [2592606,127125] 0 2024-03-14T13:02:19.428 INFO:tasks.workunit.client.0.smithi111.stdout:0/727: mknod d6/d9/d5b/c104 0 2024-03-14T13:02:19.428 INFO:tasks.workunit.client.1.smithi137.stdout:0/481: mkdir d1/d5/d1b/d1e/d25/d9f 0 2024-03-14T13:02:19.429 INFO:tasks.workunit.client.0.smithi111.stdout:3/535: dwrite d0/f20 [0,4194304] 0 2024-03-14T13:02:19.429 INFO:tasks.workunit.client.1.smithi137.stdout:6/524: truncate d2/d54/d55/d6d/d76/f8f 657259 0 2024-03-14T13:02:19.431 INFO:tasks.workunit.client.1.smithi137.stdout:5/440: write d7/d16/d18/d25/d28/d3c/f6f [166141,27266] 0 2024-03-14T13:02:19.432 INFO:tasks.workunit.client.1.smithi137.stdout:6/525: rmdir d2/d6/d1d 39 2024-03-14T13:02:19.435 INFO:tasks.workunit.client.1.smithi137.stdout:0/482: write d1/d5/d1b/d1e/d25/d55/f67 [1187929,74973] 0 2024-03-14T13:02:19.437 INFO:tasks.workunit.client.1.smithi137.stdout:6/526: readlink d2/d6/de/d45/d59/d39/d3c/d83/l9e 0 2024-03-14T13:02:19.438 INFO:tasks.workunit.client.1.smithi137.stdout:9/631: truncate d6/df/d19/f45 1510067 0 2024-03-14T13:02:19.441 INFO:tasks.workunit.client.1.smithi137.stdout:5/441: dread d7/d16/d18/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:19.441 INFO:tasks.workunit.client.1.smithi137.stdout:0/483: rmdir d1/d5/d1b/d1e/d25/d55/d73/d83 0 2024-03-14T13:02:19.441 INFO:tasks.workunit.client.1.smithi137.stdout:6/527: mknod d2/d6/cb9 0 2024-03-14T13:02:19.442 INFO:tasks.workunit.client.0.smithi111.stdout:4/899: dread f11 [0,4194304] 0 2024-03-14T13:02:19.444 INFO:tasks.workunit.client.1.smithi137.stdout:6/528: write d2/f5a [2349356,126361] 0 2024-03-14T13:02:19.446 INFO:tasks.workunit.client.1.smithi137.stdout:6/529: read - d2/d6/de/d45/d97/d9f/fb1 zero size 2024-03-14T13:02:19.447 INFO:tasks.workunit.client.1.smithi137.stdout:6/530: readlink d2/d54/d55/d33/l7c 0 2024-03-14T13:02:19.447 INFO:tasks.workunit.client.1.smithi137.stdout:0/484: dread d1/d44/d56/f9c [0,4194304] 0 2024-03-14T13:02:19.448 INFO:tasks.workunit.client.1.smithi137.stdout:6/531: stat d2/d6/cb9 0 2024-03-14T13:02:19.448 INFO:tasks.workunit.client.0.smithi111.stdout:4/900: dread d1d/d22/d30/d7c/ff6 [0,4194304] 0 2024-03-14T13:02:19.449 INFO:tasks.workunit.client.1.smithi137.stdout:0/485: fsync d1/d5/d1b/d1e/d25/d8a/f14 0 2024-03-14T13:02:19.451 INFO:tasks.workunit.client.1.smithi137.stdout:6/532: write d2/d6/de/f5d [5005404,129638] 0 2024-03-14T13:02:19.451 INFO:tasks.workunit.client.1.smithi137.stdout:9/632: truncate d6/df/d19/d3a/d40/f5c 269199 0 2024-03-14T13:02:19.452 INFO:tasks.workunit.client.1.smithi137.stdout:5/442: truncate d7/d16/d18/d25/f46 5688291 0 2024-03-14T13:02:19.453 INFO:tasks.workunit.client.1.smithi137.stdout:0/486: creat d1/d5/d1b/d1e/d25/d55/fa0 x:0 0 0 2024-03-14T13:02:19.453 INFO:tasks.workunit.client.0.smithi111.stdout:4/901: creat d1d/d22/d30/f142 x:0 0 0 2024-03-14T13:02:19.453 INFO:tasks.workunit.client.1.smithi137.stdout:5/443: chown d7/d33/d65/f7c 324 1 2024-03-14T13:02:19.454 INFO:tasks.workunit.client.1.smithi137.stdout:6/533: write d2/d6/de/d45/f77 [335320,23448] 0 2024-03-14T13:02:19.456 INFO:tasks.workunit.client.0.smithi111.stdout:0/728: dwrite d6/d9/d5b/d9e/fa8 [0,4194304] 0 2024-03-14T13:02:19.456 INFO:tasks.workunit.client.1.smithi137.stdout:6/534: write d2/f25 [3239958,41160] 0 2024-03-14T13:02:19.456 INFO:tasks.workunit.client.1.smithi137.stdout:0/487: creat d1/d5/d1b/d42/fa1 x:0 0 0 2024-03-14T13:02:19.456 INFO:tasks.workunit.client.1.smithi137.stdout:5/444: chown d7/d16/d18/d25/d28/d3c/d76/f90 51634442 1 2024-03-14T13:02:19.457 INFO:tasks.workunit.client.0.smithi111.stdout:3/536: dwrite d0/d2d/d35/d50/d5e/d19/f1a [0,4194304] 0 2024-03-14T13:02:19.459 INFO:tasks.workunit.client.1.smithi137.stdout:0/488: write d1/d5/d1b/d42/d3a/f9e [795216,127267] 0 2024-03-14T13:02:19.459 INFO:tasks.workunit.client.1.smithi137.stdout:5/445: truncate d7/f96 9633 0 2024-03-14T13:02:19.460 INFO:tasks.workunit.client.0.smithi111.stdout:4/902: fdatasync d1d/d22/d30/d87/dd1/f120 0 2024-03-14T13:02:19.461 INFO:tasks.workunit.client.1.smithi137.stdout:5/446: readlink d7/d16/d18/l1f 0 2024-03-14T13:02:19.462 INFO:tasks.workunit.client.1.smithi137.stdout:6/535: fsync d2/d6/de/f13 0 2024-03-14T13:02:19.465 INFO:tasks.workunit.client.1.smithi137.stdout:5/447: mkdir d7/d33/d65/d9c 0 2024-03-14T13:02:19.465 INFO:tasks.workunit.client.0.smithi111.stdout:4/903: fsync d1d/d22/d30/d7c/fa1 0 2024-03-14T13:02:19.468 INFO:tasks.workunit.client.1.smithi137.stdout:9/633: truncate d6/d26/f93 681191 0 2024-03-14T13:02:19.468 INFO:tasks.workunit.client.0.smithi111.stdout:3/537: dwrite d0/d30/f95 [4194304,4194304] 0 2024-03-14T13:02:19.471 INFO:tasks.workunit.client.0.smithi111.stdout:4/904: mkdir d1d/d22/d30/d143 0 2024-03-14T13:02:19.471 INFO:tasks.workunit.client.0.smithi111.stdout:0/729: creat d6/d9/d15/da9/f105 x:0 0 0 2024-03-14T13:02:19.471 INFO:tasks.workunit.client.0.smithi111.stdout:4/905: readlink d1d/lf3 0 2024-03-14T13:02:19.472 INFO:tasks.workunit.client.1.smithi137.stdout:5/448: unlink d7/d16/d18/f20 0 2024-03-14T13:02:19.472 INFO:tasks.workunit.client.0.smithi111.stdout:3/538: symlink d0/d2d/d35/d50/d6c/d98/lca 0 2024-03-14T13:02:19.472 INFO:tasks.workunit.client.0.smithi111.stdout:3/539: stat d0/d2d/d35/d50/d5e 0 2024-03-14T13:02:19.473 INFO:tasks.workunit.client.1.smithi137.stdout:5/449: dread - d7/d16/d18/d25/d28/d3c/d76/f90 zero size 2024-03-14T13:02:19.474 INFO:tasks.workunit.client.0.smithi111.stdout:3/540: write d0/d2d/d35/d69/d84/f99 [751228,66936] 0 2024-03-14T13:02:19.474 INFO:tasks.workunit.client.1.smithi137.stdout:6/536: symlink d2/lba 0 2024-03-14T13:02:19.475 INFO:tasks.workunit.client.0.smithi111.stdout:4/906: getdents d1d/d22/d30/d7c/df8 0 2024-03-14T13:02:19.476 INFO:tasks.workunit.client.1.smithi137.stdout:6/537: stat d2/d54/d55/d6d/d76/d57/f9a 0 2024-03-14T13:02:19.476 INFO:tasks.workunit.client.1.smithi137.stdout:9/634: write d6/df/d19/d1f/f66 [4067167,54764] 0 2024-03-14T13:02:19.478 INFO:tasks.workunit.client.0.smithi111.stdout:3/541: mknod d0/d2d/d35/d50/da2/db3/ccb 0 2024-03-14T13:02:19.478 INFO:tasks.workunit.client.1.smithi137.stdout:5/450: mkdir d7/d16/d18/d48/d9d 0 2024-03-14T13:02:19.480 INFO:tasks.workunit.client.1.smithi137.stdout:9/635: write d6/df/d19/d1f/f35 [7985202,89859] 0 2024-03-14T13:02:19.481 INFO:tasks.workunit.client.1.smithi137.stdout:6/538: creat d2/d6/d86/d8a/fbb x:0 0 0 2024-03-14T13:02:19.482 INFO:tasks.workunit.client.1.smithi137.stdout:6/539: fsync d2/d6/de/fa6 0 2024-03-14T13:02:19.483 INFO:tasks.workunit.client.1.smithi137.stdout:5/451: mkdir d7/d16/d18/d25/d28/d9e 0 2024-03-14T13:02:19.484 INFO:tasks.workunit.client.1.smithi137.stdout:9/636: fdatasync d6/df/f10 0 2024-03-14T13:02:19.485 INFO:tasks.workunit.client.0.smithi111.stdout:4/907: dwrite d1d/d22/d30/d7c/f11f [0,4194304] 0 2024-03-14T13:02:19.486 INFO:tasks.workunit.client.0.smithi111.stdout:3/542: symlink d0/d2d/d35/d4c/lcc 0 2024-03-14T13:02:19.486 INFO:tasks.workunit.client.1.smithi137.stdout:9/637: write d6/df/d71/fa5 [4588960,73727] 0 2024-03-14T13:02:19.487 INFO:tasks.workunit.client.1.smithi137.stdout:5/452: creat d7/d16/d18/d25/d28/d3c/d76/f9f x:0 0 0 2024-03-14T13:02:19.489 INFO:tasks.workunit.client.0.smithi111.stdout:4/908: symlink d1d/d22/d30/d5e/d6f/d7e/d66/l144 0 2024-03-14T13:02:19.491 INFO:tasks.workunit.client.0.smithi111.stdout:3/543: unlink d0/d2d/d35/d4c/db0/lc6 0 2024-03-14T13:02:19.491 INFO:tasks.workunit.client.1.smithi137.stdout:9/638: mknod d6/d91/cd0 0 2024-03-14T13:02:19.491 INFO:tasks.workunit.client.0.smithi111.stdout:4/909: mkdir d1d/d22/d30/d7c/d145 0 2024-03-14T13:02:19.493 INFO:tasks.workunit.client.1.smithi137.stdout:5/453: write d7/d16/d18/f8a [548960,52556] 0 2024-03-14T13:02:19.494 INFO:tasks.workunit.client.0.smithi111.stdout:4/910: creat d1d/d48/f146 x:0 0 0 2024-03-14T13:02:19.495 INFO:tasks.workunit.client.0.smithi111.stdout:3/544: creat d0/d2d/d35/fcd x:0 0 0 2024-03-14T13:02:19.496 INFO:tasks.workunit.client.1.smithi137.stdout:0/489: truncate d1/d5/d1b/d1e/d25/d55/f60 1314968 0 2024-03-14T13:02:19.496 INFO:tasks.workunit.client.0.smithi111.stdout:4/911: mknod d1d/d22/d23/c147 0 2024-03-14T13:02:19.498 INFO:tasks.workunit.client.1.smithi137.stdout:5/454: creat d7/d16/d18/d25/d28/fa0 x:0 0 0 2024-03-14T13:02:19.498 INFO:tasks.workunit.client.0.smithi111.stdout:3/545: unlink d0/d2d/d35/d69/d84/f99 0 2024-03-14T13:02:19.498 INFO:tasks.workunit.client.1.smithi137.stdout:9/639: dread d6/f49 [0,4194304] 0 2024-03-14T13:02:19.500 INFO:tasks.workunit.client.0.smithi111.stdout:4/912: rmdir d1d/d22/d30/d87 39 2024-03-14T13:02:19.500 INFO:tasks.workunit.client.0.smithi111.stdout:4/913: readlink d1d/d22/d40/d69/l129 0 2024-03-14T13:02:19.501 INFO:tasks.workunit.client.1.smithi137.stdout:9/640: write d6/f6d [762832,111526] 0 2024-03-14T13:02:19.501 INFO:tasks.workunit.client.1.smithi137.stdout:5/455: symlink d7/d16/d18/d25/d28/la1 0 2024-03-14T13:02:19.504 INFO:tasks.workunit.client.0.smithi111.stdout:3/546: getdents d0 0 2024-03-14T13:02:19.504 INFO:tasks.workunit.client.1.smithi137.stdout:0/490: dwrite d1/d1f/d80/f86 [0,4194304] 0 2024-03-14T13:02:19.505 INFO:tasks.workunit.client.0.smithi111.stdout:3/547: symlink d0/d2d/d35/d50/d6c/lce 0 2024-03-14T13:02:19.507 INFO:tasks.workunit.client.1.smithi137.stdout:0/491: dread d1/d44/d72/d7e/f78 [0,4194304] 0 2024-03-14T13:02:19.507 INFO:tasks.workunit.client.0.smithi111.stdout:3/548: symlink d0/d2d/d35/d4c/db0/lcf 0 2024-03-14T13:02:19.508 INFO:tasks.workunit.client.0.smithi111.stdout:4/914: dread d1d/d22/d30/d7c/ff9 [0,4194304] 0 2024-03-14T13:02:19.511 INFO:tasks.workunit.client.1.smithi137.stdout:0/492: dread d1/d44/f65 [0,4194304] 0 2024-03-14T13:02:19.514 INFO:tasks.workunit.client.0.smithi111.stdout:3/549: getdents d0/d2d/d35/d4c 0 2024-03-14T13:02:19.515 INFO:tasks.workunit.client.0.smithi111.stdout:3/550: dread - d0/d2d/d35/d50/d5e/d19/f44 zero size 2024-03-14T13:02:19.515 INFO:tasks.workunit.client.0.smithi111.stdout:3/551: fsync d0/d2d/d35/d4c/d82/fab 0 2024-03-14T13:02:19.516 INFO:tasks.workunit.client.1.smithi137.stdout:5/456: link d7/d1d/d74/d83/f86 d7/fa2 0 2024-03-14T13:02:19.517 INFO:tasks.workunit.client.1.smithi137.stdout:5/457: chown d7/d16/f98 111184583 1 2024-03-14T13:02:19.518 INFO:tasks.workunit.client.1.smithi137.stdout:5/458: fdatasync d7/d33/d65/f7c 0 2024-03-14T13:02:19.519 INFO:tasks.workunit.client.1.smithi137.stdout:9/641: truncate d6/d56/fb3 2119555 0 2024-03-14T13:02:19.520 INFO:tasks.workunit.client.1.smithi137.stdout:9/642: chown d6/df/d71/c8e 6368622 1 2024-03-14T13:02:19.520 INFO:tasks.workunit.client.1.smithi137.stdout:0/493: link d1/d5/d1b/d1e/d25/d55/fa0 d1/d5/d1b/d1e/d25/d9f/fa2 0 2024-03-14T13:02:19.523 INFO:tasks.workunit.client.1.smithi137.stdout:0/494: read d1/d5/d1b/d42/f4f [3457718,106107] 0 2024-03-14T13:02:19.524 INFO:tasks.workunit.client.1.smithi137.stdout:9/643: chown d6/df/d19/d33/d8f/fbd 180873 1 2024-03-14T13:02:19.526 INFO:tasks.workunit.client.1.smithi137.stdout:5/459: dwrite d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:19.527 INFO:tasks.workunit.client.1.smithi137.stdout:5/460: dread - d7/d36/f95 zero size 2024-03-14T13:02:19.528 INFO:tasks.workunit.client.1.smithi137.stdout:9/644: creat d6/d53/db7/fd1 x:0 0 0 2024-03-14T13:02:19.531 INFO:tasks.workunit.client.1.smithi137.stdout:5/461: mkdir d7/d1d/d74/d83/da3 0 2024-03-14T13:02:19.532 INFO:tasks.workunit.client.1.smithi137.stdout:5/462: write d7/d1d/d74/f78 [692233,62334] 0 2024-03-14T13:02:19.535 INFO:tasks.workunit.client.1.smithi137.stdout:9/645: rename d6/df/d19/d3a/d40/l98 to d6/df/d88/ld2 0 2024-03-14T13:02:19.535 INFO:tasks.workunit.client.1.smithi137.stdout:5/463: chown d7/d1d/d74/d83/f9a 200 1 2024-03-14T13:02:19.536 INFO:tasks.workunit.client.1.smithi137.stdout:9/646: stat d6/df/lc1 0 2024-03-14T13:02:19.537 INFO:tasks.workunit.client.1.smithi137.stdout:0/495: link d1/d5/d1b/d1e/d25/d8a/l40 d1/d5/d1b/d1e/d25/d8a/la3 0 2024-03-14T13:02:19.543 INFO:tasks.workunit.client.1.smithi137.stdout:5/464: dwrite d7/d16/d18/d25/d28/d3c/f77 [0,4194304] 0 2024-03-14T13:02:19.545 INFO:tasks.workunit.client.1.smithi137.stdout:0/496: rename d1/d5/d1b/d1e/d25/d9f/fa2 to d1/d44/d79/d84/fa4 0 2024-03-14T13:02:19.550 INFO:tasks.workunit.client.1.smithi137.stdout:5/465: link d7/d49/d75/f80 d7/d49/d75/fa4 0 2024-03-14T13:02:19.555 INFO:tasks.workunit.client.1.smithi137.stdout:0/497: dwrite d1/d5/d1b/d42/f77 [0,4194304] 0 2024-03-14T13:02:19.557 INFO:tasks.workunit.client.1.smithi137.stdout:0/498: creat d1/d44/d79/d84/fa5 x:0 0 0 2024-03-14T13:02:19.559 INFO:tasks.workunit.client.1.smithi137.stdout:0/499: mkdir d1/da6 0 2024-03-14T13:02:19.560 INFO:tasks.workunit.client.1.smithi137.stdout:0/500: chown d1/d44/d79 376092 1 2024-03-14T13:02:19.569 INFO:tasks.workunit.client.1.smithi137.stdout:9/647: dwrite d6/d56/fb3 [0,4194304] 0 2024-03-14T13:02:19.571 INFO:tasks.workunit.client.0.smithi111.stdout:4/915: truncate d1d/d22/d30/d7c/ff9 846765 0 2024-03-14T13:02:19.571 INFO:tasks.workunit.client.0.smithi111.stdout:9/518: sync 2024-03-14T13:02:19.571 INFO:tasks.workunit.client.1.smithi137.stdout:5/466: dread d7/d1d/f57 [0,4194304] 0 2024-03-14T13:02:19.571 INFO:tasks.workunit.client.1.smithi137.stdout:9/648: read d6/df/d19/d33/f3e [411917,79055] 0 2024-03-14T13:02:19.573 INFO:tasks.workunit.client.1.smithi137.stdout:5/467: chown d7/d16/d18/d2d/c5d 248 1 2024-03-14T13:02:19.574 INFO:tasks.workunit.client.0.smithi111.stdout:9/519: mknod d5/d42/da4/cb3 0 2024-03-14T13:02:19.574 INFO:tasks.workunit.client.1.smithi137.stdout:9/649: symlink d6/d91/dc8/ld3 0 2024-03-14T13:02:19.575 INFO:tasks.workunit.client.0.smithi111.stdout:3/552: dwrite d0/d2d/f86 [0,4194304] 0 2024-03-14T13:02:19.579 INFO:tasks.workunit.client.0.smithi111.stdout:9/520: write d5/d6/d2a/d2d/f66 [812009,63357] 0 2024-03-14T13:02:19.580 INFO:tasks.workunit.client.1.smithi137.stdout:9/650: truncate d6/df/d19/d3a/d40/f7f 13144 0 2024-03-14T13:02:19.583 INFO:tasks.workunit.client.0.smithi111.stdout:4/916: dwrite d1d/d22/d30/d5e/d6f/f10d [0,4194304] 0 2024-03-14T13:02:19.584 INFO:tasks.workunit.client.1.smithi137.stdout:0/501: dread d1/f3e [0,4194304] 0 2024-03-14T13:02:19.585 INFO:tasks.workunit.client.0.smithi111.stdout:3/553: rename d0/f8 to d0/d2d/d35/d50/da2/db3/d71/dbc/fd0 0 2024-03-14T13:02:19.586 INFO:tasks.workunit.client.0.smithi111.stdout:9/521: creat d5/d42/d52/d83/fb4 x:0 0 0 2024-03-14T13:02:19.586 INFO:tasks.workunit.client.1.smithi137.stdout:0/502: chown d1/d5/d1b/d1e/d25/d8a 23114 1 2024-03-14T13:02:19.589 INFO:tasks.workunit.client.0.smithi111.stdout:4/917: fsync d1d/d22/d30/d87/dd1/f120 0 2024-03-14T13:02:19.593 INFO:tasks.workunit.client.0.smithi111.stdout:9/522: read d5/d6/d2a/d5a/f9f [1084137,32986] 0 2024-03-14T13:02:19.596 INFO:tasks.workunit.client.1.smithi137.stdout:0/503: dwrite d1/d5/d1b/d1e/d25/d55/f7a [0,4194304] 0 2024-03-14T13:02:19.596 INFO:tasks.workunit.client.0.smithi111.stdout:9/523: write d5/d6/d2a/d5a/fad [712418,94612] 0 2024-03-14T13:02:19.596 INFO:tasks.workunit.client.1.smithi137.stdout:4/483: sync 2024-03-14T13:02:19.596 INFO:tasks.workunit.client.1.smithi137.stdout:2/555: sync 2024-03-14T13:02:19.599 INFO:tasks.workunit.client.0.smithi111.stdout:4/918: dwrite d1d/d22/fd8 [0,4194304] 0 2024-03-14T13:02:19.599 INFO:tasks.workunit.client.0.smithi111.stdout:9/524: write d5/d6/d8/d5b/d87/f8c [382402,115903] 0 2024-03-14T13:02:19.600 INFO:tasks.workunit.client.1.smithi137.stdout:0/504: fdatasync d1/d44/d56/f9c 0 2024-03-14T13:02:19.603 INFO:tasks.workunit.client.1.smithi137.stdout:4/484: unlink f2 0 2024-03-14T13:02:19.605 INFO:tasks.workunit.client.1.smithi137.stdout:0/505: symlink d1/d44/d56/la7 0 2024-03-14T13:02:19.606 INFO:tasks.workunit.client.0.smithi111.stdout:3/554: getdents d0/d2d/d35/d50/d5e/d19 0 2024-03-14T13:02:19.606 INFO:tasks.workunit.client.1.smithi137.stdout:0/506: stat d1/d44/d56/f9c 0 2024-03-14T13:02:19.607 INFO:tasks.workunit.client.0.smithi111.stdout:3/555: write d0/d2d/d35/d50/da2/db3/d6f/fb7 [909272,101145] 0 2024-03-14T13:02:19.609 INFO:tasks.workunit.client.0.smithi111.stdout:3/556: dread - d0/d2d/d35/d69/f94 zero size 2024-03-14T13:02:19.611 INFO:tasks.workunit.client.1.smithi137.stdout:5/468: dread d7/d16/d18/f39 [0,4194304] 0 2024-03-14T13:02:19.612 INFO:tasks.workunit.client.1.smithi137.stdout:2/556: symlink d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/lb3 0 2024-03-14T13:02:19.613 INFO:tasks.workunit.client.1.smithi137.stdout:0/507: creat d1/d5/d1b/d1e/d25/fa8 x:0 0 0 2024-03-14T13:02:19.614 INFO:tasks.workunit.client.1.smithi137.stdout:4/485: dwrite d4/d10/d26/f6f [0,4194304] 0 2024-03-14T13:02:19.617 INFO:tasks.workunit.client.0.smithi111.stdout:3/557: dwrite d0/d2d/d35/d50/da2/db3/d71/fa6 [4194304,4194304] 0 2024-03-14T13:02:19.617 INFO:tasks.workunit.client.1.smithi137.stdout:5/469: symlink d7/d16/d18/d25/d28/d3c/d76/la5 0 2024-03-14T13:02:19.618 INFO:tasks.workunit.client.0.smithi111.stdout:3/558: chown d0/d2d/d35/d50/d5e/f37 282 1 2024-03-14T13:02:19.620 INFO:tasks.workunit.client.0.smithi111.stdout:3/559: fsync d0/d2d/d35/d50/d5e/d19/f26 0 2024-03-14T13:02:19.621 INFO:tasks.workunit.client.1.smithi137.stdout:0/508: stat d1/d5/d1b/d1e/d25/d8a/l40 0 2024-03-14T13:02:19.621 INFO:tasks.workunit.client.1.smithi137.stdout:9/651: dread d6/df/d19/d3a/d40/f41 [0,4194304] 0 2024-03-14T13:02:19.621 INFO:tasks.workunit.client.1.smithi137.stdout:2/557: write d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 [619901,105311] 0 2024-03-14T13:02:19.625 INFO:tasks.workunit.client.0.smithi111.stdout:3/560: creat d0/d2d/d35/d50/d6c/da3/daa/fd1 x:0 0 0 2024-03-14T13:02:19.626 INFO:tasks.workunit.client.0.smithi111.stdout:9/525: dread d5/d6/d2a/d5a/f76 [0,4194304] 0 2024-03-14T13:02:19.626 INFO:tasks.workunit.client.0.smithi111.stdout:3/561: write d0/d2d/d35/d50/f54 [261631,20863] 0 2024-03-14T13:02:19.628 INFO:tasks.workunit.client.0.smithi111.stdout:9/526: chown d5/d6/d8/d74/cab 2557 1 2024-03-14T13:02:19.629 INFO:tasks.workunit.client.1.smithi137.stdout:9/652: dread d6/f6d [0,4194304] 0 2024-03-14T13:02:19.629 INFO:tasks.workunit.client.1.smithi137.stdout:2/558: mkdir d2/d7/d2e/d3e/d44/d6b/d22/da5/db4 0 2024-03-14T13:02:19.633 INFO:tasks.workunit.client.1.smithi137.stdout:5/470: dwrite d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:19.633 INFO:tasks.workunit.client.1.smithi137.stdout:2/559: write d2/d7/db/d2f/f89 [1014566,37128] 0 2024-03-14T13:02:19.636 INFO:tasks.workunit.client.0.smithi111.stdout:3/562: dwrite d0/d2d/d35/d50/da2/db3/f47 [0,4194304] 0 2024-03-14T13:02:19.637 INFO:tasks.workunit.client.1.smithi137.stdout:0/509: dread d1/d5/d1b/d1e/d25/d26/f41 [0,4194304] 0 2024-03-14T13:02:19.640 INFO:tasks.workunit.client.1.smithi137.stdout:2/560: creat d2/d7/d2e/d3e/d44/d6b/fb5 x:0 0 0 2024-03-14T13:02:19.644 INFO:tasks.workunit.client.1.smithi137.stdout:2/561: fdatasync d2/d7/db/d2f/d66/f73 0 2024-03-14T13:02:19.644 INFO:tasks.workunit.client.1.smithi137.stdout:0/510: write d1/d44/d56/f5f [843977,127567] 0 2024-03-14T13:02:19.646 INFO:tasks.workunit.client.0.smithi111.stdout:9/527: dread d5/d6/f56 [0,4194304] 0 2024-03-14T13:02:19.646 INFO:tasks.workunit.client.1.smithi137.stdout:0/511: chown d1/d5/d1b/d42/d3a/c91 16 1 2024-03-14T13:02:19.648 INFO:tasks.workunit.client.1.smithi137.stdout:2/562: creat d2/d7/db/d74/fb6 x:0 0 0 2024-03-14T13:02:19.649 INFO:tasks.workunit.client.0.smithi111.stdout:4/919: dread d1d/fbb [0,4194304] 0 2024-03-14T13:02:19.649 INFO:tasks.workunit.client.0.smithi111.stdout:9/528: dread - d5/d6/d2a/d5a/f6f zero size 2024-03-14T13:02:19.650 INFO:tasks.workunit.client.0.smithi111.stdout:3/563: dwrite d0/d2d/d35/d50/d5e/d19/fb9 [0,4194304] 0 2024-03-14T13:02:19.651 INFO:tasks.workunit.client.0.smithi111.stdout:9/529: chown d5/d6/d14/f54 4069516 1 2024-03-14T13:02:19.652 INFO:tasks.workunit.client.0.smithi111.stdout:3/564: dread - d0/d2d/d35/d69/d84/fb2 zero size 2024-03-14T13:02:19.652 INFO:tasks.workunit.client.1.smithi137.stdout:0/512: dread d1/d5/d1b/d1e/d25/d55/d73/f8f [0,4194304] 0 2024-03-14T13:02:19.653 INFO:tasks.workunit.client.0.smithi111.stdout:4/920: creat d1d/d22/d30/d5e/d6f/f148 x:0 0 0 2024-03-14T13:02:19.658 INFO:tasks.workunit.client.1.smithi137.stdout:0/513: rename d1/d5/d1b/d1e/c2e to d1/d5/d1b/d1e/d25/d55/ca9 0 2024-03-14T13:02:19.659 INFO:tasks.workunit.client.1.smithi137.stdout:4/486: dread d4/d9/d3a/d41/f21 [0,4194304] 0 2024-03-14T13:02:19.661 INFO:tasks.workunit.client.1.smithi137.stdout:0/514: creat d1/d5/d1b/d1e/d53/d6d/faa x:0 0 0 2024-03-14T13:02:19.668 INFO:tasks.workunit.client.1.smithi137.stdout:9/653: dread d6/df/f20 [0,4194304] 0 2024-03-14T13:02:19.671 INFO:tasks.workunit.client.1.smithi137.stdout:9/654: read d6/f57 [579996,6599] 0 2024-03-14T13:02:19.671 INFO:tasks.workunit.client.0.smithi111.stdout:3/565: mknod d0/d2d/d35/d4c/dad/cd2 0 2024-03-14T13:02:19.672 INFO:tasks.workunit.client.0.smithi111.stdout:9/530: truncate d5/d6/ff 1815221 0 2024-03-14T13:02:19.672 INFO:tasks.workunit.client.0.smithi111.stdout:9/531: readlink d5/l4c 0 2024-03-14T13:02:19.673 INFO:tasks.workunit.client.1.smithi137.stdout:0/515: dwrite d1/d44/d79/d84/f8c [0,4194304] 0 2024-03-14T13:02:19.673 INFO:tasks.workunit.client.0.smithi111.stdout:4/921: mkdir d1d/d22/d30/d5e/d6f/d149 0 2024-03-14T13:02:19.674 INFO:tasks.workunit.client.0.smithi111.stdout:3/566: creat d0/d2d/d35/d50/da2/db3/d71/dbc/fd3 x:0 0 0 2024-03-14T13:02:19.675 INFO:tasks.workunit.client.1.smithi137.stdout:4/487: stat d4/d9/d3a/d41/d38/d55 0 2024-03-14T13:02:19.676 INFO:tasks.workunit.client.0.smithi111.stdout:9/532: unlink d5/d42/f79 0 2024-03-14T13:02:19.679 INFO:tasks.workunit.client.0.smithi111.stdout:9/533: mknod d5/d6/d8/d1a/cb5 0 2024-03-14T13:02:19.679 INFO:tasks.workunit.client.1.smithi137.stdout:9/655: unlink d6/d56/c87 0 2024-03-14T13:02:19.682 INFO:tasks.workunit.client.1.smithi137.stdout:2/563: dwrite d2/d7/db/f16 [0,4194304] 0 2024-03-14T13:02:19.682 INFO:tasks.workunit.client.1.smithi137.stdout:0/516: write d1/d5/d1b/d42/f4b [4033237,61878] 0 2024-03-14T13:02:19.682 INFO:tasks.workunit.client.0.smithi111.stdout:9/534: mkdir d5/d6/d8/d5b/d84/db6 0 2024-03-14T13:02:19.683 INFO:tasks.workunit.client.0.smithi111.stdout:4/922: dwrite d1d/d22/d30/d5e/d6f/d7e/fc8 [4194304,4194304] 0 2024-03-14T13:02:19.684 INFO:tasks.workunit.client.1.smithi137.stdout:5/471: dwrite d7/d49/d75/f80 [0,4194304] 0 2024-03-14T13:02:19.686 INFO:tasks.workunit.client.0.smithi111.stdout:9/535: fsync d5/d6/d8/f30 0 2024-03-14T13:02:19.687 INFO:tasks.workunit.client.1.smithi137.stdout:9/656: readlink d6/df/d88/ld2 0 2024-03-14T13:02:19.687 INFO:tasks.workunit.client.0.smithi111.stdout:4/923: creat d1d/d22/d30/d5e/d6f/f14a x:0 0 0 2024-03-14T13:02:19.688 INFO:tasks.workunit.client.0.smithi111.stdout:9/536: symlink d5/d6/d8/d74/lb7 0 2024-03-14T13:02:19.689 INFO:tasks.workunit.client.1.smithi137.stdout:0/517: creat d1/d5/d1b/d1e/d25/d26/fab x:0 0 0 2024-03-14T13:02:19.690 INFO:tasks.workunit.client.0.smithi111.stdout:4/924: symlink d1d/d22/l14b 0 2024-03-14T13:02:19.692 INFO:tasks.workunit.client.1.smithi137.stdout:5/472: mknod d7/d16/ca6 0 2024-03-14T13:02:19.693 INFO:tasks.workunit.client.0.smithi111.stdout:4/925: truncate d1d/d48/f146 984581 0 2024-03-14T13:02:19.695 INFO:tasks.workunit.client.0.smithi111.stdout:4/926: mknod d1d/d48/c14c 0 2024-03-14T13:02:19.695 INFO:tasks.workunit.client.0.smithi111.stdout:4/927: stat d1d/cd4 0 2024-03-14T13:02:19.696 INFO:tasks.workunit.client.0.smithi111.stdout:9/537: dwrite d5/d6/d8/d5b/d7c/f90 [0,4194304] 0 2024-03-14T13:02:19.699 INFO:tasks.workunit.client.1.smithi137.stdout:2/564: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/fb7 x:0 0 0 2024-03-14T13:02:19.699 INFO:tasks.workunit.client.1.smithi137.stdout:5/473: rename d7/l27 to d7/d33/d65/la7 0 2024-03-14T13:02:19.706 INFO:tasks.workunit.client.1.smithi137.stdout:0/518: getdents d1/d5/d1b/d1e/d25 0 2024-03-14T13:02:19.706 INFO:tasks.workunit.client.1.smithi137.stdout:2/565: rename d2/d7/db/d2f/cae to d2/d7/d2e/d3e/d44/d6b/d22/cb8 0 2024-03-14T13:02:19.706 INFO:tasks.workunit.client.0.smithi111.stdout:9/538: dwrite d5/d6/d8/d5b/d7c/f90 [4194304,4194304] 0 2024-03-14T13:02:19.707 INFO:tasks.workunit.client.0.smithi111.stdout:9/539: chown d5/d6/l62 1 1 2024-03-14T13:02:19.709 INFO:tasks.workunit.client.1.smithi137.stdout:0/519: write d1/d5/d1b/d1e/d25/d26/fab [900122,122186] 0 2024-03-14T13:02:19.709 INFO:tasks.workunit.client.1.smithi137.stdout:5/474: getdents d7/d16/d18/d25/d28/d3c 0 2024-03-14T13:02:19.710 INFO:tasks.workunit.client.0.smithi111.stdout:4/928: dread d1d/d22/d30/d7c/d59/dc4/d123/d115/f71 [0,4194304] 0 2024-03-14T13:02:19.710 INFO:tasks.workunit.client.1.smithi137.stdout:2/566: creat d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/fb9 x:0 0 0 2024-03-14T13:02:19.712 INFO:tasks.workunit.client.1.smithi137.stdout:2/567: read - d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/fb7 zero size 2024-03-14T13:02:19.714 INFO:tasks.workunit.client.1.smithi137.stdout:5/475: fdatasync d7/d16/d18/d25/d28/f3d 0 2024-03-14T13:02:19.716 INFO:tasks.workunit.client.0.smithi111.stdout:3/567: dread d0/f7 [0,4194304] 0 2024-03-14T13:02:19.716 INFO:tasks.workunit.client.1.smithi137.stdout:2/568: creat d2/d7/d2e/d3e/daa/fba x:0 0 0 2024-03-14T13:02:19.717 INFO:tasks.workunit.client.0.smithi111.stdout:4/929: symlink d1d/d22/d30/d7c/d59/dc4/d123/de6/d107/l14d 0 2024-03-14T13:02:19.717 INFO:tasks.workunit.client.1.smithi137.stdout:5/476: symlink d7/d1d/la8 0 2024-03-14T13:02:19.719 INFO:tasks.workunit.client.1.smithi137.stdout:2/569: write d2/d7/db/f2c [3597059,97351] 0 2024-03-14T13:02:19.720 INFO:tasks.workunit.client.1.smithi137.stdout:5/477: write d7/d16/d18/f66 [4361284,84823] 0 2024-03-14T13:02:19.721 INFO:tasks.workunit.client.1.smithi137.stdout:5/478: fsync d7/f1b 0 2024-03-14T13:02:19.722 INFO:tasks.workunit.client.0.smithi111.stdout:4/930: dread d1d/d22/d30/d87/fcb [0,4194304] 0 2024-03-14T13:02:19.724 INFO:tasks.workunit.client.1.smithi137.stdout:4/488: dread d4/d10/d5a/f63 [0,4194304] 0 2024-03-14T13:02:19.724 INFO:tasks.workunit.client.0.smithi111.stdout:4/931: symlink d1d/d105/l14e 0 2024-03-14T13:02:19.725 INFO:tasks.workunit.client.1.smithi137.stdout:2/570: mkdir d2/d7/d2e/d3e/d44/d6b/d22/dbb 0 2024-03-14T13:02:19.726 INFO:tasks.workunit.client.1.smithi137.stdout:4/489: write d4/d27/d5c/d6d/d58/f7e [1849,97201] 0 2024-03-14T13:02:19.727 INFO:tasks.workunit.client.0.smithi111.stdout:4/932: mkdir d1d/d22/d23/d14f 0 2024-03-14T13:02:19.727 INFO:tasks.workunit.client.1.smithi137.stdout:9/657: truncate d6/df/d19/f62 72946 0 2024-03-14T13:02:19.727 INFO:tasks.workunit.client.1.smithi137.stdout:4/490: readlink d4/d4d/d6e/la2 0 2024-03-14T13:02:19.728 INFO:tasks.workunit.client.0.smithi111.stdout:4/933: truncate d1d/d22/d40/d69/f138 19280 0 2024-03-14T13:02:19.730 INFO:tasks.workunit.client.1.smithi137.stdout:9/658: rmdir d6/d56 39 2024-03-14T13:02:19.730 INFO:tasks.workunit.client.1.smithi137.stdout:4/491: dread d4/d9/d3a/f56 [0,4194304] 0 2024-03-14T13:02:19.731 INFO:tasks.workunit.client.0.smithi111.stdout:4/934: unlink d1d/d22/f54 0 2024-03-14T13:02:19.732 INFO:tasks.workunit.client.1.smithi137.stdout:9/659: chown d6/df/d19/d3a/d40/f5c 26 1 2024-03-14T13:02:19.734 INFO:tasks.workunit.client.1.smithi137.stdout:4/492: mkdir d4/d27/d5c/d6d/d58/daa 0 2024-03-14T13:02:19.735 INFO:tasks.workunit.client.0.smithi111.stdout:3/568: write d0/d2d/d35/d50/d5e/f27 [1951634,13154] 0 2024-03-14T13:02:19.737 INFO:tasks.workunit.client.0.smithi111.stdout:4/935: dwrite d1d/d22/d30/d87/d134/fbf [0,4194304] 0 2024-03-14T13:02:19.738 INFO:tasks.workunit.client.0.smithi111.stdout:4/936: chown d1d/d22/d30/d87/d134 334 1 2024-03-14T13:02:19.738 INFO:tasks.workunit.client.0.smithi111.stdout:4/937: chown d1d/d22/d30/de5/fb0 1794 1 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:5/692: sync 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:1/584: sync 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:2/531: sync 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:7/470: sync 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:8/752: sync 2024-03-14T13:02:19.739 INFO:tasks.workunit.client.0.smithi111.stdout:6/550: sync 2024-03-14T13:02:19.740 INFO:tasks.workunit.client.0.smithi111.stdout:3/569: creat d0/d2d/d35/d50/d5e/fd4 x:0 0 0 2024-03-14T13:02:19.746 INFO:tasks.workunit.client.0.smithi111.stdout:7/471: chown d0/d1/d22/l2b 212 1 2024-03-14T13:02:19.747 INFO:tasks.workunit.client.0.smithi111.stdout:6/551: chown d0/d17/d21/d69/d3d/d53/d81/fa4 119 1 2024-03-14T13:02:19.748 INFO:tasks.workunit.client.0.smithi111.stdout:2/532: mkdir dd/d11/d1a/d56/dab 0 2024-03-14T13:02:19.753 INFO:tasks.workunit.client.0.smithi111.stdout:4/938: creat d1d/d22/d30/d5e/db8/ddf/de3/f150 x:0 0 0 2024-03-14T13:02:19.753 INFO:tasks.workunit.client.0.smithi111.stdout:8/753: mknod d3/d1b/d20/d35/d40/d48/d77/cf5 0 2024-03-14T13:02:19.756 INFO:tasks.workunit.client.0.smithi111.stdout:5/693: dwrite d2/d45/f20 [0,4194304] 0 2024-03-14T13:02:19.757 INFO:tasks.workunit.client.0.smithi111.stdout:2/533: symlink dd/d17/d34/d73/d75/d59/lac 0 2024-03-14T13:02:19.759 INFO:tasks.workunit.client.0.smithi111.stdout:5/694: creat d2/d16/d49/d4c/d9c/fe4 x:0 0 0 2024-03-14T13:02:19.761 INFO:tasks.workunit.client.0.smithi111.stdout:2/534: symlink dd/d17/d34/d74/lad 0 2024-03-14T13:02:19.761 INFO:tasks.workunit.client.0.smithi111.stdout:8/754: write d3/d1b/d20/d35/fb2 [1288571,53632] 0 2024-03-14T13:02:19.764 INFO:tasks.workunit.client.0.smithi111.stdout:5/695: read d2/d16/d49/d44/f8e [1267219,99100] 0 2024-03-14T13:02:19.764 INFO:tasks.workunit.client.0.smithi111.stdout:7/472: dwrite d0/d1/d21/f3f [0,4194304] 0 2024-03-14T13:02:19.765 INFO:tasks.workunit.client.0.smithi111.stdout:1/585: dread f3 [0,4194304] 0 2024-03-14T13:02:19.767 INFO:tasks.workunit.client.0.smithi111.stdout:1/586: chown d20/d21/d3f/d59/f76 0 1 2024-03-14T13:02:19.767 INFO:tasks.workunit.client.0.smithi111.stdout:8/755: mkdir d3/d1b/d20/d35/d40/df6 0 2024-03-14T13:02:19.768 INFO:tasks.workunit.client.0.smithi111.stdout:7/473: dread d0/d1/d5/d38/f7f [0,4194304] 0 2024-03-14T13:02:19.768 INFO:tasks.workunit.client.0.smithi111.stdout:2/535: mknod dd/d17/d34/d73/d75/d59/cae 0 2024-03-14T13:02:19.772 INFO:tasks.workunit.client.0.smithi111.stdout:7/474: mknod d0/d1/d21/d2a/d8c/c90 0 2024-03-14T13:02:19.773 INFO:tasks.workunit.client.0.smithi111.stdout:5/696: dread d2/d45/d76/fe3 [0,4194304] 0 2024-03-14T13:02:19.775 INFO:tasks.workunit.client.0.smithi111.stdout:8/756: creat d3/d1b/d20/ff7 x:0 0 0 2024-03-14T13:02:19.776 INFO:tasks.workunit.client.0.smithi111.stdout:2/536: creat dd/d17/d34/d73/faf x:0 0 0 2024-03-14T13:02:19.782 INFO:tasks.workunit.client.0.smithi111.stdout:2/537: write dd/d17/d34/d73/f5e [1132443,89387] 0 2024-03-14T13:02:19.782 INFO:tasks.workunit.client.0.smithi111.stdout:1/587: dwrite d20/d25/fc0 [0,4194304] 0 2024-03-14T13:02:19.784 INFO:tasks.workunit.client.0.smithi111.stdout:2/538: dread - dd/d17/d34/d73/d75/f8e zero size 2024-03-14T13:02:19.786 INFO:tasks.workunit.client.0.smithi111.stdout:5/697: dwrite d2/d16/d49/d7c/d95/d36/f3c [0,4194304] 0 2024-03-14T13:02:19.788 INFO:tasks.workunit.client.0.smithi111.stdout:5/698: fsync d2/d16/d49/d44/f50 0 2024-03-14T13:02:19.793 INFO:tasks.workunit.client.0.smithi111.stdout:8/757: dread d3/d1b/d20/f9e [0,4194304] 0 2024-03-14T13:02:19.793 INFO:tasks.workunit.client.0.smithi111.stdout:2/539: unlink dd/d17/d53/f8c 0 2024-03-14T13:02:19.793 INFO:tasks.workunit.client.0.smithi111.stdout:1/588: truncate d20/d21/f42 1630816 0 2024-03-14T13:02:19.793 INFO:tasks.workunit.client.0.smithi111.stdout:5/699: chown d2/d16/d49/d33/d68/fc3 14 1 2024-03-14T13:02:19.794 INFO:tasks.workunit.client.1.smithi137.stdout:7/400: sync 2024-03-14T13:02:19.794 INFO:tasks.workunit.client.0.smithi111.stdout:2/540: mknod dd/d11/d1a/cb0 0 2024-03-14T13:02:19.795 INFO:tasks.workunit.client.0.smithi111.stdout:5/700: mkdir d2/d16/d49/d33/de5 0 2024-03-14T13:02:19.795 INFO:tasks.workunit.client.0.smithi111.stdout:2/541: readlink dd/d17/d34/d73/d75/la8 0 2024-03-14T13:02:19.796 INFO:tasks.workunit.client.1.smithi137.stdout:1/568: sync 2024-03-14T13:02:19.797 INFO:tasks.workunit.client.0.smithi111.stdout:2/542: rmdir dd/d11 39 2024-03-14T13:02:19.799 INFO:tasks.workunit.client.1.smithi137.stdout:0/520: dwrite d1/d5/d1b/d1e/d25/d26/f39 [0,4194304] 0 2024-03-14T13:02:19.799 INFO:tasks.workunit.client.1.smithi137.stdout:3/456: sync 2024-03-14T13:02:19.805 INFO:tasks.workunit.client.0.smithi111.stdout:5/701: dwrite d2/d16/d49/d4c/f73 [0,4194304] 0 2024-03-14T13:02:19.806 INFO:tasks.workunit.client.0.smithi111.stdout:7/475: dread d0/d1/d21/d2a/f8a [0,4194304] 0 2024-03-14T13:02:19.807 INFO:tasks.workunit.client.1.smithi137.stdout:8/566: sync 2024-03-14T13:02:19.807 INFO:tasks.workunit.client.0.smithi111.stdout:2/543: dwrite f1 [0,4194304] 0 2024-03-14T13:02:19.807 INFO:tasks.workunit.client.1.smithi137.stdout:2/571: write d2/f12 [2140768,102310] 0 2024-03-14T13:02:19.810 INFO:tasks.workunit.client.1.smithi137.stdout:9/660: dread d6/df/d19/d3a/d40/f41 [0,4194304] 0 2024-03-14T13:02:19.812 INFO:tasks.workunit.client.1.smithi137.stdout:9/661: dread d6/df/d71/f72 [0,4194304] 0 2024-03-14T13:02:19.813 INFO:tasks.workunit.client.0.smithi111.stdout:5/702: rmdir d2/d45/d76/d9a/ddd 0 2024-03-14T13:02:19.814 INFO:tasks.workunit.client.1.smithi137.stdout:9/662: readlink d6/df/l42 0 2024-03-14T13:02:19.816 INFO:tasks.workunit.client.0.smithi111.stdout:5/703: read d2/d16/d49/d44/f8e [1541551,652] 0 2024-03-14T13:02:19.819 INFO:tasks.workunit.client.1.smithi137.stdout:0/521: symlink d1/d1f/d82/lac 0 2024-03-14T13:02:19.820 INFO:tasks.workunit.client.0.smithi111.stdout:2/544: dread dd/d11/d1a/f62 [0,4194304] 0 2024-03-14T13:02:19.821 INFO:tasks.workunit.client.0.smithi111.stdout:5/704: truncate d2/d16/d49/d7c/d95/fa3 88512 0 2024-03-14T13:02:19.822 INFO:tasks.workunit.client.1.smithi137.stdout:2/572: rename d2/d7/db to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc 0 2024-03-14T13:02:19.823 INFO:tasks.workunit.client.0.smithi111.stdout:4/939: stat d1d/d22/d30/d7c/d59/dc4/d123/c130 0 2024-03-14T13:02:19.824 INFO:tasks.workunit.client.1.smithi137.stdout:9/663: fsync d6/df/d19/d33/f80 0 2024-03-14T13:02:19.825 INFO:tasks.workunit.client.0.smithi111.stdout:5/705: creat d2/d16/d49/d44/d4a/fe6 x:0 0 0 2024-03-14T13:02:19.825 INFO:tasks.workunit.client.1.smithi137.stdout:3/457: dwrite d4/de/d16/f6a [0,4194304] 0 2024-03-14T13:02:19.825 INFO:tasks.workunit.client.0.smithi111.stdout:2/545: read dd/d17/d34/f42 [2504575,82070] 0 2024-03-14T13:02:19.827 INFO:tasks.workunit.client.1.smithi137.stdout:8/567: creat d1/d5e/daf/fc2 x:0 0 0 2024-03-14T13:02:19.828 INFO:tasks.workunit.client.0.smithi111.stdout:5/706: write d2/d16/d49/d44/d4a/f63 [3834709,97133] 0 2024-03-14T13:02:19.828 INFO:tasks.workunit.client.0.smithi111.stdout:4/940: dread - d1d/d22/d30/d87/d134/d13f/f117 zero size 2024-03-14T13:02:19.829 INFO:tasks.workunit.client.0.smithi111.stdout:2/546: mknod dd/d17/d34/d73/d75/cb1 0 2024-03-14T13:02:19.832 INFO:tasks.workunit.client.0.smithi111.stdout:4/941: symlink d1d/d22/d30/d87/d134/d13f/l151 0 2024-03-14T13:02:19.833 INFO:tasks.workunit.client.1.smithi137.stdout:0/522: symlink d1/d5/d1b/lad 0 2024-03-14T13:02:19.833 INFO:tasks.workunit.client.1.smithi137.stdout:7/401: getdents d3/d5/d23/d3f 0 2024-03-14T13:02:19.833 INFO:tasks.workunit.client.1.smithi137.stdout:2/573: symlink d2/d7/d2e/d3e/d44/d6b/d22/da5/lbd 0 2024-03-14T13:02:19.834 INFO:tasks.workunit.client.1.smithi137.stdout:9/664: write d6/df/d19/d33/f36 [44818,122854] 0 2024-03-14T13:02:19.834 INFO:tasks.workunit.client.1.smithi137.stdout:3/458: symlink d4/de/d93/l9a 0 2024-03-14T13:02:19.835 INFO:tasks.workunit.client.1.smithi137.stdout:7/402: chown d3/d54/c65 374601921 1 2024-03-14T13:02:19.837 INFO:tasks.workunit.client.1.smithi137.stdout:3/459: fdatasync d4/d22/f64 0 2024-03-14T13:02:19.838 INFO:tasks.workunit.client.0.smithi111.stdout:4/942: mkdir d1d/d22/d30/d5e/db8/ddf/d10f/d152 0 2024-03-14T13:02:19.839 INFO:tasks.workunit.client.1.smithi137.stdout:2/574: symlink d2/d7/d2e/d3e/d44/lbe 0 2024-03-14T13:02:19.839 INFO:tasks.workunit.client.1.smithi137.stdout:3/460: fsync d4/d22/f95 0 2024-03-14T13:02:19.839 INFO:tasks.workunit.client.1.smithi137.stdout:9/665: fsync d6/f49 0 2024-03-14T13:02:19.840 INFO:tasks.workunit.client.0.smithi111.stdout:3/570: truncate d0/d2d/d35/d50/d5e/f27 540316 0 2024-03-14T13:02:19.841 INFO:tasks.workunit.client.1.smithi137.stdout:8/568: dwrite d1/f49 [4194304,4194304] 0 2024-03-14T13:02:19.844 INFO:tasks.workunit.client.1.smithi137.stdout:9/666: mknod d6/d91/cd4 0 2024-03-14T13:02:19.844 INFO:tasks.workunit.client.0.smithi111.stdout:2/547: dwrite dd/d17/d34/d73/faf [0,4194304] 0 2024-03-14T13:02:19.845 INFO:tasks.workunit.client.1.smithi137.stdout:3/461: creat d4/de/d16/d74/f9b x:0 0 0 2024-03-14T13:02:19.847 INFO:tasks.workunit.client.1.smithi137.stdout:3/462: chown d4/de/d16/d1c/d5d/d4b/l94 4559502 1 2024-03-14T13:02:19.852 INFO:tasks.workunit.client.0.smithi111.stdout:2/548: creat dd/da9/fb2 x:0 0 0 2024-03-14T13:02:19.854 INFO:tasks.workunit.client.1.smithi137.stdout:8/569: dwrite d1/dc/d3c/d87/d34/f9a [0,4194304] 0 2024-03-14T13:02:19.854 INFO:tasks.workunit.client.0.smithi111.stdout:2/549: fsync dd/d11/d1a/f3d 0 2024-03-14T13:02:19.855 INFO:tasks.workunit.client.0.smithi111.stdout:4/943: rename d1d/d48/f146 to d1d/d105/f153 0 2024-03-14T13:02:19.856 INFO:tasks.workunit.client.0.smithi111.stdout:4/944: fsync d1d/d22/f44 0 2024-03-14T13:02:19.858 INFO:tasks.workunit.client.0.smithi111.stdout:3/571: creat d0/d2d/d35/d69/fd5 x:0 0 0 2024-03-14T13:02:19.861 INFO:tasks.workunit.client.0.smithi111.stdout:2/550: truncate dd/d17/d34/d4f/f66 84522 0 2024-03-14T13:02:19.862 INFO:tasks.workunit.client.1.smithi137.stdout:3/463: rename d4/d19/l7a to d4/l9c 0 2024-03-14T13:02:19.864 INFO:tasks.workunit.client.0.smithi111.stdout:5/707: dread d2/d13/f99 [0,4194304] 0 2024-03-14T13:02:19.865 INFO:tasks.workunit.client.1.smithi137.stdout:4/493: dread d4/d10/d42/f4e [4194304,4194304] 0 2024-03-14T13:02:19.865 INFO:tasks.workunit.client.1.smithi137.stdout:1/569: truncate d7/d4d/f56 3350482 0 2024-03-14T13:02:19.866 INFO:tasks.workunit.client.0.smithi111.stdout:3/572: creat d0/d2d/d35/d50/d6c/d98/fd6 x:0 0 0 2024-03-14T13:02:19.866 INFO:tasks.workunit.client.0.smithi111.stdout:2/551: truncate dd/f4c 966516 0 2024-03-14T13:02:19.867 INFO:tasks.workunit.client.1.smithi137.stdout:4/494: mknod d4/d9/d8e/d95/cab 0 2024-03-14T13:02:19.868 INFO:tasks.workunit.client.0.smithi111.stdout:8/758: dwrite d3/d5/fb [4194304,4194304] 0 2024-03-14T13:02:19.868 INFO:tasks.workunit.client.1.smithi137.stdout:8/570: symlink d1/dc/d3c/d87/d34/d86/lc3 0 2024-03-14T13:02:19.871 INFO:tasks.workunit.client.1.smithi137.stdout:1/570: mkdir d7/d9/d19/d1f/d89/d90/db8 0 2024-03-14T13:02:19.874 INFO:tasks.workunit.client.1.smithi137.stdout:8/571: readlink d1/dc/d3c/d87/d34/l53 0 2024-03-14T13:02:19.874 INFO:tasks.workunit.client.0.smithi111.stdout:1/589: dwrite d20/d21/d3f/f83 [0,4194304] 0 2024-03-14T13:02:19.874 INFO:tasks.workunit.client.0.smithi111.stdout:8/759: fdatasync d3/dc/d31/f4b 0 2024-03-14T13:02:19.876 INFO:tasks.workunit.client.0.smithi111.stdout:5/708: mknod d2/d16/d49/d33/de5/ce7 0 2024-03-14T13:02:19.876 INFO:tasks.workunit.client.0.smithi111.stdout:3/573: rename d0/d2d/d35 to d0/d2d/d35/d50/d6c/da3/dd7 22 2024-03-14T13:02:19.878 INFO:tasks.workunit.client.1.smithi137.stdout:1/571: dread - d7/d9/d19/d5f/f7b zero size 2024-03-14T13:02:19.878 INFO:tasks.workunit.client.1.smithi137.stdout:0/523: dwrite d1/d5/d1b/d1e/d25/d55/f7a [4194304,4194304] 0 2024-03-14T13:02:19.878 INFO:tasks.workunit.client.1.smithi137.stdout:8/572: dread - d1/dc/d3c/d87/d34/d97/d13/fab zero size 2024-03-14T13:02:19.881 INFO:tasks.workunit.client.1.smithi137.stdout:8/573: write d1/dc/d3c/d87/f48 [7698147,56086] 0 2024-03-14T13:02:19.882 INFO:tasks.workunit.client.0.smithi111.stdout:1/590: mknod d20/d21/d3f/cc7 0 2024-03-14T13:02:19.883 INFO:tasks.workunit.client.1.smithi137.stdout:4/495: dwrite d4/d10/d42/f52 [0,4194304] 0 2024-03-14T13:02:19.883 INFO:tasks.workunit.client.1.smithi137.stdout:0/524: write d1/f3e [4120525,13033] 0 2024-03-14T13:02:19.884 INFO:tasks.workunit.client.1.smithi137.stdout:8/574: chown d1/dc/d10/f32 0 1 2024-03-14T13:02:19.884 INFO:tasks.workunit.client.1.smithi137.stdout:1/572: symlink d7/d9/d19/d1f/d89/lb9 0 2024-03-14T13:02:19.885 INFO:tasks.workunit.client.0.smithi111.stdout:5/709: unlink d2/d45/c7e 0 2024-03-14T13:02:19.887 INFO:tasks.workunit.client.0.smithi111.stdout:3/574: dwrite d0/d2d/d35/d50/da2/db3/da7/fbb [0,4194304] 0 2024-03-14T13:02:19.888 INFO:tasks.workunit.client.1.smithi137.stdout:8/575: rmdir d1/d4a 39 2024-03-14T13:02:19.890 INFO:tasks.workunit.client.0.smithi111.stdout:8/760: rename d3/d49/c60 to d3/cf8 0 2024-03-14T13:02:19.891 INFO:tasks.workunit.client.1.smithi137.stdout:4/496: mkdir d4/d9/d3a/d41/d6a/dac 0 2024-03-14T13:02:19.891 INFO:tasks.workunit.client.0.smithi111.stdout:3/575: write d0/d2d/d35/d50/d5e/d19/f5a [136318,45118] 0 2024-03-14T13:02:19.892 INFO:tasks.workunit.client.0.smithi111.stdout:8/761: write d3/f65 [489032,112758] 0 2024-03-14T13:02:19.892 INFO:tasks.workunit.client.1.smithi137.stdout:1/573: creat d7/d9/d19/d40/d42/d9d/fba x:0 0 0 2024-03-14T13:02:19.895 INFO:tasks.workunit.client.1.smithi137.stdout:6/540: sync 2024-03-14T13:02:19.895 INFO:tasks.workunit.client.1.smithi137.stdout:8/576: mkdir d1/d4a/dc4 0 2024-03-14T13:02:19.896 INFO:tasks.workunit.client.1.smithi137.stdout:8/577: stat d1 0 2024-03-14T13:02:19.896 INFO:tasks.workunit.client.1.smithi137.stdout:4/497: fdatasync d4/f8 0 2024-03-14T13:02:19.898 INFO:tasks.workunit.client.1.smithi137.stdout:0/525: dwrite d1/d5/d1b/d42/f4f [4194304,4194304] 0 2024-03-14T13:02:19.899 INFO:tasks.workunit.client.0.smithi111.stdout:2/552: dread dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:19.902 INFO:tasks.workunit.client.0.smithi111.stdout:5/710: fsync d2/d45/f5c 0 2024-03-14T13:02:19.905 INFO:tasks.workunit.client.1.smithi137.stdout:1/574: rename d7/d9/c5c to d7/d9/d19/d1f/d89/d90/cbb 0 2024-03-14T13:02:19.905 INFO:tasks.workunit.client.1.smithi137.stdout:1/575: fsync d7/d31/f76 0 2024-03-14T13:02:19.906 INFO:tasks.workunit.client.1.smithi137.stdout:3/464: getdents d4/d19 0 2024-03-14T13:02:19.907 INFO:tasks.workunit.client.1.smithi137.stdout:3/465: chown d4/fd 1 1 2024-03-14T13:02:19.908 INFO:tasks.workunit.client.0.smithi111.stdout:2/553: unlink dd/d11/d1a/d56/f5b 0 2024-03-14T13:02:19.908 INFO:tasks.workunit.client.1.smithi137.stdout:4/498: creat d4/d9/d3a/d70/d80/d8d/fad x:0 0 0 2024-03-14T13:02:19.909 INFO:tasks.workunit.client.1.smithi137.stdout:4/499: read d4/f89 [98,9841] 0 2024-03-14T13:02:19.912 INFO:tasks.workunit.client.0.smithi111.stdout:8/762: mkdir d3/d1b/d20/df9 0 2024-03-14T13:02:19.913 INFO:tasks.workunit.client.0.smithi111.stdout:4/945: write d1d/d22/d30/d5e/f8d [149936,27575] 0 2024-03-14T13:02:19.914 INFO:tasks.workunit.client.0.smithi111.stdout:4/946: chown d1d/d48 51002 1 2024-03-14T13:02:19.914 INFO:tasks.workunit.client.1.smithi137.stdout:8/578: creat d1/dc/d3c/d87/d34/fc5 x:0 0 0 2024-03-14T13:02:19.915 INFO:tasks.workunit.client.0.smithi111.stdout:1/591: getdents d20/d25/d31 0 2024-03-14T13:02:19.917 INFO:tasks.workunit.client.0.smithi111.stdout:4/947: rmdir d1d/d22/d23 39 2024-03-14T13:02:19.918 INFO:tasks.workunit.client.1.smithi137.stdout:3/466: fdatasync d4/de/d16/d27/f73 0 2024-03-14T13:02:19.920 INFO:tasks.workunit.client.0.smithi111.stdout:1/592: dread d20/d25/fa1 [0,4194304] 0 2024-03-14T13:02:19.921 INFO:tasks.workunit.client.1.smithi137.stdout:8/579: creat d1/dc/d3c/d87/d34/fc6 x:0 0 0 2024-03-14T13:02:19.922 INFO:tasks.workunit.client.0.smithi111.stdout:1/593: truncate d20/d25/d31/f45 1311622 0 2024-03-14T13:02:19.922 INFO:tasks.workunit.client.1.smithi137.stdout:3/467: chown d4/de/d16/f31 25417 1 2024-03-14T13:02:19.922 INFO:tasks.workunit.client.1.smithi137.stdout:4/500: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:19.923 INFO:tasks.workunit.client.0.smithi111.stdout:8/763: rename d3/d1b/d20/f8e to d3/d22/d4a/ffa 0 2024-03-14T13:02:19.926 INFO:tasks.workunit.client.0.smithi111.stdout:1/594: creat d20/d21/d3f/d59/fc8 x:0 0 0 2024-03-14T13:02:19.927 INFO:tasks.workunit.client.1.smithi137.stdout:3/468: chown d4/de/d16/c7b 20234774 1 2024-03-14T13:02:19.927 INFO:tasks.workunit.client.0.smithi111.stdout:4/948: link d1d/d22/d30/d87/cea d1d/d22/d30/d7c/df8/c154 0 2024-03-14T13:02:19.927 INFO:tasks.workunit.client.1.smithi137.stdout:3/469: fsync d4/f18 0 2024-03-14T13:02:19.930 INFO:tasks.workunit.client.0.smithi111.stdout:4/949: write d1d/d22/d30/d5e/f84 [686252,88809] 0 2024-03-14T13:02:19.932 INFO:tasks.workunit.client.1.smithi137.stdout:4/501: dwrite d4/f13 [0,4194304] 0 2024-03-14T13:02:19.933 INFO:tasks.workunit.client.0.smithi111.stdout:1/595: getdents d20/d25/d75 0 2024-03-14T13:02:19.934 INFO:tasks.workunit.client.1.smithi137.stdout:3/470: read d4/de/d16/d1c/d5d/f43 [438235,5182] 0 2024-03-14T13:02:19.935 INFO:tasks.workunit.client.0.smithi111.stdout:1/596: write f3 [2012824,100575] 0 2024-03-14T13:02:19.938 INFO:tasks.workunit.client.1.smithi137.stdout:1/576: dread d7/d9/d19/d1f/d26/d6c/f6f [0,4194304] 0 2024-03-14T13:02:19.938 INFO:tasks.workunit.client.1.smithi137.stdout:3/471: symlink d4/d22/l9d 0 2024-03-14T13:02:19.938 INFO:tasks.workunit.client.1.smithi137.stdout:4/502: chown d4/f13 104757899 1 2024-03-14T13:02:19.946 INFO:tasks.workunit.client.1.smithi137.stdout:6/541: dwrite d2/d6/fc [4194304,4194304] 0 2024-03-14T13:02:19.946 INFO:tasks.workunit.client.1.smithi137.stdout:3/472: symlink d4/de/d16/d1c/d5d/d85/l9e 0 2024-03-14T13:02:19.949 INFO:tasks.workunit.client.1.smithi137.stdout:0/526: dwrite d1/d5/d1b/d1e/d25/d55/f60 [0,4194304] 0 2024-03-14T13:02:19.955 INFO:tasks.workunit.client.1.smithi137.stdout:6/542: symlink d2/d6/de/d45/d97/lbc 0 2024-03-14T13:02:19.957 INFO:tasks.workunit.client.0.smithi111.stdout:8/764: dread d3/d1b/dc1/de4/fe5 [0,4194304] 0 2024-03-14T13:02:19.958 INFO:tasks.workunit.client.0.smithi111.stdout:4/950: dread d1d/d22/d40/d69/d86/f10b [0,4194304] 0 2024-03-14T13:02:19.958 INFO:tasks.workunit.client.0.smithi111.stdout:8/765: fdatasync d3/d1b/d20/d35/fb2 0 2024-03-14T13:02:19.958 INFO:tasks.workunit.client.1.smithi137.stdout:1/577: link d7/d31/d82/c94 d7/d9/d19/d1f/da0/cbc 0 2024-03-14T13:02:19.959 INFO:tasks.workunit.client.1.smithi137.stdout:4/503: dwrite d4/d9/d3a/d41/f97 [4194304,4194304] 0 2024-03-14T13:02:19.960 INFO:tasks.workunit.client.0.smithi111.stdout:4/951: getdents d1d/d22/d30/d5e/d6f/d149 0 2024-03-14T13:02:19.960 INFO:tasks.workunit.client.1.smithi137.stdout:6/543: fdatasync d2/d6/de/d45/d97/d9f/fb8 0 2024-03-14T13:02:19.961 INFO:tasks.workunit.client.1.smithi137.stdout:3/473: dwrite d4/f18 [0,4194304] 0 2024-03-14T13:02:19.982 INFO:tasks.workunit.client.0.smithi111.stdout:0/730: sync 2024-03-14T13:02:19.987 INFO:tasks.workunit.client.0.smithi111.stdout:4/952: write d1d/d22/d30/d5e/d6f/d7e/d66/df4/d136/f121 [385597,23504] 0 2024-03-14T13:02:19.987 INFO:tasks.workunit.client.0.smithi111.stdout:0/731: write d6/d9/d15/d55/da0/ff0 [855377,30102] 0 2024-03-14T13:02:19.988 INFO:tasks.workunit.client.0.smithi111.stdout:8/766: dwrite d3/d1b/d20/ff7 [0,4194304] 0 2024-03-14T13:02:19.994 INFO:tasks.workunit.client.0.smithi111.stdout:1/597: dread f1a [4194304,4194304] 0 2024-03-14T13:02:19.995 INFO:tasks.workunit.client.0.smithi111.stdout:8/767: fsync d3/d22/f88 0 2024-03-14T13:02:19.996 INFO:tasks.workunit.client.0.smithi111.stdout:4/953: creat d1d/d22/d23/d14f/f155 x:0 0 0 2024-03-14T13:02:19.998 INFO:tasks.workunit.client.0.smithi111.stdout:4/954: symlink d1d/d48/l156 0 2024-03-14T13:02:20.000 INFO:tasks.workunit.client.0.smithi111.stdout:1/598: dwrite f3 [0,4194304] 0 2024-03-14T13:02:20.001 INFO:tasks.workunit.client.0.smithi111.stdout:1/599: stat d20/l95 0 2024-03-14T13:02:20.001 INFO:tasks.workunit.client.0.smithi111.stdout:1/600: readlink d20/d21/d3f/d59/l72 0 2024-03-14T13:02:20.009 INFO:tasks.workunit.client.0.smithi111.stdout:1/601: dwrite d20/d21/d28/da0/db6/fb9 [0,4194304] 0 2024-03-14T13:02:20.013 INFO:tasks.workunit.client.0.smithi111.stdout:1/602: mknod d20/d21/cc9 0 2024-03-14T13:02:20.027 INFO:tasks.workunit.client.1.smithi137.stdout:0/527: mkdir d1/d5/d1b/d42/d61/dae 0 2024-03-14T13:02:20.027 INFO:tasks.workunit.client.0.smithi111.stdout:1/603: dwrite d20/d25/d31/f3a [0,4194304] 0 2024-03-14T13:02:20.028 INFO:tasks.workunit.client.0.smithi111.stdout:1/604: readlink l1 0 2024-03-14T13:02:20.029 INFO:tasks.workunit.client.1.smithi137.stdout:4/504: mkdir d4/d9/d8e/dae 0 2024-03-14T13:02:20.029 INFO:tasks.workunit.client.0.smithi111.stdout:1/605: dread - d20/d25/f7b zero size 2024-03-14T13:02:20.030 INFO:tasks.workunit.client.0.smithi111.stdout:1/606: chown d20/d25/f3d 36 1 2024-03-14T13:02:20.030 INFO:tasks.workunit.client.0.smithi111.stdout:1/607: stat d20/d25/c68 0 2024-03-14T13:02:20.031 INFO:tasks.workunit.client.1.smithi137.stdout:1/578: dread d7/d4d/d6b/fa3 [0,4194304] 0 2024-03-14T13:02:20.032 INFO:tasks.workunit.client.1.smithi137.stdout:0/528: truncate d1/d5/d1b/d1e/d25/d55/d73/f98 744881 0 2024-03-14T13:02:20.032 INFO:tasks.workunit.client.1.smithi137.stdout:6/544: unlink d2/d54/d55/d33/fac 0 2024-03-14T13:02:20.032 INFO:tasks.workunit.client.1.smithi137.stdout:1/579: dread - d7/d9/fad zero size 2024-03-14T13:02:20.032 INFO:tasks.workunit.client.0.smithi111.stdout:1/608: getdents d20/d21/d28/d5c 0 2024-03-14T13:02:20.033 INFO:tasks.workunit.client.0.smithi111.stdout:1/609: write d20/d21/d3f/f83 [1585243,33540] 0 2024-03-14T13:02:20.034 INFO:tasks.workunit.client.0.smithi111.stdout:1/610: chown l1e 119 1 2024-03-14T13:02:20.037 INFO:tasks.workunit.client.0.smithi111.stdout:1/611: getdents d20/d21/d28 0 2024-03-14T13:02:20.038 INFO:tasks.workunit.client.1.smithi137.stdout:6/545: creat d2/d4c/fbd x:0 0 0 2024-03-14T13:02:20.039 INFO:tasks.workunit.client.1.smithi137.stdout:0/529: symlink d1/d5/d1b/d1e/d25/laf 0 2024-03-14T13:02:20.044 INFO:tasks.workunit.client.1.smithi137.stdout:6/546: mknod d2/d6/d1d/cbe 0 2024-03-14T13:02:20.046 INFO:tasks.workunit.client.1.smithi137.stdout:1/580: dwrite d7/d9/d19/d40/d9c/fa5 [0,4194304] 0 2024-03-14T13:02:20.048 INFO:tasks.workunit.client.1.smithi137.stdout:0/530: rename d1/d44/d79 to d1/d5/d1b/d42/d3a/db0 0 2024-03-14T13:02:20.052 INFO:tasks.workunit.client.1.smithi137.stdout:4/505: dwrite d4/d10/f3f [0,4194304] 0 2024-03-14T13:02:20.052 INFO:tasks.workunit.client.1.smithi137.stdout:6/547: write d2/d6/f5c [423156,125673] 0 2024-03-14T13:02:20.055 INFO:tasks.workunit.client.1.smithi137.stdout:3/474: dwrite d4/de/f1f [0,4194304] 0 2024-03-14T13:02:20.055 INFO:tasks.workunit.client.1.smithi137.stdout:0/531: mknod d1/d5/d1b/d42/d61/cb1 0 2024-03-14T13:02:20.063 INFO:tasks.workunit.client.1.smithi137.stdout:4/506: mkdir d4/d10/d42/daf 0 2024-03-14T13:02:20.065 INFO:tasks.workunit.client.1.smithi137.stdout:1/581: creat d7/d9/d19/d1f/d89/d90/db8/fbd x:0 0 0 2024-03-14T13:02:20.068 INFO:tasks.workunit.client.1.smithi137.stdout:1/582: creat d7/d9/d19/d40/d50/fbe x:0 0 0 2024-03-14T13:02:20.068 INFO:tasks.workunit.client.1.smithi137.stdout:3/475: dwrite d4/de/d16/f31 [0,4194304] 0 2024-03-14T13:02:20.072 INFO:tasks.workunit.client.1.smithi137.stdout:3/476: creat d4/de/d13/f9f x:0 0 0 2024-03-14T13:02:20.072 INFO:tasks.workunit.client.1.smithi137.stdout:5/479: sync 2024-03-14T13:02:20.072 INFO:tasks.workunit.client.1.smithi137.stdout:1/583: mknod d7/d9/d19/d1f/d89/d90/db8/cbf 0 2024-03-14T13:02:20.074 INFO:tasks.workunit.client.1.smithi137.stdout:3/477: write d4/f80 [77849,96683] 0 2024-03-14T13:02:20.076 INFO:tasks.workunit.client.1.smithi137.stdout:5/480: mknod d7/d16/d18/d25/ca9 0 2024-03-14T13:02:20.077 INFO:tasks.workunit.client.0.smithi111.stdout:1/612: dread d20/d21/d3f/f47 [0,4194304] 0 2024-03-14T13:02:20.078 INFO:tasks.workunit.client.0.smithi111.stdout:1/613: chown d20/d21/d28/l35 1664 1 2024-03-14T13:02:20.078 INFO:tasks.workunit.client.1.smithi137.stdout:1/584: creat d7/d9/d19/fc0 x:0 0 0 2024-03-14T13:02:20.078 INFO:tasks.workunit.client.0.smithi111.stdout:1/614: write d20/d25/f2e [4875418,28121] 0 2024-03-14T13:02:20.081 INFO:tasks.workunit.client.0.smithi111.stdout:1/615: mkdir d20/dca 0 2024-03-14T13:02:20.081 INFO:tasks.workunit.client.1.smithi137.stdout:3/478: mknod d4/df/ca0 0 2024-03-14T13:02:20.081 INFO:tasks.workunit.client.1.smithi137.stdout:1/585: symlink d7/d31/d82/lc1 0 2024-03-14T13:02:20.081 INFO:tasks.workunit.client.1.smithi137.stdout:4/507: dread d4/d27/f2b [0,4194304] 0 2024-03-14T13:02:20.082 INFO:tasks.workunit.client.1.smithi137.stdout:1/586: rmdir d7/d4d/d6b 39 2024-03-14T13:02:20.083 INFO:tasks.workunit.client.1.smithi137.stdout:3/479: read d4/d22/f95 [642525,5100] 0 2024-03-14T13:02:20.083 INFO:tasks.workunit.client.0.smithi111.stdout:1/616: mknod d20/d21/ccb 0 2024-03-14T13:02:20.086 INFO:tasks.workunit.client.0.smithi111.stdout:1/617: rename d20/d25/c2b to d20/d21/d28/da0/ccc 0 2024-03-14T13:02:20.086 INFO:tasks.workunit.client.1.smithi137.stdout:0/532: write d1/d44/f65 [3403719,34520] 0 2024-03-14T13:02:20.087 INFO:tasks.workunit.client.0.smithi111.stdout:1/618: readlink d20/d25/d87/l8d 0 2024-03-14T13:02:20.089 INFO:tasks.workunit.client.0.smithi111.stdout:1/619: symlink d20/d25/da8/lcd 0 2024-03-14T13:02:20.089 INFO:tasks.workunit.client.1.smithi137.stdout:4/508: creat d4/d9/d3a/d41/d6a/dac/fb0 x:0 0 0 2024-03-14T13:02:20.089 INFO:tasks.workunit.client.1.smithi137.stdout:0/533: dread - d1/d5/d1b/d42/fa1 zero size 2024-03-14T13:02:20.089 INFO:tasks.workunit.client.1.smithi137.stdout:6/548: truncate d2/d54/d55/d6d/fb0 573879 0 2024-03-14T13:02:20.090 INFO:tasks.workunit.client.0.smithi111.stdout:1/620: symlink d20/lce 0 2024-03-14T13:02:20.091 INFO:tasks.workunit.client.1.smithi137.stdout:1/587: link d7/d9/d19/fae d7/d9/d19/d1f/d26/d7d/fc2 0 2024-03-14T13:02:20.093 INFO:tasks.workunit.client.1.smithi137.stdout:0/534: symlink d1/d5/d1b/d42/d3a/lb2 0 2024-03-14T13:02:20.096 INFO:tasks.workunit.client.1.smithi137.stdout:4/509: rename d4/d27/d5c/d6d/d58/daa to d4/d27/db1 0 2024-03-14T13:02:20.097 INFO:tasks.workunit.client.1.smithi137.stdout:6/549: mkdir d2/d54/d55/d9c/dbf 0 2024-03-14T13:02:20.102 INFO:tasks.workunit.client.1.smithi137.stdout:5/481: write d7/d16/d18/d25/d28/f43 [2190142,63473] 0 2024-03-14T13:02:20.105 INFO:tasks.workunit.client.0.smithi111.stdout:1/621: dread d20/f57 [0,4194304] 0 2024-03-14T13:02:20.105 INFO:tasks.workunit.client.1.smithi137.stdout:6/550: mkdir d2/d54/d55/d9c/dc0 0 2024-03-14T13:02:20.107 INFO:tasks.workunit.client.0.smithi111.stdout:1/622: dread d20/d21/da7/fbe [0,4194304] 0 2024-03-14T13:02:20.107 INFO:tasks.workunit.client.1.smithi137.stdout:5/482: mknod d7/d49/caa 0 2024-03-14T13:02:20.108 INFO:tasks.workunit.client.1.smithi137.stdout:6/551: chown d2/d6/de/d45/d59 9009758 1 2024-03-14T13:02:20.110 INFO:tasks.workunit.client.1.smithi137.stdout:6/552: chown d2/d6/f34 48499 1 2024-03-14T13:02:20.111 INFO:tasks.workunit.client.1.smithi137.stdout:3/480: dwrite d4/de/f2a [0,4194304] 0 2024-03-14T13:02:20.111 INFO:tasks.workunit.client.0.smithi111.stdout:1/623: stat d20/d21/d28/da0/ccc 0 2024-03-14T13:02:20.113 INFO:tasks.workunit.client.1.smithi137.stdout:1/588: rename d7/d9/d19/d1f/d26/l4e to d7/d9/d19/d40/d58/lc3 0 2024-03-14T13:02:20.114 INFO:tasks.workunit.client.0.smithi111.stdout:1/624: rename d20/d21/d28/fc3 to d20/d21/d28/da0/fcf 0 2024-03-14T13:02:20.115 INFO:tasks.workunit.client.0.smithi111.stdout:1/625: mkdir d20/d25/dd0 0 2024-03-14T13:02:20.119 INFO:tasks.workunit.client.1.smithi137.stdout:1/589: truncate d7/f10 2335836 0 2024-03-14T13:02:20.121 INFO:tasks.workunit.client.0.smithi111.stdout:9/540: sync 2024-03-14T13:02:20.122 INFO:tasks.workunit.client.0.smithi111.stdout:6/552: sync 2024-03-14T13:02:20.122 INFO:tasks.workunit.client.0.smithi111.stdout:1/626: write d20/d25/f7b [74268,88843] 0 2024-03-14T13:02:20.123 INFO:tasks.workunit.client.0.smithi111.stdout:6/553: readlink d0/d17/d20/d94/daa/lac 0 2024-03-14T13:02:20.123 INFO:tasks.workunit.client.0.smithi111.stdout:9/541: dread - d5/d6/d8/d5b/d7c/f94 zero size 2024-03-14T13:02:20.124 INFO:tasks.workunit.client.1.smithi137.stdout:3/481: unlink d4/de/d16/d1c/c4a 0 2024-03-14T13:02:20.127 INFO:tasks.workunit.client.0.smithi111.stdout:6/554: unlink d0/d17/d20/f30 0 2024-03-14T13:02:20.129 INFO:tasks.workunit.client.1.smithi137.stdout:5/483: getdents d7/d36 0 2024-03-14T13:02:20.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/590: creat d7/d9/d19/d40/fc4 x:0 0 0 2024-03-14T13:02:20.134 INFO:tasks.workunit.client.0.smithi111.stdout:9/542: write d5/d6/d2a/f3e [2025104,103168] 0 2024-03-14T13:02:20.134 INFO:tasks.workunit.client.1.smithi137.stdout:0/535: dwrite d1/d5/d1b/d1e/d25/d8a/f21 [0,4194304] 0 2024-03-14T13:02:20.137 INFO:tasks.workunit.client.0.smithi111.stdout:1/627: dwrite d20/f24 [0,4194304] 0 2024-03-14T13:02:20.138 INFO:tasks.workunit.client.0.smithi111.stdout:9/543: mkdir d5/d6/d2a/d2d/db8 0 2024-03-14T13:02:20.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:19 smithi111 ceph-mon[39688]: pgmap v24: 65 pgs: 65 active+clean; 2.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 66 MiB/s rd, 306 MiB/s wr, 530 op/s 2024-03-14T13:02:20.142 INFO:tasks.workunit.client.1.smithi137.stdout:3/482: rename d4/d19/d5c to d4/d19/da1 0 2024-03-14T13:02:20.142 INFO:tasks.workunit.client.1.smithi137.stdout:5/484: rename d7/d33/d65/d9c to d7/d33/dab 0 2024-03-14T13:02:20.144 INFO:tasks.workunit.client.0.smithi111.stdout:1/628: write d20/d21/d28/d5c/f93 [120861,62515] 0 2024-03-14T13:02:20.145 INFO:tasks.workunit.client.1.smithi137.stdout:0/536: readlink d1/d1f/l6a 0 2024-03-14T13:02:20.147 INFO:tasks.workunit.client.1.smithi137.stdout:4/510: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:20.148 INFO:tasks.workunit.client.1.smithi137.stdout:5/485: stat d7/d16/d18/d25/c69 0 2024-03-14T13:02:20.148 INFO:tasks.workunit.client.0.smithi111.stdout:1/629: unlink d20/d25/l48 0 2024-03-14T13:02:20.148 INFO:tasks.workunit.client.0.smithi111.stdout:9/544: link d5/d6/l63 d5/d6/lb9 0 2024-03-14T13:02:20.149 INFO:tasks.workunit.client.1.smithi137.stdout:1/591: rename d7/d9/d19/d40/d58 to d7/d4d/d6b/dc5 0 2024-03-14T13:02:20.150 INFO:tasks.workunit.client.0.smithi111.stdout:9/545: write d5/d6/d8/d5b/d87/f8c [1359771,116031] 0 2024-03-14T13:02:20.150 INFO:tasks.workunit.client.0.smithi111.stdout:1/630: rmdir d20/d21/d3f/d59/daf 39 2024-03-14T13:02:20.151 INFO:tasks.workunit.client.1.smithi137.stdout:3/483: creat d4/d22/fa2 x:0 0 0 2024-03-14T13:02:20.153 INFO:tasks.workunit.client.0.smithi111.stdout:1/631: creat d20/d25/d31/d80/fd1 x:0 0 0 2024-03-14T13:02:20.154 INFO:tasks.workunit.client.1.smithi137.stdout:0/537: rmdir d1/d44/d56 39 2024-03-14T13:02:20.154 INFO:tasks.workunit.client.1.smithi137.stdout:5/486: truncate d7/d1d/d74/d83/f9a 400116 0 2024-03-14T13:02:20.157 INFO:tasks.workunit.client.0.smithi111.stdout:9/546: dwrite d5/d6/d8/f93 [0,4194304] 0 2024-03-14T13:02:20.159 INFO:tasks.workunit.client.1.smithi137.stdout:0/538: stat d1/d5/d1b/d1e/d25/d55/f67 0 2024-03-14T13:02:20.160 INFO:tasks.workunit.client.1.smithi137.stdout:4/511: chown d4/d9/d3a/d41/d38/d79/c8f 20986 1 2024-03-14T13:02:20.160 INFO:tasks.workunit.client.1.smithi137.stdout:3/484: creat d4/df/d82/fa3 x:0 0 0 2024-03-14T13:02:20.162 INFO:tasks.workunit.client.0.smithi111.stdout:9/547: link d5/d6/d8/f3a d5/d6/d14/d7a/fba 0 2024-03-14T13:02:20.166 INFO:tasks.workunit.client.1.smithi137.stdout:1/592: dread d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:20.166 INFO:tasks.workunit.client.0.smithi111.stdout:9/548: link d5/d6/l4d d5/d6/d8/d5b/d7c/lbb 0 2024-03-14T13:02:20.167 INFO:tasks.workunit.client.1.smithi137.stdout:5/487: rename d7/d33/d65/l6d to d7/d16/d18/d25/d28/d3c/d76/lac 0 2024-03-14T13:02:20.168 INFO:tasks.workunit.client.1.smithi137.stdout:3/485: mknod d4/de/d16/ca4 0 2024-03-14T13:02:20.169 INFO:tasks.workunit.client.0.smithi111.stdout:9/549: getdents d5/d6/d8/d5b/d84 0 2024-03-14T13:02:20.170 INFO:tasks.workunit.client.1.smithi137.stdout:1/593: creat d7/d9/d19/d1f/d89/d90/fc6 x:0 0 0 2024-03-14T13:02:20.170 INFO:tasks.workunit.client.0.smithi111.stdout:9/550: fsync d5/d6/d2a/f3e 0 2024-03-14T13:02:20.171 INFO:tasks.workunit.client.1.smithi137.stdout:4/512: read d4/d27/d5c/d6d/d58/d65/f7b [946172,122342] 0 2024-03-14T13:02:20.172 INFO:tasks.workunit.client.1.smithi137.stdout:1/594: chown d7/d9/d19/d1f/d26/d6c/l7c 1536260 1 2024-03-14T13:02:20.173 INFO:tasks.workunit.client.0.smithi111.stdout:9/551: mkdir d5/d6/d8/dbc 0 2024-03-14T13:02:20.173 INFO:tasks.workunit.client.1.smithi137.stdout:3/486: rename d4/d22/f95 to d4/d22/fa5 0 2024-03-14T13:02:20.173 INFO:tasks.workunit.client.1.smithi137.stdout:3/487: readlink d4/d22/l9d 0 2024-03-14T13:02:20.174 INFO:tasks.workunit.client.0.smithi111.stdout:9/552: mkdir d5/d42/d71/dbd 0 2024-03-14T13:02:20.174 INFO:tasks.workunit.client.1.smithi137.stdout:0/539: link d1/d5/d1b/d42/fa1 d1/fb3 0 2024-03-14T13:02:20.177 INFO:tasks.workunit.client.0.smithi111.stdout:9/553: creat d5/d6/d8/d5b/fbe x:0 0 0 2024-03-14T13:02:20.178 INFO:tasks.workunit.client.1.smithi137.stdout:3/488: rename d4/de/d16/d1c/d5d/d85/l84 to d4/de/d16/d1c/d5d/d85/d6b/la6 0 2024-03-14T13:02:20.180 INFO:tasks.workunit.client.1.smithi137.stdout:4/513: dread d4/d27/f28 [0,4194304] 0 2024-03-14T13:02:20.181 INFO:tasks.workunit.client.1.smithi137.stdout:3/489: write d4/de/f3c [1429742,78521] 0 2024-03-14T13:02:20.183 INFO:tasks.workunit.client.1.smithi137.stdout:3/490: write d4/d19/f23 [3950388,120388] 0 2024-03-14T13:02:20.183 INFO:tasks.workunit.client.0.smithi111.stdout:9/554: dwrite d5/d6/d8/d5b/d84/fa2 [0,4194304] 0 2024-03-14T13:02:20.184 INFO:tasks.workunit.client.1.smithi137.stdout:4/514: mknod d4/d27/cb2 0 2024-03-14T13:02:20.185 INFO:tasks.workunit.client.0.smithi111.stdout:9/555: chown d5/d42/d52/d83/f88 23 1 2024-03-14T13:02:20.185 INFO:tasks.workunit.client.0.smithi111.stdout:9/556: write d5/d6/d14/f54 [1143917,121126] 0 2024-03-14T13:02:20.186 INFO:tasks.workunit.client.1.smithi137.stdout:4/515: write d4/f2e [4293076,86788] 0 2024-03-14T13:02:20.186 INFO:tasks.workunit.client.1.smithi137.stdout:3/491: chown d4/de/d16/d1c/d5d/f7c 6305 1 2024-03-14T13:02:20.189 INFO:tasks.workunit.client.1.smithi137.stdout:3/492: write d4/d19/f54 [1950318,63925] 0 2024-03-14T13:02:20.189 INFO:tasks.workunit.client.1.smithi137.stdout:4/516: chown d4/d4d/f6b 836 1 2024-03-14T13:02:20.191 INFO:tasks.workunit.client.1.smithi137.stdout:1/595: write d7/d31/faf [124467,89475] 0 2024-03-14T13:02:20.194 INFO:tasks.workunit.client.0.smithi111.stdout:9/557: dread d5/d6/d2a/d5a/fad [0,4194304] 0 2024-03-14T13:02:20.195 INFO:tasks.workunit.client.1.smithi137.stdout:1/596: fsync d7/f88 0 2024-03-14T13:02:20.195 INFO:tasks.workunit.client.1.smithi137.stdout:4/517: unlink d4/f32 0 2024-03-14T13:02:20.195 INFO:tasks.workunit.client.1.smithi137.stdout:5/488: dread d7/d16/d18/d2d/f53 [0,4194304] 0 2024-03-14T13:02:20.197 INFO:tasks.workunit.client.1.smithi137.stdout:4/518: fdatasync d4/d4d/f6b 0 2024-03-14T13:02:20.197 INFO:tasks.workunit.client.1.smithi137.stdout:3/493: symlink d4/de/d16/d1c/la7 0 2024-03-14T13:02:20.197 INFO:tasks.workunit.client.0.smithi111.stdout:9/558: rename d5/d6/d2a/d2d/c3f to d5/d6/d2a/d5a/cbf 0 2024-03-14T13:02:20.198 INFO:tasks.workunit.client.0.smithi111.stdout:9/559: fsync d5/d6/d2a/d2d/f66 0 2024-03-14T13:02:20.198 INFO:tasks.workunit.client.1.smithi137.stdout:7/403: sync 2024-03-14T13:02:20.198 INFO:tasks.workunit.client.1.smithi137.stdout:2/575: sync 2024-03-14T13:02:20.199 INFO:tasks.workunit.client.1.smithi137.stdout:1/597: symlink d7/d9/d19/d5f/lc7 0 2024-03-14T13:02:20.199 INFO:tasks.workunit.client.1.smithi137.stdout:5/489: truncate d7/d49/d75/f8c 535325 0 2024-03-14T13:02:20.199 INFO:tasks.workunit.client.1.smithi137.stdout:3/494: readlink d4/de/d16/l8c 0 2024-03-14T13:02:20.199 INFO:tasks.workunit.client.0.smithi111.stdout:9/560: readlink d5/d6/d14/l72 0 2024-03-14T13:02:20.200 INFO:tasks.workunit.client.0.smithi111.stdout:9/561: truncate d5/d6/faa 576829 0 2024-03-14T13:02:20.201 INFO:tasks.workunit.client.1.smithi137.stdout:4/519: creat d4/d9/d3a/d41/d6a/fb3 x:0 0 0 2024-03-14T13:02:20.203 INFO:tasks.workunit.client.1.smithi137.stdout:7/404: fsync d3/d6/d14/f1a 0 2024-03-14T13:02:20.203 INFO:tasks.workunit.client.0.smithi111.stdout:9/562: link d5/d6/l62 d5/d42/lc0 0 2024-03-14T13:02:20.205 INFO:tasks.workunit.client.0.smithi111.stdout:9/563: mknod d5/d6/d8/cc1 0 2024-03-14T13:02:20.205 INFO:tasks.workunit.client.1.smithi137.stdout:7/405: fsync d3/d5/f37 0 2024-03-14T13:02:20.206 INFO:tasks.workunit.client.1.smithi137.stdout:3/495: chown d4/d19/da1/c91 7 1 2024-03-14T13:02:20.208 INFO:tasks.workunit.client.1.smithi137.stdout:5/490: link d7/d16/d18/f34 d7/d16/d18/d25/d28/fad 0 2024-03-14T13:02:20.208 INFO:tasks.workunit.client.0.smithi111.stdout:9/564: rmdir d5/d6/d8/dbc 0 2024-03-14T13:02:20.208 INFO:tasks.workunit.client.1.smithi137.stdout:0/540: dread d1/d5/d1b/d1e/d25/d8a/f14 [0,4194304] 0 2024-03-14T13:02:20.209 INFO:tasks.workunit.client.0.smithi111.stdout:9/565: stat d5/d6/d8/d5b/d84/l85 0 2024-03-14T13:02:20.209 INFO:tasks.workunit.client.1.smithi137.stdout:1/598: rename d7/d9/d19/d1f/d26/d84 to d7/d9/d19/d1f/d26/d6c/dc8 0 2024-03-14T13:02:20.210 INFO:tasks.workunit.client.0.smithi111.stdout:9/566: chown d5/d6/d8/c59 6482089 1 2024-03-14T13:02:20.210 INFO:tasks.workunit.client.1.smithi137.stdout:7/406: mkdir d3/d4a/d7d 0 2024-03-14T13:02:20.213 INFO:tasks.workunit.client.1.smithi137.stdout:4/520: dread d4/f48 [0,4194304] 0 2024-03-14T13:02:20.214 INFO:tasks.workunit.client.1.smithi137.stdout:1/599: write d7/d4d/d6b/f9e [743665,32715] 0 2024-03-14T13:02:20.215 INFO:tasks.workunit.client.1.smithi137.stdout:0/541: creat d1/d5/d1b/d1e/d25/d55/fb4 x:0 0 0 2024-03-14T13:02:20.215 INFO:tasks.workunit.client.1.smithi137.stdout:7/407: mknod d3/d54/c7e 0 2024-03-14T13:02:20.216 INFO:tasks.workunit.client.1.smithi137.stdout:5/491: creat d7/d16/fae x:0 0 0 2024-03-14T13:02:20.216 INFO:tasks.workunit.client.1.smithi137.stdout:4/521: chown d4/d9/d3a/d41/d6a 508 1 2024-03-14T13:02:20.218 INFO:tasks.workunit.client.1.smithi137.stdout:3/496: dwrite d4/f80 [4194304,4194304] 0 2024-03-14T13:02:20.220 INFO:tasks.workunit.client.1.smithi137.stdout:1/600: creat d7/d29/fc9 x:0 0 0 2024-03-14T13:02:20.221 INFO:tasks.workunit.client.1.smithi137.stdout:4/522: symlink d4/d27/lb4 0 2024-03-14T13:02:20.223 INFO:tasks.workunit.client.1.smithi137.stdout:2/576: dread d2/d7/d2e/d3e/d44/d6b/f3f [0,4194304] 0 2024-03-14T13:02:20.225 INFO:tasks.workunit.client.1.smithi137.stdout:5/492: creat d7/d16/d18/d25/d28/d3c/faf x:0 0 0 2024-03-14T13:02:20.225 INFO:tasks.workunit.client.1.smithi137.stdout:7/408: dread d3/d5/f46 [0,4194304] 0 2024-03-14T13:02:20.226 INFO:tasks.workunit.client.1.smithi137.stdout:4/523: creat d4/d10/d42/d73/fb5 x:0 0 0 2024-03-14T13:02:20.229 INFO:tasks.workunit.client.1.smithi137.stdout:3/497: dwrite d4/de/d16/d27/f65 [0,4194304] 0 2024-03-14T13:02:20.231 INFO:tasks.workunit.client.1.smithi137.stdout:0/542: getdents d1/d5/d1b/d1e/d53/d6d 0 2024-03-14T13:02:20.231 INFO:tasks.workunit.client.1.smithi137.stdout:1/601: creat d7/d4d/fca x:0 0 0 2024-03-14T13:02:20.232 INFO:tasks.workunit.client.1.smithi137.stdout:2/577: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f45 [903699,94307] 0 2024-03-14T13:02:20.233 INFO:tasks.workunit.client.1.smithi137.stdout:4/524: mkdir d4/d9/d3a/d41/d38/d55/db6 0 2024-03-14T13:02:20.237 INFO:tasks.workunit.client.1.smithi137.stdout:7/409: creat d3/d5/d5a/f7f x:0 0 0 2024-03-14T13:02:20.238 INFO:tasks.workunit.client.1.smithi137.stdout:1/602: mknod d7/d9/d19/d1f/d89/ccb 0 2024-03-14T13:02:20.239 INFO:tasks.workunit.client.1.smithi137.stdout:2/578: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 [378333,111091] 0 2024-03-14T13:02:20.242 INFO:tasks.workunit.client.1.smithi137.stdout:0/543: creat d1/d5/d1b/d1e/d25/d8a/fb5 x:0 0 0 2024-03-14T13:02:20.244 INFO:tasks.workunit.client.1.smithi137.stdout:3/498: dwrite d4/de/f1f [0,4194304] 0 2024-03-14T13:02:20.246 INFO:tasks.workunit.client.1.smithi137.stdout:7/410: mkdir d3/d5/d5a/d80 0 2024-03-14T13:02:20.248 INFO:tasks.workunit.client.1.smithi137.stdout:4/525: dread d4/d9/d3a/d41/d38/f3b [0,4194304] 0 2024-03-14T13:02:20.250 INFO:tasks.workunit.client.1.smithi137.stdout:5/493: getdents d7/d1d/d74/d81 0 2024-03-14T13:02:20.254 INFO:tasks.workunit.client.1.smithi137.stdout:0/544: unlink d1/f13 0 2024-03-14T13:02:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:19 smithi137 ceph-mon[50336]: pgmap v24: 65 pgs: 65 active+clean; 2.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 66 MiB/s rd, 306 MiB/s wr, 530 op/s 2024-03-14T13:02:20.256 INFO:tasks.workunit.client.1.smithi137.stdout:7/411: dwrite d3/d5/d23/d3f/f76 [0,4194304] 0 2024-03-14T13:02:20.264 INFO:tasks.workunit.client.1.smithi137.stdout:0/545: dwrite d1/d5/d1b/d1e/d25/d55/d73/f98 [0,4194304] 0 2024-03-14T13:02:20.266 INFO:tasks.workunit.client.1.smithi137.stdout:7/412: mkdir d3/d5/d23/d2c/d81 0 2024-03-14T13:02:20.270 INFO:tasks.workunit.client.1.smithi137.stdout:3/499: creat d4/fa8 x:0 0 0 2024-03-14T13:02:20.270 INFO:tasks.workunit.client.1.smithi137.stdout:5/494: dread d7/d16/f1a [4194304,4194304] 0 2024-03-14T13:02:20.272 INFO:tasks.workunit.client.1.smithi137.stdout:5/495: fdatasync d7/d1d/d74/d83/f9a 0 2024-03-14T13:02:20.273 INFO:tasks.workunit.client.1.smithi137.stdout:2/579: rename d2/d7/d2e/f47 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf 0 2024-03-14T13:02:20.273 INFO:tasks.workunit.client.1.smithi137.stdout:7/413: rmdir d3/d6/d64 39 2024-03-14T13:02:20.277 INFO:tasks.workunit.client.1.smithi137.stdout:5/496: creat d7/d33/d65/fb0 x:0 0 0 2024-03-14T13:02:20.279 INFO:tasks.workunit.client.1.smithi137.stdout:4/526: rename d4/d10/d3d to d4/d9/d3a/d70/d80/db7 0 2024-03-14T13:02:20.279 INFO:tasks.workunit.client.1.smithi137.stdout:1/603: dwrite d7/f53 [0,4194304] 0 2024-03-14T13:02:20.283 INFO:tasks.workunit.client.1.smithi137.stdout:4/527: read - d4/d9/d3a/d70/d80/fa6 zero size 2024-03-14T13:02:20.286 INFO:tasks.workunit.client.1.smithi137.stdout:1/604: truncate d7/d4d/fca 958760 0 2024-03-14T13:02:20.286 INFO:tasks.workunit.client.1.smithi137.stdout:5/497: creat d7/d33/fb1 x:0 0 0 2024-03-14T13:02:20.287 INFO:tasks.workunit.client.1.smithi137.stdout:3/500: dwrite d4/de/d16/d1c/d5d/f43 [0,4194304] 0 2024-03-14T13:02:20.288 INFO:tasks.workunit.client.1.smithi137.stdout:0/546: rename d1/d44/f93 to d1/d5/d1b/d1e/d25/d8a/d4a/fb6 0 2024-03-14T13:02:20.290 INFO:tasks.workunit.client.1.smithi137.stdout:0/547: truncate d1/f9b 508663 0 2024-03-14T13:02:20.291 INFO:tasks.workunit.client.1.smithi137.stdout:1/605: write d7/d9/d19/d1f/d26/d6c/fb4 [848464,70746] 0 2024-03-14T13:02:20.292 INFO:tasks.workunit.client.1.smithi137.stdout:3/501: truncate d4/d22/f8e 433621 0 2024-03-14T13:02:20.293 INFO:tasks.workunit.client.1.smithi137.stdout:1/606: fsync d7/f9a 0 2024-03-14T13:02:20.294 INFO:tasks.workunit.client.1.smithi137.stdout:5/498: creat d7/d49/d75/fb2 x:0 0 0 2024-03-14T13:02:20.294 INFO:tasks.workunit.client.1.smithi137.stdout:3/502: fdatasync d4/fa8 0 2024-03-14T13:02:20.294 INFO:tasks.workunit.client.1.smithi137.stdout:8/580: sync 2024-03-14T13:02:20.294 INFO:tasks.workunit.client.1.smithi137.stdout:9/667: sync 2024-03-14T13:02:20.296 INFO:tasks.workunit.client.0.smithi111.stdout:7/476: sync 2024-03-14T13:02:20.296 INFO:tasks.workunit.client.1.smithi137.stdout:5/499: fsync d7/d16/fae 0 2024-03-14T13:02:20.297 INFO:tasks.workunit.client.1.smithi137.stdout:9/668: write d6/fa3 [905424,124087] 0 2024-03-14T13:02:20.298 INFO:tasks.workunit.client.0.smithi111.stdout:7/477: readlink d0/d1/d5/d38/d5c/l86 0 2024-03-14T13:02:20.299 INFO:tasks.workunit.client.0.smithi111.stdout:7/478: write d0/d11/f73 [532553,66305] 0 2024-03-14T13:02:20.300 INFO:tasks.workunit.client.0.smithi111.stdout:7/479: symlink d0/d1/d5/d38/d5c/l91 0 2024-03-14T13:02:20.302 INFO:tasks.workunit.client.1.smithi137.stdout:7/414: getdents d3/d6/d64 0 2024-03-14T13:02:20.302 INFO:tasks.workunit.client.1.smithi137.stdout:4/528: getdents d4/d10/d5a 0 2024-03-14T13:02:20.302 INFO:tasks.workunit.client.1.smithi137.stdout:5/500: chown d7/d16/d18/d48 1610 1 2024-03-14T13:02:20.305 INFO:tasks.workunit.client.1.smithi137.stdout:1/607: mknod d7/d9/d19/d1f/d2e/d63/ccc 0 2024-03-14T13:02:20.307 INFO:tasks.workunit.client.1.smithi137.stdout:8/581: dread d1/dc/f5c [0,4194304] 0 2024-03-14T13:02:20.309 INFO:tasks.workunit.client.1.smithi137.stdout:5/501: creat d7/d49/d75/fb3 x:0 0 0 2024-03-14T13:02:20.316 INFO:tasks.workunit.client.1.smithi137.stdout:2/580: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f1c [0,4194304] 0 2024-03-14T13:02:20.316 INFO:tasks.workunit.client.1.smithi137.stdout:9/669: dwrite d6/df/f70 [0,4194304] 0 2024-03-14T13:02:20.317 INFO:tasks.workunit.client.1.smithi137.stdout:8/582: write d1/dc/d4e/d6d/f74 [299147,1175] 0 2024-03-14T13:02:20.319 INFO:tasks.workunit.client.1.smithi137.stdout:5/502: write d7/d16/d18/d25/d28/f60 [471430,103232] 0 2024-03-14T13:02:20.321 INFO:tasks.workunit.client.0.smithi111.stdout:7/480: dwrite d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:20.322 INFO:tasks.workunit.client.1.smithi137.stdout:0/548: dread d1/d5/d1b/d42/d3a/f48 [0,4194304] 0 2024-03-14T13:02:20.324 INFO:tasks.workunit.client.1.smithi137.stdout:5/503: chown d7/d49/d75/f8c 86776 1 2024-03-14T13:02:20.325 INFO:tasks.workunit.client.1.smithi137.stdout:4/529: dwrite d4/d10/d42/d73/fb5 [0,4194304] 0 2024-03-14T13:02:20.326 INFO:tasks.workunit.client.0.smithi111.stdout:7/481: creat d0/d1/d5/d38/d1b/f92 x:0 0 0 2024-03-14T13:02:20.330 INFO:tasks.workunit.client.1.smithi137.stdout:8/583: unlink d1/dc/d3c/d87/d34/d86/lc3 0 2024-03-14T13:02:20.332 INFO:tasks.workunit.client.1.smithi137.stdout:7/415: dread d3/d5/d23/d2c/d5d/f6b [0,4194304] 0 2024-03-14T13:02:20.335 INFO:tasks.workunit.client.1.smithi137.stdout:7/416: chown d3/d5/d68 151244039 1 2024-03-14T13:02:20.335 INFO:tasks.workunit.client.1.smithi137.stdout:4/530: creat d4/d10/d42/d73/fb8 x:0 0 0 2024-03-14T13:02:20.343 INFO:tasks.workunit.client.1.smithi137.stdout:8/584: dwrite d1/dc/d3c/d87/f6e [0,4194304] 0 2024-03-14T13:02:20.348 INFO:tasks.workunit.client.1.smithi137.stdout:0/549: link d1/d1f/d82/f8b d1/d1f/d82/fb7 0 2024-03-14T13:02:20.348 INFO:tasks.workunit.client.1.smithi137.stdout:4/531: rename d4/d9/d3a/d70/d80/f8c to d4/d27/db1/fb9 0 2024-03-14T13:02:20.349 INFO:tasks.workunit.client.0.smithi111.stdout:7/482: dread d0/d7e/fb [0,4194304] 0 2024-03-14T13:02:20.349 INFO:tasks.workunit.client.1.smithi137.stdout:2/581: dwrite d2/d7/d2e/d3e/d44/d6b/d22/f91 [0,4194304] 0 2024-03-14T13:02:20.353 INFO:tasks.workunit.client.1.smithi137.stdout:8/585: unlink d1/dc/d10/f27 0 2024-03-14T13:02:20.355 INFO:tasks.workunit.client.1.smithi137.stdout:7/417: dwrite d3/d5/d23/d2c/d5d/f6b [0,4194304] 0 2024-03-14T13:02:20.355 INFO:tasks.workunit.client.1.smithi137.stdout:2/582: mknod d2/d7/d2e/d3e/daa/cc0 0 2024-03-14T13:02:20.362 INFO:tasks.workunit.client.1.smithi137.stdout:8/586: unlink d1/dc/d3c/d87/d34/d97/l92 0 2024-03-14T13:02:20.362 INFO:tasks.workunit.client.1.smithi137.stdout:7/418: symlink d3/d5/d23/d2c/d5d/l82 0 2024-03-14T13:02:20.363 INFO:tasks.workunit.client.0.smithi111.stdout:7/483: rmdir d0 39 2024-03-14T13:02:20.363 INFO:tasks.workunit.client.1.smithi137.stdout:2/583: write d2/d7/d2e/d3e/d44/d6b/fb5 [981427,45840] 0 2024-03-14T13:02:20.364 INFO:tasks.workunit.client.1.smithi137.stdout:7/419: stat d3/d6/d14/f57 0 2024-03-14T13:02:20.365 INFO:tasks.workunit.client.1.smithi137.stdout:9/670: dread d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:20.367 INFO:tasks.workunit.client.1.smithi137.stdout:9/671: chown d6/d91/cd4 989344 1 2024-03-14T13:02:20.368 INFO:tasks.workunit.client.1.smithi137.stdout:5/504: truncate d7/fb 47869 0 2024-03-14T13:02:20.369 INFO:tasks.workunit.client.0.smithi111.stdout:7/484: dwrite d0/d7e/f3d [0,4194304] 0 2024-03-14T13:02:20.371 INFO:tasks.workunit.client.1.smithi137.stdout:2/584: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/fc1 x:0 0 0 2024-03-14T13:02:20.371 INFO:tasks.workunit.client.0.smithi111.stdout:7/485: write d0/d1/d5/f1a [3641508,59639] 0 2024-03-14T13:02:20.371 INFO:tasks.workunit.client.1.smithi137.stdout:9/672: mkdir d6/df/d88/dd5 0 2024-03-14T13:02:20.373 INFO:tasks.workunit.client.1.smithi137.stdout:8/587: link d1/dc/d77/c8e d1/dc/d3c/cc7 0 2024-03-14T13:02:20.377 INFO:tasks.workunit.client.1.smithi137.stdout:8/588: chown d1/dc/d3c/d87/d34/d97/f2a 8575171 1 2024-03-14T13:02:20.378 INFO:tasks.workunit.client.1.smithi137.stdout:9/673: creat d6/df/d19/d3a/da9/fd6 x:0 0 0 2024-03-14T13:02:20.380 INFO:tasks.workunit.client.1.smithi137.stdout:2/585: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/f9b 606792 0 2024-03-14T13:02:20.380 INFO:tasks.workunit.client.1.smithi137.stdout:8/589: fsync d1/dc/d3c/d87/d34/d97/d13/f22 0 2024-03-14T13:02:20.380 INFO:tasks.workunit.client.1.smithi137.stdout:7/420: dread d3/d6/d14/d22/f43 [0,4194304] 0 2024-03-14T13:02:20.382 INFO:tasks.workunit.client.1.smithi137.stdout:7/421: fsync d3/d6/d2e/f3e 0 2024-03-14T13:02:20.391 INFO:tasks.workunit.client.1.smithi137.stdout:5/505: write d7/fa2 [829997,82497] 0 2024-03-14T13:02:20.392 INFO:tasks.workunit.client.0.smithi111.stdout:7/486: dread d0/d1/d5/d56/f5e [0,4194304] 0 2024-03-14T13:02:20.393 INFO:tasks.workunit.client.1.smithi137.stdout:9/674: dwrite d6/df/d19/d33/f46 [4194304,4194304] 0 2024-03-14T13:02:20.394 INFO:tasks.workunit.client.1.smithi137.stdout:2/586: write d2/f4 [330365,95612] 0 2024-03-14T13:02:20.394 INFO:tasks.workunit.client.0.smithi111.stdout:7/487: creat d0/d1/d21/d2a/f93 x:0 0 0 2024-03-14T13:02:20.399 INFO:tasks.workunit.client.0.smithi111.stdout:7/488: dread d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:20.401 INFO:tasks.workunit.client.1.smithi137.stdout:4/532: dwrite d4/f13 [0,4194304] 0 2024-03-14T13:02:20.403 INFO:tasks.workunit.client.1.smithi137.stdout:2/587: dread - d2/d34/d3c/d9d/fa1 zero size 2024-03-14T13:02:20.403 INFO:tasks.workunit.client.1.smithi137.stdout:6/553: sync 2024-03-14T13:02:20.405 INFO:tasks.workunit.client.1.smithi137.stdout:5/506: rmdir d7/d1d/d74/d81 39 2024-03-14T13:02:20.405 INFO:tasks.workunit.client.1.smithi137.stdout:8/590: dwrite d1/dc/d3c/d87/d34/d97/fc0 [0,4194304] 0 2024-03-14T13:02:20.405 INFO:tasks.workunit.client.0.smithi111.stdout:7/489: link d0/d7e/f81 d0/d1/d5/d38/f94 0 2024-03-14T13:02:20.408 INFO:tasks.workunit.client.1.smithi137.stdout:2/588: dread - d2/d7/d2e/d3e/d44/d6b/fa6 zero size 2024-03-14T13:02:20.411 INFO:tasks.workunit.client.0.smithi111.stdout:7/490: dwrite d0/f2e [0,4194304] 0 2024-03-14T13:02:20.412 INFO:tasks.workunit.client.1.smithi137.stdout:7/422: link d3/d5/f67 d3/d5/d23/d2c/d5d/d6a/d75/d78/f83 0 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:2/554: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:0/732: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:5/711: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:8/768: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:4/955: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:6/555: sync 2024-03-14T13:02:20.413 INFO:tasks.workunit.client.0.smithi111.stdout:3/576: sync 2024-03-14T13:02:20.414 INFO:tasks.workunit.client.1.smithi137.stdout:7/423: readlink d3/d6/d59/l62 0 2024-03-14T13:02:20.415 INFO:tasks.workunit.client.1.smithi137.stdout:8/591: mknod d1/dc/d3c/d87/d34/d86/cc8 0 2024-03-14T13:02:20.416 INFO:tasks.workunit.client.1.smithi137.stdout:4/533: symlink d4/d9/d3a/d41/d38/lba 0 2024-03-14T13:02:20.418 INFO:tasks.workunit.client.0.smithi111.stdout:5/712: dread - d2/d45/d79/fc5 zero size 2024-03-14T13:02:20.419 INFO:tasks.workunit.client.0.smithi111.stdout:0/733: stat d6/d9/d15/d49/cf2 0 2024-03-14T13:02:20.419 INFO:tasks.workunit.client.0.smithi111.stdout:7/491: rename d0/d1/d4a/f5a to d0/d1/d5/d56/d70/f95 0 2024-03-14T13:02:20.420 INFO:tasks.workunit.client.1.smithi137.stdout:8/592: mkdir d1/d5e/dc9 0 2024-03-14T13:02:20.420 INFO:tasks.workunit.client.0.smithi111.stdout:4/956: write d1d/d22/d40/d69/d86/f9f [1427932,94268] 0 2024-03-14T13:02:20.420 INFO:tasks.workunit.client.0.smithi111.stdout:8/769: mkdir d3/d49/dfb 0 2024-03-14T13:02:20.421 INFO:tasks.workunit.client.0.smithi111.stdout:3/577: write d0/d2d/d35/d69/f9d [3646062,85098] 0 2024-03-14T13:02:20.423 INFO:tasks.workunit.client.1.smithi137.stdout:9/675: rename d6/df/d19/d3a/da9/lbb to d6/df/d19/ld7 0 2024-03-14T13:02:20.423 INFO:tasks.workunit.client.1.smithi137.stdout:2/589: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f5f [0,4194304] 0 2024-03-14T13:02:20.424 INFO:tasks.workunit.client.0.smithi111.stdout:8/770: dread - d3/d1b/d34/fd6 zero size 2024-03-14T13:02:20.424 INFO:tasks.workunit.client.0.smithi111.stdout:0/734: chown d6/d9/de/d77/da4/dca/dd0/de8 28 1 2024-03-14T13:02:20.425 INFO:tasks.workunit.client.1.smithi137.stdout:4/534: symlink d4/d9/d3a/d41/d38/d79/lbb 0 2024-03-14T13:02:20.426 INFO:tasks.workunit.client.1.smithi137.stdout:7/424: rmdir d3/d5/d23/d2c/d81 0 2024-03-14T13:02:20.426 INFO:tasks.workunit.client.1.smithi137.stdout:2/590: write d2/d7/d2e/d3e/d44/d6b/d22/dac/faf [1125324,105920] 0 2024-03-14T13:02:20.427 INFO:tasks.workunit.client.0.smithi111.stdout:8/771: symlink d3/dc/lfc 0 2024-03-14T13:02:20.427 INFO:tasks.workunit.client.1.smithi137.stdout:8/593: mkdir d1/dc/d71/db0/dca 0 2024-03-14T13:02:20.429 INFO:tasks.workunit.client.1.smithi137.stdout:4/535: fdatasync d4/f11 0 2024-03-14T13:02:20.431 INFO:tasks.workunit.client.0.smithi111.stdout:6/556: creat d0/d17/d20/d4a/d59/d82/fcf x:0 0 0 2024-03-14T13:02:20.431 INFO:tasks.workunit.client.1.smithi137.stdout:9/676: mknod d6/df/cd8 0 2024-03-14T13:02:20.432 INFO:tasks.workunit.client.0.smithi111.stdout:0/735: readlink d6/d9/de/l20 0 2024-03-14T13:02:20.433 INFO:tasks.workunit.client.1.smithi137.stdout:7/425: mknod d3/d6/d14/d27/c84 0 2024-03-14T13:02:20.433 INFO:tasks.workunit.client.0.smithi111.stdout:6/557: chown d0/c4c 7968036 1 2024-03-14T13:02:20.433 INFO:tasks.workunit.client.1.smithi137.stdout:6/554: write d2/d6/de/fb3 [310911,18420] 0 2024-03-14T13:02:20.436 INFO:tasks.workunit.client.1.smithi137.stdout:8/594: rmdir d1/dc/da9 39 2024-03-14T13:02:20.437 INFO:tasks.workunit.client.1.smithi137.stdout:8/595: read - d1/dc/d3c/d87/d34/d97/d13/fab zero size 2024-03-14T13:02:20.437 INFO:tasks.workunit.client.1.smithi137.stdout:4/536: rmdir d4/d9/d3a/d41/d6a 39 2024-03-14T13:02:20.438 INFO:tasks.workunit.client.0.smithi111.stdout:4/957: dwrite d1d/d22/d40/d69/d86/f9f [0,4194304] 0 2024-03-14T13:02:20.438 INFO:tasks.workunit.client.0.smithi111.stdout:8/772: rename d3/d1b/d20/d35/d40/df6 to d3/d1b/d20/d35/d40/d48/d7e/db3/dfd 0 2024-03-14T13:02:20.438 INFO:tasks.workunit.client.1.smithi137.stdout:5/507: write d7/d33/f61 [652752,78775] 0 2024-03-14T13:02:20.439 INFO:tasks.workunit.client.1.smithi137.stdout:8/596: chown d1/dc/d3c/d87/d34 335070708 1 2024-03-14T13:02:20.440 INFO:tasks.workunit.client.1.smithi137.stdout:5/508: stat d7/d36/c5c 0 2024-03-14T13:02:20.441 INFO:tasks.workunit.client.1.smithi137.stdout:8/597: chown d1/l42 519619896 1 2024-03-14T13:02:20.441 INFO:tasks.workunit.client.1.smithi137.stdout:5/509: fsync d7/d16/d18/d25/f50 0 2024-03-14T13:02:20.441 INFO:tasks.workunit.client.1.smithi137.stdout:6/555: mknod d2/d6/de/d45/d59/d39/cc1 0 2024-03-14T13:02:20.442 INFO:tasks.workunit.client.0.smithi111.stdout:3/578: dwrite d0/d2d/d35/d50/d6c/fba [0,4194304] 0 2024-03-14T13:02:20.442 INFO:tasks.workunit.client.0.smithi111.stdout:6/558: unlink d0/f6e 0 2024-03-14T13:02:20.443 INFO:tasks.workunit.client.1.smithi137.stdout:6/556: fsync d2/d6/de/d45/f5b 0 2024-03-14T13:02:20.444 INFO:tasks.workunit.client.1.smithi137.stdout:8/598: fsync d1/dc/d3c/d87/d34/fc5 0 2024-03-14T13:02:20.444 INFO:tasks.workunit.client.1.smithi137.stdout:5/510: write d7/d16/f98 [473162,42667] 0 2024-03-14T13:02:20.445 INFO:tasks.workunit.client.1.smithi137.stdout:6/557: chown f0 6037674 1 2024-03-14T13:02:20.446 INFO:tasks.workunit.client.1.smithi137.stdout:5/511: readlink d7/d36/l3a 0 2024-03-14T13:02:20.448 INFO:tasks.workunit.client.1.smithi137.stdout:9/677: rename d6/df/f10 to d6/df/fd9 0 2024-03-14T13:02:20.448 INFO:tasks.workunit.client.0.smithi111.stdout:0/736: rename d6/d21/fa2 to d6/d3d/d97/f106 0 2024-03-14T13:02:20.449 INFO:tasks.workunit.client.1.smithi137.stdout:7/426: getdents d3 0 2024-03-14T13:02:20.452 INFO:tasks.workunit.client.0.smithi111.stdout:2/555: dwrite dd/d11/d1a/f1b [0,4194304] 0 2024-03-14T13:02:20.453 INFO:tasks.workunit.client.0.smithi111.stdout:8/773: dread d3/dc/d45/f7c [0,4194304] 0 2024-03-14T13:02:20.455 INFO:tasks.workunit.client.1.smithi137.stdout:4/537: dread d4/d9/d3a/d41/f21 [0,4194304] 0 2024-03-14T13:02:20.458 INFO:tasks.workunit.client.0.smithi111.stdout:3/579: mkdir d0/d91/dd8 0 2024-03-14T13:02:20.459 INFO:tasks.workunit.client.0.smithi111.stdout:2/556: mknod dd/d17/d53/d9f/cb3 0 2024-03-14T13:02:20.460 INFO:tasks.workunit.client.1.smithi137.stdout:9/678: rename d6/df/f90 to d6/fda 0 2024-03-14T13:02:20.460 INFO:tasks.workunit.client.0.smithi111.stdout:6/559: dwrite d0/f4 [0,4194304] 0 2024-03-14T13:02:20.463 INFO:tasks.workunit.client.1.smithi137.stdout:5/512: dwrite d7/d36/f82 [0,4194304] 0 2024-03-14T13:02:20.463 INFO:tasks.workunit.client.1.smithi137.stdout:9/679: dread - d6/df/d19/d33/f6e zero size 2024-03-14T13:02:20.464 INFO:tasks.workunit.client.1.smithi137.stdout:4/538: dwrite f3 [0,4194304] 0 2024-03-14T13:02:20.465 INFO:tasks.workunit.client.1.smithi137.stdout:8/599: dread d1/dc/d10/f32 [0,4194304] 0 2024-03-14T13:02:20.465 INFO:tasks.workunit.client.0.smithi111.stdout:8/774: getdents d3/d22/d30 0 2024-03-14T13:02:20.466 INFO:tasks.workunit.client.0.smithi111.stdout:3/580: dread - d0/d2d/d35/d50/da2/db3/d71/dbc/fd3 zero size 2024-03-14T13:02:20.466 INFO:tasks.workunit.client.0.smithi111.stdout:0/737: dwrite d6/d3d/f52 [0,4194304] 0 2024-03-14T13:02:20.466 INFO:tasks.workunit.client.1.smithi137.stdout:9/680: chown d6/df/d19/d33/f46 109 1 2024-03-14T13:02:20.468 INFO:tasks.workunit.client.1.smithi137.stdout:7/427: truncate f1 962118 0 2024-03-14T13:02:20.470 INFO:tasks.workunit.client.0.smithi111.stdout:3/581: truncate d0/d2d/d35/d4c/d82/f77 1046776 0 2024-03-14T13:02:20.472 INFO:tasks.workunit.client.1.smithi137.stdout:8/600: mknod d1/dc/d71/da3/ccb 0 2024-03-14T13:02:20.473 INFO:tasks.workunit.client.0.smithi111.stdout:8/775: creat d3/d1b/dc1/ffe x:0 0 0 2024-03-14T13:02:20.473 INFO:tasks.workunit.client.1.smithi137.stdout:7/428: symlink d3/d4a/l85 0 2024-03-14T13:02:20.477 INFO:tasks.workunit.client.0.smithi111.stdout:6/560: symlink d0/d17/d21/d69/d3d/d53/d81/da1/ld0 0 2024-03-14T13:02:20.477 INFO:tasks.workunit.client.0.smithi111.stdout:2/557: getdents dd/d17/d53/da4 0 2024-03-14T13:02:20.477 INFO:tasks.workunit.client.0.smithi111.stdout:8/776: write d3/d1b/d20/d35/d40/d48/dcd/fe3 [716202,17093] 0 2024-03-14T13:02:20.478 INFO:tasks.workunit.client.0.smithi111.stdout:2/558: chown dd/c15 1651405 1 2024-03-14T13:02:20.479 INFO:tasks.workunit.client.0.smithi111.stdout:6/561: write d0/d17/d21/d69/f45 [1026157,45526] 0 2024-03-14T13:02:20.479 INFO:tasks.workunit.client.1.smithi137.stdout:8/601: fsync d1/dc/d10/fa4 0 2024-03-14T13:02:20.480 INFO:tasks.workunit.client.1.smithi137.stdout:9/681: creat d6/df/d19/d3a/da9/fdb x:0 0 0 2024-03-14T13:02:20.483 INFO:tasks.workunit.client.1.smithi137.stdout:4/539: rename d4/d27/db1 to d4/d10/d42/d73/d9d/dbc 0 2024-03-14T13:02:20.484 INFO:tasks.workunit.client.0.smithi111.stdout:2/559: rename dd/l16 to dd/d17/d53/da4/lb4 0 2024-03-14T13:02:20.485 INFO:tasks.workunit.client.1.smithi137.stdout:7/429: mkdir d3/d5/d5a/d86 0 2024-03-14T13:02:20.486 INFO:tasks.workunit.client.0.smithi111.stdout:8/777: creat d3/d1b/d20/fff x:0 0 0 2024-03-14T13:02:20.487 INFO:tasks.workunit.client.1.smithi137.stdout:7/430: stat d3/c25 0 2024-03-14T13:02:20.489 INFO:tasks.workunit.client.0.smithi111.stdout:6/562: dwrite d0/f3e [0,4194304] 0 2024-03-14T13:02:20.491 INFO:tasks.workunit.client.0.smithi111.stdout:6/563: fdatasync d0/d17/d20/f6b 0 2024-03-14T13:02:20.492 INFO:tasks.workunit.client.1.smithi137.stdout:8/602: chown d1/dc/d3c/d87/d34/d97/c8f 25 1 2024-03-14T13:02:20.494 INFO:tasks.workunit.client.1.smithi137.stdout:7/431: truncate d3/d4a/f5e 935155 0 2024-03-14T13:02:20.495 INFO:tasks.workunit.client.1.smithi137.stdout:5/513: dwrite d7/d33/fb1 [0,4194304] 0 2024-03-14T13:02:20.497 INFO:tasks.workunit.client.1.smithi137.stdout:9/682: dwrite d6/df/d19/d1f/f35 [8388608,4194304] 0 2024-03-14T13:02:20.500 INFO:tasks.workunit.client.0.smithi111.stdout:8/778: dwrite d3/d1b/d20/f9e [0,4194304] 0 2024-03-14T13:02:20.500 INFO:tasks.workunit.client.1.smithi137.stdout:8/603: chown d1/d5e/f63 0 1 2024-03-14T13:02:20.502 INFO:tasks.workunit.client.0.smithi111.stdout:0/738: dread d6/d9/f58 [0,4194304] 0 2024-03-14T13:02:20.508 INFO:tasks.workunit.client.1.smithi137.stdout:7/432: readlink d3/d6/l51 0 2024-03-14T13:02:20.509 INFO:tasks.workunit.client.1.smithi137.stdout:4/540: dwrite d4/d9/d3a/d41/d6a/dac/fb0 [0,4194304] 0 2024-03-14T13:02:20.511 INFO:tasks.workunit.client.1.smithi137.stdout:7/433: readlink d3/d6/l51 0 2024-03-14T13:02:20.513 INFO:tasks.workunit.client.1.smithi137.stdout:8/604: creat d1/dc/d77/d9f/fcc x:0 0 0 2024-03-14T13:02:20.513 INFO:tasks.workunit.client.1.smithi137.stdout:6/558: dread d2/d6/de/d45/d59/d39/f73 [0,4194304] 0 2024-03-14T13:02:20.518 INFO:tasks.workunit.client.1.smithi137.stdout:9/683: rename d6/d91/fc6 to d6/df/d19/d3a/da9/fdc 0 2024-03-14T13:02:20.521 INFO:tasks.workunit.client.1.smithi137.stdout:4/541: fsync d4/d9/d3a/f4f 0 2024-03-14T13:02:20.523 INFO:tasks.workunit.client.1.smithi137.stdout:5/514: link d7/d36/c5c d7/d33/d65/cb4 0 2024-03-14T13:02:20.525 INFO:tasks.workunit.client.1.smithi137.stdout:7/434: dwrite d3/d6/d2e/f3e [0,4194304] 0 2024-03-14T13:02:20.526 INFO:tasks.workunit.client.0.smithi111.stdout:2/560: dread dd/d17/d34/f6f [0,4194304] 0 2024-03-14T13:02:20.530 INFO:tasks.workunit.client.0.smithi111.stdout:8/779: dread d3/d22/fbb [0,4194304] 0 2024-03-14T13:02:20.533 INFO:tasks.workunit.client.1.smithi137.stdout:8/605: unlink d1/d4a/c90 0 2024-03-14T13:02:20.533 INFO:tasks.workunit.client.1.smithi137.stdout:4/542: write d4/d9/f18 [3953780,88157] 0 2024-03-14T13:02:20.535 INFO:tasks.workunit.client.1.smithi137.stdout:5/515: creat d7/d33/fb5 x:0 0 0 2024-03-14T13:02:20.535 INFO:tasks.workunit.client.1.smithi137.stdout:7/435: mknod d3/d6/d64/c87 0 2024-03-14T13:02:20.535 INFO:tasks.workunit.client.1.smithi137.stdout:7/436: chown d3/d5/d23 482319269 1 2024-03-14T13:02:20.536 INFO:tasks.workunit.client.1.smithi137.stdout:4/543: creat d4/d27/d5c/d77/fbd x:0 0 0 2024-03-14T13:02:20.537 INFO:tasks.workunit.client.1.smithi137.stdout:4/544: chown d4/ld 333908 1 2024-03-14T13:02:20.539 INFO:tasks.workunit.client.1.smithi137.stdout:4/545: write d4/d27/d5c/d6d/d58/f7e [51051,24568] 0 2024-03-14T13:02:20.539 INFO:tasks.workunit.client.1.smithi137.stdout:8/606: rename d1/dc/d3c/d87/d34/d97/d13/f26 to d1/dc/fcd 0 2024-03-14T13:02:20.540 INFO:tasks.workunit.client.0.smithi111.stdout:2/561: dread dd/d17/d34/f2e [8388608,4194304] 0 2024-03-14T13:02:20.543 INFO:tasks.workunit.client.1.smithi137.stdout:8/607: dread d1/dc/d10/f32 [0,4194304] 0 2024-03-14T13:02:20.544 INFO:tasks.workunit.client.1.smithi137.stdout:5/516: dwrite d7/d16/d18/d25/f55 [4194304,4194304] 0 2024-03-14T13:02:20.544 INFO:tasks.workunit.client.1.smithi137.stdout:8/608: chown d1/dc/d10 515420152 1 2024-03-14T13:02:20.546 INFO:tasks.workunit.client.1.smithi137.stdout:5/517: write d7/d16/d18/d25/f50 [3865791,1187] 0 2024-03-14T13:02:20.548 INFO:tasks.workunit.client.1.smithi137.stdout:8/609: getdents d1/dc/d71/db0 0 2024-03-14T13:02:20.548 INFO:tasks.workunit.client.1.smithi137.stdout:8/610: chown d1 287 1 2024-03-14T13:02:20.552 INFO:tasks.workunit.client.1.smithi137.stdout:8/611: creat d1/dc/fce x:0 0 0 2024-03-14T13:02:20.553 INFO:tasks.workunit.client.1.smithi137.stdout:8/612: chown d1/dc/d3c/d87/d34/l53 961972 1 2024-03-14T13:02:20.555 INFO:tasks.workunit.client.1.smithi137.stdout:8/613: mkdir d1/dc/d10/dcf 0 2024-03-14T13:02:20.565 INFO:tasks.workunit.client.1.smithi137.stdout:9/684: truncate d6/d56/fb3 3648989 0 2024-03-14T13:02:20.567 INFO:tasks.workunit.client.1.smithi137.stdout:6/559: dwrite d2/d6/de/f4f [0,4194304] 0 2024-03-14T13:02:20.570 INFO:tasks.workunit.client.1.smithi137.stdout:9/685: symlink d6/df/d19/d3a/da9/ldd 0 2024-03-14T13:02:20.571 INFO:tasks.workunit.client.1.smithi137.stdout:9/686: chown d6/d91 1911858674 1 2024-03-14T13:02:20.575 INFO:tasks.workunit.client.1.smithi137.stdout:6/560: creat d2/d54/d55/d6d/d76/fc2 x:0 0 0 2024-03-14T13:02:20.576 INFO:tasks.workunit.client.0.smithi111.stdout:9/567: sync 2024-03-14T13:02:20.577 INFO:tasks.workunit.client.0.smithi111.stdout:1/632: sync 2024-03-14T13:02:20.577 INFO:tasks.workunit.client.1.smithi137.stdout:4/546: dwrite d4/d10/d5a/f71 [0,4194304] 0 2024-03-14T13:02:20.578 INFO:tasks.workunit.client.0.smithi111.stdout:9/568: fsync d5/d6/d8/d1a/f64 0 2024-03-14T13:02:20.578 INFO:tasks.workunit.client.0.smithi111.stdout:3/582: write d0/d30/f48 [988254,126669] 0 2024-03-14T13:02:20.582 INFO:tasks.workunit.client.0.smithi111.stdout:1/633: fdatasync d20/d21/d3f/d59/daf/fbd 0 2024-03-14T13:02:20.583 INFO:tasks.workunit.client.1.smithi137.stdout:5/518: dwrite d7/d16/d18/f34 [0,4194304] 0 2024-03-14T13:02:20.584 INFO:tasks.workunit.client.0.smithi111.stdout:6/564: write d0/f14 [1327728,24106] 0 2024-03-14T13:02:20.584 INFO:tasks.workunit.client.1.smithi137.stdout:6/561: mkdir d2/d54/d55/d6d/dc3 0 2024-03-14T13:02:20.585 INFO:tasks.workunit.client.1.smithi137.stdout:6/562: stat d2/d6/d1d/c96 0 2024-03-14T13:02:20.586 INFO:tasks.workunit.client.1.smithi137.stdout:7/437: dread f1 [0,4194304] 0 2024-03-14T13:02:20.586 INFO:tasks.workunit.client.1.smithi137.stdout:9/687: truncate d6/df/d19/d3a/d40/f41 3791109 0 2024-03-14T13:02:20.588 INFO:tasks.workunit.client.1.smithi137.stdout:7/438: fsync d3/d6/d14/f57 0 2024-03-14T13:02:20.590 INFO:tasks.workunit.client.1.smithi137.stdout:5/519: write d7/d16/f98 [967087,89611] 0 2024-03-14T13:02:20.590 INFO:tasks.workunit.client.1.smithi137.stdout:4/547: readlink d4/d9/l51 0 2024-03-14T13:02:20.590 INFO:tasks.workunit.client.1.smithi137.stdout:6/563: write d2/d6/de/d45/f60 [726288,35645] 0 2024-03-14T13:02:20.592 INFO:tasks.workunit.client.1.smithi137.stdout:5/520: write d7/f96 [679674,19078] 0 2024-03-14T13:02:20.593 INFO:tasks.workunit.client.0.smithi111.stdout:2/562: creat dd/d11/d1a/fb5 x:0 0 0 2024-03-14T13:02:20.593 INFO:tasks.workunit.client.0.smithi111.stdout:6/565: mknod d0/d17/d21/d42/da5/cd1 0 2024-03-14T13:02:20.595 INFO:tasks.workunit.client.0.smithi111.stdout:1/634: mkdir d20/d25/dd0/dd2 0 2024-03-14T13:02:20.595 INFO:tasks.workunit.client.0.smithi111.stdout:9/569: link d5/d6/d14/c15 d5/d6/d14/cc2 0 2024-03-14T13:02:20.595 INFO:tasks.workunit.client.0.smithi111.stdout:2/563: chown dd/d17/d34/d73/d75/f5d 718535504 1 2024-03-14T13:02:20.596 INFO:tasks.workunit.client.1.smithi137.stdout:4/548: rmdir d4/d10/d42 39 2024-03-14T13:02:20.596 INFO:tasks.workunit.client.1.smithi137.stdout:5/521: fsync d7/d16/d18/f66 0 2024-03-14T13:02:20.598 INFO:tasks.workunit.client.0.smithi111.stdout:3/583: write d0/d2d/d35/d50/d5e/f5f [3053144,91912] 0 2024-03-14T13:02:20.598 INFO:tasks.workunit.client.0.smithi111.stdout:6/566: write d0/d17/d20/d65/fc7 [337697,72282] 0 2024-03-14T13:02:20.599 INFO:tasks.workunit.client.1.smithi137.stdout:7/439: rename d3/d6/f63 to d3/d5/d5a/d80/f88 0 2024-03-14T13:02:20.600 INFO:tasks.workunit.client.0.smithi111.stdout:0/739: dwrite d6/d9/de/f3c [4194304,4194304] 0 2024-03-14T13:02:20.603 INFO:tasks.workunit.client.0.smithi111.stdout:9/570: truncate d5/d6/d8/d1a/f80 346508 0 2024-03-14T13:02:20.604 INFO:tasks.workunit.client.1.smithi137.stdout:5/522: dread d7/d16/d18/d2d/f53 [0,4194304] 0 2024-03-14T13:02:20.604 INFO:tasks.workunit.client.1.smithi137.stdout:7/440: fdatasync d3/d5/d5a/d80/f88 0 2024-03-14T13:02:20.604 INFO:tasks.workunit.client.1.smithi137.stdout:4/549: dread d4/d10/d5a/f71 [0,4194304] 0 2024-03-14T13:02:20.605 INFO:tasks.workunit.client.1.smithi137.stdout:5/523: chown d7/d1d/d74/d83 7163928 1 2024-03-14T13:02:20.606 INFO:tasks.workunit.client.0.smithi111.stdout:6/567: write d0/d17/d21/d69/d3d/d53/d81/da1/dc8/fc9 [529412,51008] 0 2024-03-14T13:02:20.606 INFO:tasks.workunit.client.1.smithi137.stdout:9/688: write d6/d26/f93 [1005888,63953] 0 2024-03-14T13:02:20.607 INFO:tasks.workunit.client.0.smithi111.stdout:9/571: fsync d5/d42/d52/d83/f88 0 2024-03-14T13:02:20.607 INFO:tasks.workunit.client.0.smithi111.stdout:0/740: getdents d6/d3d/d97 0 2024-03-14T13:02:20.608 INFO:tasks.workunit.client.0.smithi111.stdout:2/564: read dd/d17/d34/d73/d75/f95 [4081462,74736] 0 2024-03-14T13:02:20.608 INFO:tasks.workunit.client.1.smithi137.stdout:5/524: write d7/d16/d18/f8a [873131,128913] 0 2024-03-14T13:02:20.611 INFO:tasks.workunit.client.1.smithi137.stdout:7/441: fdatasync d3/f18 0 2024-03-14T13:02:20.612 INFO:tasks.workunit.client.1.smithi137.stdout:5/525: write d7/d33/fb1 [1965693,69635] 0 2024-03-14T13:02:20.615 INFO:tasks.workunit.client.1.smithi137.stdout:9/689: read d6/df/d19/d1f/f24 [1240333,20740] 0 2024-03-14T13:02:20.616 INFO:tasks.workunit.client.1.smithi137.stdout:4/550: dread d4/d9/d3a/d41/f97 [4194304,4194304] 0 2024-03-14T13:02:20.618 INFO:tasks.workunit.client.1.smithi137.stdout:5/526: write d7/d1d/f57 [3892204,36097] 0 2024-03-14T13:02:20.619 INFO:tasks.workunit.client.1.smithi137.stdout:7/442: symlink d3/d5/d23/l89 0 2024-03-14T13:02:20.620 INFO:tasks.workunit.client.1.smithi137.stdout:4/551: mkdir d4/d9/d3a/d41/d38/d79/dbe 0 2024-03-14T13:02:20.621 INFO:tasks.workunit.client.1.smithi137.stdout:5/527: write d7/d16/d18/f34 [3198059,36999] 0 2024-03-14T13:02:20.622 INFO:tasks.workunit.client.1.smithi137.stdout:7/443: rmdir d3/d52 39 2024-03-14T13:02:20.624 INFO:tasks.workunit.client.1.smithi137.stdout:5/528: fsync d7/d16/d18/d25/d28/fad 0 2024-03-14T13:02:20.624 INFO:tasks.workunit.client.1.smithi137.stdout:4/552: rmdir d4/d27 39 2024-03-14T13:02:20.625 INFO:tasks.workunit.client.1.smithi137.stdout:6/564: dread d2/d54/d55/d33/fa0 [0,4194304] 0 2024-03-14T13:02:20.629 INFO:tasks.workunit.client.1.smithi137.stdout:5/529: creat d7/d16/d18/d25/d28/fb6 x:0 0 0 2024-03-14T13:02:20.630 INFO:tasks.workunit.client.1.smithi137.stdout:4/553: symlink d4/d9/d3a/d41/d38/d79/da1/lbf 0 2024-03-14T13:02:20.631 INFO:tasks.workunit.client.1.smithi137.stdout:6/565: mkdir d2/d54/d55/d9c/dc0/dc4 0 2024-03-14T13:02:20.637 INFO:tasks.workunit.client.1.smithi137.stdout:7/444: dwrite d3/d5/d23/d2c/f53 [0,4194304] 0 2024-03-14T13:02:20.638 INFO:tasks.workunit.client.1.smithi137.stdout:9/690: dread d6/df/f13 [0,4194304] 0 2024-03-14T13:02:20.639 INFO:tasks.workunit.client.1.smithi137.stdout:4/554: fsync d4/d27/d5c/d6d/d58/d65/f7b 0 2024-03-14T13:02:20.640 INFO:tasks.workunit.client.1.smithi137.stdout:5/530: mknod d7/d16/d18/d25/d28/d3c/cb7 0 2024-03-14T13:02:20.641 INFO:tasks.workunit.client.1.smithi137.stdout:4/555: chown d4/d9/f18 119 1 2024-03-14T13:02:20.643 INFO:tasks.workunit.client.1.smithi137.stdout:5/531: symlink d7/d33/lb8 0 2024-03-14T13:02:20.644 INFO:tasks.workunit.client.1.smithi137.stdout:9/691: mkdir d6/df/d19/d3a/da9/dde 0 2024-03-14T13:02:20.644 INFO:tasks.workunit.client.1.smithi137.stdout:5/532: chown d7/d36/l3a 4022 1 2024-03-14T13:02:20.645 INFO:tasks.workunit.client.1.smithi137.stdout:5/533: write d7/d16/d18/d25/d28/d3c/faf [70484,44307] 0 2024-03-14T13:02:20.646 INFO:tasks.workunit.client.1.smithi137.stdout:6/566: link d2/c21 d2/d6/cc5 0 2024-03-14T13:02:20.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/556: fsync d4/d10/d5a/f71 0 2024-03-14T13:02:20.651 INFO:tasks.workunit.client.1.smithi137.stdout:5/534: creat d7/d16/fb9 x:0 0 0 2024-03-14T13:02:20.660 INFO:tasks.workunit.client.1.smithi137.stdout:6/567: dwrite d2/d6/de/d45/fa8 [0,4194304] 0 2024-03-14T13:02:20.663 INFO:tasks.workunit.client.1.smithi137.stdout:4/557: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:20.667 INFO:tasks.workunit.client.1.smithi137.stdout:7/445: dwrite d3/d5/f67 [0,4194304] 0 2024-03-14T13:02:20.671 INFO:tasks.workunit.client.1.smithi137.stdout:5/535: dwrite d7/d16/fae [0,4194304] 0 2024-03-14T13:02:20.671 INFO:tasks.workunit.client.1.smithi137.stdout:6/568: write d2/d54/d55/d6d/d76/d57/d58/fa5 [278485,118914] 0 2024-03-14T13:02:20.673 INFO:tasks.workunit.client.1.smithi137.stdout:4/558: mkdir d4/d9/d3a/d70/d80/d8d/dc0 0 2024-03-14T13:02:20.675 INFO:tasks.workunit.client.1.smithi137.stdout:5/536: write d7/d1d/f57 [916080,27527] 0 2024-03-14T13:02:20.675 INFO:tasks.workunit.client.0.smithi111.stdout:8/780: dread d3/f3e [0,4194304] 0 2024-03-14T13:02:20.676 INFO:tasks.workunit.client.1.smithi137.stdout:7/446: write d3/d6/d14/f57 [1103826,87605] 0 2024-03-14T13:02:20.678 INFO:tasks.workunit.client.1.smithi137.stdout:6/569: read d2/d6/de/f5d [2245409,55333] 0 2024-03-14T13:02:20.680 INFO:tasks.workunit.client.1.smithi137.stdout:6/570: creat d2/d6/de/d45/d97/fc6 x:0 0 0 2024-03-14T13:02:20.680 INFO:tasks.workunit.client.1.smithi137.stdout:7/447: symlink d3/d5/d23/d2c/d5d/d6a/d75/l8a 0 2024-03-14T13:02:20.681 INFO:tasks.workunit.client.0.smithi111.stdout:0/741: truncate d6/d9/de/d3a/d46/f83 1000972 0 2024-03-14T13:02:20.684 INFO:tasks.workunit.client.0.smithi111.stdout:3/584: dwrite d0/d2d/d35/d50/d5e/d19/f26 [0,4194304] 0 2024-03-14T13:02:20.685 INFO:tasks.workunit.client.0.smithi111.stdout:3/585: dread - d0/d2d/d35/d69/f94 zero size 2024-03-14T13:02:20.686 INFO:tasks.workunit.client.1.smithi137.stdout:4/559: dwrite d4/d10/d42/f6c [0,4194304] 0 2024-03-14T13:02:20.690 INFO:tasks.workunit.client.1.smithi137.stdout:0/550: sync 2024-03-14T13:02:20.691 INFO:tasks.workunit.client.0.smithi111.stdout:6/568: mknod d0/d17/d21/d69/d3d/d53/d81/da1/cd2 0 2024-03-14T13:02:20.692 INFO:tasks.workunit.client.0.smithi111.stdout:9/572: unlink d5/d6/d8/d5b/d87/f8c 0 2024-03-14T13:02:20.693 INFO:tasks.workunit.client.1.smithi137.stdout:1/608: sync 2024-03-14T13:02:20.693 INFO:tasks.workunit.client.1.smithi137.stdout:2/591: sync 2024-03-14T13:02:20.693 INFO:tasks.workunit.client.0.smithi111.stdout:2/565: creat dd/d17/d34/d74/fb6 x:0 0 0 2024-03-14T13:02:20.693 INFO:tasks.workunit.client.0.smithi111.stdout:9/573: chown d5/d6/f56 15 1 2024-03-14T13:02:20.694 INFO:tasks.workunit.client.1.smithi137.stdout:6/571: dread d2/f2d [0,4194304] 0 2024-03-14T13:02:20.694 INFO:tasks.workunit.client.0.smithi111.stdout:2/566: read f1 [2105301,7222] 0 2024-03-14T13:02:20.695 INFO:tasks.workunit.client.0.smithi111.stdout:0/742: rename d6/d9/de/d32/l2b to d6/d9/de/d3a/l107 0 2024-03-14T13:02:20.696 INFO:tasks.workunit.client.1.smithi137.stdout:7/448: dread d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:20.697 INFO:tasks.workunit.client.1.smithi137.stdout:4/560: creat d4/d27/d5c/d6d/d58/fc1 x:0 0 0 2024-03-14T13:02:20.697 INFO:tasks.workunit.client.0.smithi111.stdout:0/743: dread - d6/d9/de/d77/da4/fff zero size 2024-03-14T13:02:20.698 INFO:tasks.workunit.client.1.smithi137.stdout:3/503: sync 2024-03-14T13:02:20.701 INFO:tasks.workunit.client.0.smithi111.stdout:2/567: mkdir dd/d8d/db7 0 2024-03-14T13:02:20.701 INFO:tasks.workunit.client.0.smithi111.stdout:3/586: truncate d0/d2d/d35/d50/d5e/f12 1582419 0 2024-03-14T13:02:20.703 INFO:tasks.workunit.client.0.smithi111.stdout:3/587: fsync d0/d2d/d35/d50/fc2 0 2024-03-14T13:02:20.703 INFO:tasks.workunit.client.0.smithi111.stdout:9/574: creat d5/d6/d2a/d2d/db8/fc3 x:0 0 0 2024-03-14T13:02:20.704 INFO:tasks.workunit.client.1.smithi137.stdout:3/504: symlink d4/df/la9 0 2024-03-14T13:02:20.704 INFO:tasks.workunit.client.1.smithi137.stdout:6/572: write d2/d54/d55/d38/f51 [9348,119855] 0 2024-03-14T13:02:20.706 INFO:tasks.workunit.client.0.smithi111.stdout:9/575: chown d5/d6/d8/d74/f77 5751 1 2024-03-14T13:02:20.707 INFO:tasks.workunit.client.1.smithi137.stdout:1/609: link d7/d9/d19/d1f/d26/d7d/la4 d7/d9/d19/d1f/da0/lcd 0 2024-03-14T13:02:20.708 INFO:tasks.workunit.client.0.smithi111.stdout:3/588: creat d0/d2d/d35/d50/da2/fd9 x:0 0 0 2024-03-14T13:02:20.709 INFO:tasks.workunit.client.1.smithi137.stdout:6/573: stat d2/d6/da7 0 2024-03-14T13:02:20.710 INFO:tasks.workunit.client.0.smithi111.stdout:2/568: dwrite dd/d17/d34/d74/fb6 [0,4194304] 0 2024-03-14T13:02:20.711 INFO:tasks.workunit.client.1.smithi137.stdout:0/551: dwrite d1/d44/f4c [0,4194304] 0 2024-03-14T13:02:20.713 INFO:tasks.workunit.client.1.smithi137.stdout:6/574: write d2/d6/fc [5668581,27306] 0 2024-03-14T13:02:20.713 INFO:tasks.workunit.client.0.smithi111.stdout:2/569: chown dd/d17/d34/d73/l40 5371181 1 2024-03-14T13:02:20.715 INFO:tasks.workunit.client.1.smithi137.stdout:6/575: write d2/d54/d55/f78 [767242,55400] 0 2024-03-14T13:02:20.715 INFO:tasks.workunit.client.0.smithi111.stdout:9/576: dwrite f4 [4194304,4194304] 0 2024-03-14T13:02:20.719 INFO:tasks.workunit.client.1.smithi137.stdout:3/505: symlink d4/de/d16/d1c/d8d/laa 0 2024-03-14T13:02:20.720 INFO:tasks.workunit.client.1.smithi137.stdout:3/506: chown d4/de/l89 93612398 1 2024-03-14T13:02:20.724 INFO:tasks.workunit.client.0.smithi111.stdout:9/577: dwrite d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:20.725 INFO:tasks.workunit.client.1.smithi137.stdout:4/561: dwrite d4/d4d/f6b [0,4194304] 0 2024-03-14T13:02:20.725 INFO:tasks.workunit.client.1.smithi137.stdout:2/592: dwrite d2/d7/d2e/d3e/d44/d6b/f1b [0,4194304] 0 2024-03-14T13:02:20.725 INFO:tasks.workunit.client.1.smithi137.stdout:3/507: write d4/de/d13/f9f [738140,107844] 0 2024-03-14T13:02:20.727 INFO:tasks.workunit.client.0.smithi111.stdout:9/578: symlink d5/lc4 0 2024-03-14T13:02:20.728 INFO:tasks.workunit.client.0.smithi111.stdout:9/579: readlink d5/d6/l62 0 2024-03-14T13:02:20.730 INFO:tasks.workunit.client.1.smithi137.stdout:6/576: chown d2/d54/d55/d38/f85 0 1 2024-03-14T13:02:20.730 INFO:tasks.workunit.client.1.smithi137.stdout:1/610: mknod d7/d9/d19/d1f/d2e/d63/cce 0 2024-03-14T13:02:20.730 INFO:tasks.workunit.client.1.smithi137.stdout:1/611: read - d7/d9/d19/fc0 zero size 2024-03-14T13:02:20.736 INFO:tasks.workunit.client.1.smithi137.stdout:1/612: stat d7/d9/d19/d40/d9c 0 2024-03-14T13:02:20.736 INFO:tasks.workunit.client.1.smithi137.stdout:0/552: dread - d1/d5/d1b/d1e/d25/d55/fa0 zero size 2024-03-14T13:02:20.736 INFO:tasks.workunit.client.1.smithi137.stdout:3/508: rmdir d4/d19/da1 39 2024-03-14T13:02:20.736 INFO:tasks.workunit.client.1.smithi137.stdout:0/553: stat d1/d5/d1b/d1e/d25/d55/fb4 0 2024-03-14T13:02:20.737 INFO:tasks.workunit.client.1.smithi137.stdout:1/613: fsync d7/d9/d19/d1f/f95 0 2024-03-14T13:02:20.740 INFO:tasks.workunit.client.1.smithi137.stdout:2/593: dread d2/d7/d2e/d3e/d44/d6b/f3f [0,4194304] 0 2024-03-14T13:02:20.746 INFO:tasks.workunit.client.0.smithi111.stdout:9/580: dread d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:20.746 INFO:tasks.workunit.client.1.smithi137.stdout:7/449: dwrite d3/d6/d14/f2f [0,4194304] 0 2024-03-14T13:02:20.747 INFO:tasks.workunit.client.1.smithi137.stdout:4/562: write d4/d9/d3a/d41/f49 [2343804,110664] 0 2024-03-14T13:02:20.748 INFO:tasks.workunit.client.1.smithi137.stdout:6/577: creat d2/d54/d55/d6d/fc7 x:0 0 0 2024-03-14T13:02:20.749 INFO:tasks.workunit.client.0.smithi111.stdout:9/581: fsync d5/d6/d8/f3a 0 2024-03-14T13:02:20.750 INFO:tasks.workunit.client.1.smithi137.stdout:7/450: stat d3/f1c 0 2024-03-14T13:02:20.751 INFO:tasks.workunit.client.1.smithi137.stdout:1/614: symlink d7/lcf 0 2024-03-14T13:02:20.752 INFO:tasks.workunit.client.0.smithi111.stdout:9/582: mknod d5/d42/da4/cc5 0 2024-03-14T13:02:20.752 INFO:tasks.workunit.client.0.smithi111.stdout:8/781: dwrite d3/d5/f9c [0,4194304] 0 2024-03-14T13:02:20.753 INFO:tasks.workunit.client.1.smithi137.stdout:3/509: creat d4/d19/da1/fab x:0 0 0 2024-03-14T13:02:20.754 INFO:tasks.workunit.client.1.smithi137.stdout:4/563: stat d4/d9/d3a/d41/d38/d79/dbe 0 2024-03-14T13:02:20.755 INFO:tasks.workunit.client.1.smithi137.stdout:7/451: fdatasync d3/d5/f46 0 2024-03-14T13:02:20.756 INFO:tasks.workunit.client.0.smithi111.stdout:9/583: unlink d5/d42/l78 0 2024-03-14T13:02:20.757 INFO:tasks.workunit.client.1.smithi137.stdout:5/537: dwrite d7/d16/d18/f39 [0,4194304] 0 2024-03-14T13:02:20.761 INFO:tasks.workunit.client.1.smithi137.stdout:3/510: unlink d4/d19/l97 0 2024-03-14T13:02:20.762 INFO:tasks.workunit.client.1.smithi137.stdout:2/594: link d2/d7/d2e/d3e/d44/d6b/f1b d2/d7/d2e/d3e/d44/d6b/fc2 0 2024-03-14T13:02:20.763 INFO:tasks.workunit.client.0.smithi111.stdout:3/589: truncate d0/d2d/d35/d50/d5e/d19/f26 657738 0 2024-03-14T13:02:20.764 INFO:tasks.workunit.client.1.smithi137.stdout:0/554: dread d1/f7b [0,4194304] 0 2024-03-14T13:02:20.766 INFO:tasks.workunit.client.0.smithi111.stdout:3/590: creat d0/d2d/d35/d50/da2/db3/d71/fda x:0 0 0 2024-03-14T13:02:20.770 INFO:tasks.workunit.client.1.smithi137.stdout:3/511: mknod d4/de/d16/d1c/d8d/cac 0 2024-03-14T13:02:20.772 INFO:tasks.workunit.client.1.smithi137.stdout:7/452: dwrite d3/d5/d5a/f7f [0,4194304] 0 2024-03-14T13:02:20.772 INFO:tasks.workunit.client.1.smithi137.stdout:1/615: creat d7/d9/d19/d1f/d26/fd0 x:0 0 0 2024-03-14T13:02:20.774 INFO:tasks.workunit.client.1.smithi137.stdout:1/616: stat d7/d9/d19/d1f/f23 0 2024-03-14T13:02:20.774 INFO:tasks.workunit.client.1.smithi137.stdout:5/538: mknod d7/d16/d18/d25/d28/d3c/d7a/cba 0 2024-03-14T13:02:20.774 INFO:tasks.workunit.client.1.smithi137.stdout:0/555: write d1/d5/d1b/d42/d3a/db0/d84/f8c [2132464,47924] 0 2024-03-14T13:02:20.777 INFO:tasks.workunit.client.1.smithi137.stdout:1/617: fsync d7/d9/fad 0 2024-03-14T13:02:20.780 INFO:tasks.workunit.client.1.smithi137.stdout:3/512: symlink d4/lad 0 2024-03-14T13:02:20.781 INFO:tasks.workunit.client.1.smithi137.stdout:7/453: creat d3/d6/d2e/f8b x:0 0 0 2024-03-14T13:02:20.781 INFO:tasks.workunit.client.0.smithi111.stdout:8/782: dread d3/d1b/d20/d35/d40/f90 [0,4194304] 0 2024-03-14T13:02:20.784 INFO:tasks.workunit.client.1.smithi137.stdout:2/595: dwrite d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/fb9 [0,4194304] 0 2024-03-14T13:02:20.784 INFO:tasks.workunit.client.0.smithi111.stdout:8/783: mkdir d3/d1b/d20/d35/d40/d48/d7e/db3/d100 0 2024-03-14T13:02:20.787 INFO:tasks.workunit.client.1.smithi137.stdout:5/539: chown d7/d16/d18/d25/d28/la1 226836 1 2024-03-14T13:02:20.787 INFO:tasks.workunit.client.1.smithi137.stdout:1/618: symlink d7/d9/d19/d40/d50/ld1 0 2024-03-14T13:02:20.787 INFO:tasks.workunit.client.1.smithi137.stdout:3/513: rename d4/d19/l24 to d4/df/lae 0 2024-03-14T13:02:20.791 INFO:tasks.workunit.client.0.smithi111.stdout:8/784: dread d3/f65 [0,4194304] 0 2024-03-14T13:02:20.794 INFO:tasks.workunit.client.1.smithi137.stdout:7/454: mknod d3/d5/d23/d2c/d5d/d6a/c8c 0 2024-03-14T13:02:20.795 INFO:tasks.workunit.client.0.smithi111.stdout:8/785: getdents d3/d1b/d34 0 2024-03-14T13:02:20.796 INFO:tasks.workunit.client.1.smithi137.stdout:5/540: dread d7/d36/f82 [0,4194304] 0 2024-03-14T13:02:20.797 INFO:tasks.workunit.client.0.smithi111.stdout:8/786: truncate d3/d1b/d20/d35/d40/d48/d7e/db3/db5/fbe 870825 0 2024-03-14T13:02:20.798 INFO:tasks.workunit.client.1.smithi137.stdout:5/541: stat d7/d16/d18/d25/d28 0 2024-03-14T13:02:20.798 INFO:tasks.workunit.client.1.smithi137.stdout:7/455: fdatasync d3/d5/f46 0 2024-03-14T13:02:20.799 INFO:tasks.workunit.client.0.smithi111.stdout:8/787: mknod d3/d1b/d20/d35/d40/d48/d77/c101 0 2024-03-14T13:02:20.800 INFO:tasks.workunit.client.1.smithi137.stdout:2/596: link d2/f1e d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/fc3 0 2024-03-14T13:02:20.800 INFO:tasks.workunit.client.0.smithi111.stdout:8/788: write d3/d1b/d34/f3b [1647986,31014] 0 2024-03-14T13:02:20.802 INFO:tasks.workunit.client.1.smithi137.stdout:1/619: truncate d7/d9/d19/d1f/d22/f25 3169038 0 2024-03-14T13:02:20.804 INFO:tasks.workunit.client.0.smithi111.stdout:3/591: write d0/d2d/d35/d69/d84/f87 [81206,15588] 0 2024-03-14T13:02:20.808 INFO:tasks.workunit.client.0.smithi111.stdout:9/584: dwrite d5/f4f [0,4194304] 0 2024-03-14T13:02:20.811 INFO:tasks.workunit.client.0.smithi111.stdout:3/592: getdents d0/d2d/d35/d50/da2/db3/d71 0 2024-03-14T13:02:20.812 INFO:tasks.workunit.client.1.smithi137.stdout:8/614: sync 2024-03-14T13:02:20.814 INFO:tasks.workunit.client.0.smithi111.stdout:3/593: dread - d0/d2d/d35/d69/fa0 zero size 2024-03-14T13:02:20.816 INFO:tasks.workunit.client.1.smithi137.stdout:3/514: link d4/d22/fa5 d4/de/d13/faf 0 2024-03-14T13:02:20.816 INFO:tasks.workunit.client.1.smithi137.stdout:2/597: write d2/f1e [3588912,9119] 0 2024-03-14T13:02:20.817 INFO:tasks.workunit.client.1.smithi137.stdout:4/564: truncate d4/f48 1743132 0 2024-03-14T13:02:20.818 INFO:tasks.workunit.client.1.smithi137.stdout:5/542: mknod d7/d16/d18/d25/cbb 0 2024-03-14T13:02:20.819 INFO:tasks.workunit.client.0.smithi111.stdout:8/789: dwrite d3/f78 [0,4194304] 0 2024-03-14T13:02:20.823 INFO:tasks.workunit.client.1.smithi137.stdout:8/615: write d1/dc/d3c/d87/d34/d82/d96/fae [188394,39094] 0 2024-03-14T13:02:20.824 INFO:tasks.workunit.client.1.smithi137.stdout:6/578: dread d2/d6/f65 [0,4194304] 0 2024-03-14T13:02:20.826 INFO:tasks.workunit.client.1.smithi137.stdout:1/620: link d7/d9/d19/d40/d42/d9d/fba d7/d31/d54/fd2 0 2024-03-14T13:02:20.828 INFO:tasks.workunit.client.1.smithi137.stdout:0/556: truncate d1/d5/d1b/d1e/d25/d55/f60 4027502 0 2024-03-14T13:02:20.830 INFO:tasks.workunit.client.1.smithi137.stdout:8/616: fdatasync d1/dc/d3c/d87/d34/d82/fa7 0 2024-03-14T13:02:20.831 INFO:tasks.workunit.client.1.smithi137.stdout:5/543: write d7/d16/d18/d25/f5b [5232771,78459] 0 2024-03-14T13:02:20.831 INFO:tasks.workunit.client.1.smithi137.stdout:4/565: mkdir d4/d9/d3a/d41/d38/d79/dc2 0 2024-03-14T13:02:20.831 INFO:tasks.workunit.client.1.smithi137.stdout:7/456: dwrite d3/d6/d14/d22/f43 [0,4194304] 0 2024-03-14T13:02:20.832 INFO:tasks.workunit.client.0.smithi111.stdout:9/585: dread d5/d6/d8/d5b/d7c/f90 [4194304,4194304] 0 2024-03-14T13:02:20.832 INFO:tasks.workunit.client.1.smithi137.stdout:6/579: readlink d2/d54/d55/d6d/d76/l89 0 2024-03-14T13:02:20.834 INFO:tasks.workunit.client.0.smithi111.stdout:9/586: rename d5/d6/d14/l9d to d5/lc6 0 2024-03-14T13:02:20.837 INFO:tasks.workunit.client.1.smithi137.stdout:9/692: sync 2024-03-14T13:02:20.837 INFO:tasks.workunit.client.1.smithi137.stdout:3/515: dwrite d4/d22/f64 [0,4194304] 0 2024-03-14T13:02:20.839 INFO:tasks.workunit.client.1.smithi137.stdout:2/598: creat d2/d7/d2e/d3e/d44/d6b/d22/dbb/fc4 x:0 0 0 2024-03-14T13:02:20.842 INFO:tasks.workunit.client.1.smithi137.stdout:5/544: write d7/d36/f95 [187540,41148] 0 2024-03-14T13:02:20.843 INFO:tasks.workunit.client.1.smithi137.stdout:8/617: creat d1/dc/d71/da3/fd0 x:0 0 0 2024-03-14T13:02:20.843 INFO:tasks.workunit.client.1.smithi137.stdout:2/599: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f45 0 2024-03-14T13:02:20.844 INFO:tasks.workunit.client.1.smithi137.stdout:1/621: creat d7/d9/d19/d1f/d2e/d63/fd3 x:0 0 0 2024-03-14T13:02:20.846 INFO:tasks.workunit.client.1.smithi137.stdout:6/580: creat d2/d6/de/d45/d59/d39/d3c/fc8 x:0 0 0 2024-03-14T13:02:20.846 INFO:tasks.workunit.client.1.smithi137.stdout:4/566: fsync d4/f11 0 2024-03-14T13:02:20.847 INFO:tasks.workunit.client.0.smithi111.stdout:5/713: sync 2024-03-14T13:02:20.847 INFO:tasks.workunit.client.0.smithi111.stdout:8/790: truncate d3/d5/fb 2712900 0 2024-03-14T13:02:20.849 INFO:tasks.workunit.client.1.smithi137.stdout:1/622: dread - d7/d9/d19/d5f/f7f zero size 2024-03-14T13:02:20.850 INFO:tasks.workunit.client.0.smithi111.stdout:5/714: write d2/d16/d49/d7c/fc6 [367861,130805] 0 2024-03-14T13:02:20.850 INFO:tasks.workunit.client.0.smithi111.stdout:8/791: symlink d3/d5/l102 0 2024-03-14T13:02:20.852 INFO:tasks.workunit.client.1.smithi137.stdout:2/600: write d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 [1024535,17586] 0 2024-03-14T13:02:20.853 INFO:tasks.workunit.client.0.smithi111.stdout:3/594: dwrite d0/d2d/d35/d50/d5e/f12 [0,4194304] 0 2024-03-14T13:02:20.853 INFO:tasks.workunit.client.0.smithi111.stdout:5/715: chown d2/d16/d49/f11 18500 1 2024-03-14T13:02:20.854 INFO:tasks.workunit.client.0.smithi111.stdout:8/792: write d3/d22/fbb [2254811,83309] 0 2024-03-14T13:02:20.855 INFO:tasks.workunit.client.1.smithi137.stdout:7/457: creat d3/d52/d77/f8d x:0 0 0 2024-03-14T13:02:20.857 INFO:tasks.workunit.client.0.smithi111.stdout:3/595: creat d0/d2d/d35/d50/da2/db3/d6f/fdb x:0 0 0 2024-03-14T13:02:20.857 INFO:tasks.workunit.client.0.smithi111.stdout:5/716: dread d2/d13/fd3 [0,4194304] 0 2024-03-14T13:02:20.861 INFO:tasks.workunit.client.0.smithi111.stdout:5/717: mknod d2/d16/d49/d44/d72/de0/ce8 0 2024-03-14T13:02:20.868 INFO:tasks.workunit.client.0.smithi111.stdout:5/718: link d2/d13/cc9 d2/d45/d76/d8d/ce9 0 2024-03-14T13:02:20.869 INFO:tasks.workunit.client.0.smithi111.stdout:5/719: chown d2/d16/d49/d7c/fc6 420982003 1 2024-03-14T13:02:20.870 INFO:tasks.workunit.client.1.smithi137.stdout:7/458: mknod d3/d6/d59/c8e 0 2024-03-14T13:02:20.871 INFO:tasks.workunit.client.0.smithi111.stdout:5/720: write d2/d16/d49/d44/f8e [1020067,32675] 0 2024-03-14T13:02:20.872 INFO:tasks.workunit.client.1.smithi137.stdout:1/623: dwrite d7/d9/d19/d40/d50/f80 [0,4194304] 0 2024-03-14T13:02:20.872 INFO:tasks.workunit.client.0.smithi111.stdout:5/721: write d2/d13/f99 [22027,11661] 0 2024-03-14T13:02:20.876 INFO:tasks.workunit.client.1.smithi137.stdout:3/516: dread d4/de/d13/f21 [0,4194304] 0 2024-03-14T13:02:20.877 INFO:tasks.workunit.client.1.smithi137.stdout:4/567: dwrite d4/d4d/d6e/f9b [0,4194304] 0 2024-03-14T13:02:20.879 INFO:tasks.workunit.client.1.smithi137.stdout:8/618: dwrite d1/dc/d77/f78 [0,4194304] 0 2024-03-14T13:02:20.879 INFO:tasks.workunit.client.1.smithi137.stdout:1/624: fdatasync d7/d9/d19/d1f/d89/fb2 0 2024-03-14T13:02:20.882 INFO:tasks.workunit.client.0.smithi111.stdout:5/722: dread d2/d16/d49/d7c/d95/f5e [0,4194304] 0 2024-03-14T13:02:20.887 INFO:tasks.workunit.client.0.smithi111.stdout:5/723: truncate d2/d16/d49/d44/d4a/f8c 426362 0 2024-03-14T13:02:20.888 INFO:tasks.workunit.client.1.smithi137.stdout:2/601: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 x:0 0 0 2024-03-14T13:02:20.888 INFO:tasks.workunit.client.1.smithi137.stdout:7/459: mknod d3/d5/d23/d2c/d5d/d6a/d75/c8f 0 2024-03-14T13:02:20.888 INFO:tasks.workunit.client.1.smithi137.stdout:3/517: fsync d4/de/d16/d1c/d5d/d85/f5a 0 2024-03-14T13:02:20.889 INFO:tasks.workunit.client.0.smithi111.stdout:5/724: creat d2/d16/d49/d4c/d9c/fea x:0 0 0 2024-03-14T13:02:20.892 INFO:tasks.workunit.client.1.smithi137.stdout:8/619: symlink d1/d4a/ld1 0 2024-03-14T13:02:20.892 INFO:tasks.workunit.client.0.smithi111.stdout:5/725: chown d2/d16/d49/d33/d68 53310739 1 2024-03-14T13:02:20.893 INFO:tasks.workunit.client.1.smithi137.stdout:5/545: rename d7/d36/f71 to d7/d16/fbc 0 2024-03-14T13:02:20.894 INFO:tasks.workunit.client.1.smithi137.stdout:4/568: symlink d4/d9/d3a/d70/d80/d8d/dc0/lc3 0 2024-03-14T13:02:20.896 INFO:tasks.workunit.client.1.smithi137.stdout:7/460: creat d3/d4a/f90 x:0 0 0 2024-03-14T13:02:20.896 INFO:tasks.workunit.client.1.smithi137.stdout:3/518: chown d4/d19/f23 12793 1 2024-03-14T13:02:20.897 INFO:tasks.workunit.client.1.smithi137.stdout:2/602: write d2/d34/d3c/d53/fa7 [24873,102148] 0 2024-03-14T13:02:20.897 INFO:tasks.workunit.client.0.smithi111.stdout:8/793: dread d3/f6b [0,4194304] 0 2024-03-14T13:02:20.899 INFO:tasks.workunit.client.1.smithi137.stdout:0/557: dwrite d1/d1f/f28 [0,4194304] 0 2024-03-14T13:02:20.900 INFO:tasks.workunit.client.0.smithi111.stdout:8/794: rmdir d3/dc/d45/de1 0 2024-03-14T13:02:20.901 INFO:tasks.workunit.client.1.smithi137.stdout:9/693: dwrite d6/df/d19/f62 [0,4194304] 0 2024-03-14T13:02:20.901 INFO:tasks.workunit.client.1.smithi137.stdout:2/603: fsync d2/f4 0 2024-03-14T13:02:20.901 INFO:tasks.workunit.client.1.smithi137.stdout:8/620: truncate d1/dc/da9/fbe 155792 0 2024-03-14T13:02:20.902 INFO:tasks.workunit.client.0.smithi111.stdout:8/795: chown d3/d1b/d20/d35/d40/d48/d77 388243 1 2024-03-14T13:02:20.907 INFO:tasks.workunit.client.0.smithi111.stdout:8/796: getdents d3/dc 0 2024-03-14T13:02:20.907 INFO:tasks.workunit.client.0.smithi111.stdout:5/726: dwrite d2/f35 [0,4194304] 0 2024-03-14T13:02:20.916 INFO:tasks.workunit.client.0.smithi111.stdout:5/727: link d2/d16/d49/d7c/d95/l75 d2/d45/d79/leb 0 2024-03-14T13:02:20.918 INFO:tasks.workunit.client.0.smithi111.stdout:8/797: getdents d3/d1b/d20/d35/d40 0 2024-03-14T13:02:20.919 INFO:tasks.workunit.client.0.smithi111.stdout:5/728: mknod d2/d45/d76/d8d/cec 0 2024-03-14T13:02:20.920 INFO:tasks.workunit.client.0.smithi111.stdout:8/798: mknod d3/d1b/d20/df9/c103 0 2024-03-14T13:02:20.922 INFO:tasks.workunit.client.0.smithi111.stdout:8/799: readlink d3/d1b/d20/l71 0 2024-03-14T13:02:20.929 INFO:tasks.workunit.client.0.smithi111.stdout:8/800: dwrite d3/d22/fc4 [0,4194304] 0 2024-03-14T13:02:20.931 INFO:tasks.workunit.client.0.smithi111.stdout:8/801: read d3/d22/fc4 [2411318,11378] 0 2024-03-14T13:02:20.938 INFO:tasks.workunit.client.0.smithi111.stdout:8/802: rename d3/d22/fbb to d3/d22/da1/de6/f104 0 2024-03-14T13:02:20.939 INFO:tasks.workunit.client.0.smithi111.stdout:8/803: write d3/fec [789471,8596] 0 2024-03-14T13:02:20.940 INFO:tasks.workunit.client.0.smithi111.stdout:8/804: stat d3/dc/d45/lb8 0 2024-03-14T13:02:20.940 INFO:tasks.workunit.client.0.smithi111.stdout:5/729: dread d2/d16/d49/f27 [0,4194304] 0 2024-03-14T13:02:20.941 INFO:tasks.workunit.client.0.smithi111.stdout:8/805: readlink d3/d1b/d66/ld3 0 2024-03-14T13:02:20.943 INFO:tasks.workunit.client.0.smithi111.stdout:5/730: mknod d2/d13/ced 0 2024-03-14T13:02:20.946 INFO:tasks.workunit.client.0.smithi111.stdout:8/806: creat d3/d1b/d20/d35/d40/d48/d7e/db3/f105 x:0 0 0 2024-03-14T13:02:20.947 INFO:tasks.workunit.client.0.smithi111.stdout:5/731: creat d2/d16/d49/d7c/d95/fee x:0 0 0 2024-03-14T13:02:20.947 INFO:tasks.workunit.client.0.smithi111.stdout:8/807: chown d3/d1b/d20/l5b 2277215 1 2024-03-14T13:02:20.948 INFO:tasks.workunit.client.0.smithi111.stdout:5/732: readlink d2/d16/d49/d7c/d95/d6a/ld4 0 2024-03-14T13:02:20.949 INFO:tasks.workunit.client.0.smithi111.stdout:8/808: symlink d3/d1b/d66/l106 0 2024-03-14T13:02:20.951 INFO:tasks.workunit.client.0.smithi111.stdout:5/733: rename d2/d45/d76/d9a/d9f/lb6 to d2/d45/d76/db2/lef 0 2024-03-14T13:02:20.951 INFO:tasks.workunit.client.0.smithi111.stdout:8/809: creat d3/d1b/d34/f107 x:0 0 0 2024-03-14T13:02:20.952 INFO:tasks.workunit.client.0.smithi111.stdout:8/810: stat d3/d1b/d20 0 2024-03-14T13:02:20.954 INFO:tasks.workunit.client.1.smithi137.stdout:1/625: rename d7/f10 to d7/d9/d19/d1f/d26/d7d/fd4 0 2024-03-14T13:02:20.954 INFO:tasks.workunit.client.1.smithi137.stdout:7/461: read d3/d6/d14/f1a [1143738,105830] 0 2024-03-14T13:02:20.956 INFO:tasks.workunit.client.1.smithi137.stdout:3/519: unlink d4/d22/f2e 0 2024-03-14T13:02:20.956 INFO:tasks.workunit.client.0.smithi111.stdout:5/734: dread d2/d16/d49/d7c/d95/fa3 [0,4194304] 0 2024-03-14T13:02:20.957 INFO:tasks.workunit.client.0.smithi111.stdout:8/811: dread - d3/d22/d30/f95 zero size 2024-03-14T13:02:20.958 INFO:tasks.workunit.client.1.smithi137.stdout:6/581: dread d2/f5a [0,4194304] 0 2024-03-14T13:02:20.958 INFO:tasks.workunit.client.1.smithi137.stdout:5/546: rename d7/d16/d18/d25/d28/d3c/faf to d7/d49/fbd 0 2024-03-14T13:02:20.960 INFO:tasks.workunit.client.0.smithi111.stdout:8/812: mknod d3/d1b/d20/d35/d40/d48/d7e/db3/db5/c108 0 2024-03-14T13:02:20.964 INFO:tasks.workunit.client.0.smithi111.stdout:8/813: rmdir d3/d1b/d20/d35/d40/d48/d7e/db3/d100 0 2024-03-14T13:02:20.966 INFO:tasks.workunit.client.0.smithi111.stdout:8/814: fdatasync d3/d1b/d20/f2b 0 2024-03-14T13:02:20.969 INFO:tasks.workunit.client.0.smithi111.stdout:5/735: dread d2/d16/d49/d33/d68/fc3 [0,4194304] 0 2024-03-14T13:02:20.970 INFO:tasks.workunit.client.0.smithi111.stdout:8/815: mknod d3/d1b/d20/d35/d40/d48/d77/c109 0 2024-03-14T13:02:20.971 INFO:tasks.workunit.client.0.smithi111.stdout:5/736: creat d2/d16/d49/d44/ff0 x:0 0 0 2024-03-14T13:02:20.972 INFO:tasks.workunit.client.0.smithi111.stdout:5/737: chown d2/d13/f99 2233525 1 2024-03-14T13:02:20.973 INFO:tasks.workunit.client.0.smithi111.stdout:5/738: fsync d2/d45/f66 0 2024-03-14T13:02:20.974 INFO:tasks.workunit.client.0.smithi111.stdout:5/739: stat d2/d45/d76/fab 0 2024-03-14T13:02:20.974 INFO:tasks.workunit.client.0.smithi111.stdout:8/816: getdents d3/d1b/d20/d35/d40/d48/d7e/db3/db5 0 2024-03-14T13:02:20.976 INFO:tasks.workunit.client.0.smithi111.stdout:8/817: creat d3/d22/d30/f10a x:0 0 0 2024-03-14T13:02:20.980 INFO:tasks.workunit.client.1.smithi137.stdout:8/621: rmdir d1/d5e/daf 39 2024-03-14T13:02:20.980 INFO:tasks.workunit.client.1.smithi137.stdout:9/694: mkdir d6/df/d19/d33/ddf 0 2024-03-14T13:02:20.980 INFO:tasks.workunit.client.0.smithi111.stdout:8/818: link d3/d1b/d20/d35/d40/d48/d7e/db3/db5/cf3 d3/d49/dfb/c10b 0 2024-03-14T13:02:20.980 INFO:tasks.workunit.client.1.smithi137.stdout:5/547: dread - d7/d16/fb9 zero size 2024-03-14T13:02:20.981 INFO:tasks.workunit.client.1.smithi137.stdout:7/462: unlink d3/d5/d23/d2c/d5d/l82 0 2024-03-14T13:02:20.982 INFO:tasks.workunit.client.0.smithi111.stdout:8/819: creat d3/dc/ddb/f10c x:0 0 0 2024-03-14T13:02:20.983 INFO:tasks.workunit.client.1.smithi137.stdout:1/626: mknod d7/d9/d19/d1f/d89/d90/db1/cd5 0 2024-03-14T13:02:20.984 INFO:tasks.workunit.client.1.smithi137.stdout:8/622: chown d1/dc/d3c/d87/d34/d82/d94 6058 1 2024-03-14T13:02:20.985 INFO:tasks.workunit.client.0.smithi111.stdout:8/820: symlink d3/d1b/d20/df9/l10d 0 2024-03-14T13:02:20.986 INFO:tasks.workunit.client.0.smithi111.stdout:8/821: chown d3/d1b/d66/ff2 45 1 2024-03-14T13:02:20.987 INFO:tasks.workunit.client.1.smithi137.stdout:2/604: getdents d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99 0 2024-03-14T13:02:20.990 INFO:tasks.workunit.client.0.smithi111.stdout:5/740: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:20.990 INFO:tasks.workunit.client.1.smithi137.stdout:9/695: mknod d6/df/d19/d3a/ce0 0 2024-03-14T13:02:20.990 INFO:tasks.workunit.client.0.smithi111.stdout:8/822: fdatasync d3/d5/fb 0 2024-03-14T13:02:20.992 INFO:tasks.workunit.client.0.smithi111.stdout:8/823: readlink d3/dc/d31/la6 0 2024-03-14T13:02:20.992 INFO:tasks.workunit.client.1.smithi137.stdout:5/548: mkdir d7/d16/d18/d2d/dbe 0 2024-03-14T13:02:20.996 INFO:tasks.workunit.client.0.smithi111.stdout:8/824: creat d3/d22/d4a/f10e x:0 0 0 2024-03-14T13:02:20.997 INFO:tasks.workunit.client.1.smithi137.stdout:9/696: stat d6/df/d19/d3a/l97 0 2024-03-14T13:02:20.997 INFO:tasks.workunit.client.1.smithi137.stdout:6/582: rename d2/d6/de/d45/d59/d39/d3c/l91 to d2/d6/de/d45/d59/d39/lc9 0 2024-03-14T13:02:20.998 INFO:tasks.workunit.client.0.smithi111.stdout:5/741: rename d2/d16/d49/d7c/d95/f24 to d2/d16/d49/d44/d72/ff1 0 2024-03-14T13:02:20.999 INFO:tasks.workunit.client.1.smithi137.stdout:3/520: dwrite d4/de/d16/d1c/d5d/d85/d6b/f8b [0,4194304] 0 2024-03-14T13:02:20.999 INFO:tasks.workunit.client.1.smithi137.stdout:8/623: chown d1/d5e/daf 294 1 2024-03-14T13:02:20.999 INFO:tasks.workunit.client.1.smithi137.stdout:0/558: dwrite d1/d44/d56/f5f [0,4194304] 0 2024-03-14T13:02:21.000 INFO:tasks.workunit.client.0.smithi111.stdout:8/825: creat d3/d5/f10f x:0 0 0 2024-03-14T13:02:21.001 INFO:tasks.workunit.client.1.smithi137.stdout:3/521: chown d4/de/d13/c92 1 1 2024-03-14T13:02:21.004 INFO:tasks.workunit.client.0.smithi111.stdout:8/826: unlink d3/d22/d30/f10a 0 2024-03-14T13:02:21.004 INFO:tasks.workunit.client.1.smithi137.stdout:3/522: fsync d4/de/d16/d74/f9b 0 2024-03-14T13:02:21.004 INFO:tasks.workunit.client.1.smithi137.stdout:1/627: rename d7/d4d/f85 to d7/d9/d19/d1f/d89/d90/db1/fd6 0 2024-03-14T13:02:21.005 INFO:tasks.workunit.client.1.smithi137.stdout:3/523: chown d4/f18 3916 1 2024-03-14T13:02:21.005 INFO:tasks.workunit.client.0.smithi111.stdout:8/827: mkdir d3/dc/d31/d9d/d110 0 2024-03-14T13:02:21.006 INFO:tasks.workunit.client.1.smithi137.stdout:6/583: chown d2/d6/l2f 122141936 1 2024-03-14T13:02:21.009 INFO:tasks.workunit.client.1.smithi137.stdout:5/549: getdents d7/d16/d18/d25/d28/d9e 0 2024-03-14T13:02:21.009 INFO:tasks.workunit.client.1.smithi137.stdout:5/550: chown d7/d16 133305 1 2024-03-14T13:02:21.009 INFO:tasks.workunit.client.1.smithi137.stdout:5/551: stat d7/f96 0 2024-03-14T13:02:21.011 INFO:tasks.workunit.client.0.smithi111.stdout:8/828: mkdir d3/dc/d31/d9d/d110/d111 0 2024-03-14T13:02:21.011 INFO:tasks.workunit.client.1.smithi137.stdout:0/559: fdatasync d1/d5/d1b/d1e/f3d 0 2024-03-14T13:02:21.011 INFO:tasks.workunit.client.0.smithi111.stdout:8/829: chown d3/d1b/d34 1968409 1 2024-03-14T13:02:21.014 INFO:tasks.workunit.client.1.smithi137.stdout:7/463: truncate d3/d5/f7c 2783853 0 2024-03-14T13:02:21.016 INFO:tasks.workunit.client.1.smithi137.stdout:1/628: truncate d7/d9/d19/d1f/d26/f48 423573 0 2024-03-14T13:02:21.016 INFO:tasks.workunit.client.1.smithi137.stdout:7/464: fdatasync d3/d5/d23/d2c/f53 0 2024-03-14T13:02:21.017 INFO:tasks.workunit.client.1.smithi137.stdout:8/624: mkdir d1/dc/d10/dcf/dd2 0 2024-03-14T13:02:21.018 INFO:tasks.workunit.client.0.smithi111.stdout:5/742: dwrite d2/d13/fd3 [0,4194304] 0 2024-03-14T13:02:21.019 INFO:tasks.workunit.client.1.smithi137.stdout:1/629: write d7/d9/d19/d5f/f7f [1011433,19559] 0 2024-03-14T13:02:21.020 INFO:tasks.workunit.client.1.smithi137.stdout:0/560: mknod d1/d5/d1b/d42/d3a/db0/cb8 0 2024-03-14T13:02:21.021 INFO:tasks.workunit.client.1.smithi137.stdout:8/625: write d1/dc/d71/fba [690754,125598] 0 2024-03-14T13:02:21.023 INFO:tasks.workunit.client.1.smithi137.stdout:3/524: rename d4/d22/f98 to d4/de/d93/fb0 0 2024-03-14T13:02:21.023 INFO:tasks.workunit.client.0.smithi111.stdout:5/743: read d2/d16/d49/d44/d72/ff1 [3738224,117238] 0 2024-03-14T13:02:21.023 INFO:tasks.workunit.client.0.smithi111.stdout:5/744: stat d2/d13/cde 0 2024-03-14T13:02:21.024 INFO:tasks.workunit.client.1.smithi137.stdout:8/626: readlink d1/dc/d3c/d87/d34/d97/d13/l69 0 2024-03-14T13:02:21.024 INFO:tasks.workunit.client.1.smithi137.stdout:6/584: symlink d2/d54/lca 0 2024-03-14T13:02:21.028 INFO:tasks.workunit.client.1.smithi137.stdout:2/605: truncate d2/f14 521649 0 2024-03-14T13:02:21.031 INFO:tasks.workunit.client.0.smithi111.stdout:8/830: dwrite d3/d1b/d20/f37 [0,4194304] 0 2024-03-14T13:02:21.031 INFO:tasks.workunit.client.1.smithi137.stdout:1/630: dread - d7/d4d/f64 zero size 2024-03-14T13:02:21.032 INFO:tasks.workunit.client.0.smithi111.stdout:8/831: readlink d3/l54 0 2024-03-14T13:02:21.032 INFO:tasks.workunit.client.1.smithi137.stdout:3/525: mknod d4/d22/cb1 0 2024-03-14T13:02:21.033 INFO:tasks.workunit.client.1.smithi137.stdout:6/585: readlink d2/d54/d55/d6d/lae 0 2024-03-14T13:02:21.035 INFO:tasks.workunit.client.0.smithi111.stdout:8/832: fdatasync d3/d1b/d20/f9e 0 2024-03-14T13:02:21.035 INFO:tasks.workunit.client.1.smithi137.stdout:2/606: dread d2/d7/d2e/d3e/d44/d6b/f1b [0,4194304] 0 2024-03-14T13:02:21.036 INFO:tasks.workunit.client.1.smithi137.stdout:5/552: dread d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:21.038 INFO:tasks.workunit.client.1.smithi137.stdout:8/627: creat d1/d4a/dc4/fd3 x:0 0 0 2024-03-14T13:02:21.039 INFO:tasks.workunit.client.0.smithi111.stdout:5/745: dwrite d2/d16/d49/d33/d68/fc3 [0,4194304] 0 2024-03-14T13:02:21.040 INFO:tasks.workunit.client.0.smithi111.stdout:8/833: creat d3/d1b/f112 x:0 0 0 2024-03-14T13:02:21.043 INFO:tasks.workunit.client.0.smithi111.stdout:4/958: sync 2024-03-14T13:02:21.043 INFO:tasks.workunit.client.0.smithi111.stdout:7/492: sync 2024-03-14T13:02:21.043 INFO:tasks.workunit.client.0.smithi111.stdout:5/746: truncate d2/d45/f22 722506 0 2024-03-14T13:02:21.044 INFO:tasks.workunit.client.0.smithi111.stdout:1/635: sync 2024-03-14T13:02:21.047 INFO:tasks.workunit.client.0.smithi111.stdout:1/636: mknod d20/d25/cd3 0 2024-03-14T13:02:21.047 INFO:tasks.workunit.client.1.smithi137.stdout:1/631: rmdir d7/d9/d19 39 2024-03-14T13:02:21.048 INFO:tasks.workunit.client.0.smithi111.stdout:5/747: dread d2/d16/d49/d7c/d95/fa3 [0,4194304] 0 2024-03-14T13:02:21.051 INFO:tasks.workunit.client.0.smithi111.stdout:7/493: getdents d0/d1/d5/d38/d5c/d6d 0 2024-03-14T13:02:21.054 INFO:tasks.workunit.client.1.smithi137.stdout:1/632: creat d7/d9/d19/d40/d50/fd7 x:0 0 0 2024-03-14T13:02:21.054 INFO:tasks.workunit.client.0.smithi111.stdout:4/959: link d1d/d22/d40/d69/l129 d1d/d22/d30/d5e/db8/ddf/de3/l157 0 2024-03-14T13:02:21.055 INFO:tasks.workunit.client.0.smithi111.stdout:7/494: chown d0/d1/d4a/c5f 6026 1 2024-03-14T13:02:21.056 INFO:tasks.workunit.client.1.smithi137.stdout:3/526: rmdir d4 39 2024-03-14T13:02:21.057 INFO:tasks.workunit.client.1.smithi137.stdout:7/465: truncate d3/d5/f67 4051161 0 2024-03-14T13:02:21.058 INFO:tasks.workunit.client.0.smithi111.stdout:1/637: dwrite d20/d40/f9b [0,4194304] 0 2024-03-14T13:02:21.059 INFO:tasks.workunit.client.1.smithi137.stdout:3/527: chown d4/df/d82 0 1 2024-03-14T13:02:21.063 INFO:tasks.workunit.client.0.smithi111.stdout:5/748: dwrite d2/d16/d49/d44/d4a/f63 [4194304,4194304] 0 2024-03-14T13:02:21.064 INFO:tasks.workunit.client.1.smithi137.stdout:7/466: fdatasync d3/fe 0 2024-03-14T13:02:21.065 INFO:tasks.workunit.client.0.smithi111.stdout:1/638: truncate d20/d21/d28/d5c/fae 938164 0 2024-03-14T13:02:21.065 INFO:tasks.workunit.client.1.smithi137.stdout:8/628: dread d1/dc/f4b [0,4194304] 0 2024-03-14T13:02:21.066 INFO:tasks.workunit.client.0.smithi111.stdout:4/960: mkdir d1d/d22/d30/d5e/db8/ddf/d10f/d158 0 2024-03-14T13:02:21.066 INFO:tasks.workunit.client.1.smithi137.stdout:0/561: dwrite d1/d1f/d80/f96 [0,4194304] 0 2024-03-14T13:02:21.066 INFO:tasks.workunit.client.0.smithi111.stdout:5/749: write d2/d16/d49/d4c/f73 [1562594,80576] 0 2024-03-14T13:02:21.067 INFO:tasks.workunit.client.1.smithi137.stdout:7/467: read - d3/d52/d77/f8d zero size 2024-03-14T13:02:21.069 INFO:tasks.workunit.client.0.smithi111.stdout:8/834: dwrite d3/d22/f72 [0,4194304] 0 2024-03-14T13:02:21.069 INFO:tasks.workunit.client.1.smithi137.stdout:2/607: truncate d2/d34/d3c/f5d 3468440 0 2024-03-14T13:02:21.077 INFO:tasks.workunit.client.0.smithi111.stdout:8/835: getdents d3/d5 0 2024-03-14T13:02:21.081 INFO:tasks.workunit.client.0.smithi111.stdout:8/836: creat d3/dc/ddb/f113 x:0 0 0 2024-03-14T13:02:21.082 INFO:tasks.workunit.client.1.smithi137.stdout:0/562: creat d1/d1f/d82/fb9 x:0 0 0 2024-03-14T13:02:21.083 INFO:tasks.workunit.client.1.smithi137.stdout:3/528: write d4/de/d13/faf [214719,83223] 0 2024-03-14T13:02:21.084 INFO:tasks.workunit.client.0.smithi111.stdout:4/961: rename d1d/d48/d112/c12a to d1d/d22/d30/d7c/d59/dc4/d123/d115/c159 0 2024-03-14T13:02:21.086 INFO:tasks.workunit.client.0.smithi111.stdout:1/639: dread fa [0,4194304] 0 2024-03-14T13:02:21.087 INFO:tasks.workunit.client.0.smithi111.stdout:8/837: mknod d3/d22/c114 0 2024-03-14T13:02:21.087 INFO:tasks.workunit.client.1.smithi137.stdout:8/629: creat d1/dc/d4e/d6d/db5/fd4 x:0 0 0 2024-03-14T13:02:21.087 INFO:tasks.workunit.client.0.smithi111.stdout:5/750: dwrite d2/d16/d49/d44/d4a/d9e/db7/dbb/fcf [0,4194304] 0 2024-03-14T13:02:21.089 INFO:tasks.workunit.client.1.smithi137.stdout:2/608: link d2/d7/l15 d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/lc6 0 2024-03-14T13:02:21.089 INFO:tasks.workunit.client.1.smithi137.stdout:1/633: dwrite d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:21.089 INFO:tasks.workunit.client.1.smithi137.stdout:8/630: stat d1/dc/d71/fb6 0 2024-03-14T13:02:21.090 INFO:tasks.workunit.client.0.smithi111.stdout:4/962: unlink d1d/d22/d40/d69/fef 0 2024-03-14T13:02:21.090 INFO:tasks.workunit.client.0.smithi111.stdout:5/751: write d2/d45/f5c [1364601,123751] 0 2024-03-14T13:02:21.090 INFO:tasks.workunit.client.0.smithi111.stdout:8/838: write d3/d1b/d20/d35/d40/fdc [1590550,67263] 0 2024-03-14T13:02:21.091 INFO:tasks.workunit.client.1.smithi137.stdout:0/563: truncate d1/d5/d1b/d1e/d25/d26/f41 370687 0 2024-03-14T13:02:21.093 INFO:tasks.workunit.client.1.smithi137.stdout:4/569: sync 2024-03-14T13:02:21.093 INFO:tasks.workunit.client.1.smithi137.stdout:8/631: write d1/dc/d77/f78 [705828,60244] 0 2024-03-14T13:02:21.094 INFO:tasks.workunit.client.1.smithi137.stdout:1/634: symlink d7/d9/d19/d1f/d26/d7d/ld8 0 2024-03-14T13:02:21.094 INFO:tasks.workunit.client.1.smithi137.stdout:8/632: readlink d1/l9c 0 2024-03-14T13:02:21.097 INFO:tasks.workunit.client.1.smithi137.stdout:2/609: read d2/d7/d2e/d3e/d44/d6b/f39 [1388877,75788] 0 2024-03-14T13:02:21.098 INFO:tasks.workunit.client.0.smithi111.stdout:5/752: rename d2/d45/d79/lc2 to d2/d16/d49/d7c/d95/d36/lf2 0 2024-03-14T13:02:21.098 INFO:tasks.workunit.client.0.smithi111.stdout:1/640: symlink d20/d40/db4/ld4 0 2024-03-14T13:02:21.098 INFO:tasks.workunit.client.0.smithi111.stdout:4/963: chown d1d/d22/d30/d87/dd1/cda 9270081 1 2024-03-14T13:02:21.098 INFO:tasks.workunit.client.1.smithi137.stdout:8/633: write d1/d4a/dc4/fd3 [812638,56604] 0 2024-03-14T13:02:21.101 INFO:tasks.workunit.client.1.smithi137.stdout:8/634: fdatasync d1/dc/d3c/d87/d34/d97/d13/f9e 0 2024-03-14T13:02:21.103 INFO:tasks.workunit.client.0.smithi111.stdout:4/964: stat d1d/d22/d30/f32 0 2024-03-14T13:02:21.104 INFO:tasks.workunit.client.0.smithi111.stdout:8/839: getdents d3/d1b/d20/d35/d40/d48/d7e/db3/dfd 0 2024-03-14T13:02:21.104 INFO:tasks.workunit.client.0.smithi111.stdout:4/965: chown d1d/d22/d30/d7c/f11b 1883 1 2024-03-14T13:02:21.104 INFO:tasks.workunit.client.0.smithi111.stdout:5/753: dread d2/d16/d49/f27 [0,4194304] 0 2024-03-14T13:02:21.106 INFO:tasks.workunit.client.1.smithi137.stdout:1/635: fsync d7/d4d/d6b/fa3 0 2024-03-14T13:02:21.106 INFO:tasks.workunit.client.1.smithi137.stdout:0/564: fdatasync d1/d5/d1b/d1e/d25/d26/f41 0 2024-03-14T13:02:21.106 INFO:tasks.workunit.client.0.smithi111.stdout:1/641: link d20/f9d d20/d25/d31/d80/fd5 0 2024-03-14T13:02:21.107 INFO:tasks.workunit.client.0.smithi111.stdout:8/840: fdatasync d3/d22/d30/f74 0 2024-03-14T13:02:21.107 INFO:tasks.workunit.client.0.smithi111.stdout:4/966: mkdir d1d/d22/d30/d5e/d6f/d15a 0 2024-03-14T13:02:21.108 INFO:tasks.workunit.client.1.smithi137.stdout:2/610: rename d2/d7/l1f to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/lc7 0 2024-03-14T13:02:21.108 INFO:tasks.workunit.client.0.smithi111.stdout:5/754: mknod d2/d16/d49/d4c/cf3 0 2024-03-14T13:02:21.108 INFO:tasks.workunit.client.1.smithi137.stdout:0/565: write d1/f3e [3511130,14959] 0 2024-03-14T13:02:21.109 INFO:tasks.workunit.client.0.smithi111.stdout:1/642: unlink d20/d21/l2f 0 2024-03-14T13:02:21.110 INFO:tasks.workunit.client.1.smithi137.stdout:2/611: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/f63 0 2024-03-14T13:02:21.112 INFO:tasks.workunit.client.0.smithi111.stdout:8/841: creat d3/d1b/dc1/de4/f115 x:0 0 0 2024-03-14T13:02:21.113 INFO:tasks.workunit.client.0.smithi111.stdout:1/643: creat d20/d25/fd6 x:0 0 0 2024-03-14T13:02:21.113 INFO:tasks.workunit.client.0.smithi111.stdout:7/495: dread d0/d1/d21/f3f [8388608,4194304] 0 2024-03-14T13:02:21.114 INFO:tasks.workunit.client.1.smithi137.stdout:1/636: symlink d7/d9/d19/d1f/d26/d6c/dc8/ld9 0 2024-03-14T13:02:21.114 INFO:tasks.workunit.client.0.smithi111.stdout:5/755: rename d2/d16/d49/d44/l9d to d2/d16/d49/d33/lf4 0 2024-03-14T13:02:21.115 INFO:tasks.workunit.client.0.smithi111.stdout:4/967: mkdir d1d/d22/d30/d143/d15b 0 2024-03-14T13:02:21.116 INFO:tasks.workunit.client.1.smithi137.stdout:0/566: rename d1/d5/d1b/d42/f77 to d1/d5/d1b/d1e/d53/d6d/fba 0 2024-03-14T13:02:21.116 INFO:tasks.workunit.client.1.smithi137.stdout:7/468: write d3/d5/d23/f2b [754524,6075] 0 2024-03-14T13:02:21.118 INFO:tasks.workunit.client.0.smithi111.stdout:7/496: truncate d0/d1/d22/f7b 530018 0 2024-03-14T13:02:21.119 INFO:tasks.workunit.client.0.smithi111.stdout:4/968: symlink d1d/d105/l15c 0 2024-03-14T13:02:21.120 INFO:tasks.workunit.client.1.smithi137.stdout:8/635: link d1/dc/d3c/d87/d34/d97/f2a d1/dc/d3c/d87/d34/d82/fd5 0 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:6/569: sync 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:9/587: sync 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:7/497: fdatasync d0/d1/d5/f4c 0 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:0/744: sync 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:2/570: sync 2024-03-14T13:02:21.121 INFO:tasks.workunit.client.0.smithi111.stdout:3/596: sync 2024-03-14T13:02:21.124 INFO:tasks.workunit.client.0.smithi111.stdout:8/842: creat d3/d49/f116 x:0 0 0 2024-03-14T13:02:21.126 INFO:tasks.workunit.client.1.smithi137.stdout:4/570: write d4/d9/d3a/d41/d38/d55/f59 [3760324,95483] 0 2024-03-14T13:02:21.127 INFO:tasks.workunit.client.0.smithi111.stdout:3/597: readlink d0/d91/l96 0 2024-03-14T13:02:21.129 INFO:tasks.workunit.client.1.smithi137.stdout:0/567: rename d1/d1f/f3b to d1/d1f/d80/fbb 0 2024-03-14T13:02:21.129 INFO:tasks.workunit.client.1.smithi137.stdout:0/568: stat d1/da6 0 2024-03-14T13:02:21.130 INFO:tasks.workunit.client.0.smithi111.stdout:6/570: chown d0/l23 5086563 1 2024-03-14T13:02:21.130 INFO:tasks.workunit.client.1.smithi137.stdout:7/469: symlink d3/d5/d5a/d86/l91 0 2024-03-14T13:02:21.131 INFO:tasks.workunit.client.0.smithi111.stdout:2/571: chown dd/d17/d53/da4 48541054 1 2024-03-14T13:02:21.137 INFO:tasks.workunit.client.0.smithi111.stdout:1/644: dwrite d20/d25/d75/f7d [0,4194304] 0 2024-03-14T13:02:21.138 INFO:tasks.workunit.client.0.smithi111.stdout:5/756: dwrite d2/d16/d49/d44/f50 [4194304,4194304] 0 2024-03-14T13:02:21.139 INFO:tasks.workunit.client.1.smithi137.stdout:1/637: getdents d7/d9/d19/d1f/d26 0 2024-03-14T13:02:21.139 INFO:tasks.workunit.client.0.smithi111.stdout:7/498: fdatasync d0/d1/d22/f46 0 2024-03-14T13:02:21.139 INFO:tasks.workunit.client.1.smithi137.stdout:8/636: dread d1/dc/d3c/d87/d34/d97/d13/f45 [0,4194304] 0 2024-03-14T13:02:21.141 INFO:tasks.workunit.client.0.smithi111.stdout:6/571: write d0/d17/d20/d5a/fcb [392887,11146] 0 2024-03-14T13:02:21.143 INFO:tasks.workunit.client.1.smithi137.stdout:2/612: write d2/d7/d2e/d3e/d44/d6b/d1d/d41/f6e [877589,119091] 0 2024-03-14T13:02:21.145 INFO:tasks.workunit.client.0.smithi111.stdout:0/745: dwrite d6/fd [4194304,4194304] 0 2024-03-14T13:02:21.146 INFO:tasks.workunit.client.0.smithi111.stdout:8/843: mknod d3/dc/d31/d9d/c117 0 2024-03-14T13:02:21.147 INFO:tasks.workunit.client.1.smithi137.stdout:7/470: read d3/d5/f67 [1062710,104417] 0 2024-03-14T13:02:21.147 INFO:tasks.workunit.client.1.smithi137.stdout:8/637: creat d1/dc/d3c/fd6 x:0 0 0 2024-03-14T13:02:21.147 INFO:tasks.workunit.client.0.smithi111.stdout:5/757: readlink d2/d16/d49/d33/l39 0 2024-03-14T13:02:21.147 INFO:tasks.workunit.client.0.smithi111.stdout:9/588: rmdir d5/d42/d71 39 2024-03-14T13:02:21.148 INFO:tasks.workunit.client.1.smithi137.stdout:8/638: read - d1/dc/d3c/f8c zero size 2024-03-14T13:02:21.150 INFO:tasks.workunit.client.1.smithi137.stdout:7/471: chown d3/f1c 0 1 2024-03-14T13:02:21.150 INFO:tasks.workunit.client.0.smithi111.stdout:3/598: dwrite d0/d2d/d35/d50/d5e/d19/f1a [0,4194304] 0 2024-03-14T13:02:21.150 INFO:tasks.workunit.client.0.smithi111.stdout:6/572: fsync d0/d17/d21/d42/da5/fb4 0 2024-03-14T13:02:21.151 INFO:tasks.workunit.client.1.smithi137.stdout:4/571: dread d4/d27/d5c/d6d/d58/d65/f7b [0,4194304] 0 2024-03-14T13:02:21.151 INFO:tasks.workunit.client.1.smithi137.stdout:2/613: unlink d2/f28 0 2024-03-14T13:02:21.153 INFO:tasks.workunit.client.1.smithi137.stdout:7/472: write d3/d6/d14/f57 [4989805,16326] 0 2024-03-14T13:02:21.154 INFO:tasks.workunit.client.1.smithi137.stdout:0/569: link d1/d5/d1b/d1e/d25/d26/f2d d1/d5/d1b/d42/fbc 0 2024-03-14T13:02:21.155 INFO:tasks.workunit.client.1.smithi137.stdout:7/473: chown d3/d6/d2e 104373487 1 2024-03-14T13:02:21.157 INFO:tasks.workunit.client.0.smithi111.stdout:0/746: chown d6/d21/d74 15345 1 2024-03-14T13:02:21.157 INFO:tasks.workunit.client.0.smithi111.stdout:2/572: fdatasync dd/d11/d1a/f38 0 2024-03-14T13:02:21.158 INFO:tasks.workunit.client.1.smithi137.stdout:8/639: unlink d1/d4a/dc4/fd3 0 2024-03-14T13:02:21.161 INFO:tasks.workunit.client.1.smithi137.stdout:1/638: getdents d7/d9/d19 0 2024-03-14T13:02:21.162 INFO:tasks.workunit.client.1.smithi137.stdout:0/570: symlink d1/d5/d1b/d1e/d25/d9f/lbd 0 2024-03-14T13:02:21.163 INFO:tasks.workunit.client.1.smithi137.stdout:0/571: chown d1/d44 35 1 2024-03-14T13:02:21.163 INFO:tasks.workunit.client.1.smithi137.stdout:2/614: symlink d2/d7/d2e/d3e/d44/d6b/d22/d97/lc8 0 2024-03-14T13:02:21.164 INFO:tasks.workunit.client.0.smithi111.stdout:5/758: read d2/d16/d49/f11 [481637,41325] 0 2024-03-14T13:02:21.164 INFO:tasks.workunit.client.1.smithi137.stdout:4/572: rename d4/d9/d3a/d70/d80/d8d/c96 to d4/d9/d3a/d41/d38/d79/dc2/cc4 0 2024-03-14T13:02:21.165 INFO:tasks.workunit.client.0.smithi111.stdout:7/499: rmdir d0/d1/d21/d5d 39 2024-03-14T13:02:21.165 INFO:tasks.workunit.client.0.smithi111.stdout:8/844: mknod d3/c118 0 2024-03-14T13:02:21.166 INFO:tasks.workunit.client.0.smithi111.stdout:0/747: mkdir d6/d9/d15/d55/d108 0 2024-03-14T13:02:21.166 INFO:tasks.workunit.client.0.smithi111.stdout:1/645: dwrite d20/d25/f7b [0,4194304] 0 2024-03-14T13:02:21.168 INFO:tasks.workunit.client.1.smithi137.stdout:0/572: fsync d1/f7b 0 2024-03-14T13:02:21.169 INFO:tasks.workunit.client.0.smithi111.stdout:6/573: unlink d0/d17/d20/fc5 0 2024-03-14T13:02:21.169 INFO:tasks.workunit.client.1.smithi137.stdout:8/640: rename d1/d5e/l6f to d1/dc/d71/da3/ld7 0 2024-03-14T13:02:21.169 INFO:tasks.workunit.client.0.smithi111.stdout:7/500: write d0/d1/f28 [6794926,109117] 0 2024-03-14T13:02:21.170 INFO:tasks.workunit.client.0.smithi111.stdout:2/573: write dd/d11/d1a/d88/f89 [3123091,120596] 0 2024-03-14T13:02:21.172 INFO:tasks.workunit.client.0.smithi111.stdout:3/599: creat d0/d2d/fdc x:0 0 0 2024-03-14T13:02:21.173 INFO:tasks.workunit.client.0.smithi111.stdout:1/646: readlink d20/d21/d28/d5c/l67 0 2024-03-14T13:02:21.174 INFO:tasks.workunit.client.1.smithi137.stdout:2/615: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/lc9 0 2024-03-14T13:02:21.175 INFO:tasks.workunit.client.0.smithi111.stdout:3/600: readlink d0/d2d/d35/d50/lc0 0 2024-03-14T13:02:21.175 INFO:tasks.workunit.client.0.smithi111.stdout:0/748: mkdir d6/d3d/d97/d109 0 2024-03-14T13:02:21.176 INFO:tasks.workunit.client.1.smithi137.stdout:0/573: rename d1/d5/l47 to d1/d5/d1b/d1e/d25/d8a/d9d/lbe 0 2024-03-14T13:02:21.177 INFO:tasks.workunit.client.0.smithi111.stdout:3/601: write d0/d2d/d35/d50/d6c/fba [2174278,15549] 0 2024-03-14T13:02:21.178 INFO:tasks.workunit.client.1.smithi137.stdout:4/573: dread d4/d4d/f6b [0,4194304] 0 2024-03-14T13:02:21.180 INFO:tasks.workunit.client.0.smithi111.stdout:8/845: unlink d3/d22/d4a/fb6 0 2024-03-14T13:02:21.181 INFO:tasks.workunit.client.0.smithi111.stdout:4/969: dwrite d1d/d22/d30/d7c/d59/dc4/d123/d115/f7f [0,4194304] 0 2024-03-14T13:02:21.182 INFO:tasks.workunit.client.1.smithi137.stdout:8/641: dwrite d1/dc/d3c/d87/fc1 [0,4194304] 0 2024-03-14T13:02:21.183 INFO:tasks.workunit.client.0.smithi111.stdout:9/589: link d5/d42/d52/c61 d5/d6/d8/d1a/cc7 0 2024-03-14T13:02:21.183 INFO:tasks.workunit.client.1.smithi137.stdout:8/642: chown d1/dc/d10/fa4 3 1 2024-03-14T13:02:21.186 INFO:tasks.workunit.client.0.smithi111.stdout:8/846: write d3/dc/f99 [415881,117796] 0 2024-03-14T13:02:21.189 INFO:tasks.workunit.client.1.smithi137.stdout:8/643: mknod d1/dc/d3c/d87/d34/d97/d7f/cd8 0 2024-03-14T13:02:21.192 INFO:tasks.workunit.client.1.smithi137.stdout:0/574: getdents d1/d44 0 2024-03-14T13:02:21.194 INFO:tasks.workunit.client.0.smithi111.stdout:6/574: mkdir d0/d17/d20/d4a/d68/dd3 0 2024-03-14T13:02:21.194 INFO:tasks.workunit.client.1.smithi137.stdout:8/644: unlink d1/dc/d77/d9f/l91 0 2024-03-14T13:02:21.195 INFO:tasks.workunit.client.0.smithi111.stdout:3/602: symlink d0/d2d/d35/d50/da2/ldd 0 2024-03-14T13:02:21.195 INFO:tasks.workunit.client.0.smithi111.stdout:1/647: mkdir d20/d21/dd7 0 2024-03-14T13:02:21.196 INFO:tasks.workunit.client.1.smithi137.stdout:8/645: write d1/d5e/f61 [1363887,68374] 0 2024-03-14T13:02:21.198 INFO:tasks.workunit.client.1.smithi137.stdout:2/616: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/ff [0,4194304] 0 2024-03-14T13:02:21.199 INFO:tasks.workunit.client.0.smithi111.stdout:9/590: stat d5/d42/d71/d96 0 2024-03-14T13:02:21.201 INFO:tasks.workunit.client.0.smithi111.stdout:6/575: chown d0/d17/d20/f48 1193653 1 2024-03-14T13:02:21.203 INFO:tasks.workunit.client.1.smithi137.stdout:8/646: mknod d1/dc/d71/db0/cd9 0 2024-03-14T13:02:21.203 INFO:tasks.workunit.client.0.smithi111.stdout:0/749: dread d6/d9/de/f30 [0,4194304] 0 2024-03-14T13:02:21.205 INFO:tasks.workunit.client.0.smithi111.stdout:7/501: dwrite d0/d1/d21/d2a/f8a [0,4194304] 0 2024-03-14T13:02:21.206 INFO:tasks.workunit.client.1.smithi137.stdout:2/617: fsync d2/d34/f7f 0 2024-03-14T13:02:21.207 INFO:tasks.workunit.client.1.smithi137.stdout:1/639: dwrite d7/d31/d54/fd2 [0,4194304] 0 2024-03-14T13:02:21.208 INFO:tasks.workunit.client.0.smithi111.stdout:9/591: chown d5/d6/d8/c8a 8354650 1 2024-03-14T13:02:21.208 INFO:tasks.workunit.client.0.smithi111.stdout:8/847: creat d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f119 x:0 0 0 2024-03-14T13:02:21.208 INFO:tasks.workunit.client.0.smithi111.stdout:3/603: dread d0/d2d/d35/f61 [0,4194304] 0 2024-03-14T13:02:21.210 INFO:tasks.workunit.client.1.smithi137.stdout:8/647: mkdir d1/d5e/dc9/dda 0 2024-03-14T13:02:21.210 INFO:tasks.workunit.client.0.smithi111.stdout:2/574: truncate dd/f76 327717 0 2024-03-14T13:02:21.212 INFO:tasks.workunit.client.1.smithi137.stdout:1/640: fsync d7/d9/d19/d1f/f95 0 2024-03-14T13:02:21.214 INFO:tasks.workunit.client.1.smithi137.stdout:8/648: read - d1/dc/d3c/d87/d34/fc6 zero size 2024-03-14T13:02:21.215 INFO:tasks.workunit.client.0.smithi111.stdout:1/648: mknod d20/d25/d31/d80/cd8 0 2024-03-14T13:02:21.217 INFO:tasks.workunit.client.1.smithi137.stdout:2/618: rmdir d2/d7/d2e/d3e/d44/d6b/d1d 39 2024-03-14T13:02:21.217 INFO:tasks.workunit.client.0.smithi111.stdout:8/848: chown d3/d22/d4a/f5c 2227745 1 2024-03-14T13:02:21.218 INFO:tasks.workunit.client.0.smithi111.stdout:6/576: write d0/d17/d21/d69/f4d [2648837,8915] 0 2024-03-14T13:02:21.221 INFO:tasks.workunit.client.0.smithi111.stdout:1/649: fsync d20/d25/f3d 0 2024-03-14T13:02:21.222 INFO:tasks.workunit.client.1.smithi137.stdout:0/575: dread d1/d5/d1b/d1e/f3d [0,4194304] 0 2024-03-14T13:02:21.223 INFO:tasks.workunit.client.0.smithi111.stdout:0/750: creat d6/d45/d87/f10a x:0 0 0 2024-03-14T13:02:21.223 INFO:tasks.workunit.client.1.smithi137.stdout:1/641: link d7/d9/d19/d1f/d2e/f34 d7/d9/d19/d1f/d89/fda 0 2024-03-14T13:02:21.224 INFO:tasks.workunit.client.0.smithi111.stdout:3/604: stat d0/d30/c49 0 2024-03-14T13:02:21.225 INFO:tasks.workunit.client.0.smithi111.stdout:7/502: symlink d0/d1/d22/d3e/l96 0 2024-03-14T13:02:21.225 INFO:tasks.workunit.client.0.smithi111.stdout:4/970: dwrite d1d/f11e [0,4194304] 0 2024-03-14T13:02:21.225 INFO:tasks.workunit.client.0.smithi111.stdout:5/759: dread d2/d16/d49/d44/f50 [0,4194304] 0 2024-03-14T13:02:21.227 INFO:tasks.workunit.client.1.smithi137.stdout:0/576: creat d1/d5/d1b/d1e/d25/d8a/d4a/fbf x:0 0 0 2024-03-14T13:02:21.228 INFO:tasks.workunit.client.1.smithi137.stdout:9/697: sync 2024-03-14T13:02:21.228 INFO:tasks.workunit.client.1.smithi137.stdout:6/586: sync 2024-03-14T13:02:21.228 INFO:tasks.workunit.client.1.smithi137.stdout:3/529: sync 2024-03-14T13:02:21.228 INFO:tasks.workunit.client.1.smithi137.stdout:5/553: sync 2024-03-14T13:02:21.230 INFO:tasks.workunit.client.1.smithi137.stdout:1/642: unlink d7/d9/d19/d1f/d26/d7d/ld8 0 2024-03-14T13:02:21.230 INFO:tasks.workunit.client.0.smithi111.stdout:3/605: chown d0/f23 1668267 1 2024-03-14T13:02:21.231 INFO:tasks.workunit.client.0.smithi111.stdout:7/503: chown d0/d11/l7a 1322882 1 2024-03-14T13:02:21.233 INFO:tasks.workunit.client.0.smithi111.stdout:1/650: creat d20/d25/d87/fd9 x:0 0 0 2024-03-14T13:02:21.234 INFO:tasks.workunit.client.0.smithi111.stdout:5/760: write d2/d45/f5c [2490686,22250] 0 2024-03-14T13:02:21.234 INFO:tasks.workunit.client.0.smithi111.stdout:8/849: rename d3/d22/d4a to d3/dc/d45/d11a 0 2024-03-14T13:02:21.234 INFO:tasks.workunit.client.1.smithi137.stdout:0/577: creat d1/d44/d72/fc0 x:0 0 0 2024-03-14T13:02:21.236 INFO:tasks.workunit.client.0.smithi111.stdout:4/971: creat d1d/d22/d30/d5e/db8/ddf/f15d x:0 0 0 2024-03-14T13:02:21.237 INFO:tasks.workunit.client.1.smithi137.stdout:2/619: dwrite d2/f1e [0,4194304] 0 2024-03-14T13:02:21.238 INFO:tasks.workunit.client.1.smithi137.stdout:1/643: chown d7/d31/d82/cb0 52 1 2024-03-14T13:02:21.240 INFO:tasks.workunit.client.0.smithi111.stdout:3/606: mkdir d0/d91/dd8/dde 0 2024-03-14T13:02:21.240 INFO:tasks.workunit.client.1.smithi137.stdout:0/578: dread - d1/d44/d72/d7e/f81 zero size 2024-03-14T13:02:21.242 INFO:tasks.workunit.client.0.smithi111.stdout:5/761: write d2/d16/d49/d4c/f73 [4068199,74965] 0 2024-03-14T13:02:21.244 INFO:tasks.workunit.client.1.smithi137.stdout:4/574: dread d4/f13 [0,4194304] 0 2024-03-14T13:02:21.245 INFO:tasks.workunit.client.1.smithi137.stdout:8/649: truncate d1/dc/d3c/d87/d34/d97/f2d 3096975 0 2024-03-14T13:02:21.246 INFO:tasks.workunit.client.1.smithi137.stdout:1/644: chown d7/d9/d19/d1f/l70 65860588 1 2024-03-14T13:02:21.246 INFO:tasks.workunit.client.0.smithi111.stdout:6/577: link d0/d17/d21/d69/ccd d0/d17/d20/d65/cd4 0 2024-03-14T13:02:21.247 INFO:tasks.workunit.client.1.smithi137.stdout:7/474: sync 2024-03-14T13:02:21.247 INFO:tasks.workunit.client.1.smithi137.stdout:4/575: stat d4/d9/d3a/c5f 0 2024-03-14T13:02:21.248 INFO:tasks.workunit.client.1.smithi137.stdout:3/530: symlink d4/de/d16/lb2 0 2024-03-14T13:02:21.248 INFO:tasks.workunit.client.0.smithi111.stdout:7/504: rename d0/d1/d21/f15 to d0/d1/d4a/f97 0 2024-03-14T13:02:21.252 INFO:tasks.workunit.client.1.smithi137.stdout:1/645: mkdir d7/d9/d19/d5f/ddb 0 2024-03-14T13:02:21.252 INFO:tasks.workunit.client.0.smithi111.stdout:8/850: creat d3/d22/da1/f11b x:0 0 0 2024-03-14T13:02:21.252 INFO:tasks.workunit.client.0.smithi111.stdout:0/751: dwrite d6/d9/de/f30 [0,4194304] 0 2024-03-14T13:02:21.256 INFO:tasks.workunit.client.1.smithi137.stdout:9/698: dwrite d6/fe [0,4194304] 0 2024-03-14T13:02:21.256 INFO:tasks.workunit.client.0.smithi111.stdout:4/972: mkdir d1d/d22/d30/d15e 0 2024-03-14T13:02:21.256 INFO:tasks.workunit.client.0.smithi111.stdout:7/505: dread d0/d1/d22/f7b [0,4194304] 0 2024-03-14T13:02:21.256 INFO:tasks.workunit.client.1.smithi137.stdout:6/587: dread d2/d54/d55/d33/fa0 [0,4194304] 0 2024-03-14T13:02:21.260 INFO:tasks.workunit.client.0.smithi111.stdout:4/973: fdatasync d1d/d22/d30/d5e/d6f/f14a 0 2024-03-14T13:02:21.260 INFO:tasks.workunit.client.0.smithi111.stdout:7/506: write d0/d7e/f19 [761453,54451] 0 2024-03-14T13:02:21.261 INFO:tasks.workunit.client.1.smithi137.stdout:4/576: chown d4/d27/f2b 19027 1 2024-03-14T13:02:21.261 INFO:tasks.workunit.client.1.smithi137.stdout:3/531: symlink d4/de/d16/d1c/d5d/lb3 0 2024-03-14T13:02:21.261 INFO:tasks.workunit.client.1.smithi137.stdout:6/588: read d2/f17 [517023,38389] 0 2024-03-14T13:02:21.262 INFO:tasks.workunit.client.0.smithi111.stdout:4/974: chown d1d/d22/d30/d7c/d145 1 1 2024-03-14T13:02:21.262 INFO:tasks.workunit.client.1.smithi137.stdout:5/554: dwrite d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:21.263 INFO:tasks.workunit.client.1.smithi137.stdout:8/650: creat d1/dc/d3c/d87/d34/d82/d94/fdb x:0 0 0 2024-03-14T13:02:21.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/578: symlink d0/d17/d20/d4a/d59/d80/ld5 0 2024-03-14T13:02:21.263 INFO:tasks.workunit.client.0.smithi111.stdout:5/762: symlink d2/d16/d49/d4c/d9c/lf5 0 2024-03-14T13:02:21.263 INFO:tasks.workunit.client.0.smithi111.stdout:0/752: chown d6/d9/de/f6a 11905900 1 2024-03-14T13:02:21.265 INFO:tasks.workunit.client.1.smithi137.stdout:2/620: dwrite d2/d7/d2e/d3e/d44/d6b/f1b [0,4194304] 0 2024-03-14T13:02:21.269 INFO:tasks.workunit.client.1.smithi137.stdout:6/589: truncate d2/d6/f5c 1421994 0 2024-03-14T13:02:21.271 INFO:tasks.workunit.client.1.smithi137.stdout:8/651: rmdir d1/d5e/daf 39 2024-03-14T13:02:21.273 INFO:tasks.workunit.client.0.smithi111.stdout:9/592: dread d5/fa [0,4194304] 0 2024-03-14T13:02:21.273 INFO:tasks.workunit.client.0.smithi111.stdout:4/975: chown l6 2653 1 2024-03-14T13:02:21.274 INFO:tasks.workunit.client.1.smithi137.stdout:6/590: write d2/d6/de/d45/d59/f5e [3702663,2941] 0 2024-03-14T13:02:21.279 INFO:tasks.workunit.client.1.smithi137.stdout:6/591: read d2/d54/d55/d6d/d76/d57/f70 [728952,2689] 0 2024-03-14T13:02:21.280 INFO:tasks.workunit.client.1.smithi137.stdout:2/621: unlink d2/d7/d2e/d3e/d44/f60 0 2024-03-14T13:02:21.281 INFO:tasks.workunit.client.0.smithi111.stdout:0/753: readlink d6/d5a/l8e 0 2024-03-14T13:02:21.281 INFO:tasks.workunit.client.0.smithi111.stdout:5/763: unlink d2/d16/d49/d7c/f90 0 2024-03-14T13:02:21.281 INFO:tasks.workunit.client.0.smithi111.stdout:3/607: dwrite d0/d2d/d35/d50/da2/db3/d71/dbc/fd0 [0,4194304] 0 2024-03-14T13:02:21.282 INFO:tasks.workunit.client.1.smithi137.stdout:9/699: dread d6/df/d19/d3a/d40/f4b [0,4194304] 0 2024-03-14T13:02:21.284 INFO:tasks.workunit.client.1.smithi137.stdout:2/622: readlink d2/d34/d3c/d53/l5b 0 2024-03-14T13:02:21.284 INFO:tasks.workunit.client.1.smithi137.stdout:7/475: write d3/d6/f45 [840979,63233] 0 2024-03-14T13:02:21.284 INFO:tasks.workunit.client.1.smithi137.stdout:6/592: fdatasync d2/d6/f34 0 2024-03-14T13:02:21.285 INFO:tasks.workunit.client.1.smithi137.stdout:8/652: creat d1/d5e/dc9/dda/fdc x:0 0 0 2024-03-14T13:02:21.287 INFO:tasks.workunit.client.0.smithi111.stdout:8/851: dwrite d3/d22/da1/de6/f104 [0,4194304] 0 2024-03-14T13:02:21.288 INFO:tasks.workunit.client.0.smithi111.stdout:9/593: readlink d5/d6/d8/d5b/d84/l85 0 2024-03-14T13:02:21.292 INFO:tasks.workunit.client.1.smithi137.stdout:8/653: chown d1/dc/d3c/d87/d34/d82/fa7 7979 1 2024-03-14T13:02:21.292 INFO:tasks.workunit.client.1.smithi137.stdout:0/579: dwrite d1/d1f/d82/fb7 [0,4194304] 0 2024-03-14T13:02:21.293 INFO:tasks.workunit.client.1.smithi137.stdout:1/646: write d7/d9/d19/d1f/d26/d7d/fd4 [170336,127272] 0 2024-03-14T13:02:21.297 INFO:tasks.workunit.client.0.smithi111.stdout:6/579: creat d0/d17/d20/d4a/d59/dc3/fd6 x:0 0 0 2024-03-14T13:02:21.297 INFO:tasks.workunit.client.0.smithi111.stdout:0/754: symlink d6/d9/de/d77/da4/dca/dd0/l10b 0 2024-03-14T13:02:21.300 INFO:tasks.workunit.client.0.smithi111.stdout:1/651: dwrite d20/d21/d3f/d59/f76 [0,4194304] 0 2024-03-14T13:02:21.302 INFO:tasks.workunit.client.1.smithi137.stdout:3/532: read d4/de/d93/fb0 [333222,82599] 0 2024-03-14T13:02:21.302 INFO:tasks.workunit.client.1.smithi137.stdout:7/476: creat d3/d5/d23/d2c/d5d/f92 x:0 0 0 2024-03-14T13:02:21.302 INFO:tasks.workunit.client.1.smithi137.stdout:6/593: symlink d2/d54/d55/d6d/d76/d57/lcb 0 2024-03-14T13:02:21.302 INFO:tasks.workunit.client.1.smithi137.stdout:2/623: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/dca 0 2024-03-14T13:02:21.303 INFO:tasks.workunit.client.1.smithi137.stdout:3/533: chown d4/d22/cb1 37523 1 2024-03-14T13:02:21.303 INFO:tasks.workunit.client.1.smithi137.stdout:8/654: chown d1/dc/d3c/d87/d34/d97/f2a 4 1 2024-03-14T13:02:21.303 INFO:tasks.workunit.client.1.smithi137.stdout:7/477: fdatasync d3/d5/d5a/f7f 0 2024-03-14T13:02:21.305 INFO:tasks.workunit.client.1.smithi137.stdout:3/534: stat d4/d19/f90 0 2024-03-14T13:02:21.306 INFO:tasks.workunit.client.0.smithi111.stdout:9/594: creat d5/d42/d71/dbd/fc8 x:0 0 0 2024-03-14T13:02:21.307 INFO:tasks.workunit.client.0.smithi111.stdout:0/755: creat d6/d3d/d97/f10c x:0 0 0 2024-03-14T13:02:21.307 INFO:tasks.workunit.client.0.smithi111.stdout:9/595: chown d5/d6/d8/d1a 269320 1 2024-03-14T13:02:21.307 INFO:tasks.workunit.client.0.smithi111.stdout:6/580: mkdir d0/d17/d21/d69/d3d/d53/d81/da1/dd7 0 2024-03-14T13:02:21.308 INFO:tasks.workunit.client.0.smithi111.stdout:1/652: truncate d20/d25/d31/f3b 4346672 0 2024-03-14T13:02:21.309 INFO:tasks.workunit.client.0.smithi111.stdout:3/608: mkdir d0/d2d/d35/d50/ddf 0 2024-03-14T13:02:21.310 INFO:tasks.workunit.client.1.smithi137.stdout:8/655: rmdir d1/dc/d10 39 2024-03-14T13:02:21.311 INFO:tasks.workunit.client.1.smithi137.stdout:4/577: dread d4/d10/d42/f52 [0,4194304] 0 2024-03-14T13:02:21.313 INFO:tasks.workunit.client.1.smithi137.stdout:7/478: stat d3/d5/d23/d3f/d5b/f79 0 2024-03-14T13:02:21.314 INFO:tasks.workunit.client.0.smithi111.stdout:6/581: symlink d0/ld8 0 2024-03-14T13:02:21.314 INFO:tasks.workunit.client.0.smithi111.stdout:0/756: mknod d6/d9/de/d3a/c10d 0 2024-03-14T13:02:21.314 INFO:tasks.workunit.client.0.smithi111.stdout:9/596: symlink d5/d6/d2a/d2d/db8/lc9 0 2024-03-14T13:02:21.315 INFO:tasks.workunit.client.0.smithi111.stdout:3/609: rmdir d0/d2d/d35/d4c/dad 39 2024-03-14T13:02:21.319 INFO:tasks.workunit.client.0.smithi111.stdout:0/757: write d6/d9/de/d3a/f95 [1618704,56725] 0 2024-03-14T13:02:21.321 INFO:tasks.workunit.client.1.smithi137.stdout:4/578: rmdir d4/d9/d3a/d41 39 2024-03-14T13:02:21.322 INFO:tasks.workunit.client.1.smithi137.stdout:1/647: dread d7/d9/d19/d40/d50/f80 [0,4194304] 0 2024-03-14T13:02:21.323 INFO:tasks.workunit.client.0.smithi111.stdout:7/507: truncate d0/d11/f73 57176 0 2024-03-14T13:02:21.324 INFO:tasks.workunit.client.0.smithi111.stdout:3/610: fsync d0/d2d/d35/d69/fa0 0 2024-03-14T13:02:21.325 INFO:tasks.workunit.client.1.smithi137.stdout:6/594: dwrite d2/d54/d55/d6d/d76/d57/f9a [0,4194304] 0 2024-03-14T13:02:21.326 INFO:tasks.workunit.client.1.smithi137.stdout:3/535: truncate d4/fd 1040601 0 2024-03-14T13:02:21.327 INFO:tasks.workunit.client.0.smithi111.stdout:7/508: fsync d0/d7e/f19 0 2024-03-14T13:02:21.328 INFO:tasks.workunit.client.0.smithi111.stdout:0/758: mknod d6/d9/d5b/c10e 0 2024-03-14T13:02:21.334 INFO:tasks.workunit.client.1.smithi137.stdout:2/624: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f80 [207464,120786] 0 2024-03-14T13:02:21.334 INFO:tasks.workunit.client.0.smithi111.stdout:1/653: link d20/d21/d28/l88 d20/d25/dd0/dd2/lda 0 2024-03-14T13:02:21.334 INFO:tasks.workunit.client.0.smithi111.stdout:7/509: truncate d0/d1/d5/f4c 4513937 0 2024-03-14T13:02:21.337 INFO:tasks.workunit.client.1.smithi137.stdout:0/580: dwrite d1/d5/d1b/d42/f4b [0,4194304] 0 2024-03-14T13:02:21.338 INFO:tasks.workunit.client.0.smithi111.stdout:6/582: dwrite d0/d17/d20/d5a/fa2 [0,4194304] 0 2024-03-14T13:02:21.339 INFO:tasks.workunit.client.0.smithi111.stdout:5/764: dread d2/d16/d49/d7c/d8b/f91 [0,4194304] 0 2024-03-14T13:02:21.341 INFO:tasks.workunit.client.0.smithi111.stdout:5/765: write d2/d16/d49/d44/d4a/faf [449875,130598] 0 2024-03-14T13:02:21.341 INFO:tasks.workunit.client.1.smithi137.stdout:3/536: chown d4/de/d16/d1c/d5d/d85/d6b 57 1 2024-03-14T13:02:21.342 INFO:tasks.workunit.client.0.smithi111.stdout:7/510: readlink d0/d1/d21/d5d/l71 0 2024-03-14T13:02:21.343 INFO:tasks.workunit.client.1.smithi137.stdout:1/648: unlink d7/d9/d19/d1f/f69 0 2024-03-14T13:02:21.345 INFO:tasks.workunit.client.0.smithi111.stdout:4/976: dread d1d/d22/d30/d87/d134/fbf [0,4194304] 0 2024-03-14T13:02:21.346 INFO:tasks.workunit.client.0.smithi111.stdout:6/583: mkdir d0/d17/d20/d65/d93/dd9 0 2024-03-14T13:02:21.346 INFO:tasks.workunit.client.0.smithi111.stdout:7/511: stat d0/d1/d5/d38/f7f 0 2024-03-14T13:02:21.348 INFO:tasks.workunit.client.1.smithi137.stdout:5/555: dwrite d7/d1d/d74/d81/f89 [0,4194304] 0 2024-03-14T13:02:21.348 INFO:tasks.workunit.client.1.smithi137.stdout:2/625: read - d2/d7/f87 zero size 2024-03-14T13:02:21.349 INFO:tasks.workunit.client.0.smithi111.stdout:8/852: dread d3/d1b/f6f [0,4194304] 0 2024-03-14T13:02:21.349 INFO:tasks.workunit.client.0.smithi111.stdout:5/766: unlink d2/d16/d49/d44/l61 0 2024-03-14T13:02:21.350 INFO:tasks.workunit.client.0.smithi111.stdout:5/767: chown d2/d45/l2a 2425481 1 2024-03-14T13:02:21.351 INFO:tasks.workunit.client.1.smithi137.stdout:9/700: dwrite d6/df/d19/d33/f6e [0,4194304] 0 2024-03-14T13:02:21.351 INFO:tasks.workunit.client.1.smithi137.stdout:6/595: rmdir d2/d54/d55/d9c/dc0 39 2024-03-14T13:02:21.351 INFO:tasks.workunit.client.0.smithi111.stdout:4/977: chown d1d/d22/d30/f38 33 1 2024-03-14T13:02:21.351 INFO:tasks.workunit.client.0.smithi111.stdout:4/978: dread - d1d/d22/d30/f142 zero size 2024-03-14T13:02:21.352 INFO:tasks.workunit.client.1.smithi137.stdout:4/579: mknod d4/d9/d8e/dae/cc5 0 2024-03-14T13:02:21.352 INFO:tasks.workunit.client.1.smithi137.stdout:1/649: dread - d7/d9/d19/d1f/d22/fa2 zero size 2024-03-14T13:02:21.353 INFO:tasks.workunit.client.0.smithi111.stdout:5/768: write d2/d16/d49/d44/ff0 [986934,20857] 0 2024-03-14T13:02:21.353 INFO:tasks.workunit.client.0.smithi111.stdout:7/512: rmdir d0/d1/d5 39 2024-03-14T13:02:21.357 INFO:tasks.workunit.client.1.smithi137.stdout:8/656: dwrite d1/dc/d3c/d87/d34/d82/f8d [0,4194304] 0 2024-03-14T13:02:21.358 INFO:tasks.workunit.client.1.smithi137.stdout:3/537: creat d4/de/d16/d1c/d8d/fb4 x:0 0 0 2024-03-14T13:02:21.358 INFO:tasks.workunit.client.1.smithi137.stdout:5/556: dread d7/d49/fbd [0,4194304] 0 2024-03-14T13:02:21.358 INFO:tasks.workunit.client.0.smithi111.stdout:5/769: fsync d2/d16/f70 0 2024-03-14T13:02:21.359 INFO:tasks.workunit.client.1.smithi137.stdout:2/626: creat d2/d34/fcb x:0 0 0 2024-03-14T13:02:21.360 INFO:tasks.workunit.client.0.smithi111.stdout:5/770: chown d2/d13/fb9 2730064 1 2024-03-14T13:02:21.360 INFO:tasks.workunit.client.0.smithi111.stdout:8/853: symlink d3/d1b/l11c 0 2024-03-14T13:02:21.362 INFO:tasks.workunit.client.1.smithi137.stdout:3/538: fsync d4/de/d16/d1c/d5d/d85/f42 0 2024-03-14T13:02:21.363 INFO:tasks.workunit.client.0.smithi111.stdout:8/854: readlink d3/dc/d45/d11a/l98 0 2024-03-14T13:02:21.364 INFO:tasks.workunit.client.0.smithi111.stdout:6/584: creat d0/d17/d20/d4a/d59/fda x:0 0 0 2024-03-14T13:02:21.366 INFO:tasks.workunit.client.0.smithi111.stdout:4/979: dwrite d1d/d22/d30/d7c/f11b [0,4194304] 0 2024-03-14T13:02:21.366 INFO:tasks.workunit.client.1.smithi137.stdout:5/557: unlink d7/d16/d18/d48/c91 0 2024-03-14T13:02:21.367 INFO:tasks.workunit.client.0.smithi111.stdout:5/771: chown d2/d16/d49/d44/d72/ff1 0 1 2024-03-14T13:02:21.367 INFO:tasks.workunit.client.0.smithi111.stdout:7/513: mkdir d0/d11/d98 0 2024-03-14T13:02:21.367 INFO:tasks.workunit.client.0.smithi111.stdout:5/772: chown d2/d16/c2d 50 1 2024-03-14T13:02:21.368 INFO:tasks.workunit.client.1.smithi137.stdout:4/580: dwrite d4/d10/d26/f6f [4194304,4194304] 0 2024-03-14T13:02:21.369 INFO:tasks.workunit.client.0.smithi111.stdout:9/597: truncate d5/d6/d8/d5b/d7c/f90 1666460 0 2024-03-14T13:02:21.370 INFO:tasks.workunit.client.1.smithi137.stdout:3/539: chown d4/de/d16/d1c/d5d/f43 6019083 1 2024-03-14T13:02:21.370 INFO:tasks.workunit.client.0.smithi111.stdout:8/855: rmdir d3/d22/da1/de6 39 2024-03-14T13:02:21.372 INFO:tasks.workunit.client.1.smithi137.stdout:2/627: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc 0 2024-03-14T13:02:21.376 INFO:tasks.workunit.client.1.smithi137.stdout:0/581: write d1/d1f/d80/fbb [369644,18696] 0 2024-03-14T13:02:21.378 INFO:tasks.workunit.client.1.smithi137.stdout:9/701: write d6/df/d19/d33/f3e [1366563,70459] 0 2024-03-14T13:02:21.379 INFO:tasks.workunit.client.1.smithi137.stdout:1/650: write d7/d4d/f64 [198625,50345] 0 2024-03-14T13:02:21.381 INFO:tasks.workunit.client.0.smithi111.stdout:3/611: truncate d0/d2d/d35/d50/d5e/f5f 914203 0 2024-03-14T13:02:21.381 INFO:tasks.workunit.client.0.smithi111.stdout:1/654: truncate d20/d25/f7c 473838 0 2024-03-14T13:02:21.383 INFO:tasks.workunit.client.1.smithi137.stdout:5/558: dwrite d7/d16/f38 [0,4194304] 0 2024-03-14T13:02:21.385 INFO:tasks.workunit.client.0.smithi111.stdout:5/773: dwrite d2/d16/d49/d7c/d95/f28 [0,4194304] 0 2024-03-14T13:02:21.386 INFO:tasks.workunit.client.0.smithi111.stdout:8/856: rename d3/d1b/d20/d35/d40/d48/d7e/l9b to d3/d1b/dc1/de0/l11d 0 2024-03-14T13:02:21.388 INFO:tasks.workunit.client.0.smithi111.stdout:4/980: dread d1d/d22/d30/d87/fcb [0,4194304] 0 2024-03-14T13:02:21.389 INFO:tasks.workunit.client.1.smithi137.stdout:3/540: dwrite d4/f18 [4194304,4194304] 0 2024-03-14T13:02:21.389 INFO:tasks.workunit.client.1.smithi137.stdout:9/702: creat d6/fe1 x:0 0 0 2024-03-14T13:02:21.390 INFO:tasks.workunit.client.0.smithi111.stdout:7/514: dread - d0/d1/d5/d38/d1b/f92 zero size 2024-03-14T13:02:21.390 INFO:tasks.workunit.client.0.smithi111.stdout:5/774: symlink d2/lf6 0 2024-03-14T13:02:21.392 INFO:tasks.workunit.client.0.smithi111.stdout:1/655: creat d20/d25/dd0/dd2/fdb x:0 0 0 2024-03-14T13:02:21.392 INFO:tasks.workunit.client.0.smithi111.stdout:8/857: stat d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f119 0 2024-03-14T13:02:21.392 INFO:tasks.workunit.client.0.smithi111.stdout:7/515: mkdir d0/d1/d5/d38/d1b/d99 0 2024-03-14T13:02:21.393 INFO:tasks.workunit.client.1.smithi137.stdout:1/651: dread - d7/d9/d19/d1f/d2e/d63/fd3 zero size 2024-03-14T13:02:21.393 INFO:tasks.workunit.client.1.smithi137.stdout:3/541: creat d4/df/d82/fb5 x:0 0 0 2024-03-14T13:02:21.396 INFO:tasks.workunit.client.0.smithi111.stdout:5/775: mkdir d2/d45/d76/db2/df7 0 2024-03-14T13:02:21.398 INFO:tasks.workunit.client.0.smithi111.stdout:4/981: write d1d/d22/d30/de5/fb0 [6014904,128216] 0 2024-03-14T13:02:21.398 INFO:tasks.workunit.client.0.smithi111.stdout:3/612: rename d0/d2d/d35/d50/lc0 to d0/d2d/d35/d50/da2/db3/d67/le0 0 2024-03-14T13:02:21.400 INFO:tasks.workunit.client.1.smithi137.stdout:9/703: creat d6/df/d19/d3a/fe2 x:0 0 0 2024-03-14T13:02:21.400 INFO:tasks.workunit.client.1.smithi137.stdout:1/652: chown d7/d9/d19/d40/d42/d9d/fba 20933537 1 2024-03-14T13:02:21.401 INFO:tasks.workunit.client.1.smithi137.stdout:5/559: symlink d7/d16/d18/d25/d28/d3c/d76/lbf 0 2024-03-14T13:02:21.401 INFO:tasks.workunit.client.0.smithi111.stdout:8/858: symlink d3/d1b/dc1/de4/l11e 0 2024-03-14T13:02:21.402 INFO:tasks.workunit.client.1.smithi137.stdout:4/581: truncate d4/d9/d3a/d41/f21 3982999 0 2024-03-14T13:02:21.402 INFO:tasks.workunit.client.1.smithi137.stdout:0/582: dread d1/d1f/d80/fbb [0,4194304] 0 2024-03-14T13:02:21.403 INFO:tasks.workunit.client.1.smithi137.stdout:9/704: fsync d6/df/d19/fcc 0 2024-03-14T13:02:21.406 INFO:tasks.workunit.client.0.smithi111.stdout:3/613: truncate d0/d2d/d35/d69/f9d 4257241 0 2024-03-14T13:02:21.407 INFO:tasks.workunit.client.0.smithi111.stdout:5/776: unlink d2/d45/f22 0 2024-03-14T13:02:21.407 INFO:tasks.workunit.client.1.smithi137.stdout:1/653: symlink d7/d4d/ldc 0 2024-03-14T13:02:21.410 INFO:tasks.workunit.client.0.smithi111.stdout:1/656: dread d20/d40/f9b [0,4194304] 0 2024-03-14T13:02:21.411 INFO:tasks.workunit.client.1.smithi137.stdout:9/705: fsync d6/df/d71/f72 0 2024-03-14T13:02:21.411 INFO:tasks.workunit.client.0.smithi111.stdout:7/516: mknod d0/d1/d5/d38/c9a 0 2024-03-14T13:02:21.411 INFO:tasks.workunit.client.0.smithi111.stdout:7/517: stat d0/d1/d5/d56/f7c 0 2024-03-14T13:02:21.413 INFO:tasks.workunit.client.0.smithi111.stdout:6/585: write d0/d50/f67 [978875,15648] 0 2024-03-14T13:02:21.413 INFO:tasks.workunit.client.1.smithi137.stdout:5/560: link d7/d16/d18/f34 d7/d16/d18/d25/d28/d3c/d7a/fc0 0 2024-03-14T13:02:21.416 INFO:tasks.workunit.client.1.smithi137.stdout:4/582: dwrite d4/d27/d5c/d77/f7c [0,4194304] 0 2024-03-14T13:02:21.417 INFO:tasks.workunit.client.1.smithi137.stdout:5/561: chown d7/d16/d18/d25/f5b 240716 1 2024-03-14T13:02:21.418 INFO:tasks.workunit.client.1.smithi137.stdout:1/654: mkdir d7/d9/d19/d1f/d2e/da6/ddd 0 2024-03-14T13:02:21.419 INFO:tasks.workunit.client.0.smithi111.stdout:9/598: write d5/d6/d14/d7a/fba [310320,94982] 0 2024-03-14T13:02:21.421 INFO:tasks.workunit.client.0.smithi111.stdout:7/518: chown d0/d1/d21/d2a/c2d 206614 1 2024-03-14T13:02:21.423 INFO:tasks.workunit.client.1.smithi137.stdout:7/479: dread d3/d5/d23/f2b [0,4194304] 0 2024-03-14T13:02:21.426 INFO:tasks.workunit.client.1.smithi137.stdout:4/583: creat d4/d9/d3a/d41/d38/d55/db6/fc6 x:0 0 0 2024-03-14T13:02:21.426 INFO:tasks.workunit.client.0.smithi111.stdout:8/859: fdatasync d3/d1b/d20/d35/d40/d48/f50 0 2024-03-14T13:02:21.427 INFO:tasks.workunit.client.1.smithi137.stdout:5/562: read d7/d36/f95 [51059,73371] 0 2024-03-14T13:02:21.427 INFO:tasks.workunit.client.0.smithi111.stdout:6/586: dread - d0/d17/d20/f4b zero size 2024-03-14T13:02:21.428 INFO:tasks.workunit.client.0.smithi111.stdout:9/599: creat d5/d42/d71/fca x:0 0 0 2024-03-14T13:02:21.428 INFO:tasks.workunit.client.1.smithi137.stdout:1/655: rmdir d7/d9/d19/d5f/ddb 0 2024-03-14T13:02:21.429 INFO:tasks.workunit.client.0.smithi111.stdout:7/519: creat d0/d1/d21/d2a/f9b x:0 0 0 2024-03-14T13:02:21.429 INFO:tasks.workunit.client.1.smithi137.stdout:5/563: fsync d7/d16/fb9 0 2024-03-14T13:02:21.430 INFO:tasks.workunit.client.0.smithi111.stdout:6/587: write d0/d50/f67 [1576094,36548] 0 2024-03-14T13:02:21.430 INFO:tasks.workunit.client.0.smithi111.stdout:4/982: link d1d/d48/c62 d1d/d22/d30/d7c/c15f 0 2024-03-14T13:02:21.430 INFO:tasks.workunit.client.0.smithi111.stdout:9/600: fdatasync d5/d6/d2a/d2d/f4a 0 2024-03-14T13:02:21.431 INFO:tasks.workunit.client.1.smithi137.stdout:5/564: chown d7/d49/fbd 955 1 2024-03-14T13:02:21.432 INFO:tasks.workunit.client.0.smithi111.stdout:1/657: link d20/d25/d75/f5e d20/d21/fdc 0 2024-03-14T13:02:21.437 INFO:tasks.workunit.client.1.smithi137.stdout:3/542: dread d4/de/d16/f17 [0,4194304] 0 2024-03-14T13:02:21.437 INFO:tasks.workunit.client.1.smithi137.stdout:2/628: dread d2/d7/d2e/f6a [0,4194304] 0 2024-03-14T13:02:21.437 INFO:tasks.workunit.client.1.smithi137.stdout:5/565: rename d7/d1d/d74/d83/da3 to d7/d16/d41/dc1 0 2024-03-14T13:02:21.437 INFO:tasks.workunit.client.0.smithi111.stdout:7/520: mkdir d0/d1/d5/d38/d1b/d9c 0 2024-03-14T13:02:21.437 INFO:tasks.workunit.client.1.smithi137.stdout:4/584: symlink d4/d9/d3a/d70/d80/db7/lc7 0 2024-03-14T13:02:21.438 INFO:tasks.workunit.client.0.smithi111.stdout:7/521: readlink d0/d1/d5/d38/l40 0 2024-03-14T13:02:21.438 INFO:tasks.workunit.client.1.smithi137.stdout:3/543: mknod d4/de/d93/cb6 0 2024-03-14T13:02:21.441 INFO:tasks.workunit.client.1.smithi137.stdout:4/585: write d4/d10/d42/f6c [1049072,112717] 0 2024-03-14T13:02:21.441 INFO:tasks.workunit.client.1.smithi137.stdout:1/656: creat d7/d9/d19/d1f/d22/fde x:0 0 0 2024-03-14T13:02:21.441 INFO:tasks.workunit.client.0.smithi111.stdout:8/860: symlink d3/d1b/d20/d35/d40/d48/d7e/db3/db5/l11f 0 2024-03-14T13:02:21.442 INFO:tasks.workunit.client.0.smithi111.stdout:8/861: stat d3/d1b/d34/f107 0 2024-03-14T13:02:21.442 INFO:tasks.workunit.client.1.smithi137.stdout:5/566: mkdir d7/d33/d65/dc2 0 2024-03-14T13:02:21.443 INFO:tasks.workunit.client.0.smithi111.stdout:9/601: creat d5/d6/d2a/d2d/d8b/fcb x:0 0 0 2024-03-14T13:02:21.443 INFO:tasks.workunit.client.1.smithi137.stdout:4/586: write d4/d9/d3a/d41/d6a/dac/fb0 [233126,41611] 0 2024-03-14T13:02:21.445 INFO:tasks.workunit.client.0.smithi111.stdout:6/588: rename d0/d17/d21/d69/f4d to d0/d17/d21/d69/d3d/d53/d81/da1/dd7/fdb 0 2024-03-14T13:02:21.446 INFO:tasks.workunit.client.1.smithi137.stdout:0/583: truncate d1/d5/d1b/d42/f4b 3986535 0 2024-03-14T13:02:21.447 INFO:tasks.workunit.client.1.smithi137.stdout:9/706: dread d6/df/d19/d3a/f47 [0,4194304] 0 2024-03-14T13:02:21.447 INFO:tasks.workunit.client.1.smithi137.stdout:5/567: dread d7/d33/f93 [0,4194304] 0 2024-03-14T13:02:21.449 INFO:tasks.workunit.client.1.smithi137.stdout:1/657: rmdir d7/d4d 39 2024-03-14T13:02:21.450 INFO:tasks.workunit.client.0.smithi111.stdout:9/602: write d5/d6/d2a/d5a/f76 [3193077,69614] 0 2024-03-14T13:02:21.451 INFO:tasks.workunit.client.0.smithi111.stdout:7/522: mknod d0/d1/c9d 0 2024-03-14T13:02:21.452 INFO:tasks.workunit.client.0.smithi111.stdout:1/658: rename d20/d21/d3f/d59/daf to d20/d25/da8/ddd 0 2024-03-14T13:02:21.454 INFO:tasks.workunit.client.0.smithi111.stdout:4/983: creat d1d/d22/f160 x:0 0 0 2024-03-14T13:02:21.455 INFO:tasks.workunit.client.0.smithi111.stdout:1/659: truncate d20/d25/dd0/dd2/fdb 614779 0 2024-03-14T13:02:21.456 INFO:tasks.workunit.client.0.smithi111.stdout:9/603: creat d5/d42/da4/fcc x:0 0 0 2024-03-14T13:02:21.457 INFO:tasks.workunit.client.0.smithi111.stdout:4/984: stat d1d/d22/d30/d87/d134/c116 0 2024-03-14T13:02:21.457 INFO:tasks.workunit.client.1.smithi137.stdout:9/707: dread d6/fe [0,4194304] 0 2024-03-14T13:02:21.458 INFO:tasks.workunit.client.1.smithi137.stdout:4/587: unlink d4/d9/d3a/d70/d80/db7/lc7 0 2024-03-14T13:02:21.460 INFO:tasks.workunit.client.0.smithi111.stdout:3/614: dread d0/d2d/d35/d4c/d82/f7e [0,4194304] 0 2024-03-14T13:02:21.460 INFO:tasks.workunit.client.0.smithi111.stdout:9/604: rename d5/d42/d52/d83 to d5/d42/d71/dbd/dcd 0 2024-03-14T13:02:21.461 INFO:tasks.workunit.client.1.smithi137.stdout:0/584: dwrite d1/d44/d56/f5f [4194304,4194304] 0 2024-03-14T13:02:21.462 INFO:tasks.workunit.client.0.smithi111.stdout:4/985: symlink d1d/d22/d30/l161 0 2024-03-14T13:02:21.463 INFO:tasks.workunit.client.1.smithi137.stdout:7/480: dread d3/d6/d59/f5f [0,4194304] 0 2024-03-14T13:02:21.466 INFO:tasks.workunit.client.0.smithi111.stdout:3/615: rename d0/d2d/d35/d50/d5e/l2f to d0/d91/le1 0 2024-03-14T13:02:21.467 INFO:tasks.workunit.client.1.smithi137.stdout:0/585: truncate d1/d5/d1b/d1e/d25/d55/f67 1606288 0 2024-03-14T13:02:21.467 INFO:tasks.workunit.client.1.smithi137.stdout:3/544: creat d4/d19/fb7 x:0 0 0 2024-03-14T13:02:21.468 INFO:tasks.workunit.client.0.smithi111.stdout:4/986: symlink d1d/d22/l162 0 2024-03-14T13:02:21.470 INFO:tasks.workunit.client.1.smithi137.stdout:2/629: truncate d2/d7/d2e/d3e/d44/d6b/f20 5137518 0 2024-03-14T13:02:21.471 INFO:tasks.workunit.client.0.smithi111.stdout:4/987: stat d1d/d22/d30/c3a 0 2024-03-14T13:02:21.471 INFO:tasks.workunit.client.1.smithi137.stdout:9/708: mknod d6/df/d71/d94/ce3 0 2024-03-14T13:02:21.471 INFO:tasks.workunit.client.0.smithi111.stdout:9/605: link d5/d6/d8/d74/lb7 d5/d6/d2a/d2d/lce 0 2024-03-14T13:02:21.472 INFO:tasks.workunit.client.1.smithi137.stdout:4/588: readlink d4/d10/d26/la9 0 2024-03-14T13:02:21.473 INFO:tasks.workunit.client.0.smithi111.stdout:9/606: dread - d5/d6/d2a/d5a/f6f zero size 2024-03-14T13:02:21.474 INFO:tasks.workunit.client.1.smithi137.stdout:5/568: dread d7/d16/d18/f34 [0,4194304] 0 2024-03-14T13:02:21.474 INFO:tasks.workunit.client.1.smithi137.stdout:1/658: mkdir d7/d9/d19/d1f/ddf 0 2024-03-14T13:02:21.475 INFO:tasks.workunit.client.0.smithi111.stdout:4/988: rename d1d/d22/d23 to d1d/d22/d30/d87/dd1/de1/d163 0 2024-03-14T13:02:21.476 INFO:tasks.workunit.client.0.smithi111.stdout:4/989: write d1d/d22/fd7 [3013431,63623] 0 2024-03-14T13:02:21.476 INFO:tasks.workunit.client.0.smithi111.stdout:9/607: write d5/d6/d2a/d5a/fad [1058946,106238] 0 2024-03-14T13:02:21.477 INFO:tasks.workunit.client.1.smithi137.stdout:7/481: unlink d3/d6/d59/c71 0 2024-03-14T13:02:21.479 INFO:tasks.workunit.client.1.smithi137.stdout:5/569: write d7/d1d/d74/f78 [341168,19011] 0 2024-03-14T13:02:21.479 INFO:tasks.workunit.client.1.smithi137.stdout:3/545: mkdir d4/de/d16/d1c/db8 0 2024-03-14T13:02:21.479 INFO:tasks.workunit.client.0.smithi111.stdout:9/608: symlink d5/d42/d71/d96/lcf 0 2024-03-14T13:02:21.480 INFO:tasks.workunit.client.0.smithi111.stdout:4/990: truncate d1d/d22/d40/f63 2539249 0 2024-03-14T13:02:21.481 INFO:tasks.workunit.client.1.smithi137.stdout:4/589: truncate d4/f3c 4475197 0 2024-03-14T13:02:21.481 INFO:tasks.workunit.client.0.smithi111.stdout:8/862: truncate d3/d22/d30/f7d 2353207 0 2024-03-14T13:02:21.482 INFO:tasks.workunit.client.0.smithi111.stdout:8/863: fsync d3/d1b/f1d 0 2024-03-14T13:02:21.482 INFO:tasks.workunit.client.1.smithi137.stdout:0/586: symlink d1/d5/d1b/d42/d61/dae/lc1 0 2024-03-14T13:02:21.482 INFO:tasks.workunit.client.1.smithi137.stdout:1/659: read d7/d9/d19/d1f/d26/d7d/fc2 [526866,127466] 0 2024-03-14T13:02:21.483 INFO:tasks.workunit.client.0.smithi111.stdout:9/609: symlink d5/d42/d52/ld0 0 2024-03-14T13:02:21.484 INFO:tasks.workunit.client.1.smithi137.stdout:7/482: symlink d3/d5/d23/d2c/l93 0 2024-03-14T13:02:21.484 INFO:tasks.workunit.client.0.smithi111.stdout:7/523: truncate d0/d1/d5/d56/f5e 1462667 0 2024-03-14T13:02:21.485 INFO:tasks.workunit.client.1.smithi137.stdout:9/709: dwrite d6/df/d19/d33/d8f/fbd [0,4194304] 0 2024-03-14T13:02:21.486 INFO:tasks.workunit.client.1.smithi137.stdout:4/590: write d4/d10/f3f [1421798,121996] 0 2024-03-14T13:02:21.487 INFO:tasks.workunit.client.0.smithi111.stdout:9/610: creat d5/d6/d2a/d2d/db8/fd1 x:0 0 0 2024-03-14T13:02:21.488 INFO:tasks.workunit.client.1.smithi137.stdout:7/483: readlink d3/d5/d5a/d86/l91 0 2024-03-14T13:02:21.488 INFO:tasks.workunit.client.0.smithi111.stdout:9/611: chown d5/d42/d52/l89 51 1 2024-03-14T13:02:21.489 INFO:tasks.workunit.client.1.smithi137.stdout:0/587: creat d1/d5/d1b/d42/d3a/fc2 x:0 0 0 2024-03-14T13:02:21.491 INFO:tasks.workunit.client.0.smithi111.stdout:4/991: dwrite d1d/d22/d30/d87/dd1/de1/d163/d14f/f155 [0,4194304] 0 2024-03-14T13:02:21.492 INFO:tasks.workunit.client.1.smithi137.stdout:5/570: mknod d7/d16/d18/d25/d28/d9e/cc3 0 2024-03-14T13:02:21.493 INFO:tasks.workunit.client.1.smithi137.stdout:9/710: unlink d6/d56/c68 0 2024-03-14T13:02:21.494 INFO:tasks.workunit.client.0.smithi111.stdout:9/612: mknod d5/d6/d2a/d2d/d8b/cd2 0 2024-03-14T13:02:21.497 INFO:tasks.workunit.client.1.smithi137.stdout:1/660: creat d7/d9/d19/d1f/d89/d90/db8/fe0 x:0 0 0 2024-03-14T13:02:21.497 INFO:tasks.workunit.client.0.smithi111.stdout:8/864: rename d3/d1b/d66 to d3/dc/d45/d120 0 2024-03-14T13:02:21.500 INFO:tasks.workunit.client.0.smithi111.stdout:9/613: mkdir d5/d42/d71/d96/dd3 0 2024-03-14T13:02:21.502 INFO:tasks.workunit.client.0.smithi111.stdout:8/865: creat d3/d1b/d20/d35/d40/d48/d77/f121 x:0 0 0 2024-03-14T13:02:21.502 INFO:tasks.workunit.client.1.smithi137.stdout:1/661: stat d7/d9/d19/d1f/ddf 0 2024-03-14T13:02:21.503 INFO:tasks.workunit.client.0.smithi111.stdout:8/866: fsync d3/d22/d30/fa4 0 2024-03-14T13:02:21.504 INFO:tasks.workunit.client.0.smithi111.stdout:4/992: mkdir d1d/d22/d40/d69/d164 0 2024-03-14T13:02:21.505 INFO:tasks.workunit.client.1.smithi137.stdout:3/546: rename d4/de/d16/d1c/d5d/d85/f44 to d4/de/d16/fb9 0 2024-03-14T13:02:21.506 INFO:tasks.workunit.client.1.smithi137.stdout:5/571: mkdir d7/d1d/d74/d83/dc4 0 2024-03-14T13:02:21.508 INFO:tasks.workunit.client.1.smithi137.stdout:9/711: creat d6/df/d19/d3a/da9/fe4 x:0 0 0 2024-03-14T13:02:21.508 INFO:tasks.workunit.client.0.smithi111.stdout:4/993: write d1d/d22/d30/d5e/d6f/d7e/d66/df4/d136/f121 [1276855,90801] 0 2024-03-14T13:02:21.509 INFO:tasks.workunit.client.1.smithi137.stdout:0/588: dwrite d1/d1f/d80/fbb [0,4194304] 0 2024-03-14T13:02:21.510 INFO:tasks.workunit.client.1.smithi137.stdout:5/572: chown d7/d36/f95 27282 1 2024-03-14T13:02:21.511 INFO:tasks.workunit.client.0.smithi111.stdout:9/614: mkdir d5/d42/dd4 0 2024-03-14T13:02:21.512 INFO:tasks.workunit.client.1.smithi137.stdout:1/662: fdatasync d7/d9/d19/d1f/d26/d6c/f6f 0 2024-03-14T13:02:21.512 INFO:tasks.workunit.client.0.smithi111.stdout:9/615: stat f2 0 2024-03-14T13:02:21.514 INFO:tasks.workunit.client.1.smithi137.stdout:9/712: dread - d6/df/d19/d33/f65 zero size 2024-03-14T13:02:21.514 INFO:tasks.workunit.client.1.smithi137.stdout:1/663: chown d7/d31 117 1 2024-03-14T13:02:21.514 INFO:tasks.workunit.client.1.smithi137.stdout:9/713: readlink d6/df/d19/d1f/l9b 0 2024-03-14T13:02:21.514 INFO:tasks.workunit.client.1.smithi137.stdout:3/547: creat d4/d22/fba x:0 0 0 2024-03-14T13:02:21.517 INFO:tasks.workunit.client.1.smithi137.stdout:9/714: stat d6/df/d19/d1f/l21 0 2024-03-14T13:02:21.518 INFO:tasks.workunit.client.1.smithi137.stdout:7/484: write d3/d5/d23/d2c/d5d/d6a/d75/d78/f83 [3708567,30507] 0 2024-03-14T13:02:21.519 INFO:tasks.workunit.client.1.smithi137.stdout:0/589: mknod d1/d5/d1b/d1e/d25/d55/d73/cc3 0 2024-03-14T13:02:21.520 INFO:tasks.workunit.client.0.smithi111.stdout:8/867: dwrite d3/d1b/d20/f37 [0,4194304] 0 2024-03-14T13:02:21.522 INFO:tasks.workunit.client.0.smithi111.stdout:9/616: dread d5/d6/d8/d1a/f64 [0,4194304] 0 2024-03-14T13:02:21.522 INFO:tasks.workunit.client.1.smithi137.stdout:1/664: creat d7/d9/d19/d1f/d89/d90/db1/fe1 x:0 0 0 2024-03-14T13:02:21.522 INFO:tasks.workunit.client.1.smithi137.stdout:7/485: chown d3/d5/d23/d2c/d5d/d6a/c8c 17133098 1 2024-03-14T13:02:21.522 INFO:tasks.workunit.client.0.smithi111.stdout:7/524: dwrite d0/d11/f53 [4194304,4194304] 0 2024-03-14T13:02:21.523 INFO:tasks.workunit.client.1.smithi137.stdout:5/573: dread d7/d16/d18/d25/d28/f60 [0,4194304] 0 2024-03-14T13:02:21.523 INFO:tasks.workunit.client.1.smithi137.stdout:7/486: readlink d3/d5/d23/d3f/d5b/l69 0 2024-03-14T13:02:21.525 INFO:tasks.workunit.client.1.smithi137.stdout:4/591: dread d4/d9/d3a/d41/f49 [0,4194304] 0 2024-03-14T13:02:21.526 INFO:tasks.workunit.client.1.smithi137.stdout:9/715: mkdir d6/df/d88/de5 0 2024-03-14T13:02:21.527 INFO:tasks.workunit.client.1.smithi137.stdout:1/665: readlink d7/l12 0 2024-03-14T13:02:21.527 INFO:tasks.workunit.client.1.smithi137.stdout:4/592: write d4/d9/d3a/d70/d80/fa6 [928009,47780] 0 2024-03-14T13:02:21.529 INFO:tasks.workunit.client.0.smithi111.stdout:8/868: rmdir d3/d1b/d20/d35/d40/d48/d7e 39 2024-03-14T13:02:21.531 INFO:tasks.workunit.client.1.smithi137.stdout:3/548: symlink d4/de/d16/d1c/d5d/d85/lbb 0 2024-03-14T13:02:21.531 INFO:tasks.workunit.client.0.smithi111.stdout:9/617: getdents d5/d6/d8/d5b/d84 0 2024-03-14T13:02:21.533 INFO:tasks.workunit.client.1.smithi137.stdout:4/593: rmdir d4/d10/d26 39 2024-03-14T13:02:21.533 INFO:tasks.workunit.client.0.smithi111.stdout:8/869: write d3/d22/f88 [1464386,115760] 0 2024-03-14T13:02:21.533 INFO:tasks.workunit.client.1.smithi137.stdout:0/590: creat d1/d1f/fc4 x:0 0 0 2024-03-14T13:02:21.535 INFO:tasks.workunit.client.0.smithi111.stdout:9/618: creat d5/d6/fd5 x:0 0 0 2024-03-14T13:02:21.537 INFO:tasks.workunit.client.0.smithi111.stdout:4/994: dwrite d1d/d22/d30/d7c/d59/dc4/fce [0,4194304] 0 2024-03-14T13:02:21.537 INFO:tasks.workunit.client.1.smithi137.stdout:1/666: creat d7/d4d/d6b/fe2 x:0 0 0 2024-03-14T13:02:21.538 INFO:tasks.workunit.client.1.smithi137.stdout:0/591: chown d1/d5/d1b/d42/l8d 43 1 2024-03-14T13:02:21.542 INFO:tasks.workunit.client.0.smithi111.stdout:8/870: dread d3/d1b/dc1/fda [0,4194304] 0 2024-03-14T13:02:21.543 INFO:tasks.workunit.client.1.smithi137.stdout:4/594: rename d4/l19 to d4/d9/d3a/d41/d38/d79/lc8 0 2024-03-14T13:02:21.546 INFO:tasks.workunit.client.0.smithi111.stdout:8/871: rename d3/d49/c87 to d3/dc/c122 0 2024-03-14T13:02:21.546 INFO:tasks.workunit.client.1.smithi137.stdout:4/595: write d4/d10/d26/f6f [7307883,106373] 0 2024-03-14T13:02:21.552 INFO:tasks.workunit.client.1.smithi137.stdout:4/596: dread d4/d27/d5c/d6d/d58/d65/f7b [0,4194304] 0 2024-03-14T13:02:21.554 INFO:tasks.workunit.client.0.smithi111.stdout:8/872: rename d3/d1b/d20/lbc to d3/d22/da1/de6/l123 0 2024-03-14T13:02:21.555 INFO:tasks.workunit.client.1.smithi137.stdout:9/716: dwrite d6/df/fd9 [0,4194304] 0 2024-03-14T13:02:21.558 INFO:tasks.workunit.client.0.smithi111.stdout:8/873: dread d3/d22/d30/f7d [0,4194304] 0 2024-03-14T13:02:21.559 INFO:tasks.workunit.client.1.smithi137.stdout:9/717: write d6/f69 [2547995,26125] 0 2024-03-14T13:02:21.561 INFO:tasks.workunit.client.0.smithi111.stdout:9/619: dread d5/f20 [0,4194304] 0 2024-03-14T13:02:21.561 INFO:tasks.workunit.client.1.smithi137.stdout:3/549: truncate d4/de/d13/f9f 814280 0 2024-03-14T13:02:21.562 INFO:tasks.workunit.client.1.smithi137.stdout:4/597: getdents d4/d10/d42/daf 0 2024-03-14T13:02:21.563 INFO:tasks.workunit.client.0.smithi111.stdout:7/525: dread d0/d1/d5/f4c [0,4194304] 0 2024-03-14T13:02:21.564 INFO:tasks.workunit.client.0.smithi111.stdout:9/620: mknod d5/d6/d8/d5b/d84/db6/cd6 0 2024-03-14T13:02:21.565 INFO:tasks.workunit.client.1.smithi137.stdout:9/718: rename d6/da8 to d6/df/d19/d33/ddf/de6 0 2024-03-14T13:02:21.566 INFO:tasks.workunit.client.1.smithi137.stdout:0/592: write d1/d5/d1b/d42/d3a/f48 [4531963,70195] 0 2024-03-14T13:02:21.567 INFO:tasks.workunit.client.0.smithi111.stdout:7/526: read d0/d1/d22/f7b [332644,38135] 0 2024-03-14T13:02:21.567 INFO:tasks.workunit.client.1.smithi137.stdout:3/550: write d4/de/d16/f17 [262844,57452] 0 2024-03-14T13:02:21.568 INFO:tasks.workunit.client.0.smithi111.stdout:9/621: mknod d5/d6/d2a/d5a/cd7 0 2024-03-14T13:02:21.568 INFO:tasks.workunit.client.1.smithi137.stdout:9/719: chown d6/df/l29 3 1 2024-03-14T13:02:21.568 INFO:tasks.workunit.client.0.smithi111.stdout:8/874: dwrite d3/d1b/d20/d35/d40/d48/d7e/fba [0,4194304] 0 2024-03-14T13:02:21.570 INFO:tasks.workunit.client.0.smithi111.stdout:8/875: chown d3/d1b/d20/d35/d40/d48/f50 1517 1 2024-03-14T13:02:21.571 INFO:tasks.workunit.client.0.smithi111.stdout:9/622: creat d5/d6/d8/d74/fd8 x:0 0 0 2024-03-14T13:02:21.572 INFO:tasks.workunit.client.1.smithi137.stdout:4/598: dwrite d4/d27/d5c/d6d/d58/f7e [0,4194304] 0 2024-03-14T13:02:21.572 INFO:tasks.workunit.client.1.smithi137.stdout:0/593: creat d1/d5/d1b/fc5 x:0 0 0 2024-03-14T13:02:21.573 INFO:tasks.workunit.client.1.smithi137.stdout:3/551: read d4/de/d16/d27/f65 [4271887,34562] 0 2024-03-14T13:02:21.575 INFO:tasks.workunit.client.1.smithi137.stdout:9/720: write d6/df/f20 [3850939,82287] 0 2024-03-14T13:02:21.579 INFO:tasks.workunit.client.1.smithi137.stdout:4/599: creat d4/d9/d3a/d41/fc9 x:0 0 0 2024-03-14T13:02:21.580 INFO:tasks.workunit.client.1.smithi137.stdout:9/721: creat d6/df/d19/d3a/da9/fe7 x:0 0 0 2024-03-14T13:02:21.582 INFO:tasks.workunit.client.1.smithi137.stdout:9/722: rmdir d6/df 39 2024-03-14T13:02:21.582 INFO:tasks.workunit.client.1.smithi137.stdout:4/600: mknod d4/d9/d3a/d70/d80/db7/cca 0 2024-03-14T13:02:21.582 INFO:tasks.workunit.client.0.smithi111.stdout:9/623: dwrite d5/d6/d14/f91 [0,4194304] 0 2024-03-14T13:02:21.583 INFO:tasks.workunit.client.0.smithi111.stdout:4/995: dread d1d/d22/d30/d5e/f84 [0,4194304] 0 2024-03-14T13:02:21.585 INFO:tasks.workunit.client.1.smithi137.stdout:9/723: truncate d6/df/f6a 184327 0 2024-03-14T13:02:21.588 INFO:tasks.workunit.client.1.smithi137.stdout:4/601: rename d4/d9/d3a/d41/d38/d79/lc8 to d4/d9/d3a/d70/d80/db7/lcb 0 2024-03-14T13:02:21.588 INFO:tasks.workunit.client.1.smithi137.stdout:9/724: truncate d6/df/d19/d33/d8f/f9a 6329 0 2024-03-14T13:02:21.589 INFO:tasks.workunit.client.0.smithi111.stdout:7/527: truncate d0/d1/d4a/f6a 509105 0 2024-03-14T13:02:21.590 INFO:tasks.workunit.client.0.smithi111.stdout:7/528: fsync d0/d7e/f49 0 2024-03-14T13:02:21.591 INFO:tasks.workunit.client.1.smithi137.stdout:9/725: mkdir d6/df/d19/d1f/de8 0 2024-03-14T13:02:21.594 INFO:tasks.workunit.client.0.smithi111.stdout:8/876: dread d3/d1b/f75 [0,4194304] 0 2024-03-14T13:02:21.597 INFO:tasks.workunit.client.1.smithi137.stdout:9/726: creat d6/df/d19/dcf/fe9 x:0 0 0 2024-03-14T13:02:21.597 INFO:tasks.workunit.client.1.smithi137.stdout:4/602: dwrite d4/d10/d42/d73/fb5 [0,4194304] 0 2024-03-14T13:02:21.597 INFO:tasks.workunit.client.0.smithi111.stdout:7/529: creat d0/d1/d5/d38/d5c/d6d/f9e x:0 0 0 2024-03-14T13:02:21.597 INFO:tasks.workunit.client.0.smithi111.stdout:9/624: dread d5/f4f [0,4194304] 0 2024-03-14T13:02:21.598 INFO:tasks.workunit.client.0.smithi111.stdout:8/877: mkdir d3/dc/d31/d9d/d124 0 2024-03-14T13:02:21.599 INFO:tasks.workunit.client.0.smithi111.stdout:7/530: chown d0/d1/d22/l77 941368 1 2024-03-14T13:02:21.600 INFO:tasks.workunit.client.1.smithi137.stdout:3/552: dread d4/de/d16/f1a [4194304,4194304] 0 2024-03-14T13:02:21.601 INFO:tasks.workunit.client.0.smithi111.stdout:8/878: stat d3/d22/cbd 0 2024-03-14T13:02:21.601 INFO:tasks.workunit.client.1.smithi137.stdout:3/553: stat d4/d22 0 2024-03-14T13:02:21.603 INFO:tasks.workunit.client.0.smithi111.stdout:9/625: creat d5/d6/d8/fd9 x:0 0 0 2024-03-14T13:02:21.603 INFO:tasks.workunit.client.0.smithi111.stdout:4/996: dwrite d1d/d22/d30/d5e/db8/ddf/f15d [0,4194304] 0 2024-03-14T13:02:21.605 INFO:tasks.workunit.client.1.smithi137.stdout:3/554: chown d4/lad 0 1 2024-03-14T13:02:21.606 INFO:tasks.workunit.client.1.smithi137.stdout:3/555: readlink d4/d22/l69 0 2024-03-14T13:02:21.607 INFO:tasks.workunit.client.0.smithi111.stdout:7/531: read d0/d1/d5/d56/f7c [236483,128758] 0 2024-03-14T13:02:21.607 INFO:tasks.workunit.client.1.smithi137.stdout:4/603: dwrite d4/d10/d42/d73/d9d/dbc/fb9 [0,4194304] 0 2024-03-14T13:02:21.608 INFO:tasks.workunit.client.0.smithi111.stdout:9/626: mknod d5/d6/d2a/d2d/db8/cda 0 2024-03-14T13:02:21.610 INFO:tasks.workunit.client.0.smithi111.stdout:4/997: read d1d/d22/d30/d87/dd1/de1/d163/fe7 [478568,60730] 0 2024-03-14T13:02:21.610 INFO:tasks.workunit.client.0.smithi111.stdout:9/627: fsync d5/d6/f2f 0 2024-03-14T13:02:21.613 INFO:tasks.workunit.client.0.smithi111.stdout:9/628: mkdir d5/d42/d71/dbd/ddb 0 2024-03-14T13:02:21.613 INFO:tasks.workunit.client.1.smithi137.stdout:4/604: chown d4/d9/f1f 1127 1 2024-03-14T13:02:21.613 INFO:tasks.workunit.client.1.smithi137.stdout:3/556: creat d4/de/d16/fbc x:0 0 0 2024-03-14T13:02:21.614 INFO:tasks.workunit.client.0.smithi111.stdout:4/998: creat d1d/d22/d30/d5e/d6f/d7e/d66/dba/f165 x:0 0 0 2024-03-14T13:02:21.615 INFO:tasks.workunit.client.0.smithi111.stdout:4/999: chown d1d/d22/d40/c61 4630 1 2024-03-14T13:02:21.617 INFO:tasks.workunit.client.1.smithi137.stdout:4/605: creat d4/d9/d3a/d70/d80/db7/fcc x:0 0 0 2024-03-14T13:02:21.618 INFO:tasks.workunit.client.1.smithi137.stdout:4/606: fsync d4/d10/d5a/f63 0 2024-03-14T13:02:21.619 INFO:tasks.workunit.client.1.smithi137.stdout:4/607: write d4/f8 [3563425,27995] 0 2024-03-14T13:02:21.621 INFO:tasks.workunit.client.1.smithi137.stdout:4/608: readlink d4/d9/d3a/d41/d38/l43 0 2024-03-14T13:02:21.622 INFO:tasks.workunit.client.0.smithi111.stdout:7/532: dread d0/d1/d22/f42 [0,4194304] 0 2024-03-14T13:02:21.626 INFO:tasks.workunit.client.1.smithi137.stdout:4/609: dread d4/d9/d3a/d41/d38/f53 [0,4194304] 0 2024-03-14T13:02:21.627 INFO:tasks.workunit.client.0.smithi111.stdout:7/533: link d0/d1/d5/d38/d5c/l63 d0/d1/d5/d38/d5c/d6d/l9f 0 2024-03-14T13:02:21.628 INFO:tasks.workunit.client.1.smithi137.stdout:4/610: chown d4/d9/d3a/d41/d38/l43 14 1 2024-03-14T13:02:21.629 INFO:tasks.workunit.client.0.smithi111.stdout:7/534: creat d0/d1/d5/fa0 x:0 0 0 2024-03-14T13:02:21.630 INFO:tasks.workunit.client.1.smithi137.stdout:4/611: write d4/d27/d5c/d6d/f72 [1033217,86739] 0 2024-03-14T13:02:21.631 INFO:tasks.workunit.client.0.smithi111.stdout:7/535: write d0/d1/d5/d38/d5c/f8d [336729,35690] 0 2024-03-14T13:02:21.631 INFO:tasks.workunit.client.1.smithi137.stdout:9/727: write d6/df/d19/d1f/f6c [336475,44403] 0 2024-03-14T13:02:21.632 INFO:tasks.workunit.client.0.smithi111.stdout:8/879: truncate d3/d1b/d20/d35/fa0 312188 0 2024-03-14T13:02:21.632 INFO:tasks.workunit.client.0.smithi111.stdout:7/536: write d0/d1/d21/d2a/f7d [586094,112520] 0 2024-03-14T13:02:21.635 INFO:tasks.workunit.client.0.smithi111.stdout:7/537: creat d0/d1/d5/d38/d1b/fa1 x:0 0 0 2024-03-14T13:02:21.636 INFO:tasks.workunit.client.1.smithi137.stdout:9/728: symlink d6/df/d19/d1f/de8/lea 0 2024-03-14T13:02:21.636 INFO:tasks.workunit.client.1.smithi137.stdout:4/612: dread d4/d10/d42/f52 [0,4194304] 0 2024-03-14T13:02:21.637 INFO:tasks.workunit.client.0.smithi111.stdout:7/538: write d0/d1/d5/d38/d5c/f8d [413229,89263] 0 2024-03-14T13:02:21.638 INFO:tasks.workunit.client.0.smithi111.stdout:7/539: chown d0/d7e/f81 266 1 2024-03-14T13:02:21.639 INFO:tasks.workunit.client.0.smithi111.stdout:8/880: dread d3/f79 [0,4194304] 0 2024-03-14T13:02:21.640 INFO:tasks.workunit.client.0.smithi111.stdout:7/540: rmdir d0/d1/d5/d56/d70 39 2024-03-14T13:02:21.640 INFO:tasks.workunit.client.1.smithi137.stdout:9/729: link d6/d91/cd0 d6/df/d19/d33/ddf/de6/ceb 0 2024-03-14T13:02:21.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:21 smithi111 ceph-mon[39688]: pgmap v25: 65 pgs: 65 active+clean; 2.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 64 MiB/s rd, 270 MiB/s wr, 478 op/s 2024-03-14T13:02:21.641 INFO:tasks.workunit.client.0.smithi111.stdout:8/881: stat d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f8f 0 2024-03-14T13:02:21.642 INFO:tasks.workunit.client.1.smithi137.stdout:9/730: mkdir d6/df/d19/d33/dec 0 2024-03-14T13:02:21.643 INFO:tasks.workunit.client.0.smithi111.stdout:8/882: symlink d3/dc/d45/d11a/l125 0 2024-03-14T13:02:21.643 INFO:tasks.workunit.client.1.smithi137.stdout:4/613: dwrite d4/d9/d3a/d41/d6a/fb3 [0,4194304] 0 2024-03-14T13:02:21.646 INFO:tasks.workunit.client.1.smithi137.stdout:4/614: write d4/d10/d42/d73/d9d/dbc/fb9 [950714,70363] 0 2024-03-14T13:02:21.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/615: readlink d4/d9/l51 0 2024-03-14T13:02:21.648 INFO:tasks.workunit.client.0.smithi111.stdout:8/883: getdents d3/d1b/d20/d35 0 2024-03-14T13:02:21.648 INFO:tasks.workunit.client.1.smithi137.stdout:9/731: getdents d6/df/d71 0 2024-03-14T13:02:21.650 INFO:tasks.workunit.client.1.smithi137.stdout:9/732: chown d6/df/d19/d33/f65 3166 1 2024-03-14T13:02:21.650 INFO:tasks.workunit.client.0.smithi111.stdout:8/884: dread - d3/dc/d31/d9d/fea zero size 2024-03-14T13:02:21.656 INFO:tasks.workunit.client.0.smithi111.stdout:7/541: dwrite d0/d1/d22/f7b [0,4194304] 0 2024-03-14T13:02:21.657 INFO:tasks.workunit.client.0.smithi111.stdout:7/542: read - d0/f72 zero size 2024-03-14T13:02:21.658 INFO:tasks.workunit.client.0.smithi111.stdout:8/885: truncate d3/d1b/f75 2761635 0 2024-03-14T13:02:21.660 INFO:tasks.workunit.client.0.smithi111.stdout:7/543: readlink d0/d1/d5/d38/l40 0 2024-03-14T13:02:21.662 INFO:tasks.workunit.client.0.smithi111.stdout:8/886: dread d3/d22/d30/f43 [0,4194304] 0 2024-03-14T13:02:21.666 INFO:tasks.workunit.client.0.smithi111.stdout:8/887: link d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f8f d3/d1b/f126 0 2024-03-14T13:02:21.701 INFO:tasks.workunit.client.0.smithi111.stdout:2/575: sync 2024-03-14T13:02:21.702 INFO:tasks.workunit.client.0.smithi111.stdout:2/576: chown dd/d11/d1a/f3d 1 1 2024-03-14T13:02:21.702 INFO:tasks.workunit.client.1.smithi137.stdout:6/596: sync 2024-03-14T13:02:21.703 INFO:tasks.workunit.client.0.smithi111.stdout:2/577: symlink dd/d8d/lb8 0 2024-03-14T13:02:21.708 INFO:tasks.workunit.client.0.smithi111.stdout:7/544: write d0/d1/d5/d38/f7f [886161,32733] 0 2024-03-14T13:02:21.708 INFO:tasks.workunit.client.1.smithi137.stdout:6/597: truncate d2/d6/de/f5d 1370525 0 2024-03-14T13:02:21.712 INFO:tasks.workunit.client.0.smithi111.stdout:7/545: symlink d0/d1/d22/la2 0 2024-03-14T13:02:21.712 INFO:tasks.workunit.client.1.smithi137.stdout:4/616: dwrite d4/d10/d42/f75 [0,4194304] 0 2024-03-14T13:02:21.712 INFO:tasks.workunit.client.0.smithi111.stdout:2/578: dwrite dd/d11/d1a/d56/f85 [0,4194304] 0 2024-03-14T13:02:21.726 INFO:tasks.workunit.client.1.smithi137.stdout:4/617: write d4/d27/d5c/d77/f7c [4885599,30285] 0 2024-03-14T13:02:21.731 INFO:tasks.workunit.client.1.smithi137.stdout:6/598: dwrite d2/d54/d55/d6d/fb0 [0,4194304] 0 2024-03-14T13:02:21.733 INFO:tasks.workunit.client.1.smithi137.stdout:6/599: creat d2/d54/d55/d33/fcc x:0 0 0 2024-03-14T13:02:21.734 INFO:tasks.workunit.client.0.smithi111.stdout:7/546: write d0/d11/f73 [1028459,53619] 0 2024-03-14T13:02:21.736 INFO:tasks.workunit.client.1.smithi137.stdout:6/600: mkdir d2/d6/d1d/d88/dcd 0 2024-03-14T13:02:21.738 INFO:tasks.workunit.client.0.smithi111.stdout:2/579: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:21.742 INFO:tasks.workunit.client.1.smithi137.stdout:6/601: dread d2/d54/d55/d6d/d76/d57/f70 [0,4194304] 0 2024-03-14T13:02:21.748 INFO:tasks.workunit.client.0.smithi111.stdout:2/580: getdents dd/d17/d53/da4 0 2024-03-14T13:02:21.749 INFO:tasks.workunit.client.1.smithi137.stdout:6/602: dwrite d2/d54/d55/d33/fcc [0,4194304] 0 2024-03-14T13:02:21.749 INFO:tasks.workunit.client.1.smithi137.stdout:6/603: chown d2/d6/d1d/l31 2411 1 2024-03-14T13:02:21.751 INFO:tasks.workunit.client.0.smithi111.stdout:2/581: rmdir dd/d11/d1a 39 2024-03-14T13:02:21.754 INFO:tasks.workunit.client.0.smithi111.stdout:2/582: creat dd/d17/d34/d4f/fb9 x:0 0 0 2024-03-14T13:02:21.755 INFO:tasks.workunit.client.1.smithi137.stdout:4/618: dread d4/d9/d3a/d41/d38/d79/f94 [0,4194304] 0 2024-03-14T13:02:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:21 smithi137 ceph-mon[50336]: pgmap v25: 65 pgs: 65 active+clean; 2.2 GiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 64 MiB/s rd, 270 MiB/s wr, 478 op/s 2024-03-14T13:02:21.757 INFO:tasks.workunit.client.1.smithi137.stdout:4/619: symlink d4/d9/d3a/d70/d80/lcd 0 2024-03-14T13:02:21.759 INFO:tasks.workunit.client.0.smithi111.stdout:2/583: truncate dd/f76 1012828 0 2024-03-14T13:02:21.764 INFO:tasks.workunit.client.0.smithi111.stdout:2/584: dread dd/d11/d1a/d56/f85 [0,4194304] 0 2024-03-14T13:02:21.771 INFO:tasks.workunit.client.0.smithi111.stdout:2/585: read dd/f76 [968918,55915] 0 2024-03-14T13:02:21.843 INFO:tasks.workunit.client.1.smithi137.stdout:6/604: dread d2/d6/de/fa6 [0,4194304] 0 2024-03-14T13:02:21.844 INFO:tasks.workunit.client.1.smithi137.stdout:4/620: read d4/d10/d42/d73/d9d/dbc/fb9 [1543725,26832] 0 2024-03-14T13:02:21.849 INFO:tasks.workunit.client.1.smithi137.stdout:6/605: dread d2/d54/d55/d38/f85 [0,4194304] 0 2024-03-14T13:02:21.857 INFO:tasks.workunit.client.1.smithi137.stdout:4/621: creat d4/d27/d5c/d6d/d58/d65/fce x:0 0 0 2024-03-14T13:02:21.858 INFO:tasks.workunit.client.1.smithi137.stdout:6/606: unlink d2/d4c/c6a 0 2024-03-14T13:02:21.859 INFO:tasks.workunit.client.1.smithi137.stdout:6/607: truncate d2/d54/d55/d6d/fc7 798113 0 2024-03-14T13:02:21.860 INFO:tasks.workunit.client.1.smithi137.stdout:4/622: symlink d4/d10/d26/lcf 0 2024-03-14T13:02:21.862 INFO:tasks.workunit.client.1.smithi137.stdout:8/657: sync 2024-03-14T13:02:21.862 INFO:tasks.workunit.client.1.smithi137.stdout:2/630: sync 2024-03-14T13:02:21.862 INFO:tasks.workunit.client.1.smithi137.stdout:6/608: dread d2/d54/d55/d6d/d76/d57/f70 [0,4194304] 0 2024-03-14T13:02:21.862 INFO:tasks.workunit.client.1.smithi137.stdout:4/623: creat d4/d9/d3a/d70/fd0 x:0 0 0 2024-03-14T13:02:21.863 INFO:tasks.workunit.client.1.smithi137.stdout:8/658: chown d1/cac 35 1 2024-03-14T13:02:21.865 INFO:tasks.workunit.client.1.smithi137.stdout:2/631: write d2/d7/d2e/d3e/d44/d6b/d22/dac/faf [1087411,73646] 0 2024-03-14T13:02:21.865 INFO:tasks.workunit.client.1.smithi137.stdout:6/609: unlink d2/d6/de/fa6 0 2024-03-14T13:02:21.867 INFO:tasks.workunit.client.1.smithi137.stdout:4/624: mkdir d4/d10/d42/d73/d9d/dbc/dd1 0 2024-03-14T13:02:21.868 INFO:tasks.workunit.client.1.smithi137.stdout:2/632: mkdir d2/d7/d2e/d3e/daa/dcd 0 2024-03-14T13:02:21.871 INFO:tasks.workunit.client.1.smithi137.stdout:4/625: rmdir d4/d27/d5c/d77 39 2024-03-14T13:02:21.872 INFO:tasks.workunit.client.1.smithi137.stdout:8/659: mknod d1/dc/d10/dcf/dd2/cdd 0 2024-03-14T13:02:21.875 INFO:tasks.workunit.client.1.smithi137.stdout:2/633: dread d2/d7/d2e/d3e/d44/d6b/fc2 [0,4194304] 0 2024-03-14T13:02:21.875 INFO:tasks.workunit.client.0.smithi111.stdout:5/777: sync 2024-03-14T13:02:21.875 INFO:tasks.workunit.client.0.smithi111.stdout:0/759: sync 2024-03-14T13:02:21.876 INFO:tasks.workunit.client.1.smithi137.stdout:8/660: symlink d1/dc/d3c/d87/d34/d86/lde 0 2024-03-14T13:02:21.876 INFO:tasks.workunit.client.1.smithi137.stdout:4/626: dread d4/d9/d3a/d41/d38/f3b [0,4194304] 0 2024-03-14T13:02:21.879 INFO:tasks.workunit.client.1.smithi137.stdout:8/661: unlink d1/d4a/f9d 0 2024-03-14T13:02:21.879 INFO:tasks.workunit.client.0.smithi111.stdout:5/778: dread d2/f7 [0,4194304] 0 2024-03-14T13:02:21.880 INFO:tasks.workunit.client.1.smithi137.stdout:8/662: write d1/dc/da9/fbe [732495,13896] 0 2024-03-14T13:02:21.883 INFO:tasks.workunit.client.1.smithi137.stdout:6/610: dread d2/d6/de/d45/d59/f37 [0,4194304] 0 2024-03-14T13:02:21.883 INFO:tasks.workunit.client.1.smithi137.stdout:8/663: write d1/dc/d3c/d87/f1c [3152867,129953] 0 2024-03-14T13:02:21.883 INFO:tasks.workunit.client.1.smithi137.stdout:4/627: mknod d4/d9/d8e/dae/cd2 0 2024-03-14T13:02:21.883 INFO:tasks.workunit.client.0.smithi111.stdout:0/760: dwrite d6/d21/f73 [4194304,4194304] 0 2024-03-14T13:02:21.887 INFO:tasks.workunit.client.0.smithi111.stdout:0/761: symlink d6/d9/de/d32/d64/l10f 0 2024-03-14T13:02:21.888 INFO:tasks.workunit.client.1.smithi137.stdout:2/634: dread d2/d7/d2e/d3e/d44/d6b/d22/dac/faf [0,4194304] 0 2024-03-14T13:02:21.889 INFO:tasks.workunit.client.0.smithi111.stdout:5/779: dwrite d2/d16/d49/d4c/d9c/fe4 [0,4194304] 0 2024-03-14T13:02:21.890 INFO:tasks.workunit.client.0.smithi111.stdout:0/762: unlink d6/d9/d15/d55/l76 0 2024-03-14T13:02:21.890 INFO:tasks.workunit.client.1.smithi137.stdout:8/664: dread d1/dc/f52 [0,4194304] 0 2024-03-14T13:02:21.890 INFO:tasks.workunit.client.1.smithi137.stdout:2/635: read d2/d7/d2e/d3e/d44/d6b/f3f [1217125,124959] 0 2024-03-14T13:02:21.893 INFO:tasks.workunit.client.1.smithi137.stdout:7/487: sync 2024-03-14T13:02:21.893 INFO:tasks.workunit.client.1.smithi137.stdout:1/667: sync 2024-03-14T13:02:21.893 INFO:tasks.workunit.client.1.smithi137.stdout:5/574: sync 2024-03-14T13:02:21.894 INFO:tasks.workunit.client.0.smithi111.stdout:5/780: unlink d2/d45/l2a 0 2024-03-14T13:02:21.894 INFO:tasks.workunit.client.0.smithi111.stdout:0/763: creat d6/d3d/dba/f110 x:0 0 0 2024-03-14T13:02:21.895 INFO:tasks.workunit.client.0.smithi111.stdout:5/781: mknod d2/d45/d76/d9a/d9f/cf8 0 2024-03-14T13:02:21.897 INFO:tasks.workunit.client.0.smithi111.stdout:0/764: creat d6/d9/de/d77/f111 x:0 0 0 2024-03-14T13:02:21.898 INFO:tasks.workunit.client.1.smithi137.stdout:1/668: stat d7/d9/d19/d1f/d2e/f79 0 2024-03-14T13:02:21.899 INFO:tasks.workunit.client.1.smithi137.stdout:5/575: truncate d7/d16/d18/d25/d28/f60 1445703 0 2024-03-14T13:02:21.899 INFO:tasks.workunit.client.0.smithi111.stdout:5/782: creat d2/d16/d49/d44/d4a/d9e/db7/dbb/ff9 x:0 0 0 2024-03-14T13:02:21.900 INFO:tasks.workunit.client.1.smithi137.stdout:8/665: creat d1/dc/d4e/d6d/db5/fdf x:0 0 0 2024-03-14T13:02:21.901 INFO:tasks.workunit.client.1.smithi137.stdout:8/666: chown d1/dc/c7e 49068 1 2024-03-14T13:02:21.902 INFO:tasks.workunit.client.0.smithi111.stdout:0/765: rename d6/c6b to d6/d3d/da6/dfe/c112 0 2024-03-14T13:02:21.903 INFO:tasks.workunit.client.0.smithi111.stdout:5/783: truncate d2/d16/d49/d33/d68/f8a 204380 0 2024-03-14T13:02:21.903 INFO:tasks.workunit.client.1.smithi137.stdout:2/636: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/lce 0 2024-03-14T13:02:21.904 INFO:tasks.workunit.client.0.smithi111.stdout:0/766: creat d6/d9/d5b/f113 x:0 0 0 2024-03-14T13:02:21.905 INFO:tasks.workunit.client.1.smithi137.stdout:7/488: rename d3/d5/d23/d2c/d5d/d6a/d75/l8a to d3/d5/d23/d2c/d5d/d6a/d75/l94 0 2024-03-14T13:02:21.905 INFO:tasks.workunit.client.1.smithi137.stdout:6/611: dread d2/d6/de/f5d [0,4194304] 0 2024-03-14T13:02:21.906 INFO:tasks.workunit.client.0.smithi111.stdout:0/767: mkdir d6/d45/d114 0 2024-03-14T13:02:21.907 INFO:tasks.workunit.client.0.smithi111.stdout:0/768: dread - d6/d9/de/ff9 zero size 2024-03-14T13:02:21.907 INFO:tasks.workunit.client.0.smithi111.stdout:0/769: chown d6/d45/d87/fdc 7 1 2024-03-14T13:02:21.908 INFO:tasks.workunit.client.1.smithi137.stdout:4/628: write d4/f11 [4080963,83174] 0 2024-03-14T13:02:21.911 INFO:tasks.workunit.client.1.smithi137.stdout:7/489: truncate d3/f1c 396917 0 2024-03-14T13:02:21.913 INFO:tasks.workunit.client.1.smithi137.stdout:6/612: write d2/d6/de/d45/d59/d39/f73 [2881038,123586] 0 2024-03-14T13:02:21.913 INFO:tasks.workunit.client.0.smithi111.stdout:0/770: rmdir d6/d9/de 39 2024-03-14T13:02:21.913 INFO:tasks.workunit.client.1.smithi137.stdout:2/637: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 [0,4194304] 0 2024-03-14T13:02:21.914 INFO:tasks.workunit.client.1.smithi137.stdout:6/613: chown d2/d4c/c7e 496271337 1 2024-03-14T13:02:21.915 INFO:tasks.workunit.client.0.smithi111.stdout:0/771: readlink d6/d9/de/d3a/d46/d4a/l9f 0 2024-03-14T13:02:21.916 INFO:tasks.workunit.client.1.smithi137.stdout:8/667: rename d1/dc/d3c/d87/d34/d97/d13/f45 to d1/dc/fe0 0 2024-03-14T13:02:21.916 INFO:tasks.workunit.client.0.smithi111.stdout:0/772: readlink d6/d9/de/d3a/l107 0 2024-03-14T13:02:21.917 INFO:tasks.workunit.client.1.smithi137.stdout:7/490: rmdir d3/d6/d14/d22 39 2024-03-14T13:02:21.917 INFO:tasks.workunit.client.0.smithi111.stdout:5/784: write d2/d16/d49/f27 [3493803,130938] 0 2024-03-14T13:02:21.917 INFO:tasks.workunit.client.0.smithi111.stdout:0/773: readlink d6/d45/l80 0 2024-03-14T13:02:21.920 INFO:tasks.workunit.client.0.smithi111.stdout:5/785: mkdir d2/d16/d49/d33/de5/dfa 0 2024-03-14T13:02:21.920 INFO:tasks.workunit.client.1.smithi137.stdout:8/668: dread d1/dc/fcd [0,4194304] 0 2024-03-14T13:02:21.921 INFO:tasks.workunit.client.1.smithi137.stdout:4/629: rename d4/d9/d3a/d41/d38/lba to d4/d9/d3a/d41/d38/d79/da1/ld3 0 2024-03-14T13:02:21.921 INFO:tasks.workunit.client.0.smithi111.stdout:0/774: rename d6/d9/de/d77/da4/fe6 to d6/d9/de/d3a/d46/f115 0 2024-03-14T13:02:21.923 INFO:tasks.workunit.client.1.smithi137.stdout:1/669: truncate d7/d4d/d6b/faa 317072 0 2024-03-14T13:02:21.923 INFO:tasks.workunit.client.1.smithi137.stdout:6/614: mkdir d2/d54/d55/d6d/dc3/dce 0 2024-03-14T13:02:21.924 INFO:tasks.workunit.client.0.smithi111.stdout:0/775: write d6/d3d/d97/f106 [1821129,47813] 0 2024-03-14T13:02:21.926 INFO:tasks.workunit.client.0.smithi111.stdout:0/776: fsync d6/d9/d15/d55/da0/ff0 0 2024-03-14T13:02:21.927 INFO:tasks.workunit.client.1.smithi137.stdout:2/638: creat d2/d7/d2e/d3e/d4e/fcf x:0 0 0 2024-03-14T13:02:21.927 INFO:tasks.workunit.client.0.smithi111.stdout:5/786: dwrite d2/d16/d49/d44/d72/f3e [0,4194304] 0 2024-03-14T13:02:21.928 INFO:tasks.workunit.client.1.smithi137.stdout:6/615: chown d2/d6/f34 3377 1 2024-03-14T13:02:21.929 INFO:tasks.workunit.client.0.smithi111.stdout:0/777: truncate d6/d9/d15/d55/da0/fb9 4551594 0 2024-03-14T13:02:21.935 INFO:tasks.workunit.client.0.smithi111.stdout:5/787: dwrite d2/f7 [0,4194304] 0 2024-03-14T13:02:21.937 INFO:tasks.workunit.client.1.smithi137.stdout:7/491: dwrite d3/d5/d23/d3f/d5b/f79 [0,4194304] 0 2024-03-14T13:02:21.938 INFO:tasks.workunit.client.0.smithi111.stdout:5/788: rmdir d2/d16/d49/d33/de5 39 2024-03-14T13:02:21.939 INFO:tasks.workunit.client.0.smithi111.stdout:0/778: rmdir d6 39 2024-03-14T13:02:21.939 INFO:tasks.workunit.client.1.smithi137.stdout:4/630: dread d4/d10/d42/f75 [0,4194304] 0 2024-03-14T13:02:21.940 INFO:tasks.workunit.client.1.smithi137.stdout:8/669: dwrite d1/dc/d10/f1a [0,4194304] 0 2024-03-14T13:02:21.943 INFO:tasks.workunit.client.0.smithi111.stdout:0/779: unlink d6/d9/de/f36 0 2024-03-14T13:02:21.944 INFO:tasks.workunit.client.1.smithi137.stdout:8/670: stat d1/dc/d3c/d87/d34/l53 0 2024-03-14T13:02:21.946 INFO:tasks.workunit.client.1.smithi137.stdout:7/492: fdatasync d3/d4a/f5e 0 2024-03-14T13:02:21.950 INFO:tasks.workunit.client.1.smithi137.stdout:4/631: creat d4/d9/d3a/d70/d80/d8d/fd4 x:0 0 0 2024-03-14T13:02:21.950 INFO:tasks.workunit.client.1.smithi137.stdout:4/632: stat d4/d10/l93 0 2024-03-14T13:02:21.951 INFO:tasks.workunit.client.0.smithi111.stdout:0/780: dwrite d6/d9/de/d3a/fc3 [0,4194304] 0 2024-03-14T13:02:21.952 INFO:tasks.workunit.client.1.smithi137.stdout:8/671: mknod d1/dc/d3c/d87/d34/d82/d96/ce1 0 2024-03-14T13:02:21.954 INFO:tasks.workunit.client.1.smithi137.stdout:2/639: dread d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f83 [0,4194304] 0 2024-03-14T13:02:21.954 INFO:tasks.workunit.client.1.smithi137.stdout:1/670: getdents d7/d9/d19/d1f/d26/d6c/dc8 0 2024-03-14T13:02:21.955 INFO:tasks.workunit.client.1.smithi137.stdout:7/493: mknod d3/d5/d23/c95 0 2024-03-14T13:02:21.956 INFO:tasks.workunit.client.1.smithi137.stdout:6/616: getdents d2/d6/de/d45/d97 0 2024-03-14T13:02:21.957 INFO:tasks.workunit.client.1.smithi137.stdout:4/633: unlink d4/d9/d3a/d41/c90 0 2024-03-14T13:02:21.958 INFO:tasks.workunit.client.1.smithi137.stdout:7/494: chown d3/d5/d23/d3f/c42 33 1 2024-03-14T13:02:21.958 INFO:tasks.workunit.client.1.smithi137.stdout:2/640: read d2/d7/d2e/d3e/d44/d6b/f3f [3665900,28002] 0 2024-03-14T13:02:21.965 INFO:tasks.workunit.client.1.smithi137.stdout:6/617: creat d2/d54/d55/d9c/dc0/dc4/fcf x:0 0 0 2024-03-14T13:02:21.966 INFO:tasks.workunit.client.1.smithi137.stdout:7/495: write d3/f6f [858113,77463] 0 2024-03-14T13:02:21.966 INFO:tasks.workunit.client.1.smithi137.stdout:1/671: rename d7/d29/l2c to d7/d9/d19/d40/le3 0 2024-03-14T13:02:21.968 INFO:tasks.workunit.client.1.smithi137.stdout:6/618: readlink d2/d6/d1d/l3e 0 2024-03-14T13:02:21.969 INFO:tasks.workunit.client.1.smithi137.stdout:4/634: dwrite d4/d27/d5c/d6d/d58/f76 [0,4194304] 0 2024-03-14T13:02:21.971 INFO:tasks.workunit.client.1.smithi137.stdout:7/496: symlink d3/d5/d23/d2c/l96 0 2024-03-14T13:02:21.988 INFO:tasks.workunit.client.0.smithi111.stdout:0/781: dread d6/d9/de/f3b [0,4194304] 0 2024-03-14T13:02:21.993 INFO:tasks.workunit.client.0.smithi111.stdout:0/782: write d6/d21/d74/f99 [287740,114927] 0 2024-03-14T13:02:21.996 INFO:tasks.workunit.client.0.smithi111.stdout:0/783: rmdir d6/d3d/dc1/dd6 0 2024-03-14T13:02:21.997 INFO:tasks.workunit.client.0.smithi111.stdout:0/784: symlink d6/d9/d5b/d9e/l116 0 2024-03-14T13:02:22.000 INFO:tasks.workunit.client.0.smithi111.stdout:0/785: rename d6/d9/de/d32/d64/l6c to d6/d3d/da6/dfe/l117 0 2024-03-14T13:02:22.000 INFO:tasks.workunit.client.0.smithi111.stdout:0/786: dread - d6/d9/d5b/f113 zero size 2024-03-14T13:02:22.004 INFO:tasks.workunit.client.1.smithi137.stdout:3/557: sync 2024-03-14T13:02:22.004 INFO:tasks.workunit.client.1.smithi137.stdout:0/594: sync 2024-03-14T13:02:22.004 INFO:tasks.workunit.client.1.smithi137.stdout:9/733: sync 2024-03-14T13:02:22.006 INFO:tasks.workunit.client.0.smithi111.stdout:0/787: truncate d6/d9/d15/d55/fee 2443187 0 2024-03-14T13:02:22.007 INFO:tasks.workunit.client.0.smithi111.stdout:0/788: mkdir d6/d9/de/d32/d118 0 2024-03-14T13:02:22.009 INFO:tasks.workunit.client.0.smithi111.stdout:0/789: creat d6/d3d/dc1/f119 x:0 0 0 2024-03-14T13:02:22.011 INFO:tasks.workunit.client.0.smithi111.stdout:0/790: mkdir d6/d9/de/d11a 0 2024-03-14T13:02:22.012 INFO:tasks.workunit.client.0.smithi111.stdout:0/791: mkdir d6/d9/de/d3a/d46/d4a/d11b 0 2024-03-14T13:02:22.014 INFO:tasks.workunit.client.0.smithi111.stdout:0/792: read d6/d9/d15/d55/f86 [1175623,130590] 0 2024-03-14T13:02:22.021 INFO:tasks.workunit.client.1.smithi137.stdout:2/641: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f80 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd0 0 2024-03-14T13:02:22.022 INFO:tasks.workunit.client.1.smithi137.stdout:2/642: fdatasync d2/d7/d2e/d3e/d44/d6b/d1d/f7b 0 2024-03-14T13:02:22.022 INFO:tasks.workunit.client.1.smithi137.stdout:1/672: truncate d7/d9/d19/d1f/d22/f25 2182313 0 2024-03-14T13:02:22.023 INFO:tasks.workunit.client.0.smithi111.stdout:0/793: dwrite d6/d3d/fa7 [0,4194304] 0 2024-03-14T13:02:22.025 INFO:tasks.workunit.client.1.smithi137.stdout:7/497: mkdir d3/d6/d64/d97 0 2024-03-14T13:02:22.026 INFO:tasks.workunit.client.1.smithi137.stdout:0/595: mkdir d1/d5/d1b/d42/d3a/dc6 0 2024-03-14T13:02:22.029 INFO:tasks.workunit.client.1.smithi137.stdout:3/558: rename d4/de/d16/d74/f9b to d4/d22/fbd 0 2024-03-14T13:02:22.030 INFO:tasks.workunit.client.1.smithi137.stdout:3/559: chown d4/d19/f70 0 1 2024-03-14T13:02:22.033 INFO:tasks.workunit.client.1.smithi137.stdout:2/643: rmdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99 39 2024-03-14T13:02:22.035 INFO:tasks.workunit.client.1.smithi137.stdout:2/644: write d2/d7/d2e/d3e/d44/d6b/d22/f8f [285314,70058] 0 2024-03-14T13:02:22.036 INFO:tasks.workunit.client.1.smithi137.stdout:7/498: unlink d3/d5/d23/d2c/l96 0 2024-03-14T13:02:22.039 INFO:tasks.workunit.client.1.smithi137.stdout:9/734: rename d6/df/d19/d33/f36 to d6/df/d71/fed 0 2024-03-14T13:02:22.041 INFO:tasks.workunit.client.1.smithi137.stdout:6/619: link d2/d6/de/d45/d97/d9f/l95 d2/d54/d55/d9c/dc0/ld0 0 2024-03-14T13:02:22.041 INFO:tasks.workunit.client.1.smithi137.stdout:1/673: creat d7/d92/fe4 x:0 0 0 2024-03-14T13:02:22.042 INFO:tasks.workunit.client.1.smithi137.stdout:3/560: creat d4/df/fbe x:0 0 0 2024-03-14T13:02:22.043 INFO:tasks.workunit.client.1.smithi137.stdout:2/645: mknod d2/d7/d2e/d3e/d44/d6b/d1d/cd1 0 2024-03-14T13:02:22.044 INFO:tasks.workunit.client.1.smithi137.stdout:6/620: chown d2/d6/de/d45/lb7 279532 1 2024-03-14T13:02:22.044 INFO:tasks.workunit.client.1.smithi137.stdout:7/499: dwrite d3/d6/d14/f57 [0,4194304] 0 2024-03-14T13:02:22.048 INFO:tasks.workunit.client.1.smithi137.stdout:2/646: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f 0 2024-03-14T13:02:22.051 INFO:tasks.workunit.client.1.smithi137.stdout:2/647: dread - d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/fc1 zero size 2024-03-14T13:02:22.052 INFO:tasks.workunit.client.0.smithi111.stdout:1/660: sync 2024-03-14T13:02:22.052 INFO:tasks.workunit.client.0.smithi111.stdout:6/589: sync 2024-03-14T13:02:22.052 INFO:tasks.workunit.client.0.smithi111.stdout:3/616: sync 2024-03-14T13:02:22.053 INFO:tasks.workunit.client.1.smithi137.stdout:7/500: creat d3/d5/d5a/f98 x:0 0 0 2024-03-14T13:02:22.054 INFO:tasks.workunit.client.1.smithi137.stdout:0/596: rename d1/d1f/f2f to d1/d5/d1b/d42/d3a/dc6/fc7 0 2024-03-14T13:02:22.055 INFO:tasks.workunit.client.1.smithi137.stdout:3/561: mknod d4/de/d16/d1c/d5d/d85/cbf 0 2024-03-14T13:02:22.055 INFO:tasks.workunit.client.1.smithi137.stdout:7/501: read d3/d6/d59/f5f [2733,67016] 0 2024-03-14T13:02:22.055 INFO:tasks.workunit.client.0.smithi111.stdout:6/590: unlink d0/f1a 0 2024-03-14T13:02:22.056 INFO:tasks.workunit.client.1.smithi137.stdout:6/621: symlink d2/d54/d55/d9c/dbf/ld1 0 2024-03-14T13:02:22.057 INFO:tasks.workunit.client.0.smithi111.stdout:3/617: unlink d0/d2d/d35/d50/d5e/d19/l3f 0 2024-03-14T13:02:22.059 INFO:tasks.workunit.client.1.smithi137.stdout:6/622: stat d2/d6/de/d45/d59 0 2024-03-14T13:02:22.061 INFO:tasks.workunit.client.0.smithi111.stdout:3/618: symlink d0/d2d/d35/le2 0 2024-03-14T13:02:22.063 INFO:tasks.workunit.client.0.smithi111.stdout:3/619: chown d0/d2d/d35/d50/da2/db3/f47 253400 1 2024-03-14T13:02:22.063 INFO:tasks.workunit.client.1.smithi137.stdout:3/562: mknod d4/de/cc0 0 2024-03-14T13:02:22.064 INFO:tasks.workunit.client.1.smithi137.stdout:3/563: readlink d4/de/d16/lb2 0 2024-03-14T13:02:22.067 INFO:tasks.workunit.client.0.smithi111.stdout:1/661: dwrite f11 [0,4194304] 0 2024-03-14T13:02:22.067 INFO:tasks.workunit.client.1.smithi137.stdout:6/623: mknod d2/cd2 0 2024-03-14T13:02:22.070 INFO:tasks.workunit.client.1.smithi137.stdout:2/648: link d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/l88 d2/d7/d2e/d3e/d44/d6b/d1d/ld2 0 2024-03-14T13:02:22.071 INFO:tasks.workunit.client.1.smithi137.stdout:1/674: write d7/d9/d19/d1f/d22/f45 [3413115,34029] 0 2024-03-14T13:02:22.071 INFO:tasks.workunit.client.0.smithi111.stdout:3/620: dread d0/d2d/d35/f92 [0,4194304] 0 2024-03-14T13:02:22.072 INFO:tasks.workunit.client.1.smithi137.stdout:9/735: dread d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:22.074 INFO:tasks.workunit.client.0.smithi111.stdout:6/591: dwrite d0/d17/d20/fb1 [0,4194304] 0 2024-03-14T13:02:22.074 INFO:tasks.workunit.client.0.smithi111.stdout:1/662: rename d20/d25/d87/l8d to d20/d25/da8/lde 0 2024-03-14T13:02:22.076 INFO:tasks.workunit.client.1.smithi137.stdout:3/564: dwrite d4/d19/f90 [0,4194304] 0 2024-03-14T13:02:22.077 INFO:tasks.workunit.client.0.smithi111.stdout:1/663: mkdir d20/d40/db4/ddf 0 2024-03-14T13:02:22.078 INFO:tasks.workunit.client.1.smithi137.stdout:6/624: creat d2/d54/d55/d6d/d76/d57/fd3 x:0 0 0 2024-03-14T13:02:22.079 INFO:tasks.workunit.client.1.smithi137.stdout:2/649: fsync d2/d7/d2e/d3e/d44/d6b/f1b 0 2024-03-14T13:02:22.080 INFO:tasks.workunit.client.0.smithi111.stdout:1/664: rename d20/d25/c8a to d20/d21/d28/ce0 0 2024-03-14T13:02:22.083 INFO:tasks.workunit.client.1.smithi137.stdout:6/625: dread - d2/d54/d55/d6d/d76/d57/fd3 zero size 2024-03-14T13:02:22.083 INFO:tasks.workunit.client.1.smithi137.stdout:0/597: dread d1/d5/d1b/d1e/d53/d6d/fba [0,4194304] 0 2024-03-14T13:02:22.085 INFO:tasks.workunit.client.1.smithi137.stdout:2/650: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64 570256 1 2024-03-14T13:02:22.087 INFO:tasks.workunit.client.0.smithi111.stdout:1/665: write f19 [4979699,125439] 0 2024-03-14T13:02:22.087 INFO:tasks.workunit.client.1.smithi137.stdout:3/565: rmdir d4/df/d82 39 2024-03-14T13:02:22.089 INFO:tasks.workunit.client.1.smithi137.stdout:6/626: read d2/d6/de/d45/fa8 [3013647,69452] 0 2024-03-14T13:02:22.091 INFO:tasks.workunit.client.1.smithi137.stdout:7/502: dwrite d3/d6/f15 [0,4194304] 0 2024-03-14T13:02:22.092 INFO:tasks.workunit.client.0.smithi111.stdout:3/621: dwrite d0/d2d/d35/f63 [4194304,4194304] 0 2024-03-14T13:02:22.094 INFO:tasks.workunit.client.1.smithi137.stdout:3/566: unlink d4/de/cc0 0 2024-03-14T13:02:22.094 INFO:tasks.workunit.client.0.smithi111.stdout:3/622: write d0/d2d/d35/d50/da2/db3/f47 [5232261,31922] 0 2024-03-14T13:02:22.097 INFO:tasks.workunit.client.0.smithi111.stdout:3/623: creat d0/d2d/d35/fe3 x:0 0 0 2024-03-14T13:02:22.098 INFO:tasks.workunit.client.1.smithi137.stdout:9/736: dread d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:22.099 INFO:tasks.workunit.client.0.smithi111.stdout:3/624: mknod d0/d2d/d35/d50/ddf/ce4 0 2024-03-14T13:02:22.099 INFO:tasks.workunit.client.1.smithi137.stdout:7/503: readlink d3/d5/d23/l55 0 2024-03-14T13:02:22.100 INFO:tasks.workunit.client.1.smithi137.stdout:2/651: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 4109568 1 2024-03-14T13:02:22.101 INFO:tasks.workunit.client.0.smithi111.stdout:3/625: write d0/d2d/d35/d50/d5e/f12 [1913116,115578] 0 2024-03-14T13:02:22.103 INFO:tasks.workunit.client.0.smithi111.stdout:3/626: mknod d0/d2d/d35/d69/d84/ce5 0 2024-03-14T13:02:22.104 INFO:tasks.workunit.client.1.smithi137.stdout:6/627: dwrite d2/d6/de/fb3 [0,4194304] 0 2024-03-14T13:02:22.104 INFO:tasks.workunit.client.1.smithi137.stdout:0/598: getdents d1/d44 0 2024-03-14T13:02:22.104 INFO:tasks.workunit.client.1.smithi137.stdout:3/567: read d4/de/d16/f31 [386222,101781] 0 2024-03-14T13:02:22.108 INFO:tasks.workunit.client.0.smithi111.stdout:3/627: rename d0/d91 to d0/d2d/d35/d69/de6 0 2024-03-14T13:02:22.110 INFO:tasks.workunit.client.1.smithi137.stdout:6/628: fsync d2/d6/de/d45/f60 0 2024-03-14T13:02:22.112 INFO:tasks.workunit.client.0.smithi111.stdout:1/666: write d20/f58 [1533813,47477] 0 2024-03-14T13:02:22.113 INFO:tasks.workunit.client.1.smithi137.stdout:2/652: mknod d2/d7/d2e/d3e/daa/dcd/cd3 0 2024-03-14T13:02:22.115 INFO:tasks.workunit.client.1.smithi137.stdout:3/568: write d4/df/d82/fb5 [990701,121599] 0 2024-03-14T13:02:22.121 INFO:tasks.workunit.client.1.smithi137.stdout:9/737: dwrite d6/fa3 [0,4194304] 0 2024-03-14T13:02:22.121 INFO:tasks.workunit.client.1.smithi137.stdout:7/504: dwrite d3/d6/f45 [0,4194304] 0 2024-03-14T13:02:22.123 INFO:tasks.workunit.client.0.smithi111.stdout:3/628: creat d0/d2d/d35/d50/da2/db3/da7/fe7 x:0 0 0 2024-03-14T13:02:22.123 INFO:tasks.workunit.client.0.smithi111.stdout:1/667: dwrite d20/d25/d75/f6a [4194304,4194304] 0 2024-03-14T13:02:22.125 INFO:tasks.workunit.client.0.smithi111.stdout:3/629: write d0/d2d/d35/d50/d6c/fbf [897959,119025] 0 2024-03-14T13:02:22.127 INFO:tasks.workunit.client.1.smithi137.stdout:2/653: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/dca/ld4 0 2024-03-14T13:02:22.130 INFO:tasks.workunit.client.1.smithi137.stdout:1/675: dwrite f6 [0,4194304] 0 2024-03-14T13:02:22.131 INFO:tasks.workunit.client.1.smithi137.stdout:5/576: sync 2024-03-14T13:02:22.133 INFO:tasks.workunit.client.1.smithi137.stdout:9/738: read d6/df/d71/f8b [126306,38778] 0 2024-03-14T13:02:22.134 INFO:tasks.workunit.client.0.smithi111.stdout:3/630: mknod d0/d2d/d35/d50/d5e/ce8 0 2024-03-14T13:02:22.136 INFO:tasks.workunit.client.1.smithi137.stdout:1/676: mkdir d7/d9/d19/d40/d42/d9d/de5 0 2024-03-14T13:02:22.137 INFO:tasks.workunit.client.1.smithi137.stdout:9/739: truncate d6/df/d19/d33/f80 4846874 0 2024-03-14T13:02:22.141 INFO:tasks.workunit.client.1.smithi137.stdout:3/569: getdents d4/de/d93 0 2024-03-14T13:02:22.142 INFO:tasks.workunit.client.1.smithi137.stdout:9/740: rmdir d6/df/d19/d3a/d40 39 2024-03-14T13:02:22.142 INFO:tasks.workunit.client.1.smithi137.stdout:2/654: creat d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/fd5 x:0 0 0 2024-03-14T13:02:22.146 INFO:tasks.workunit.client.1.smithi137.stdout:3/570: chown d4/df/fbe 3 1 2024-03-14T13:02:22.146 INFO:tasks.workunit.client.1.smithi137.stdout:0/599: truncate d1/d5/d1b/d42/d3a/dc6/fc7 3920634 0 2024-03-14T13:02:22.146 INFO:tasks.workunit.client.1.smithi137.stdout:6/629: write d2/d6/de/d45/d59/f37 [4445581,29357] 0 2024-03-14T13:02:22.147 INFO:tasks.workunit.client.0.smithi111.stdout:1/668: truncate d20/f24 1159828 0 2024-03-14T13:02:22.147 INFO:tasks.workunit.client.0.smithi111.stdout:1/669: stat fe 0 2024-03-14T13:02:22.149 INFO:tasks.workunit.client.1.smithi137.stdout:0/600: chown d1/d5/d1b/d1e/d25/d55/f67 25633 1 2024-03-14T13:02:22.149 INFO:tasks.workunit.client.1.smithi137.stdout:0/601: stat d1/f1c 0 2024-03-14T13:02:22.149 INFO:tasks.workunit.client.1.smithi137.stdout:3/571: rmdir d4/de/d16 39 2024-03-14T13:02:22.150 INFO:tasks.workunit.client.0.smithi111.stdout:1/670: write f1a [1686476,102898] 0 2024-03-14T13:02:22.153 INFO:tasks.workunit.client.1.smithi137.stdout:1/677: dread d7/d9/d19/d1f/d2e/f3d [0,4194304] 0 2024-03-14T13:02:22.154 INFO:tasks.workunit.client.0.smithi111.stdout:3/631: dwrite d0/d2d/d35/f92 [0,4194304] 0 2024-03-14T13:02:22.154 INFO:tasks.workunit.client.1.smithi137.stdout:2/655: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd6 x:0 0 0 2024-03-14T13:02:22.154 INFO:tasks.workunit.client.1.smithi137.stdout:6/630: creat d2/d6/da7/fd4 x:0 0 0 2024-03-14T13:02:22.155 INFO:tasks.workunit.client.1.smithi137.stdout:0/602: mkdir d1/d5/d1b/d42/d3a/dc8 0 2024-03-14T13:02:22.156 INFO:tasks.workunit.client.0.smithi111.stdout:1/671: rename d20/f57 to d20/d21/d28/fe1 0 2024-03-14T13:02:22.157 INFO:tasks.workunit.client.1.smithi137.stdout:3/572: unlink d4/df/d82/fa3 0 2024-03-14T13:02:22.158 INFO:tasks.workunit.client.0.smithi111.stdout:9/629: sync 2024-03-14T13:02:22.160 INFO:tasks.workunit.client.0.smithi111.stdout:1/672: fdatasync d20/d25/d31/f3b 0 2024-03-14T13:02:22.161 INFO:tasks.workunit.client.1.smithi137.stdout:1/678: symlink d7/d9/d19/d1f/d89/le6 0 2024-03-14T13:02:22.161 INFO:tasks.workunit.client.1.smithi137.stdout:6/631: symlink d2/d54/d55/d6d/d76/d57/ld5 0 2024-03-14T13:02:22.164 INFO:tasks.workunit.client.0.smithi111.stdout:1/673: symlink d20/d25/d75/le2 0 2024-03-14T13:02:22.164 INFO:tasks.workunit.client.1.smithi137.stdout:2/656: read - d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 zero size 2024-03-14T13:02:22.165 INFO:tasks.workunit.client.1.smithi137.stdout:0/603: symlink d1/d5/d1b/d42/d3a/db0/lc9 0 2024-03-14T13:02:22.170 INFO:tasks.workunit.client.0.smithi111.stdout:1/674: getdents d20/d21 0 2024-03-14T13:02:22.176 INFO:tasks.workunit.client.1.smithi137.stdout:6/632: mkdir d2/dd6 0 2024-03-14T13:02:22.177 INFO:tasks.workunit.client.1.smithi137.stdout:2/657: truncate d2/d7/d2e/d3e/d44/d6b/d22/f8f 996127 0 2024-03-14T13:02:22.177 INFO:tasks.workunit.client.1.smithi137.stdout:3/573: mknod d4/de/d16/d1c/db8/cc1 0 2024-03-14T13:02:22.178 INFO:tasks.workunit.client.0.smithi111.stdout:1/675: dwrite d20/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:22.178 INFO:tasks.workunit.client.1.smithi137.stdout:0/604: truncate d1/d5/d1b/d1e/d25/d55/f60 2355086 0 2024-03-14T13:02:22.179 INFO:tasks.workunit.client.1.smithi137.stdout:4/635: sync 2024-03-14T13:02:22.180 INFO:tasks.workunit.client.1.smithi137.stdout:8/672: sync 2024-03-14T13:02:22.180 INFO:tasks.workunit.client.1.smithi137.stdout:7/505: dread d3/d5/d5a/f73 [0,4194304] 0 2024-03-14T13:02:22.180 INFO:tasks.workunit.client.0.smithi111.stdout:9/630: dwrite d5/d6/d8/d74/f77 [0,4194304] 0 2024-03-14T13:02:22.181 INFO:tasks.workunit.client.0.smithi111.stdout:1/676: rmdir d20/d40/da6 39 2024-03-14T13:02:22.181 INFO:tasks.workunit.client.1.smithi137.stdout:1/679: write d7/d9/d19/d5f/f7b [857107,96308] 0 2024-03-14T13:02:22.182 INFO:tasks.workunit.client.1.smithi137.stdout:3/574: unlink d4/de/d16/d27/f65 0 2024-03-14T13:02:22.183 INFO:tasks.workunit.client.0.smithi111.stdout:1/677: rmdir d20/d25/d75 39 2024-03-14T13:02:22.183 INFO:tasks.workunit.client.0.smithi111.stdout:9/631: mkdir d5/d42/d71/ddc 0 2024-03-14T13:02:22.184 INFO:tasks.workunit.client.1.smithi137.stdout:0/605: creat d1/d44/d72/fca x:0 0 0 2024-03-14T13:02:22.185 INFO:tasks.workunit.client.0.smithi111.stdout:1/678: chown d20/d21/d28/ce0 170475 1 2024-03-14T13:02:22.185 INFO:tasks.workunit.client.0.smithi111.stdout:1/679: fsync d20/d25/d87/fb7 0 2024-03-14T13:02:22.189 INFO:tasks.workunit.client.1.smithi137.stdout:2/658: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/dd7 0 2024-03-14T13:02:22.190 INFO:tasks.workunit.client.1.smithi137.stdout:3/575: dread - d4/d19/f75 zero size 2024-03-14T13:02:22.193 INFO:tasks.workunit.client.1.smithi137.stdout:6/633: link d2/d54/d55/d38/l72 d2/d6/de/d45/d97/d9f/ld7 0 2024-03-14T13:02:22.194 INFO:tasks.workunit.client.0.smithi111.stdout:9/632: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:22.195 INFO:tasks.workunit.client.0.smithi111.stdout:1/680: dwrite d20/d25/d87/fb7 [0,4194304] 0 2024-03-14T13:02:22.195 INFO:tasks.workunit.client.1.smithi137.stdout:7/506: symlink d3/d6/d2e/l99 0 2024-03-14T13:02:22.196 INFO:tasks.workunit.client.0.smithi111.stdout:9/633: stat d5/d42/lc0 0 2024-03-14T13:02:22.196 INFO:tasks.workunit.client.0.smithi111.stdout:1/681: fsync d20/d21/d28/d5c/fae 0 2024-03-14T13:02:22.197 INFO:tasks.workunit.client.0.smithi111.stdout:9/634: stat d5/d42/da4/cc5 0 2024-03-14T13:02:22.200 INFO:tasks.workunit.client.1.smithi137.stdout:5/577: dread d7/d16/d18/d25/f5b [0,4194304] 0 2024-03-14T13:02:22.202 INFO:tasks.workunit.client.1.smithi137.stdout:9/741: dwrite d6/df/d19/d3a/d40/f7f [0,4194304] 0 2024-03-14T13:02:22.203 INFO:tasks.workunit.client.0.smithi111.stdout:9/635: rename d5/f4f to d5/d6/d8/d74/fdd 0 2024-03-14T13:02:22.206 INFO:tasks.workunit.client.0.smithi111.stdout:1/682: dwrite d20/d25/f3d [0,4194304] 0 2024-03-14T13:02:22.207 INFO:tasks.workunit.client.0.smithi111.stdout:3/632: dread d0/d2d/d35/d69/d84/f87 [0,4194304] 0 2024-03-14T13:02:22.207 INFO:tasks.workunit.client.0.smithi111.stdout:9/636: creat d5/d42/d71/d96/fde x:0 0 0 2024-03-14T13:02:22.209 INFO:tasks.workunit.client.0.smithi111.stdout:9/637: chown d5/d42/d71/d96/fac 3339 1 2024-03-14T13:02:22.209 INFO:tasks.workunit.client.1.smithi137.stdout:5/578: write d7/d16/d18/f39 [4045948,46823] 0 2024-03-14T13:02:22.209 INFO:tasks.workunit.client.1.smithi137.stdout:0/606: link d1/d5/d1b/d42/d3a/fc2 d1/d44/d72/d7e/fcb 0 2024-03-14T13:02:22.210 INFO:tasks.workunit.client.0.smithi111.stdout:3/633: write d0/f20 [5384054,64177] 0 2024-03-14T13:02:22.212 INFO:tasks.workunit.client.0.smithi111.stdout:9/638: readlink d5/d6/lb9 0 2024-03-14T13:02:22.216 INFO:tasks.workunit.client.1.smithi137.stdout:6/634: mknod d2/d54/d55/d9c/dbf/cd8 0 2024-03-14T13:02:22.218 INFO:tasks.workunit.client.0.smithi111.stdout:1/683: dwrite d20/d21/d28/d5c/f9a [0,4194304] 0 2024-03-14T13:02:22.220 INFO:tasks.workunit.client.0.smithi111.stdout:3/634: rename d0/d2d/d35/fe3 to d0/d2d/d35/d50/d6c/da3/daa/fe9 0 2024-03-14T13:02:22.221 INFO:tasks.workunit.client.1.smithi137.stdout:8/673: dwrite d1/dc/d71/fb6 [0,4194304] 0 2024-03-14T13:02:22.221 INFO:tasks.workunit.client.1.smithi137.stdout:4/636: dread d4/d27/d5c/d6d/d58/f7e [0,4194304] 0 2024-03-14T13:02:22.221 INFO:tasks.workunit.client.1.smithi137.stdout:8/674: chown d1/f6 9 1 2024-03-14T13:02:22.226 INFO:tasks.workunit.client.1.smithi137.stdout:6/635: dread - d2/d6/de/d45/d97/d9f/fb8 zero size 2024-03-14T13:02:22.226 INFO:tasks.workunit.client.1.smithi137.stdout:3/576: rename d4/de/d16/d1c/d5d/d85/f49 to d4/de/d16/d1c/d5d/d85/d6b/fc2 0 2024-03-14T13:02:22.228 INFO:tasks.workunit.client.1.smithi137.stdout:6/636: fsync d2/d54/d55/d6d/d76/f8f 0 2024-03-14T13:02:22.228 INFO:tasks.workunit.client.1.smithi137.stdout:8/675: write d1/dc/fce [621222,54854] 0 2024-03-14T13:02:22.229 INFO:tasks.workunit.client.1.smithi137.stdout:7/507: link d3/d6/d14/d27/c39 d3/d52/d77/c9a 0 2024-03-14T13:02:22.229 INFO:tasks.workunit.client.0.smithi111.stdout:1/684: rmdir d20/d21/d28/da0/db6 39 2024-03-14T13:02:22.229 INFO:tasks.workunit.client.0.smithi111.stdout:3/635: fsync d0/d2d/d35/d50/f88 0 2024-03-14T13:02:22.229 INFO:tasks.workunit.client.0.smithi111.stdout:9/639: dwrite d5/d42/d71/dbd/dcd/fb4 [0,4194304] 0 2024-03-14T13:02:22.230 INFO:tasks.workunit.client.1.smithi137.stdout:9/742: symlink d6/df/d88/dd5/lee 0 2024-03-14T13:02:22.233 INFO:tasks.workunit.client.1.smithi137.stdout:9/743: write d6/df/d19/d3a/fe2 [900030,121539] 0 2024-03-14T13:02:22.235 INFO:tasks.workunit.client.1.smithi137.stdout:5/579: mkdir d7/d16/d18/d25/d28/dc5 0 2024-03-14T13:02:22.236 INFO:tasks.workunit.client.1.smithi137.stdout:7/508: read d3/d5/d5a/f7f [2731424,94062] 0 2024-03-14T13:02:22.237 INFO:tasks.workunit.client.1.smithi137.stdout:4/637: symlink d4/d9/d3a/d41/d38/d55/db6/ld5 0 2024-03-14T13:02:22.239 INFO:tasks.workunit.client.0.smithi111.stdout:3/636: write d0/d2d/d35/d50/da2/db3/d71/fda [134258,102907] 0 2024-03-14T13:02:22.243 INFO:tasks.workunit.client.0.smithi111.stdout:1/685: rename d20/c43 to d20/d21/d28/ce3 0 2024-03-14T13:02:22.243 INFO:tasks.workunit.client.0.smithi111.stdout:3/637: write d0/d2d/d35/d50/da2/db3/d71/dbc/fd3 [789333,57293] 0 2024-03-14T13:02:22.244 INFO:tasks.workunit.client.0.smithi111.stdout:9/640: dwrite d5/d6/d2a/d5a/f76 [0,4194304] 0 2024-03-14T13:02:22.246 INFO:tasks.workunit.client.1.smithi137.stdout:4/638: dwrite d4/d9/d3a/d41/d38/d55/db6/fc6 [0,4194304] 0 2024-03-14T13:02:22.246 INFO:tasks.workunit.client.1.smithi137.stdout:5/580: readlink d7/d16/d18/l22 0 2024-03-14T13:02:22.250 INFO:tasks.workunit.client.0.smithi111.stdout:1/686: symlink d20/d21/d3f/d59/le4 0 2024-03-14T13:02:22.251 INFO:tasks.workunit.client.0.smithi111.stdout:9/641: fsync d5/d6/d2a/f75 0 2024-03-14T13:02:22.253 INFO:tasks.workunit.client.0.smithi111.stdout:3/638: creat d0/d2d/d35/fea x:0 0 0 2024-03-14T13:02:22.253 INFO:tasks.workunit.client.0.smithi111.stdout:1/687: mknod d20/d25/d31/d80/ce5 0 2024-03-14T13:02:22.254 INFO:tasks.workunit.client.0.smithi111.stdout:1/688: write d20/d40/f4f [378150,110802] 0 2024-03-14T13:02:22.255 INFO:tasks.workunit.client.1.smithi137.stdout:8/676: mknod d1/ce2 0 2024-03-14T13:02:22.257 INFO:tasks.workunit.client.1.smithi137.stdout:5/581: stat d7/d49/d75/f85 0 2024-03-14T13:02:22.257 INFO:tasks.workunit.client.0.smithi111.stdout:1/689: dread d20/d25/d87/fb8 [0,4194304] 0 2024-03-14T13:02:22.257 INFO:tasks.workunit.client.0.smithi111.stdout:3/639: rename d0/d2d/d35/fcd to d0/d2d/d35/d50/da2/feb 0 2024-03-14T13:02:22.258 INFO:tasks.workunit.client.1.smithi137.stdout:0/607: getdents d1/d5/d1b/d42/d3a/db0/d84 0 2024-03-14T13:02:22.260 INFO:tasks.workunit.client.1.smithi137.stdout:6/637: creat d2/d6/fd9 x:0 0 0 2024-03-14T13:02:22.261 INFO:tasks.workunit.client.1.smithi137.stdout:7/509: read d3/d54/f61 [603976,130225] 0 2024-03-14T13:02:22.261 INFO:tasks.workunit.client.1.smithi137.stdout:4/639: mknod d4/d9/cd6 0 2024-03-14T13:02:22.261 INFO:tasks.workunit.client.0.smithi111.stdout:3/640: dread d0/d2d/d35/d4c/d82/f7e [0,4194304] 0 2024-03-14T13:02:22.261 INFO:tasks.workunit.client.0.smithi111.stdout:1/690: chown d20/d40/da6/lb2 5599 1 2024-03-14T13:02:22.261 INFO:tasks.workunit.client.1.smithi137.stdout:9/744: dwrite d6/df/d71/fed [0,4194304] 0 2024-03-14T13:02:22.265 INFO:tasks.workunit.client.1.smithi137.stdout:5/582: write d7/f30 [1755967,118249] 0 2024-03-14T13:02:22.266 INFO:tasks.workunit.client.0.smithi111.stdout:1/691: link fa d20/d25/d75/fe6 0 2024-03-14T13:02:22.269 INFO:tasks.workunit.client.0.smithi111.stdout:1/692: fsync d20/d25/dd0/dd2/fdb 0 2024-03-14T13:02:22.269 INFO:tasks.workunit.client.0.smithi111.stdout:3/641: dwrite d0/d2d/d35/d50/d6c/fbf [0,4194304] 0 2024-03-14T13:02:22.270 INFO:tasks.workunit.client.1.smithi137.stdout:0/608: chown d1/d5/d1b/d1e/d25/d26/fab 0 1 2024-03-14T13:02:22.270 INFO:tasks.workunit.client.1.smithi137.stdout:4/640: mkdir d4/d9/d3a/d41/dd7 0 2024-03-14T13:02:22.271 INFO:tasks.workunit.client.1.smithi137.stdout:7/510: write d3/d6/d14/d27/f6c [27521,39189] 0 2024-03-14T13:02:22.274 INFO:tasks.workunit.client.1.smithi137.stdout:5/583: chown d7/d16/d18/d48/l7e 9277 1 2024-03-14T13:02:22.274 INFO:tasks.workunit.client.1.smithi137.stdout:3/577: dread d4/de/d16/f6a [0,4194304] 0 2024-03-14T13:02:22.275 INFO:tasks.workunit.client.1.smithi137.stdout:2/659: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/fa8 [0,4194304] 0 2024-03-14T13:02:22.275 INFO:tasks.workunit.client.1.smithi137.stdout:4/641: readlink d4/d10/l93 0 2024-03-14T13:02:22.276 INFO:tasks.workunit.client.1.smithi137.stdout:6/638: rename d2/d6/d1d/d88/f9d to d2/d54/d55/d6d/d76/fda 0 2024-03-14T13:02:22.276 INFO:tasks.workunit.client.1.smithi137.stdout:8/677: dwrite d1/dc/d10/f1a [0,4194304] 0 2024-03-14T13:02:22.277 INFO:tasks.workunit.client.0.smithi111.stdout:3/642: creat d0/d2d/d35/d50/da2/fec x:0 0 0 2024-03-14T13:02:22.277 INFO:tasks.workunit.client.1.smithi137.stdout:7/511: readlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:22.281 INFO:tasks.workunit.client.0.smithi111.stdout:3/643: creat d0/d2d/d35/d4c/db0/fed x:0 0 0 2024-03-14T13:02:22.281 INFO:tasks.workunit.client.0.smithi111.stdout:3/644: chown d0/d2d/d35/c78 17 1 2024-03-14T13:02:22.282 INFO:tasks.workunit.client.1.smithi137.stdout:5/584: write d7/f7b [125192,112771] 0 2024-03-14T13:02:22.283 INFO:tasks.workunit.client.1.smithi137.stdout:8/678: creat d1/dc/d3c/d87/d34/d82/d96/fe3 x:0 0 0 2024-03-14T13:02:22.284 INFO:tasks.workunit.client.1.smithi137.stdout:7/512: write d3/d6/d2e/f49 [2854152,39306] 0 2024-03-14T13:02:22.286 INFO:tasks.workunit.client.0.smithi111.stdout:3/645: mkdir d0/d2d/d35/d4c/dad/dee 0 2024-03-14T13:02:22.287 INFO:tasks.workunit.client.0.smithi111.stdout:3/646: dread - d0/d2d/d35/d50/da2/feb zero size 2024-03-14T13:02:22.288 INFO:tasks.workunit.client.1.smithi137.stdout:2/660: rename d2/d7/d2e/c32 to d2/d34/d3c/cd8 0 2024-03-14T13:02:22.288 INFO:tasks.workunit.client.1.smithi137.stdout:0/609: truncate d1/d5/d1b/d1e/d25/d8a/d4a/fbf 229350 0 2024-03-14T13:02:22.288 INFO:tasks.workunit.client.0.smithi111.stdout:3/647: stat d0/d2d/d35/d50/d6c/da3/caf 0 2024-03-14T13:02:22.289 INFO:tasks.workunit.client.1.smithi137.stdout:8/679: write d1/f83 [367354,67832] 0 2024-03-14T13:02:22.290 INFO:tasks.workunit.client.1.smithi137.stdout:9/745: dwrite d6/df/d19/d33/f65 [0,4194304] 0 2024-03-14T13:02:22.293 INFO:tasks.workunit.client.1.smithi137.stdout:8/680: stat d1/dc/d3c/d87/d34/d97/d7f/fa0 0 2024-03-14T13:02:22.294 INFO:tasks.workunit.client.1.smithi137.stdout:9/746: fsync d6/df/d19/d3a/fcb 0 2024-03-14T13:02:22.295 INFO:tasks.workunit.client.1.smithi137.stdout:7/513: rename d3/d6/d14/l2d to d3/d5/d68/l9b 0 2024-03-14T13:02:22.303 INFO:tasks.workunit.client.1.smithi137.stdout:0/610: dwrite d1/d5/d1b/d42/f32 [0,4194304] 0 2024-03-14T13:02:22.304 INFO:tasks.workunit.client.1.smithi137.stdout:7/514: creat d3/d5/d68/f9c x:0 0 0 2024-03-14T13:02:22.304 INFO:tasks.workunit.client.1.smithi137.stdout:8/681: rename d1/dc/d3c/d87/d34/d97/d13 to d1/dc/d77/d9f/de4 0 2024-03-14T13:02:22.307 INFO:tasks.workunit.client.1.smithi137.stdout:9/747: symlink d6/df/d19/d3a/da9/dde/lef 0 2024-03-14T13:02:22.308 INFO:tasks.workunit.client.1.smithi137.stdout:0/611: symlink d1/d5/d1b/d42/lcc 0 2024-03-14T13:02:22.310 INFO:tasks.workunit.client.1.smithi137.stdout:0/612: symlink d1/d5/d1b/d1e/d25/lcd 0 2024-03-14T13:02:22.314 INFO:tasks.workunit.client.1.smithi137.stdout:0/613: creat d1/d5/d1b/d1e/d25/d9f/fce x:0 0 0 2024-03-14T13:02:22.314 INFO:tasks.workunit.client.1.smithi137.stdout:8/682: dread d1/dc/d71/fba [0,4194304] 0 2024-03-14T13:02:22.315 INFO:tasks.workunit.client.1.smithi137.stdout:0/614: dread - d1/d44/d72/d7e/f94 zero size 2024-03-14T13:02:22.317 INFO:tasks.workunit.client.0.smithi111.stdout:8/888: sync 2024-03-14T13:02:22.320 INFO:tasks.workunit.client.1.smithi137.stdout:0/615: rename d1/d5/d1b/d1e/d25/d55/d73/cc3 to d1/d5/d1b/d1e/d25/ccf 0 2024-03-14T13:02:22.320 INFO:tasks.workunit.client.1.smithi137.stdout:8/683: link d1/dc/d77/d9f/de4/f16 d1/dc/d77/d9f/fe5 0 2024-03-14T13:02:22.321 INFO:tasks.workunit.client.0.smithi111.stdout:9/642: write d5/f31 [4588811,51817] 0 2024-03-14T13:02:22.321 INFO:tasks.workunit.client.1.smithi137.stdout:0/616: truncate d1/f46 3008578 0 2024-03-14T13:02:22.325 INFO:tasks.workunit.client.1.smithi137.stdout:8/684: mkdir d1/dc/d3c/d87/d34/d82/de6 0 2024-03-14T13:02:22.326 INFO:tasks.workunit.client.1.smithi137.stdout:0/617: creat d1/d5/d1b/d1e/d25/d8a/d4a/fd0 x:0 0 0 2024-03-14T13:02:22.326 INFO:tasks.workunit.client.1.smithi137.stdout:9/748: dread d6/df/f20 [4194304,4194304] 0 2024-03-14T13:02:22.327 INFO:tasks.workunit.client.1.smithi137.stdout:8/685: creat d1/dc/d77/d9f/fe7 x:0 0 0 2024-03-14T13:02:22.328 INFO:tasks.workunit.client.0.smithi111.stdout:7/547: sync 2024-03-14T13:02:22.329 INFO:tasks.workunit.client.0.smithi111.stdout:6/592: sync 2024-03-14T13:02:22.329 INFO:tasks.workunit.client.0.smithi111.stdout:5/789: sync 2024-03-14T13:02:22.329 INFO:tasks.workunit.client.0.smithi111.stdout:0/794: sync 2024-03-14T13:02:22.329 INFO:tasks.workunit.client.0.smithi111.stdout:2/586: sync 2024-03-14T13:02:22.329 INFO:tasks.workunit.client.0.smithi111.stdout:7/548: write d0/d7e/f49 [1144107,70610] 0 2024-03-14T13:02:22.330 INFO:tasks.workunit.client.0.smithi111.stdout:5/790: stat d2/f53 0 2024-03-14T13:02:22.332 INFO:tasks.workunit.client.1.smithi137.stdout:0/618: mknod d1/d5/d1b/d1e/d25/d55/d73/cd1 0 2024-03-14T13:02:22.333 INFO:tasks.workunit.client.0.smithi111.stdout:1/693: write fa [914538,20552] 0 2024-03-14T13:02:22.333 INFO:tasks.workunit.client.1.smithi137.stdout:9/749: symlink d6/df/d19/d33/lf0 0 2024-03-14T13:02:22.334 INFO:tasks.workunit.client.0.smithi111.stdout:6/593: dread - d0/d17/d21/d69/d3d/f5c zero size 2024-03-14T13:02:22.334 INFO:tasks.workunit.client.0.smithi111.stdout:0/795: mknod d6/d9/de/d32/c11c 0 2024-03-14T13:02:22.335 INFO:tasks.workunit.client.1.smithi137.stdout:0/619: rmdir d1/d5/d1b/d1e/d25/d26 39 2024-03-14T13:02:22.338 INFO:tasks.workunit.client.0.smithi111.stdout:5/791: chown d2/d16/d49/d7c/d95/l75 1 1 2024-03-14T13:02:22.338 INFO:tasks.workunit.client.0.smithi111.stdout:3/648: write d0/d2d/d35/d50/f74 [1046356,108737] 0 2024-03-14T13:02:22.340 INFO:tasks.workunit.client.0.smithi111.stdout:0/796: write d6/d9/de/d3a/d46/d4a/f81 [1715336,46209] 0 2024-03-14T13:02:22.340 INFO:tasks.workunit.client.0.smithi111.stdout:6/594: write d0/d17/d20/d4a/d59/dc3/fd6 [821632,32918] 0 2024-03-14T13:02:22.341 INFO:tasks.workunit.client.0.smithi111.stdout:7/549: creat d0/d1/d5/d38/d5c/d6d/fa3 x:0 0 0 2024-03-14T13:02:22.342 INFO:tasks.workunit.client.0.smithi111.stdout:2/587: rename c8 to dd/d17/d34/d74/cba 0 2024-03-14T13:02:22.350 INFO:tasks.workunit.client.1.smithi137.stdout:3/578: truncate d4/de/f2a 345822 0 2024-03-14T13:02:22.353 INFO:tasks.workunit.client.0.smithi111.stdout:2/588: rename f1 to dd/d11/d8b/fbb 0 2024-03-14T13:02:22.353 INFO:tasks.workunit.client.0.smithi111.stdout:5/792: creat d2/d16/d49/d7c/d95/ffb x:0 0 0 2024-03-14T13:02:22.353 INFO:tasks.workunit.client.1.smithi137.stdout:5/585: truncate d7/d33/f61 311727 0 2024-03-14T13:02:22.354 INFO:tasks.workunit.client.1.smithi137.stdout:7/515: fsync d3/d5/d68/f9c 0 2024-03-14T13:02:22.358 INFO:tasks.workunit.client.0.smithi111.stdout:7/550: creat d0/d1/d21/fa4 x:0 0 0 2024-03-14T13:02:22.358 INFO:tasks.workunit.client.0.smithi111.stdout:6/595: symlink d0/d17/d20/d94/dbd/ldc 0 2024-03-14T13:02:22.358 INFO:tasks.workunit.client.1.smithi137.stdout:3/579: rename d4/f18 to d4/df/d82/fc3 0 2024-03-14T13:02:22.364 INFO:tasks.workunit.client.0.smithi111.stdout:2/589: link dd/c22 dd/d17/d34/d74/cbc 0 2024-03-14T13:02:22.367 INFO:tasks.workunit.client.0.smithi111.stdout:3/649: dwrite d0/d2d/d35/d50/d5e/fd4 [0,4194304] 0 2024-03-14T13:02:22.367 INFO:tasks.workunit.client.0.smithi111.stdout:0/797: dwrite d6/d9/d15/d49/fc2 [0,4194304] 0 2024-03-14T13:02:22.367 INFO:tasks.workunit.client.1.smithi137.stdout:5/586: write d7/d1d/f57 [1201035,74518] 0 2024-03-14T13:02:22.367 INFO:tasks.workunit.client.1.smithi137.stdout:4/642: dwrite d4/f3c [0,4194304] 0 2024-03-14T13:02:22.367 INFO:tasks.workunit.client.1.smithi137.stdout:7/516: truncate d3/d6/d2e/f8b 386089 0 2024-03-14T13:02:22.368 INFO:tasks.workunit.client.0.smithi111.stdout:9/643: dread d5/f31 [0,4194304] 0 2024-03-14T13:02:22.368 INFO:tasks.workunit.client.1.smithi137.stdout:9/750: dread d6/df/f70 [0,4194304] 0 2024-03-14T13:02:22.369 INFO:tasks.workunit.client.1.smithi137.stdout:8/686: dread d1/dc/f39 [0,4194304] 0 2024-03-14T13:02:22.370 INFO:tasks.workunit.client.1.smithi137.stdout:0/620: dread d1/d44/d56/f70 [0,4194304] 0 2024-03-14T13:02:22.370 INFO:tasks.workunit.client.0.smithi111.stdout:5/793: write d2/d16/f70 [1771636,47317] 0 2024-03-14T13:02:22.371 INFO:tasks.workunit.client.1.smithi137.stdout:7/517: write d3/d4a/f90 [203027,118618] 0 2024-03-14T13:02:22.371 INFO:tasks.workunit.client.1.smithi137.stdout:4/643: dread - d4/d9/d3a/d70/d80/db7/fcc zero size 2024-03-14T13:02:22.372 INFO:tasks.workunit.client.0.smithi111.stdout:6/596: symlink d0/d17/d20/d5a/d5e/ldd 0 2024-03-14T13:02:22.372 INFO:tasks.workunit.client.1.smithi137.stdout:3/580: creat d4/d19/da1/fc4 x:0 0 0 2024-03-14T13:02:22.373 INFO:tasks.workunit.client.0.smithi111.stdout:7/551: mknod d0/d1/d4a/d87/ca5 0 2024-03-14T13:02:22.373 INFO:tasks.workunit.client.0.smithi111.stdout:2/590: readlink dd/d17/d34/d4f/l77 0 2024-03-14T13:02:22.375 INFO:tasks.workunit.client.0.smithi111.stdout:1/694: dwrite d20/d21/d3f/fa3 [0,4194304] 0 2024-03-14T13:02:22.376 INFO:tasks.workunit.client.1.smithi137.stdout:2/661: dwrite d2/d7/d2e/d3e/d44/d6b/d22/f4c [0,4194304] 0 2024-03-14T13:02:22.381 INFO:tasks.workunit.client.0.smithi111.stdout:5/794: chown d2/d16/d49/d44/d4a/d9e/db7/dbb/fcf 18 1 2024-03-14T13:02:22.384 INFO:tasks.workunit.client.1.smithi137.stdout:5/587: mknod d7/d16/d18/d25/d28/d3c/d7a/cc6 0 2024-03-14T13:02:22.385 INFO:tasks.workunit.client.0.smithi111.stdout:6/597: fsync d0/d17/d21/f44 0 2024-03-14T13:02:22.390 INFO:tasks.workunit.client.0.smithi111.stdout:1/695: creat d20/dca/fe7 x:0 0 0 2024-03-14T13:02:22.391 INFO:tasks.workunit.client.0.smithi111.stdout:7/552: chown d0/d1/d5/d38/c83 74068 1 2024-03-14T13:02:22.391 INFO:tasks.workunit.client.0.smithi111.stdout:3/650: creat d0/d2d/d35/fef x:0 0 0 2024-03-14T13:02:22.392 INFO:tasks.workunit.client.0.smithi111.stdout:6/598: fdatasync d0/d17/d20/d4a/d68/f88 0 2024-03-14T13:02:22.392 INFO:tasks.workunit.client.0.smithi111.stdout:0/798: dwrite d6/d9/de/f6a [0,4194304] 0 2024-03-14T13:02:22.394 INFO:tasks.workunit.client.1.smithi137.stdout:9/751: write d6/df/d19/d33/d8f/fb2 [779707,31437] 0 2024-03-14T13:02:22.394 INFO:tasks.workunit.client.1.smithi137.stdout:2/662: write d2/f4 [795074,129072] 0 2024-03-14T13:02:22.394 INFO:tasks.workunit.client.1.smithi137.stdout:7/518: fsync d3/d6/d14/f1a 0 2024-03-14T13:02:22.395 INFO:tasks.workunit.client.0.smithi111.stdout:5/795: creat d2/d16/d49/d7c/d95/ffc x:0 0 0 2024-03-14T13:02:22.395 INFO:tasks.workunit.client.0.smithi111.stdout:3/651: write d0/d2d/f86 [3238454,76091] 0 2024-03-14T13:02:22.395 INFO:tasks.workunit.client.1.smithi137.stdout:4/644: dread d4/d9/d3a/d41/d38/d79/f94 [0,4194304] 0 2024-03-14T13:02:22.399 INFO:tasks.workunit.client.0.smithi111.stdout:7/553: fsync d0/d1/d21/d2a/f9b 0 2024-03-14T13:02:22.399 INFO:tasks.workunit.client.0.smithi111.stdout:2/591: truncate dd/d17/d34/d4f/f66 331464 0 2024-03-14T13:02:22.400 INFO:tasks.workunit.client.1.smithi137.stdout:4/645: dread - d4/d9/d3a/d41/fc9 zero size 2024-03-14T13:02:22.400 INFO:tasks.workunit.client.0.smithi111.stdout:1/696: mknod d20/d21/d3f/ce8 0 2024-03-14T13:02:22.403 INFO:tasks.workunit.client.0.smithi111.stdout:7/554: fdatasync d0/d7e/f19 0 2024-03-14T13:02:22.403 INFO:tasks.workunit.client.0.smithi111.stdout:3/652: stat d0/d30/c52 0 2024-03-14T13:02:22.405 INFO:tasks.workunit.client.1.smithi137.stdout:5/588: creat d7/d16/d18/d25/d28/d9e/fc7 x:0 0 0 2024-03-14T13:02:22.405 INFO:tasks.workunit.client.0.smithi111.stdout:5/796: unlink d2/d16/d49/d7c/d95/f28 0 2024-03-14T13:02:22.406 INFO:tasks.workunit.client.0.smithi111.stdout:0/799: rename d6/d9/de/d32/f5e to d6/d9/d15/d55/d89/f11d 0 2024-03-14T13:02:22.408 INFO:tasks.workunit.client.1.smithi137.stdout:7/519: link d3/d6/d14/f2f d3/d52/f9d 0 2024-03-14T13:02:22.408 INFO:tasks.workunit.client.1.smithi137.stdout:0/621: dwrite d1/d44/d72/fca [0,4194304] 0 2024-03-14T13:02:22.408 INFO:tasks.workunit.client.1.smithi137.stdout:9/752: write d6/df/f70 [3193363,23600] 0 2024-03-14T13:02:22.410 INFO:tasks.workunit.client.1.smithi137.stdout:1/680: sync 2024-03-14T13:02:22.410 INFO:tasks.workunit.client.0.smithi111.stdout:2/592: read dd/d11/f6d [1588415,23149] 0 2024-03-14T13:02:22.411 INFO:tasks.workunit.client.1.smithi137.stdout:2/663: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/fd9 x:0 0 0 2024-03-14T13:02:22.411 INFO:tasks.workunit.client.1.smithi137.stdout:8/687: getdents d1/dc/d4e 0 2024-03-14T13:02:22.411 INFO:tasks.workunit.client.0.smithi111.stdout:0/800: write d6/d9/d15/d55/ff6 [353958,48096] 0 2024-03-14T13:02:22.414 INFO:tasks.workunit.client.0.smithi111.stdout:1/697: rename d20/d25/dd0/dd2 to d20/d25/da8/de9 0 2024-03-14T13:02:22.415 INFO:tasks.workunit.client.1.smithi137.stdout:4/646: creat d4/d9/d8e/fd8 x:0 0 0 2024-03-14T13:02:22.415 INFO:tasks.workunit.client.1.smithi137.stdout:0/622: dread - d1/d5/d1b/d1e/d25/d8a/d4a/fd0 zero size 2024-03-14T13:02:22.415 INFO:tasks.workunit.client.1.smithi137.stdout:5/589: mkdir d7/d33/d65/dc2/dc8 0 2024-03-14T13:02:22.416 INFO:tasks.workunit.client.1.smithi137.stdout:8/688: truncate d1/dc/d3c/d87/d34/fc5 372183 0 2024-03-14T13:02:22.416 INFO:tasks.workunit.client.1.smithi137.stdout:1/681: stat f5 0 2024-03-14T13:02:22.417 INFO:tasks.workunit.client.0.smithi111.stdout:3/653: dwrite d0/d2d/d35/d4c/db0/fed [0,4194304] 0 2024-03-14T13:02:22.417 INFO:tasks.workunit.client.0.smithi111.stdout:5/797: dread d2/d16/d49/d7c/d95/fa3 [0,4194304] 0 2024-03-14T13:02:22.418 INFO:tasks.workunit.client.1.smithi137.stdout:9/753: unlink d6/df/d19/d33/d8f/fc2 0 2024-03-14T13:02:22.418 INFO:tasks.workunit.client.1.smithi137.stdout:2/664: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc 1275550616 1 2024-03-14T13:02:22.421 INFO:tasks.workunit.client.1.smithi137.stdout:4/647: mknod d4/d9/d3a/d41/d38/d79/dc2/cd9 0 2024-03-14T13:02:22.424 INFO:tasks.workunit.client.0.smithi111.stdout:3/654: mknod d0/d2d/d35/d69/de6/dd8/cf0 0 2024-03-14T13:02:22.425 INFO:tasks.workunit.client.1.smithi137.stdout:2/665: write d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/fb9 [4259938,63308] 0 2024-03-14T13:02:22.426 INFO:tasks.workunit.client.1.smithi137.stdout:0/623: link d1/f1c d1/d5/d1b/d42/d61/dae/fd2 0 2024-03-14T13:02:22.429 INFO:tasks.workunit.client.0.smithi111.stdout:5/798: creat d2/d16/d49/d7c/ffd x:0 0 0 2024-03-14T13:02:22.429 INFO:tasks.workunit.client.0.smithi111.stdout:0/801: dwrite d6/d3d/dba/fdd [0,4194304] 0 2024-03-14T13:02:22.431 INFO:tasks.workunit.client.1.smithi137.stdout:8/689: rename d1/dc/f39 to d1/dc/d3c/d87/d34/d97/d7f/fe8 0 2024-03-14T13:02:22.432 INFO:tasks.workunit.client.0.smithi111.stdout:3/655: mkdir d0/d30/df1 0 2024-03-14T13:02:22.433 INFO:tasks.workunit.client.1.smithi137.stdout:0/624: chown d1/d5/d1b/d42/l8d 1 1 2024-03-14T13:02:22.433 INFO:tasks.workunit.client.0.smithi111.stdout:9/644: write d5/fa [2938412,114695] 0 2024-03-14T13:02:22.434 INFO:tasks.workunit.client.1.smithi137.stdout:3/581: dwrite d4/de/d93/fb0 [0,4194304] 0 2024-03-14T13:02:22.434 INFO:tasks.workunit.client.1.smithi137.stdout:4/648: symlink d4/d27/d5c/d77/lda 0 2024-03-14T13:02:22.434 INFO:tasks.workunit.client.0.smithi111.stdout:3/656: stat d0/d2d/f86 0 2024-03-14T13:02:22.435 INFO:tasks.workunit.client.1.smithi137.stdout:5/590: rename d7/d16/d18/d25/d28/d9e/cc3 to d7/d16/d18/d48/d9d/cc9 0 2024-03-14T13:02:22.437 INFO:tasks.workunit.client.0.smithi111.stdout:9/645: chown d5/d6/d2a/d5a/f9f 5623 1 2024-03-14T13:02:22.437 INFO:tasks.workunit.client.1.smithi137.stdout:0/625: chown d1/d5/d1b/d1e/d25/d8a/la3 14 1 2024-03-14T13:02:22.437 INFO:tasks.workunit.client.0.smithi111.stdout:7/555: truncate d0/d7e/f3d 5405418 0 2024-03-14T13:02:22.438 INFO:tasks.workunit.client.1.smithi137.stdout:8/690: creat d1/d5e/dc9/fe9 x:0 0 0 2024-03-14T13:02:22.438 INFO:tasks.workunit.client.0.smithi111.stdout:9/646: dread - d5/d42/d71/fca zero size 2024-03-14T13:02:22.439 INFO:tasks.workunit.client.1.smithi137.stdout:0/626: stat d1/d5/d1b/d42/d61/dae 0 2024-03-14T13:02:22.439 INFO:tasks.workunit.client.1.smithi137.stdout:2/666: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dda 0 2024-03-14T13:02:22.440 INFO:tasks.workunit.client.1.smithi137.stdout:3/582: fdatasync d4/de/d16/d1c/d5d/d85/f42 0 2024-03-14T13:02:22.442 INFO:tasks.workunit.client.1.smithi137.stdout:0/627: symlink d1/d5/d1b/d42/d61/dae/ld3 0 2024-03-14T13:02:22.442 INFO:tasks.workunit.client.1.smithi137.stdout:8/691: chown d1/dc/d71/da3/ccb 2224 1 2024-03-14T13:02:22.443 INFO:tasks.workunit.client.0.smithi111.stdout:5/799: mkdir d2/d16/d49/d44/d72/de0/dfe 0 2024-03-14T13:02:22.444 INFO:tasks.workunit.client.0.smithi111.stdout:6/599: dread d0/d50/f5f [0,4194304] 0 2024-03-14T13:02:22.445 INFO:tasks.workunit.client.1.smithi137.stdout:4/649: unlink d4/d27/d5c/d6d/f72 0 2024-03-14T13:02:22.445 INFO:tasks.workunit.client.0.smithi111.stdout:0/802: getdents d6/d9/de/d11a 0 2024-03-14T13:02:22.445 INFO:tasks.workunit.client.0.smithi111.stdout:7/556: fsync d0/d1/d21/d3a/f5b 0 2024-03-14T13:02:22.449 INFO:tasks.workunit.client.1.smithi137.stdout:0/628: mknod d1/d44/d72/d7e/cd4 0 2024-03-14T13:02:22.450 INFO:tasks.workunit.client.0.smithi111.stdout:1/698: rename d20/d40/lbc to d20/d21/d28/da0/lea 0 2024-03-14T13:02:22.452 INFO:tasks.workunit.client.1.smithi137.stdout:2/667: symlink d2/d7/d2e/d3e/d44/d6b/d1d/d41/ldb 0 2024-03-14T13:02:22.452 INFO:tasks.workunit.client.0.smithi111.stdout:7/557: write d0/d11/f29 [2120176,48164] 0 2024-03-14T13:02:22.453 INFO:tasks.workunit.client.1.smithi137.stdout:5/591: dread d7/f96 [0,4194304] 0 2024-03-14T13:02:22.454 INFO:tasks.workunit.client.1.smithi137.stdout:4/650: symlink d4/d9/d3a/d41/d38/d55/db6/ldb 0 2024-03-14T13:02:22.457 INFO:tasks.workunit.client.0.smithi111.stdout:5/800: dwrite d2/d45/f20 [0,4194304] 0 2024-03-14T13:02:22.458 INFO:tasks.workunit.client.1.smithi137.stdout:5/592: fdatasync d7/d16/d18/d25/d28/f43 0 2024-03-14T13:02:22.458 INFO:tasks.workunit.client.1.smithi137.stdout:0/629: creat d1/d5/d1b/d42/d3a/db0/d84/fd5 x:0 0 0 2024-03-14T13:02:22.458 INFO:tasks.workunit.client.1.smithi137.stdout:7/520: truncate d3/d6/d2e/f49 4190855 0 2024-03-14T13:02:22.459 INFO:tasks.workunit.client.0.smithi111.stdout:6/600: dread d0/d17/d20/fb1 [0,4194304] 0 2024-03-14T13:02:22.460 INFO:tasks.workunit.client.1.smithi137.stdout:4/651: rmdir d4/d9/d3a/d41/d38/d55 39 2024-03-14T13:02:22.460 INFO:tasks.workunit.client.1.smithi137.stdout:1/682: write d7/d9/d19/d1f/d2e/f79 [293533,106599] 0 2024-03-14T13:02:22.461 INFO:tasks.workunit.client.0.smithi111.stdout:2/593: dread dd/d17/f63 [0,4194304] 0 2024-03-14T13:02:22.466 INFO:tasks.workunit.client.0.smithi111.stdout:9/647: creat d5/d6/d2a/fdf x:0 0 0 2024-03-14T13:02:22.466 INFO:tasks.workunit.client.1.smithi137.stdout:0/630: creat d1/d5/d1b/d1e/d25/d8a/fd6 x:0 0 0 2024-03-14T13:02:22.468 INFO:tasks.workunit.client.1.smithi137.stdout:7/521: dwrite d3/d5/d23/d3f/d5b/f79 [0,4194304] 0 2024-03-14T13:02:22.470 INFO:tasks.workunit.client.0.smithi111.stdout:2/594: read dd/d11/d1a/f1b [742484,119611] 0 2024-03-14T13:02:22.470 INFO:tasks.workunit.client.0.smithi111.stdout:7/558: dwrite d0/d1/d21/d2a/f7d [0,4194304] 0 2024-03-14T13:02:22.471 INFO:tasks.workunit.client.0.smithi111.stdout:1/699: mknod d20/d21/d28/ceb 0 2024-03-14T13:02:22.472 INFO:tasks.workunit.client.0.smithi111.stdout:5/801: unlink d2/d16/d49/d4c/ca1 0 2024-03-14T13:02:22.472 INFO:tasks.workunit.client.1.smithi137.stdout:7/522: fdatasync d3/d5/d23/d2c/d5d/f92 0 2024-03-14T13:02:22.474 INFO:tasks.workunit.client.0.smithi111.stdout:1/700: chown d20/d40/da6/lb2 1218 1 2024-03-14T13:02:22.475 INFO:tasks.workunit.client.1.smithi137.stdout:1/683: creat d7/d9/d19/d1f/d2e/fe7 x:0 0 0 2024-03-14T13:02:22.477 INFO:tasks.workunit.client.0.smithi111.stdout:3/657: rename d0/d30/f48 to d0/d2d/d35/ff2 0 2024-03-14T13:02:22.477 INFO:tasks.workunit.client.0.smithi111.stdout:9/648: dwrite d5/d6/d2a/fdf [0,4194304] 0 2024-03-14T13:02:22.478 INFO:tasks.workunit.client.0.smithi111.stdout:1/701: dread d20/f44 [0,4194304] 0 2024-03-14T13:02:22.478 INFO:tasks.workunit.client.1.smithi137.stdout:5/593: creat d7/d16/d18/d25/fca x:0 0 0 2024-03-14T13:02:22.478 INFO:tasks.workunit.client.1.smithi137.stdout:1/684: fsync d7/d9/d19/d40/fa8 0 2024-03-14T13:02:22.479 INFO:tasks.workunit.client.1.smithi137.stdout:0/631: creat d1/d5/d1b/d42/d3a/dc6/fd7 x:0 0 0 2024-03-14T13:02:22.480 INFO:tasks.workunit.client.0.smithi111.stdout:9/649: stat d5/d6/d8/d5b/d84/db6/cd6 0 2024-03-14T13:02:22.483 INFO:tasks.workunit.client.0.smithi111.stdout:2/595: unlink dd/d17/d34/d4f/f7e 0 2024-03-14T13:02:22.486 INFO:tasks.workunit.client.0.smithi111.stdout:5/802: creat d2/d45/fff x:0 0 0 2024-03-14T13:02:22.488 INFO:tasks.workunit.client.1.smithi137.stdout:3/583: dread d4/de/d16/d1c/d5d/d85/f42 [4194304,4194304] 0 2024-03-14T13:02:22.488 INFO:tasks.workunit.client.0.smithi111.stdout:0/803: getdents d6/d3d/dc1 0 2024-03-14T13:02:22.491 INFO:tasks.workunit.client.0.smithi111.stdout:7/559: mknod d0/d1/d4a/d52/ca6 0 2024-03-14T13:02:22.491 INFO:tasks.workunit.client.0.smithi111.stdout:6/601: link d0/d17/lab d0/d17/d20/d94/lde 0 2024-03-14T13:02:22.491 INFO:tasks.workunit.client.1.smithi137.stdout:1/685: chown d7/d4d/d6b/faa 1798 1 2024-03-14T13:02:22.493 INFO:tasks.workunit.client.1.smithi137.stdout:8/692: truncate d1/f6 4902097 0 2024-03-14T13:02:22.494 INFO:tasks.workunit.client.0.smithi111.stdout:3/658: rename d0/d2d/d35/d50/da2/db3/d6f/lc8 to d0/d2d/d35/d69/de6/dd8/dde/lf3 0 2024-03-14T13:02:22.495 INFO:tasks.workunit.client.1.smithi137.stdout:1/686: creat d7/d4d/d6b/dc5/fe8 x:0 0 0 2024-03-14T13:02:22.497 INFO:tasks.workunit.client.1.smithi137.stdout:3/584: mknod d4/de/d16/d1c/d5d/d85/cc5 0 2024-03-14T13:02:22.497 INFO:tasks.workunit.client.0.smithi111.stdout:7/560: unlink d0/d11/l35 0 2024-03-14T13:02:22.498 INFO:tasks.workunit.client.0.smithi111.stdout:1/702: dwrite d20/dca/fe7 [0,4194304] 0 2024-03-14T13:02:22.498 INFO:tasks.workunit.client.1.smithi137.stdout:1/687: chown d7/d9/d19/d1f/d2e/f9b 26236 1 2024-03-14T13:02:22.499 INFO:tasks.workunit.client.1.smithi137.stdout:2/668: truncate d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 7986492 0 2024-03-14T13:02:22.500 INFO:tasks.workunit.client.0.smithi111.stdout:2/596: dwrite dd/d17/f83 [4194304,4194304] 0 2024-03-14T13:02:22.500 INFO:tasks.workunit.client.0.smithi111.stdout:9/650: getdents d5/d42/d71/d96 0 2024-03-14T13:02:22.500 INFO:tasks.workunit.client.0.smithi111.stdout:6/602: mknod d0/d17/d21/d42/da5/cdf 0 2024-03-14T13:02:22.500 INFO:tasks.workunit.client.1.smithi137.stdout:1/688: fdatasync d7/d9/d19/d1f/d22/fa2 0 2024-03-14T13:02:22.501 INFO:tasks.workunit.client.1.smithi137.stdout:0/632: dwrite d1/d44/d56/f7d [0,4194304] 0 2024-03-14T13:02:22.502 INFO:tasks.workunit.client.1.smithi137.stdout:8/693: fsync d1/dc/d3c/d87/d34/d97/f2a 0 2024-03-14T13:02:22.503 INFO:tasks.workunit.client.0.smithi111.stdout:0/804: mknod d6/d9/d15/da9/c11e 0 2024-03-14T13:02:22.506 INFO:tasks.workunit.client.0.smithi111.stdout:1/703: read - d20/d40/f84 zero size 2024-03-14T13:02:22.507 INFO:tasks.workunit.client.0.smithi111.stdout:3/659: chown d0/d2d/d35/d50/da2/db3/da7/fe7 2 1 2024-03-14T13:02:22.508 INFO:tasks.workunit.client.0.smithi111.stdout:0/805: readlink d6/d9/de/d32/d64/l10f 0 2024-03-14T13:02:22.509 INFO:tasks.workunit.client.0.smithi111.stdout:5/803: rmdir d2/d16/d49/d44/d4a/d9e 39 2024-03-14T13:02:22.510 INFO:tasks.workunit.client.1.smithi137.stdout:8/694: readlink d1/dc/d3c/lb9 0 2024-03-14T13:02:22.510 INFO:tasks.workunit.client.0.smithi111.stdout:5/804: stat d2/d16/d49/d7c/d95/d36/cd0 0 2024-03-14T13:02:22.511 INFO:tasks.workunit.client.0.smithi111.stdout:6/603: creat d0/d17/d20/d65/d93/fe0 x:0 0 0 2024-03-14T13:02:22.515 INFO:tasks.workunit.client.1.smithi137.stdout:2/669: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/lc9 73870781 1 2024-03-14T13:02:22.515 INFO:tasks.workunit.client.1.smithi137.stdout:8/695: readlink d1/dc/d77/d9f/de4/l69 0 2024-03-14T13:02:22.515 INFO:tasks.workunit.client.1.smithi137.stdout:4/652: dread d4/d10/d26/f6f [4194304,4194304] 0 2024-03-14T13:02:22.515 INFO:tasks.workunit.client.1.smithi137.stdout:8/696: readlink d1/dc/l6c 0 2024-03-14T13:02:22.516 INFO:tasks.workunit.client.0.smithi111.stdout:8/889: sync 2024-03-14T13:02:22.520 INFO:tasks.workunit.client.0.smithi111.stdout:6/604: fdatasync d0/d17/d21/d42/f77 0 2024-03-14T13:02:22.521 INFO:tasks.workunit.client.1.smithi137.stdout:7/523: write d3/d52/f9d [4083187,59585] 0 2024-03-14T13:02:22.521 INFO:tasks.workunit.client.1.smithi137.stdout:3/585: dwrite d4/d22/fbd [0,4194304] 0 2024-03-14T13:02:22.521 INFO:tasks.workunit.client.1.smithi137.stdout:0/633: rename d1/d5/d1b/d1e/d53/c90 to d1/d5/d1b/d42/d3a/cd8 0 2024-03-14T13:02:22.523 INFO:tasks.workunit.client.1.smithi137.stdout:2/670: truncate d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f8a 655085 0 2024-03-14T13:02:22.524 INFO:tasks.workunit.client.0.smithi111.stdout:9/651: unlink d5/d6/d8/d5b/d84/l8f 0 2024-03-14T13:02:22.524 INFO:tasks.workunit.client.1.smithi137.stdout:0/634: dread - d1/d5/d1b/d1e/d25/d55/fa0 zero size 2024-03-14T13:02:22.525 INFO:tasks.workunit.client.1.smithi137.stdout:7/524: mkdir d3/d6/d64/d9e 0 2024-03-14T13:02:22.527 INFO:tasks.workunit.client.1.smithi137.stdout:1/689: rename d7/d4d/d6b/f9e to d7/d9/d19/d1f/d89/d90/fe9 0 2024-03-14T13:02:22.527 INFO:tasks.workunit.client.0.smithi111.stdout:3/660: fsync d0/d2d/d35/d50/d5e/d19/f1a 0 2024-03-14T13:02:22.527 INFO:tasks.workunit.client.0.smithi111.stdout:0/806: rename d6/d21/l2a to d6/d9/de/d11a/l11f 0 2024-03-14T13:02:22.529 INFO:tasks.workunit.client.1.smithi137.stdout:2/671: dread - d2/d7/d2e/d3e/d44/d6b/d1d/d2d/f61 zero size 2024-03-14T13:02:22.530 INFO:tasks.workunit.client.1.smithi137.stdout:0/635: creat d1/d5/d1b/d1e/d25/d8a/d4a/fd9 x:0 0 0 2024-03-14T13:02:22.530 INFO:tasks.workunit.client.1.smithi137.stdout:7/525: symlink d3/l9f 0 2024-03-14T13:02:22.530 INFO:tasks.workunit.client.0.smithi111.stdout:7/561: creat d0/d1/d4a/fa7 x:0 0 0 2024-03-14T13:02:22.532 INFO:tasks.workunit.client.1.smithi137.stdout:8/697: getdents d1/d5e/dc9/dda 0 2024-03-14T13:02:22.532 INFO:tasks.workunit.client.1.smithi137.stdout:2/672: chown d2/d7/d2e/d3e/d44/d6b/f4a 7185 1 2024-03-14T13:02:22.532 INFO:tasks.workunit.client.0.smithi111.stdout:2/597: dwrite dd/d17/f83 [8388608,4194304] 0 2024-03-14T13:02:22.533 INFO:tasks.workunit.client.1.smithi137.stdout:7/526: fsync d3/d5/d5a/f73 0 2024-03-14T13:02:22.533 INFO:tasks.workunit.client.0.smithi111.stdout:8/890: mknod d3/d1b/d20/d35/d40/d48/d77/c127 0 2024-03-14T13:02:22.536 INFO:tasks.workunit.client.1.smithi137.stdout:1/690: rename d7/d9/d19/d1f/d26/l93 to d7/d9/d19/d1f/d89/lea 0 2024-03-14T13:02:22.537 INFO:tasks.workunit.client.1.smithi137.stdout:2/673: fsync d2/d7/d2e/f3d 0 2024-03-14T13:02:22.538 INFO:tasks.workunit.client.0.smithi111.stdout:1/704: dwrite d20/d21/d28/da0/fcf [0,4194304] 0 2024-03-14T13:02:22.539 INFO:tasks.workunit.client.1.smithi137.stdout:8/698: mknod d1/d5e/cea 0 2024-03-14T13:02:22.541 INFO:tasks.workunit.client.0.smithi111.stdout:5/805: truncate d2/d13/f2f 172096 0 2024-03-14T13:02:22.542 INFO:tasks.workunit.client.1.smithi137.stdout:2/674: fdatasync d2/d7/d2e/f3d 0 2024-03-14T13:02:22.544 INFO:tasks.workunit.client.0.smithi111.stdout:6/605: unlink d0/d17/d20/d5a/fa2 0 2024-03-14T13:02:22.546 INFO:tasks.workunit.client.0.smithi111.stdout:9/652: mkdir d5/d42/d71/dbd/de0 0 2024-03-14T13:02:22.546 INFO:tasks.workunit.client.0.smithi111.stdout:3/661: unlink d0/d2d/d35/d50/ddf/ce4 0 2024-03-14T13:02:22.548 INFO:tasks.workunit.client.1.smithi137.stdout:8/699: dwrite d1/dc/d3c/d87/f6e [0,4194304] 0 2024-03-14T13:02:22.548 INFO:tasks.workunit.client.1.smithi137.stdout:4/653: dread d4/f48 [0,4194304] 0 2024-03-14T13:02:22.549 INFO:tasks.workunit.client.1.smithi137.stdout:8/700: chown d1/dc/l25 219559 1 2024-03-14T13:02:22.550 INFO:tasks.workunit.client.0.smithi111.stdout:8/891: truncate d3/d1b/d20/d35/d40/d48/d7e/fca 779480 0 2024-03-14T13:02:22.553 INFO:tasks.workunit.client.0.smithi111.stdout:8/892: read d3/f79 [156126,58659] 0 2024-03-14T13:02:22.554 INFO:tasks.workunit.client.0.smithi111.stdout:5/806: mknod d2/d16/d49/d33/d68/c100 0 2024-03-14T13:02:22.555 INFO:tasks.workunit.client.0.smithi111.stdout:8/893: chown d3/d1b 11909 1 2024-03-14T13:02:22.556 INFO:tasks.workunit.client.0.smithi111.stdout:1/705: rename d20/d21/d3f/cc7 to d20/d25/da8/ddd/cec 0 2024-03-14T13:02:22.556 INFO:tasks.workunit.client.0.smithi111.stdout:6/606: dread d0/f74 [0,4194304] 0 2024-03-14T13:02:22.556 INFO:tasks.workunit.client.1.smithi137.stdout:2/675: symlink d2/d7/d2e/d3e/d44/d6b/d22/da5/ldc 0 2024-03-14T13:02:22.556 INFO:tasks.workunit.client.0.smithi111.stdout:0/807: getdents d6/d9/de/d77/da4/dca/dd0/de8 0 2024-03-14T13:02:22.557 INFO:tasks.workunit.client.0.smithi111.stdout:2/598: dwrite dd/d17/d34/d74/f9e [0,4194304] 0 2024-03-14T13:02:22.557 INFO:tasks.workunit.client.1.smithi137.stdout:7/527: getdents d3/d5/d5a 0 2024-03-14T13:02:22.559 INFO:tasks.workunit.client.1.smithi137.stdout:8/701: unlink d1/dc/d77/f78 0 2024-03-14T13:02:22.559 INFO:tasks.workunit.client.1.smithi137.stdout:0/636: write d1/d5/d1b/d42/fa1 [578401,49699] 0 2024-03-14T13:02:22.559 INFO:tasks.workunit.client.1.smithi137.stdout:1/691: link d7/d4d/d6b/dc5/l6e d7/d4d/d6b/dc5/leb 0 2024-03-14T13:02:22.562 INFO:tasks.workunit.client.0.smithi111.stdout:9/653: creat d5/d6/fe1 x:0 0 0 2024-03-14T13:02:22.563 INFO:tasks.workunit.client.0.smithi111.stdout:0/808: rmdir d6/d9/d5b 39 2024-03-14T13:02:22.563 INFO:tasks.workunit.client.1.smithi137.stdout:2/676: readlink d2/d7/d2e/d3e/d44/d6b/d1d/ld2 0 2024-03-14T13:02:22.564 INFO:tasks.workunit.client.0.smithi111.stdout:6/607: chown d0/d17/d21/d69/d3d/d53/d81/da1/dd7 2972 1 2024-03-14T13:02:22.564 INFO:tasks.workunit.client.0.smithi111.stdout:8/894: write d3/d1b/d20/d35/d40/d48/d7e/db3/db5/fbe [30909,61376] 0 2024-03-14T13:02:22.564 INFO:tasks.workunit.client.1.smithi137.stdout:2/677: write d2/d7/f87 [82386,86934] 0 2024-03-14T13:02:22.565 INFO:tasks.workunit.client.1.smithi137.stdout:7/528: symlink d3/d5/d23/d2c/la0 0 2024-03-14T13:02:22.567 INFO:tasks.workunit.client.1.smithi137.stdout:8/702: fdatasync d1/dc/f1f 0 2024-03-14T13:02:22.569 INFO:tasks.workunit.client.0.smithi111.stdout:0/809: fdatasync d6/d9/d15/d55/f86 0 2024-03-14T13:02:22.570 INFO:tasks.workunit.client.1.smithi137.stdout:1/692: unlink d7/d9/d19/d40/fc4 0 2024-03-14T13:02:22.571 INFO:tasks.workunit.client.0.smithi111.stdout:1/706: link d20/d25/d31/d80/cd8 d20/d25/d87/ced 0 2024-03-14T13:02:22.572 INFO:tasks.workunit.client.1.smithi137.stdout:0/637: mknod d1/da6/cda 0 2024-03-14T13:02:22.573 INFO:tasks.workunit.client.1.smithi137.stdout:0/638: dread - d1/d5/d1b/d1e/d25/d9f/fce zero size 2024-03-14T13:02:22.576 INFO:tasks.workunit.client.0.smithi111.stdout:9/654: mknod d5/d42/dd4/ce2 0 2024-03-14T13:02:22.577 INFO:tasks.workunit.client.1.smithi137.stdout:8/703: unlink d1/dc/l5b 0 2024-03-14T13:02:22.577 INFO:tasks.workunit.client.1.smithi137.stdout:0/639: mknod d1/d5/d1b/d1e/d53/cdb 0 2024-03-14T13:02:22.578 INFO:tasks.workunit.client.0.smithi111.stdout:8/895: chown d3/ca 2054796 1 2024-03-14T13:02:22.578 INFO:tasks.workunit.client.0.smithi111.stdout:3/662: dread d0/d2d/d35/d50/d5e/d19/f5a [4194304,4194304] 0 2024-03-14T13:02:22.578 INFO:tasks.workunit.client.1.smithi137.stdout:2/678: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f77 997638 0 2024-03-14T13:02:22.578 INFO:tasks.workunit.client.1.smithi137.stdout:0/640: readlink d1/d5/d1b/d42/d3a/db0/lc9 0 2024-03-14T13:02:22.580 INFO:tasks.workunit.client.1.smithi137.stdout:7/529: dread f1 [0,4194304] 0 2024-03-14T13:02:22.581 INFO:tasks.workunit.client.1.smithi137.stdout:8/704: mknod d1/dc/d3c/d87/d34/d82/ceb 0 2024-03-14T13:02:22.581 INFO:tasks.workunit.client.1.smithi137.stdout:7/530: readlink d3/d5/d23/l70 0 2024-03-14T13:02:22.581 INFO:tasks.workunit.client.1.smithi137.stdout:2/679: dread d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f83 [0,4194304] 0 2024-03-14T13:02:22.582 INFO:tasks.workunit.client.0.smithi111.stdout:7/562: dwrite d0/d1/d5/d38/f94 [0,4194304] 0 2024-03-14T13:02:22.583 INFO:tasks.workunit.client.0.smithi111.stdout:5/807: dwrite d2/d13/fb9 [0,4194304] 0 2024-03-14T13:02:22.583 INFO:tasks.workunit.client.1.smithi137.stdout:0/641: write d1/d5/d1b/d1e/d25/d26/fab [1033600,17696] 0 2024-03-14T13:02:22.585 INFO:tasks.workunit.client.1.smithi137.stdout:2/680: write d2/d34/d3c/d9d/fa1 [886079,116815] 0 2024-03-14T13:02:22.586 INFO:tasks.workunit.client.1.smithi137.stdout:7/531: mkdir d3/d5/d23/d2c/d5d/d6a/da1 0 2024-03-14T13:02:22.587 INFO:tasks.workunit.client.0.smithi111.stdout:6/608: creat d0/d17/d20/d65/d93/dd9/fe1 x:0 0 0 2024-03-14T13:02:22.590 INFO:tasks.workunit.client.0.smithi111.stdout:1/707: symlink d20/d25/lee 0 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.1.smithi137.stdout:6/639: sync 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.1.smithi137.stdout:7/532: symlink d3/d5/d23/d2c/d5d/la2 0 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.0.smithi111.stdout:7/563: fdatasync d0/d7e/f49 0 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.0.smithi111.stdout:3/663: mkdir d0/d2d/d35/d69/de6/df4 0 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.0.smithi111.stdout:2/599: dwrite dd/d17/d34/d73/d49/f90 [4194304,4194304] 0 2024-03-14T13:02:22.591 INFO:tasks.workunit.client.0.smithi111.stdout:9/655: fsync d5/d6/d2a/f3e 0 2024-03-14T13:02:22.592 INFO:tasks.workunit.client.1.smithi137.stdout:7/533: write d3/d6/d59/f5f [3878001,62712] 0 2024-03-14T13:02:22.593 INFO:tasks.workunit.client.1.smithi137.stdout:2/681: creat d2/d7/d2e/d3e/d44/d6b/d1d/fdd x:0 0 0 2024-03-14T13:02:22.593 INFO:tasks.workunit.client.0.smithi111.stdout:8/896: symlink d3/d1b/d20/d35/d40/d48/d7e/l128 0 2024-03-14T13:02:22.595 INFO:tasks.workunit.client.0.smithi111.stdout:0/810: dread d6/d21/f73 [4194304,4194304] 0 2024-03-14T13:02:22.596 INFO:tasks.workunit.client.0.smithi111.stdout:1/708: write d20/d21/d3f/d59/f76 [1623168,40161] 0 2024-03-14T13:02:22.597 INFO:tasks.workunit.client.0.smithi111.stdout:7/564: readlink d0/d1/d5/d38/d5c/d6d/l78 0 2024-03-14T13:02:22.600 INFO:tasks.workunit.client.0.smithi111.stdout:6/609: creat d0/d17/d21/fe2 x:0 0 0 2024-03-14T13:02:22.601 INFO:tasks.workunit.client.0.smithi111.stdout:3/664: mknod d0/d2d/d35/d50/da2/db3/d71/dbc/cf5 0 2024-03-14T13:02:22.601 INFO:tasks.workunit.client.1.smithi137.stdout:7/534: creat d3/d5/d23/d3f/d5b/fa3 x:0 0 0 2024-03-14T13:02:22.602 INFO:tasks.workunit.client.1.smithi137.stdout:0/642: getdents d1/d5/d1b/d42/d3a 0 2024-03-14T13:02:22.602 INFO:tasks.workunit.client.0.smithi111.stdout:1/709: write d20/f58 [2144493,91577] 0 2024-03-14T13:02:22.604 INFO:tasks.workunit.client.1.smithi137.stdout:6/640: creat d2/dd6/fdb x:0 0 0 2024-03-14T13:02:22.604 INFO:tasks.workunit.client.0.smithi111.stdout:5/808: rename d2/d45/d76/lc4 to d2/d16/d49/d4c/d9c/l101 0 2024-03-14T13:02:22.604 INFO:tasks.workunit.client.1.smithi137.stdout:2/682: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f5f [2077673,112492] 0 2024-03-14T13:02:22.605 INFO:tasks.workunit.client.0.smithi111.stdout:8/897: dread d3/dc/fa7 [0,4194304] 0 2024-03-14T13:02:22.605 INFO:tasks.workunit.client.1.smithi137.stdout:1/693: dwrite d7/d9/d19/d1f/d2e/f3d [0,4194304] 0 2024-03-14T13:02:22.607 INFO:tasks.workunit.client.0.smithi111.stdout:7/565: mkdir d0/d11/da8 0 2024-03-14T13:02:22.607 INFO:tasks.workunit.client.0.smithi111.stdout:7/566: stat d0/l2f 0 2024-03-14T13:02:22.608 INFO:tasks.workunit.client.1.smithi137.stdout:0/643: mknod d1/d5/d1b/cdc 0 2024-03-14T13:02:22.611 INFO:tasks.workunit.client.1.smithi137.stdout:6/641: dread d2/d54/d55/f8d [0,4194304] 0 2024-03-14T13:02:22.611 INFO:tasks.workunit.client.0.smithi111.stdout:5/809: symlink d2/d45/d76/d9a/dbd/l102 0 2024-03-14T13:02:22.612 INFO:tasks.workunit.client.1.smithi137.stdout:0/644: mknod d1/d5/d1b/d42/d3a/dc6/cdd 0 2024-03-14T13:02:22.613 INFO:tasks.workunit.client.1.smithi137.stdout:2/683: unlink d2/d34/d3c/c9a 0 2024-03-14T13:02:22.614 INFO:tasks.workunit.client.0.smithi111.stdout:6/610: mkdir d0/d17/d21/d69/d3d/dca/de3 0 2024-03-14T13:02:22.614 INFO:tasks.workunit.client.1.smithi137.stdout:0/645: dread - d1/d1f/d82/fb9 zero size 2024-03-14T13:02:22.614 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:22.615 INFO:tasks.workunit.client.1.smithi137.stdout:0/646: dread - d1/d5/d1b/d1e/d53/d6d/faa zero size 2024-03-14T13:02:22.615 INFO:tasks.workunit.client.0.smithi111.stdout:3/665: creat d0/d2d/d35/d4c/dad/dee/ff6 x:0 0 0 2024-03-14T13:02:22.616 INFO:tasks.workunit.client.1.smithi137.stdout:0/647: fdatasync d1/d44/f65 0 2024-03-14T13:02:22.616 INFO:tasks.workunit.client.0.smithi111.stdout:8/898: symlink d3/dc/d31/d9d/d124/l129 0 2024-03-14T13:02:22.618 INFO:tasks.workunit.client.1.smithi137.stdout:6/642: creat d2/d54/d55/d6d/dc3/dce/fdc x:0 0 0 2024-03-14T13:02:22.619 INFO:tasks.workunit.client.1.smithi137.stdout:1/694: link d7/l38 d7/d4d/d6b/dc5/lec 0 2024-03-14T13:02:22.619 INFO:tasks.workunit.client.0.smithi111.stdout:7/567: creat d0/d1/d5/d38/d1b/d9c/fa9 x:0 0 0 2024-03-14T13:02:22.620 INFO:tasks.workunit.client.0.smithi111.stdout:9/656: link d5/d6/d2a/d5a/cd7 d5/d6/d8/d74/ce3 0 2024-03-14T13:02:22.620 INFO:tasks.workunit.client.1.smithi137.stdout:0/648: truncate d1/d5/d1b/d42/f58 1580896 0 2024-03-14T13:02:22.620 INFO:tasks.workunit.client.0.smithi111.stdout:5/810: fdatasync d2/d16/d49/d7c/d95/fa3 0 2024-03-14T13:02:22.622 INFO:tasks.workunit.client.1.smithi137.stdout:1/695: mkdir d7/d31/d54/ded 0 2024-03-14T13:02:22.622 INFO:tasks.workunit.client.1.smithi137.stdout:8/705: dread d1/dc/d3c/d87/d34/d97/d7f/f8a [0,4194304] 0 2024-03-14T13:02:22.623 INFO:tasks.workunit.client.1.smithi137.stdout:2/684: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/cde 0 2024-03-14T13:02:22.623 INFO:tasks.workunit.client.0.smithi111.stdout:9/657: fsync d5/d42/d71/d96/fde 0 2024-03-14T13:02:22.623 INFO:tasks.workunit.client.0.smithi111.stdout:8/899: creat d3/d1b/d20/d35/d40/f12a x:0 0 0 2024-03-14T13:02:22.624 INFO:tasks.workunit.client.0.smithi111.stdout:0/811: getdents d6/d9/de/d3a/d46/d4a 0 2024-03-14T13:02:22.627 INFO:tasks.workunit.client.1.smithi137.stdout:2/685: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/ddf 0 2024-03-14T13:02:22.627 INFO:tasks.workunit.client.0.smithi111.stdout:7/568: rmdir d0/d1/d21/d5d 39 2024-03-14T13:02:22.628 INFO:tasks.workunit.client.0.smithi111.stdout:3/666: link d0/d2d/d35/d50/d5e/fd4 d0/d2d/d35/d69/d84/ff7 0 2024-03-14T13:02:22.628 INFO:tasks.workunit.client.0.smithi111.stdout:1/710: dread d20/d25/f2e [0,4194304] 0 2024-03-14T13:02:22.629 INFO:tasks.workunit.client.0.smithi111.stdout:5/811: creat d2/d16/d49/d7c/d95/d6a/f103 x:0 0 0 2024-03-14T13:02:22.629 INFO:tasks.workunit.client.0.smithi111.stdout:3/667: chown d0/d2d/d35/d50/f88 60125 1 2024-03-14T13:02:22.630 INFO:tasks.workunit.client.0.smithi111.stdout:9/658: creat d5/d6/d2a/d5a/fe4 x:0 0 0 2024-03-14T13:02:22.631 INFO:tasks.workunit.client.0.smithi111.stdout:0/812: mkdir d6/d5a/d120 0 2024-03-14T13:02:22.634 INFO:tasks.workunit.client.1.smithi137.stdout:2/686: read d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f77 [42308,8135] 0 2024-03-14T13:02:22.634 INFO:tasks.workunit.client.0.smithi111.stdout:9/659: write d5/d6/faa [384226,28744] 0 2024-03-14T13:02:22.637 INFO:tasks.workunit.client.0.smithi111.stdout:0/813: fsync d6/d9/de/d3a/d46/d4a/fb4 0 2024-03-14T13:02:22.638 INFO:tasks.workunit.client.0.smithi111.stdout:2/600: dread dd/d17/d34/d74/fb6 [0,4194304] 0 2024-03-14T13:02:22.638 INFO:tasks.workunit.client.0.smithi111.stdout:7/569: getdents d0/d1/d4a/d75 0 2024-03-14T13:02:22.639 INFO:tasks.workunit.client.0.smithi111.stdout:7/570: readlink d0/d7e/l66 0 2024-03-14T13:02:22.640 INFO:tasks.workunit.client.1.smithi137.stdout:6/643: dread d2/d6/de/f68 [0,4194304] 0 2024-03-14T13:02:22.640 INFO:tasks.workunit.client.0.smithi111.stdout:7/571: fsync d0/d1/d21/fa4 0 2024-03-14T13:02:22.641 INFO:tasks.workunit.client.1.smithi137.stdout:2/687: rename d2/d34/l49 to d2/d7/d2e/d3e/d44/d6b/d1d/d41/le0 0 2024-03-14T13:02:22.642 INFO:tasks.workunit.client.0.smithi111.stdout:1/711: rename d20/d21/d28/ceb to d20/d40/db4/ddf/cef 0 2024-03-14T13:02:22.642 INFO:tasks.workunit.client.0.smithi111.stdout:8/900: link d3/d1b/d34/ce2 d3/d22/c12b 0 2024-03-14T13:02:22.643 INFO:tasks.workunit.client.1.smithi137.stdout:2/688: read d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 [845851,73799] 0 2024-03-14T13:02:22.643 INFO:tasks.workunit.client.1.smithi137.stdout:6/644: chown d2/d4c/c94 214 1 2024-03-14T13:02:22.643 INFO:tasks.workunit.client.1.smithi137.stdout:7/535: truncate d3/d5/d23/d2c/d5d/f6b 3862196 0 2024-03-14T13:02:22.644 INFO:tasks.workunit.client.0.smithi111.stdout:8/901: truncate d3/d22/da1/f11b 31466 0 2024-03-14T13:02:22.648 INFO:tasks.workunit.client.0.smithi111.stdout:5/812: creat d2/d16/d49/d44/d4a/d9e/db7/f104 x:0 0 0 2024-03-14T13:02:22.648 INFO:tasks.workunit.client.0.smithi111.stdout:8/902: dread d3/d22/d30/fa4 [0,4194304] 0 2024-03-14T13:02:22.649 INFO:tasks.workunit.client.0.smithi111.stdout:5/813: chown d2/d16/d49/d33/fa6 338 1 2024-03-14T13:02:22.650 INFO:tasks.workunit.client.1.smithi137.stdout:0/649: write d1/d5/d1b/d42/d3a/db0/d84/fa4 [725437,121428] 0 2024-03-14T13:02:22.652 INFO:tasks.workunit.client.0.smithi111.stdout:9/660: mknod d5/d6/d8/ce5 0 2024-03-14T13:02:22.653 INFO:tasks.workunit.client.0.smithi111.stdout:0/814: symlink d6/d9/de/d32/d64/l121 0 2024-03-14T13:02:22.653 INFO:tasks.workunit.client.0.smithi111.stdout:2/601: creat dd/d8d/db7/fbd x:0 0 0 2024-03-14T13:02:22.653 INFO:tasks.workunit.client.1.smithi137.stdout:7/536: mkdir d3/d5/d23/d2c/d5d/d6a/d75/d78/da4 0 2024-03-14T13:02:22.656 INFO:tasks.workunit.client.0.smithi111.stdout:1/712: symlink d20/d40/lf0 0 2024-03-14T13:02:22.656 INFO:tasks.workunit.client.1.smithi137.stdout:0/650: unlink d1/d5/d1b/d1e/d25/d8a/d4a/fd0 0 2024-03-14T13:02:22.658 INFO:tasks.workunit.client.1.smithi137.stdout:1/696: dwrite f5 [0,4194304] 0 2024-03-14T13:02:22.660 INFO:tasks.workunit.client.0.smithi111.stdout:6/611: write d0/d17/d20/d4a/d68/f88 [140014,80416] 0 2024-03-14T13:02:22.661 INFO:tasks.workunit.client.0.smithi111.stdout:5/814: dwrite d2/f35 [0,4194304] 0 2024-03-14T13:02:22.661 INFO:tasks.workunit.client.1.smithi137.stdout:6/645: write d2/d6/de/f13 [1562418,9552] 0 2024-03-14T13:02:22.661 INFO:tasks.workunit.client.1.smithi137.stdout:2/689: write d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 [4953152,62123] 0 2024-03-14T13:02:22.662 INFO:tasks.workunit.client.1.smithi137.stdout:7/537: symlink d3/d5/d23/la5 0 2024-03-14T13:02:22.664 INFO:tasks.workunit.client.0.smithi111.stdout:3/668: rmdir d0/d2d/d35/d69/d84 39 2024-03-14T13:02:22.664 INFO:tasks.workunit.client.1.smithi137.stdout:8/706: dwrite d1/dc/d3c/d87/d34/f68 [4194304,4194304] 0 2024-03-14T13:02:22.668 INFO:tasks.workunit.client.1.smithi137.stdout:1/697: unlink d7/d9/d19/d40/d50/fd7 0 2024-03-14T13:02:22.670 INFO:tasks.workunit.client.1.smithi137.stdout:6/646: unlink d2/d6/cc5 0 2024-03-14T13:02:22.673 INFO:tasks.workunit.client.1.smithi137.stdout:8/707: mknod d1/dc/d3c/d87/d34/d86/cec 0 2024-03-14T13:02:22.673 INFO:tasks.workunit.client.1.smithi137.stdout:0/651: dread d1/d44/d72/fca [0,4194304] 0 2024-03-14T13:02:22.676 INFO:tasks.workunit.client.1.smithi137.stdout:6/647: fsync d2/f17 0 2024-03-14T13:02:22.677 INFO:tasks.workunit.client.0.smithi111.stdout:8/903: write d3/dc/d45/f7c [558118,87414] 0 2024-03-14T13:02:22.677 INFO:tasks.workunit.client.0.smithi111.stdout:5/815: dwrite d2/d13/f99 [0,4194304] 0 2024-03-14T13:02:22.677 INFO:tasks.workunit.client.1.smithi137.stdout:1/698: unlink d7/d4d/la9 0 2024-03-14T13:02:22.677 INFO:tasks.workunit.client.1.smithi137.stdout:8/708: rmdir d1/dc/d10 39 2024-03-14T13:02:22.680 INFO:tasks.workunit.client.0.smithi111.stdout:7/572: getdents d0/d1/d5/d38/d5c 0 2024-03-14T13:02:22.681 INFO:tasks.workunit.client.1.smithi137.stdout:6/648: creat d2/d6/d1d/fdd x:0 0 0 2024-03-14T13:02:22.684 INFO:tasks.workunit.client.1.smithi137.stdout:8/709: write d1/dc/d3c/d87/d34/d82/d94/fdb [392543,120040] 0 2024-03-14T13:02:22.684 INFO:tasks.workunit.client.1.smithi137.stdout:0/652: rename d1/d5/d1b/d1e/d25/d55/c88 to d1/d5/d1b/d42/d3a/db0/cde 0 2024-03-14T13:02:22.686 INFO:tasks.workunit.client.1.smithi137.stdout:6/649: symlink d2/d54/d55/d6d/d76/d57/lde 0 2024-03-14T13:02:22.688 INFO:tasks.workunit.client.1.smithi137.stdout:6/650: write d2/d6/fd9 [322002,2791] 0 2024-03-14T13:02:22.693 INFO:tasks.workunit.client.1.smithi137.stdout:0/653: creat d1/d5/d1b/d1e/d53/d6d/fdf x:0 0 0 2024-03-14T13:02:22.693 INFO:tasks.workunit.client.1.smithi137.stdout:7/538: dread d3/d5/f37 [0,4194304] 0 2024-03-14T13:02:22.696 INFO:tasks.workunit.client.0.smithi111.stdout:6/612: dwrite d0/d17/d20/d65/d93/fe0 [0,4194304] 0 2024-03-14T13:02:22.696 INFO:tasks.workunit.client.1.smithi137.stdout:1/699: link d7/d9/f14 d7/d9/d19/d1f/d26/d6c/dc8/fee 0 2024-03-14T13:02:22.697 INFO:tasks.workunit.client.1.smithi137.stdout:0/654: rmdir d1/d5/d1b/d42/d3a 39 2024-03-14T13:02:22.697 INFO:tasks.workunit.client.1.smithi137.stdout:7/539: readlink d3/d6/lc 0 2024-03-14T13:02:22.697 INFO:tasks.workunit.client.1.smithi137.stdout:6/651: chown d2/d54/d55/d9c/dc0/ld0 47919328 1 2024-03-14T13:02:22.698 INFO:tasks.workunit.client.1.smithi137.stdout:8/710: mkdir d1/d5e/daf/ded 0 2024-03-14T13:02:22.699 INFO:tasks.workunit.client.1.smithi137.stdout:0/655: fsync d1/d5/d1b/d42/f8e 0 2024-03-14T13:02:22.700 INFO:tasks.workunit.client.1.smithi137.stdout:1/700: creat d7/d31/fef x:0 0 0 2024-03-14T13:02:22.700 INFO:tasks.workunit.client.1.smithi137.stdout:7/540: rename d3/d5/d23/d2c to d3/d5/d23/d2c/d5d/da6 22 2024-03-14T13:02:22.703 INFO:tasks.workunit.client.0.smithi111.stdout:3/669: dwrite d0/d2d/d35/d4c/dad/dee/ff6 [0,4194304] 0 2024-03-14T13:02:22.703 INFO:tasks.workunit.client.1.smithi137.stdout:8/711: chown d1/f6 26219 1 2024-03-14T13:02:22.704 INFO:tasks.workunit.client.1.smithi137.stdout:1/701: chown d7/d31 4 1 2024-03-14T13:02:22.704 INFO:tasks.workunit.client.1.smithi137.stdout:2/690: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f16 [0,4194304] 0 2024-03-14T13:02:22.706 INFO:tasks.workunit.client.1.smithi137.stdout:7/541: creat d3/d5/d5a/fa7 x:0 0 0 2024-03-14T13:02:22.710 INFO:tasks.workunit.client.0.smithi111.stdout:2/602: creat dd/d17/d34/fbe x:0 0 0 2024-03-14T13:02:22.711 INFO:tasks.workunit.client.1.smithi137.stdout:6/652: rename d2/d54/lca to d2/d54/d55/d6d/d76/d57/ldf 0 2024-03-14T13:02:22.713 INFO:tasks.workunit.client.0.smithi111.stdout:1/713: rename d20/d21/d28/da0/lea to d20/d40/da6/lf1 0 2024-03-14T13:02:22.713 INFO:tasks.workunit.client.0.smithi111.stdout:1/714: write f19 [3025813,81561] 0 2024-03-14T13:02:22.714 INFO:tasks.workunit.client.1.smithi137.stdout:8/712: write d1/d5e/dc9/dda/fdc [774867,85921] 0 2024-03-14T13:02:22.714 INFO:tasks.workunit.client.0.smithi111.stdout:2/603: write dd/d17/d53/f98 [6240975,83735] 0 2024-03-14T13:02:22.714 INFO:tasks.workunit.client.0.smithi111.stdout:5/816: mknod d2/d16/d49/d44/d4a/d9e/db7/dbb/c105 0 2024-03-14T13:02:22.714 INFO:tasks.workunit.client.0.smithi111.stdout:0/815: dread d6/d9/d15/d55/f86 [0,4194304] 0 2024-03-14T13:02:22.715 INFO:tasks.workunit.client.0.smithi111.stdout:7/573: mknod d0/d1/d5/d38/d1b/caa 0 2024-03-14T13:02:22.715 INFO:tasks.workunit.client.1.smithi137.stdout:7/542: mkdir d3/d5/d5a/da8 0 2024-03-14T13:02:22.715 INFO:tasks.workunit.client.1.smithi137.stdout:8/713: dread - d1/d5e/dc9/fe9 zero size 2024-03-14T13:02:22.717 INFO:tasks.workunit.client.1.smithi137.stdout:6/653: mkdir d2/d6/d86/de0 0 2024-03-14T13:02:22.717 INFO:tasks.workunit.client.0.smithi111.stdout:2/604: stat dd/l4a 0 2024-03-14T13:02:22.718 INFO:tasks.workunit.client.1.smithi137.stdout:7/543: mkdir d3/d6/d59/da9 0 2024-03-14T13:02:22.718 INFO:tasks.workunit.client.1.smithi137.stdout:8/714: read - d1/dc/d3c/d87/d34/d97/fbb zero size 2024-03-14T13:02:22.719 INFO:tasks.workunit.client.1.smithi137.stdout:2/691: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d41/de1 0 2024-03-14T13:02:22.720 INFO:tasks.workunit.client.1.smithi137.stdout:6/654: truncate d2/d54/d55/d38/f51 4250465 0 2024-03-14T13:02:22.721 INFO:tasks.workunit.client.0.smithi111.stdout:8/904: creat d3/d1b/f12c x:0 0 0 2024-03-14T13:02:22.721 INFO:tasks.workunit.client.0.smithi111.stdout:8/905: chown d3/dc/d45/d11a/f5c 1799719194 1 2024-03-14T13:02:22.721 INFO:tasks.workunit.client.1.smithi137.stdout:7/544: mkdir d3/d6/d59/daa 0 2024-03-14T13:02:22.722 INFO:tasks.workunit.client.0.smithi111.stdout:8/906: readlink d3/d5/le9 0 2024-03-14T13:02:22.722 INFO:tasks.workunit.client.1.smithi137.stdout:6/655: fdatasync d2/d6/de/d45/f60 0 2024-03-14T13:02:22.722 INFO:tasks.workunit.client.1.smithi137.stdout:2/692: write d2/d7/d2e/d3e/d44/d6b/d1d/f7b [1190933,10698] 0 2024-03-14T13:02:22.723 INFO:tasks.workunit.client.1.smithi137.stdout:7/545: stat d3/f18 0 2024-03-14T13:02:22.724 INFO:tasks.workunit.client.1.smithi137.stdout:6/656: chown d2/d54/d55/d6d/f90 1819307 1 2024-03-14T13:02:22.724 INFO:tasks.workunit.client.1.smithi137.stdout:7/546: chown d3/d5/d5a/f7f 5113035 1 2024-03-14T13:02:22.725 INFO:tasks.workunit.client.0.smithi111.stdout:0/816: unlink d6/d9/de/d32/c8a 0 2024-03-14T13:02:22.726 INFO:tasks.workunit.client.1.smithi137.stdout:2/693: mkdir d2/d7/d2e/d3e/d44/d6b/d22/da5/de2 0 2024-03-14T13:02:22.727 INFO:tasks.workunit.client.0.smithi111.stdout:6/613: rmdir d0/d17/d20/d4a/d68/dd3 0 2024-03-14T13:02:22.727 INFO:tasks.workunit.client.1.smithi137.stdout:6/657: creat d2/d6/de/d45/d59/fe1 x:0 0 0 2024-03-14T13:02:22.728 INFO:tasks.workunit.client.1.smithi137.stdout:7/547: creat d3/d5/d23/d3f/d5b/fab x:0 0 0 2024-03-14T13:02:22.729 INFO:tasks.workunit.client.0.smithi111.stdout:6/614: chown d0/d17/d20/l75 3 1 2024-03-14T13:02:22.730 INFO:tasks.workunit.client.1.smithi137.stdout:7/548: chown d3/d5/d23/d2c/d5d/d6a/d75/d78 28781 1 2024-03-14T13:02:22.731 INFO:tasks.workunit.client.1.smithi137.stdout:6/658: truncate d2/d54/d55/d6d/d76/fc2 912290 0 2024-03-14T13:02:22.732 INFO:tasks.workunit.client.0.smithi111.stdout:0/817: creat d6/d9/de/d11a/f122 x:0 0 0 2024-03-14T13:02:22.732 INFO:tasks.workunit.client.0.smithi111.stdout:2/605: mkdir dd/d17/d34/dbf 0 2024-03-14T13:02:22.732 INFO:tasks.workunit.client.0.smithi111.stdout:2/606: fsync dd/d11/d1a/d88/f89 0 2024-03-14T13:02:22.734 INFO:tasks.workunit.client.1.smithi137.stdout:6/659: dread d2/d54/d55/f8d [0,4194304] 0 2024-03-14T13:02:22.735 INFO:tasks.workunit.client.0.smithi111.stdout:6/615: creat d0/d17/d21/d69/d3d/fe4 x:0 0 0 2024-03-14T13:02:22.737 INFO:tasks.workunit.client.1.smithi137.stdout:2/694: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf [0,4194304] 0 2024-03-14T13:02:22.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/817: dread d2/d16/d49/d4c/f73 [0,4194304] 0 2024-03-14T13:02:22.740 INFO:tasks.workunit.client.0.smithi111.stdout:2/607: getdents dd/d17/d34/d4f 0 2024-03-14T13:02:22.740 INFO:tasks.workunit.client.0.smithi111.stdout:0/818: truncate d6/d9/de/f23 2229563 0 2024-03-14T13:02:22.740 INFO:tasks.workunit.client.1.smithi137.stdout:1/702: dread d7/d9/f15 [0,4194304] 0 2024-03-14T13:02:22.742 INFO:tasks.workunit.client.1.smithi137.stdout:1/703: chown d7/d9/d19/d1f/c21 264099224 1 2024-03-14T13:02:22.742 INFO:tasks.workunit.client.0.smithi111.stdout:6/616: mkdir d0/d17/d20/d5a/d5e/dc6/de5 0 2024-03-14T13:02:22.744 INFO:tasks.workunit.client.1.smithi137.stdout:1/704: fsync d7/d31/d54/f60 0 2024-03-14T13:02:22.746 INFO:tasks.workunit.client.1.smithi137.stdout:1/705: stat d7/d9/d19/d1f/d22/fa2 0 2024-03-14T13:02:22.747 INFO:tasks.workunit.client.1.smithi137.stdout:2/695: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/d41/f6e [0,4194304] 0 2024-03-14T13:02:22.747 INFO:tasks.workunit.client.0.smithi111.stdout:2/608: unlink dd/d17/d34/d73/f5e 0 2024-03-14T13:02:22.748 INFO:tasks.workunit.client.0.smithi111.stdout:0/819: write d6/d9/de/d3a/d46/dde/ff3 [812316,37084] 0 2024-03-14T13:02:22.748 INFO:tasks.workunit.client.0.smithi111.stdout:5/818: creat d2/d45/d76/db2/df7/f106 x:0 0 0 2024-03-14T13:02:22.749 INFO:tasks.workunit.client.1.smithi137.stdout:1/706: dread d7/d4d/d6b/faa [0,4194304] 0 2024-03-14T13:02:22.749 INFO:tasks.workunit.client.0.smithi111.stdout:2/609: rmdir dd/d11 39 2024-03-14T13:02:22.751 INFO:tasks.workunit.client.0.smithi111.stdout:0/820: rmdir d6/d3d/da6/dfe 39 2024-03-14T13:02:22.752 INFO:tasks.workunit.client.1.smithi137.stdout:1/707: creat d7/d9/d19/d5f/ff0 x:0 0 0 2024-03-14T13:02:22.754 INFO:tasks.workunit.client.1.smithi137.stdout:1/708: write d7/d9/fad [532122,113343] 0 2024-03-14T13:02:22.754 INFO:tasks.workunit.client.1.smithi137.stdout:2/696: rename d2/d7/d2e/d3e/d44/d6b/d22/da5/lbd to d2/d7/d2e/d3e/daa/dcd/le3 0 2024-03-14T13:02:22.754 INFO:tasks.workunit.client.0.smithi111.stdout:0/821: write d6/d9/d5b/d9e/ff5 [466232,115543] 0 2024-03-14T13:02:22.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:22.756 INFO:tasks.workunit.client.0.smithi111.stdout:0/822: chown d6/d3d/fc0 0 1 2024-03-14T13:02:22.757 INFO:tasks.workunit.client.1.smithi137.stdout:1/709: rmdir d7/d31/d82 39 2024-03-14T13:02:22.758 INFO:tasks.workunit.client.0.smithi111.stdout:2/610: getdents dd/d17/d34/d73/d75 0 2024-03-14T13:02:22.759 INFO:tasks.workunit.client.0.smithi111.stdout:6/617: dread d0/d17/d21/d69/f76 [0,4194304] 0 2024-03-14T13:02:22.760 INFO:tasks.workunit.client.0.smithi111.stdout:0/823: creat d6/d45/d114/f123 x:0 0 0 2024-03-14T13:02:22.762 INFO:tasks.workunit.client.1.smithi137.stdout:2/697: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f77 213363 0 2024-03-14T13:02:22.762 INFO:tasks.workunit.client.0.smithi111.stdout:5/819: dwrite d2/d16/d49/d33/f7b [0,4194304] 0 2024-03-14T13:02:22.762 INFO:tasks.workunit.client.0.smithi111.stdout:6/618: symlink d0/d17/le6 0 2024-03-14T13:02:22.762 INFO:tasks.workunit.client.1.smithi137.stdout:1/710: creat d7/d9/d19/d40/d42/d9d/de5/ff1 x:0 0 0 2024-03-14T13:02:22.763 INFO:tasks.workunit.client.1.smithi137.stdout:1/711: fdatasync f6 0 2024-03-14T13:02:22.765 INFO:tasks.workunit.client.1.smithi137.stdout:2/698: mknod d2/d7/d2e/d3e/d44/d6b/d22/d97/ce4 0 2024-03-14T13:02:22.765 INFO:tasks.workunit.client.0.smithi111.stdout:0/824: stat d6/d9/de/d77/f7c 0 2024-03-14T13:02:22.766 INFO:tasks.workunit.client.1.smithi137.stdout:1/712: stat d7/d9/d19/d1f/d89/d90/cbb 0 2024-03-14T13:02:22.767 INFO:tasks.workunit.client.0.smithi111.stdout:5/820: rename d2/d16/d49/d4c/dcb to d2/d16/d49/d7c/d107 0 2024-03-14T13:02:22.770 INFO:tasks.workunit.client.0.smithi111.stdout:5/821: rmdir d2/d16/d49/d44/d4a/d9e 39 2024-03-14T13:02:22.771 INFO:tasks.workunit.client.1.smithi137.stdout:6/660: dread d2/d54/d55/d33/fcc [0,4194304] 0 2024-03-14T13:02:22.772 INFO:tasks.workunit.client.0.smithi111.stdout:0/825: getdents d6/d9/de/d77 0 2024-03-14T13:02:22.773 INFO:tasks.workunit.client.1.smithi137.stdout:2/699: rename d2/d34/d3c/d53 to d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5 0 2024-03-14T13:02:22.774 INFO:tasks.workunit.client.0.smithi111.stdout:5/822: creat d2/d16/d49/d33/d68/f108 x:0 0 0 2024-03-14T13:02:22.774 INFO:tasks.workunit.client.0.smithi111.stdout:0/826: mkdir d6/d9/de/d3a/d46/d124 0 2024-03-14T13:02:22.777 INFO:tasks.workunit.client.0.smithi111.stdout:5/823: rename d2/d16/d49/f78 to d2/d13/f109 0 2024-03-14T13:02:22.780 INFO:tasks.workunit.client.1.smithi137.stdout:6/661: write d2/f2d [169133,25593] 0 2024-03-14T13:02:22.780 INFO:tasks.workunit.client.0.smithi111.stdout:9/661: dread d5/d6/d2a/d5a/f9f [0,4194304] 0 2024-03-14T13:02:22.781 INFO:tasks.workunit.client.1.smithi137.stdout:2/700: write d2/d7/d2e/d3e/d44/d6b/d22/dac/faf [1378884,43444] 0 2024-03-14T13:02:22.782 INFO:tasks.workunit.client.0.smithi111.stdout:0/827: symlink d6/d9/d15/d49/d56/l125 0 2024-03-14T13:02:22.783 INFO:tasks.workunit.client.0.smithi111.stdout:0/828: readlink d6/d9/d15/da9/lbf 0 2024-03-14T13:02:22.783 INFO:tasks.workunit.client.0.smithi111.stdout:9/662: dread - d5/d42/f95 zero size 2024-03-14T13:02:22.783 INFO:tasks.workunit.client.0.smithi111.stdout:5/824: truncate d2/d16/d49/d44/d4a/d9e/db7/dbb/ff9 149157 0 2024-03-14T13:02:22.784 INFO:tasks.workunit.client.1.smithi137.stdout:0/656: rmdir d1/d5/d1b/d1e 39 2024-03-14T13:02:22.784 INFO:tasks.workunit.client.0.smithi111.stdout:9/663: fdatasync d5/d42/d71/d96/fac 0 2024-03-14T13:02:22.785 INFO:tasks.workunit.client.1.smithi137.stdout:0/657: chown d1/d44/d56/la7 0 1 2024-03-14T13:02:22.785 INFO:tasks.workunit.client.0.smithi111.stdout:5/825: fsync d2/d16/d49/d4c/f73 0 2024-03-14T13:02:22.786 INFO:tasks.workunit.client.1.smithi137.stdout:2/701: symlink d2/d7/d2e/d3e/le6 0 2024-03-14T13:02:22.787 INFO:tasks.workunit.client.0.smithi111.stdout:9/664: mknod d5/d6/d8/d74/ce6 0 2024-03-14T13:02:22.787 INFO:tasks.workunit.client.1.smithi137.stdout:1/713: link d7/d9/d19/d1f/d26/d6c/dc8/ca1 d7/d9/d19/d1f/d89/d90/db8/cf2 0 2024-03-14T13:02:22.788 INFO:tasks.workunit.client.0.smithi111.stdout:5/826: read d2/d45/d76/fab [3744848,19589] 0 2024-03-14T13:02:22.789 INFO:tasks.workunit.client.0.smithi111.stdout:6/619: dread d0/d17/d21/f2d [0,4194304] 0 2024-03-14T13:02:22.789 INFO:tasks.workunit.client.0.smithi111.stdout:9/665: truncate d5/d6/d2a/d5a/fad 2089526 0 2024-03-14T13:02:22.790 INFO:tasks.workunit.client.1.smithi137.stdout:1/714: dread d7/d4d/f64 [0,4194304] 0 2024-03-14T13:02:22.792 INFO:tasks.workunit.client.1.smithi137.stdout:6/662: link d2/d54/l4b d2/d54/d55/d6d/dc3/dce/le2 0 2024-03-14T13:02:22.793 INFO:tasks.workunit.client.1.smithi137.stdout:2/702: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd0 [0,4194304] 0 2024-03-14T13:02:22.793 INFO:tasks.workunit.client.0.smithi111.stdout:6/620: symlink d0/d17/d21/le7 0 2024-03-14T13:02:22.793 INFO:tasks.workunit.client.0.smithi111.stdout:1/715: write d20/d21/da7/fbe [1211149,77214] 0 2024-03-14T13:02:22.794 INFO:tasks.workunit.client.1.smithi137.stdout:6/663: write d2/d6/de/d45/d59/f75 [2707400,12762] 0 2024-03-14T13:02:22.795 INFO:tasks.workunit.client.0.smithi111.stdout:9/666: mkdir d5/d6/d8/d5b/d84/db6/de7 0 2024-03-14T13:02:22.796 INFO:tasks.workunit.client.0.smithi111.stdout:1/716: readlink d20/l95 0 2024-03-14T13:02:22.797 INFO:tasks.workunit.client.0.smithi111.stdout:6/621: chown d0/d17 2648065 1 2024-03-14T13:02:22.798 INFO:tasks.workunit.client.0.smithi111.stdout:3/670: dwrite d0/d2d/d35/d50/da2/db3/d6f/f79 [0,4194304] 0 2024-03-14T13:02:22.798 INFO:tasks.workunit.client.0.smithi111.stdout:7/574: write d0/d1/d4a/f6a [1478931,9791] 0 2024-03-14T13:02:22.799 INFO:tasks.workunit.client.1.smithi137.stdout:2/703: mkdir d2/d7/d2e/d3e/daa/de7 0 2024-03-14T13:02:22.799 INFO:tasks.workunit.client.1.smithi137.stdout:6/664: dread d2/d6/de/f5d [0,4194304] 0 2024-03-14T13:02:22.799 INFO:tasks.workunit.client.0.smithi111.stdout:9/667: chown d5/d6/d8/d74/cab 883215 1 2024-03-14T13:02:22.801 INFO:tasks.workunit.client.0.smithi111.stdout:1/717: fdatasync d20/d25/d87/f8f 0 2024-03-14T13:02:22.802 INFO:tasks.workunit.client.1.smithi137.stdout:8/715: mkdir d1/d5e/daf/dee 0 2024-03-14T13:02:22.802 INFO:tasks.workunit.client.1.smithi137.stdout:6/665: readlink d2/d54/d55/d6d/d76/d57/d58/l80 0 2024-03-14T13:02:22.805 INFO:tasks.workunit.client.1.smithi137.stdout:1/715: symlink d7/d9/d19/d1f/d26/d7d/lf3 0 2024-03-14T13:02:22.805 INFO:tasks.workunit.client.1.smithi137.stdout:7/549: fsync d3/d5/d23/d3f/d5b/fab 0 2024-03-14T13:02:22.806 INFO:tasks.workunit.client.1.smithi137.stdout:2/704: symlink d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/le8 0 2024-03-14T13:02:22.807 INFO:tasks.workunit.client.1.smithi137.stdout:7/550: write d3/d52/f9d [3868085,86271] 0 2024-03-14T13:02:22.808 INFO:tasks.workunit.client.1.smithi137.stdout:6/666: unlink d2/d54/d55/d33/fa0 0 2024-03-14T13:02:22.809 INFO:tasks.workunit.client.1.smithi137.stdout:2/705: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/cde 4794 1 2024-03-14T13:02:22.809 INFO:tasks.workunit.client.1.smithi137.stdout:0/658: getdents d1/d1f/d80 0 2024-03-14T13:02:22.810 INFO:tasks.workunit.client.0.smithi111.stdout:6/622: truncate d0/d17/d20/d65/fc7 814385 0 2024-03-14T13:02:22.810 INFO:tasks.workunit.client.0.smithi111.stdout:7/575: unlink d0/d1/d5/d38/d1b/caa 0 2024-03-14T13:02:22.812 INFO:tasks.workunit.client.1.smithi137.stdout:0/659: truncate d1/d5/d1b/fc5 494271 0 2024-03-14T13:02:22.812 INFO:tasks.workunit.client.1.smithi137.stdout:1/716: rmdir d7/d9/d19/d40/d42 39 2024-03-14T13:02:22.813 INFO:tasks.workunit.client.0.smithi111.stdout:8/907: dwrite d3/d1b/d20/d35/d40/d48/d77/fdf [0,4194304] 0 2024-03-14T13:02:22.814 INFO:tasks.workunit.client.1.smithi137.stdout:1/717: fdatasync d7/d9/d19/d5f/f7f 0 2024-03-14T13:02:22.815 INFO:tasks.workunit.client.1.smithi137.stdout:6/667: mknod d2/d54/d55/d6d/d76/d57/ce3 0 2024-03-14T13:02:22.815 INFO:tasks.workunit.client.1.smithi137.stdout:1/718: fsync d7/d9/d19/d40/f46 0 2024-03-14T13:02:22.817 INFO:tasks.workunit.client.0.smithi111.stdout:7/576: chown d0/d1/d5/d38/f7f 102 1 2024-03-14T13:02:22.818 INFO:tasks.workunit.client.0.smithi111.stdout:6/623: read - d0/d17/d20/d65/d93/dd9/fe1 zero size 2024-03-14T13:02:22.818 INFO:tasks.workunit.client.1.smithi137.stdout:6/668: chown d2/d54/l6f 12 1 2024-03-14T13:02:22.818 INFO:tasks.workunit.client.0.smithi111.stdout:3/671: dwrite d0/d2d/d35/d50/d5e/f97 [0,4194304] 0 2024-03-14T13:02:22.819 INFO:tasks.workunit.client.0.smithi111.stdout:9/668: rename d5/lc6 to d5/d6/d8/d5b/d7c/le8 0 2024-03-14T13:02:22.819 INFO:tasks.workunit.client.1.smithi137.stdout:7/551: mknod d3/cac 0 2024-03-14T13:02:22.821 INFO:tasks.workunit.client.1.smithi137.stdout:2/706: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/le9 0 2024-03-14T13:02:22.821 INFO:tasks.workunit.client.1.smithi137.stdout:0/660: dread d1/d44/d56/f5a [0,4194304] 0 2024-03-14T13:02:22.823 INFO:tasks.workunit.client.1.smithi137.stdout:8/716: dread d1/dc/d77/d9f/de4/f16 [0,4194304] 0 2024-03-14T13:02:22.825 INFO:tasks.workunit.client.0.smithi111.stdout:1/718: creat d20/d21/d28/da0/db6/ff2 x:0 0 0 2024-03-14T13:02:22.826 INFO:tasks.workunit.client.0.smithi111.stdout:5/827: write d2/d16/d49/d7c/d95/d6a/f71 [156751,119426] 0 2024-03-14T13:02:22.827 INFO:tasks.workunit.client.1.smithi137.stdout:6/669: mknod d2/d6/d1d/d88/ce4 0 2024-03-14T13:02:22.828 INFO:tasks.workunit.client.1.smithi137.stdout:7/552: write d3/d6/d2e/f35 [3268524,50627] 0 2024-03-14T13:02:22.833 INFO:tasks.workunit.client.1.smithi137.stdout:1/719: creat d7/d9/d19/d40/d42/d9d/de5/ff4 x:0 0 0 2024-03-14T13:02:22.834 INFO:tasks.workunit.client.1.smithi137.stdout:7/553: dread d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:22.834 INFO:tasks.workunit.client.1.smithi137.stdout:1/720: dread - d7/d9/d19/d1f/d2e/fe7 zero size 2024-03-14T13:02:22.835 INFO:tasks.workunit.client.1.smithi137.stdout:0/661: fdatasync d1/d5/d1b/d1e/d25/d26/f2d 0 2024-03-14T13:02:22.837 INFO:tasks.workunit.client.1.smithi137.stdout:8/717: rename d1/dc/d3c/d87/d34/d97/f2d to d1/d4a/fef 0 2024-03-14T13:02:22.838 INFO:tasks.workunit.client.1.smithi137.stdout:9/754: sync 2024-03-14T13:02:22.843 INFO:tasks.workunit.client.1.smithi137.stdout:7/554: dwrite d3/d6/d14/d27/f7b [0,4194304] 0 2024-03-14T13:02:22.851 INFO:tasks.workunit.client.1.smithi137.stdout:7/555: stat d3/f6f 0 2024-03-14T13:02:22.851 INFO:tasks.workunit.client.1.smithi137.stdout:9/755: mknod d6/df/d19/d33/ddf/cf1 0 2024-03-14T13:02:22.851 INFO:tasks.workunit.client.1.smithi137.stdout:8/718: symlink d1/d5e/lf0 0 2024-03-14T13:02:22.853 INFO:tasks.workunit.client.1.smithi137.stdout:1/721: write d7/d29/f2b [862068,87010] 0 2024-03-14T13:02:22.857 INFO:tasks.workunit.client.1.smithi137.stdout:6/670: getdents d2/d6/de/d45/d59/d39 0 2024-03-14T13:02:22.863 INFO:tasks.workunit.client.1.smithi137.stdout:7/556: dwrite d3/d5/d23/d3f/f76 [0,4194304] 0 2024-03-14T13:02:22.866 INFO:tasks.workunit.client.1.smithi137.stdout:0/662: dwrite d1/d5/d1b/d1e/d25/d55/f60 [0,4194304] 0 2024-03-14T13:02:22.869 INFO:tasks.workunit.client.1.smithi137.stdout:2/707: write d2/f14 [905577,77056] 0 2024-03-14T13:02:22.870 INFO:tasks.workunit.client.1.smithi137.stdout:9/756: mknod d6/d53/cf2 0 2024-03-14T13:02:22.874 INFO:tasks.workunit.client.1.smithi137.stdout:8/719: mknod d1/d5e/daf/dee/cf1 0 2024-03-14T13:02:22.879 INFO:tasks.workunit.client.1.smithi137.stdout:9/757: truncate d6/df/d19/d3a/da9/fe4 134016 0 2024-03-14T13:02:22.879 INFO:tasks.workunit.client.1.smithi137.stdout:1/722: rename d7/l38 to d7/d9/d19/d1f/d2e/d63/lf5 0 2024-03-14T13:02:22.879 INFO:tasks.workunit.client.1.smithi137.stdout:7/557: symlink d3/d6/d59/da9/lad 0 2024-03-14T13:02:22.880 INFO:tasks.workunit.client.1.smithi137.stdout:9/758: mknod d6/df/d88/dd5/cf3 0 2024-03-14T13:02:22.881 INFO:tasks.workunit.client.0.smithi111.stdout:6/624: creat d0/d17/d20/d4a/d59/d82/fe8 x:0 0 0 2024-03-14T13:02:22.882 INFO:tasks.workunit.client.1.smithi137.stdout:2/708: getdents d2/d7/d2e/d3e/d44/d6b/d22 0 2024-03-14T13:02:22.883 INFO:tasks.workunit.client.1.smithi137.stdout:7/558: creat d3/d5/d5a/d80/fae x:0 0 0 2024-03-14T13:02:22.883 INFO:tasks.workunit.client.0.smithi111.stdout:6/625: dread - d0/d17/d20/d4a/d59/d82/fe8 zero size 2024-03-14T13:02:22.883 INFO:tasks.workunit.client.1.smithi137.stdout:9/759: write d6/df/d19/f62 [2353511,12194] 0 2024-03-14T13:02:22.884 INFO:tasks.workunit.client.1.smithi137.stdout:1/723: creat d7/d9/d19/d1f/d89/d90/db8/ff6 x:0 0 0 2024-03-14T13:02:22.885 INFO:tasks.workunit.client.0.smithi111.stdout:3/672: creat d0/d2d/d35/d50/d5e/d19/ff8 x:0 0 0 2024-03-14T13:02:22.885 INFO:tasks.workunit.client.0.smithi111.stdout:9/669: mknod d5/d42/d71/d96/dd3/ce9 0 2024-03-14T13:02:22.887 INFO:tasks.workunit.client.1.smithi137.stdout:9/760: mkdir d6/df/d88/df4 0 2024-03-14T13:02:22.888 INFO:tasks.workunit.client.0.smithi111.stdout:7/577: getdents d0/d1/d4a/d87 0 2024-03-14T13:02:22.889 INFO:tasks.workunit.client.0.smithi111.stdout:6/626: mkdir d0/d17/d24/db8/de9 0 2024-03-14T13:02:22.889 INFO:tasks.workunit.client.0.smithi111.stdout:3/673: chown d0/d2d/d35/d50/c76 0 1 2024-03-14T13:02:22.891 INFO:tasks.workunit.client.1.smithi137.stdout:7/559: rename d3/d6/d14/d27/c36 to d3/d6/d64/d9e/caf 0 2024-03-14T13:02:22.891 INFO:tasks.workunit.client.0.smithi111.stdout:9/670: rmdir d5/d6/d8/d5b/d7c 39 2024-03-14T13:02:22.893 INFO:tasks.workunit.client.1.smithi137.stdout:2/709: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/fdd [0,4194304] 0 2024-03-14T13:02:22.893 INFO:tasks.workunit.client.0.smithi111.stdout:1/719: getdents d20/d25/da8/ddd 0 2024-03-14T13:02:22.894 INFO:tasks.workunit.client.0.smithi111.stdout:8/908: dwrite d3/dc/d45/f56 [0,4194304] 0 2024-03-14T13:02:22.894 INFO:tasks.workunit.client.1.smithi137.stdout:9/761: creat d6/df/d19/dcf/ff5 x:0 0 0 2024-03-14T13:02:22.897 INFO:tasks.workunit.client.0.smithi111.stdout:1/720: stat d20/d25/d75/c82 0 2024-03-14T13:02:22.897 INFO:tasks.workunit.client.0.smithi111.stdout:9/671: creat d5/d42/da4/fea x:0 0 0 2024-03-14T13:02:22.897 INFO:tasks.workunit.client.1.smithi137.stdout:1/724: creat d7/d9/d19/d1f/ff7 x:0 0 0 2024-03-14T13:02:22.897 INFO:tasks.workunit.client.1.smithi137.stdout:2/710: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/fb7 [876163,16617] 0 2024-03-14T13:02:22.899 INFO:tasks.workunit.client.1.smithi137.stdout:2/711: readlink d2/d34/l62 0 2024-03-14T13:02:22.899 INFO:tasks.workunit.client.0.smithi111.stdout:7/578: truncate d0/d1/d5/d38/d1b/d9c/fa9 410073 0 2024-03-14T13:02:22.900 INFO:tasks.workunit.client.0.smithi111.stdout:6/627: rename d0/d17/d21/f28 to d0/d17/d20/d5a/d5e/fea 0 2024-03-14T13:02:22.901 INFO:tasks.workunit.client.1.smithi137.stdout:2/712: fdatasync d2/d7/d2e/d3e/d44/fab 0 2024-03-14T13:02:22.903 INFO:tasks.workunit.client.1.smithi137.stdout:1/725: creat d7/d9/d19/d40/d42/d9d/de5/ff8 x:0 0 0 2024-03-14T13:02:22.903 INFO:tasks.workunit.client.0.smithi111.stdout:5/828: dwrite d2/d16/d49/f11 [0,4194304] 0 2024-03-14T13:02:22.904 INFO:tasks.workunit.client.0.smithi111.stdout:7/579: write d0/d1/d22/f46 [2544341,33653] 0 2024-03-14T13:02:22.906 INFO:tasks.workunit.client.0.smithi111.stdout:3/674: rename d0/d2d/d35/f63 to d0/d2d/d35/d50/d6c/d98/ff9 0 2024-03-14T13:02:22.906 INFO:tasks.workunit.client.0.smithi111.stdout:7/580: fdatasync d0/f2e 0 2024-03-14T13:02:22.906 INFO:tasks.workunit.client.0.smithi111.stdout:9/672: creat d5/d6/feb x:0 0 0 2024-03-14T13:02:22.909 INFO:tasks.workunit.client.0.smithi111.stdout:5/829: mkdir d2/d16/d49/d4c/d9c/d10a 0 2024-03-14T13:02:22.910 INFO:tasks.workunit.client.0.smithi111.stdout:7/581: fdatasync d0/d1/d21/d2a/f8a 0 2024-03-14T13:02:22.910 INFO:tasks.workunit.client.0.smithi111.stdout:9/673: chown d5/d6/lb9 3 1 2024-03-14T13:02:22.910 INFO:tasks.workunit.client.1.smithi137.stdout:6/671: dwrite d2/d6/de/f32 [0,4194304] 0 2024-03-14T13:02:22.918 INFO:tasks.workunit.client.1.smithi137.stdout:1/726: rename d7 to d7/d31/d82/df9 22 2024-03-14T13:02:22.919 INFO:tasks.workunit.client.0.smithi111.stdout:3/675: dread d0/d2d/d35/f92 [0,4194304] 0 2024-03-14T13:02:22.919 INFO:tasks.workunit.client.0.smithi111.stdout:9/674: write d5/d6/d8/d5b/d84/fa2 [2633538,103199] 0 2024-03-14T13:02:22.926 INFO:tasks.workunit.client.0.smithi111.stdout:7/582: dwrite d0/d1/d4a/fa7 [0,4194304] 0 2024-03-14T13:02:22.926 INFO:tasks.workunit.client.1.smithi137.stdout:7/560: truncate d3/d5/d5a/f73 1320420 0 2024-03-14T13:02:22.927 INFO:tasks.workunit.client.0.smithi111.stdout:5/830: dwrite d2/d16/d49/f11 [0,4194304] 0 2024-03-14T13:02:22.929 INFO:tasks.workunit.client.1.smithi137.stdout:7/561: rmdir d3/d5/d5a/d86 39 2024-03-14T13:02:22.929 INFO:tasks.workunit.client.0.smithi111.stdout:8/909: dread d3/d1b/f6f [0,4194304] 0 2024-03-14T13:02:22.932 INFO:tasks.workunit.client.1.smithi137.stdout:6/672: read d2/d54/d55/fa3 [126523,100941] 0 2024-03-14T13:02:22.935 INFO:tasks.workunit.client.1.smithi137.stdout:1/727: dwrite d7/d9/d19/d1f/d2e/f9b [0,4194304] 0 2024-03-14T13:02:22.936 INFO:tasks.workunit.client.1.smithi137.stdout:2/713: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/f48 [0,4194304] 0 2024-03-14T13:02:22.938 INFO:tasks.workunit.client.0.smithi111.stdout:1/721: dread d20/d25/d31/d80/fd5 [0,4194304] 0 2024-03-14T13:02:22.939 INFO:tasks.workunit.client.0.smithi111.stdout:7/583: mkdir d0/d11/d98/dab 0 2024-03-14T13:02:22.939 INFO:tasks.workunit.client.1.smithi137.stdout:6/673: fdatasync d2/d54/d55/d38/f85 0 2024-03-14T13:02:22.939 INFO:tasks.workunit.client.1.smithi137.stdout:1/728: chown d7/d9/d19/d5f 24236 1 2024-03-14T13:02:22.940 INFO:tasks.workunit.client.0.smithi111.stdout:3/676: dwrite d0/d2d/d35/d50/f54 [0,4194304] 0 2024-03-14T13:02:22.940 INFO:tasks.workunit.client.0.smithi111.stdout:8/910: truncate d3/d1b/d34/f107 258769 0 2024-03-14T13:02:22.946 INFO:tasks.workunit.client.0.smithi111.stdout:3/677: readlink d0/d2d/l7a 0 2024-03-14T13:02:22.947 INFO:tasks.workunit.client.1.smithi137.stdout:1/729: chown d7/d9/d19/d1f/d26/d6c/dc8/ca1 3754701 1 2024-03-14T13:02:22.950 INFO:tasks.workunit.client.1.smithi137.stdout:2/714: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf [0,4194304] 0 2024-03-14T13:02:22.950 INFO:tasks.workunit.client.1.smithi137.stdout:9/762: dread d6/df/d19/f62 [0,4194304] 0 2024-03-14T13:02:22.951 INFO:tasks.workunit.client.0.smithi111.stdout:9/675: dwrite d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:22.951 INFO:tasks.workunit.client.1.smithi137.stdout:1/730: readlink d7/d9/d19/d1f/d22/l75 0 2024-03-14T13:02:22.956 INFO:tasks.workunit.client.0.smithi111.stdout:1/722: rename d20/d21/d28/da0/db6/cbf to d20/d21/dd7/cf3 0 2024-03-14T13:02:22.956 INFO:tasks.workunit.client.0.smithi111.stdout:8/911: mkdir d3/d5/d12d 0 2024-03-14T13:02:22.957 INFO:tasks.workunit.client.0.smithi111.stdout:1/723: chown d20/d21/d3f/f47 1738171629 1 2024-03-14T13:02:22.958 INFO:tasks.workunit.client.0.smithi111.stdout:7/584: creat d0/d1/d21/d5d/fac x:0 0 0 2024-03-14T13:02:22.959 INFO:tasks.workunit.client.1.smithi137.stdout:2/715: creat d2/d7/d2e/d3e/d44/d6b/d22/da5/fea x:0 0 0 2024-03-14T13:02:22.959 INFO:tasks.workunit.client.0.smithi111.stdout:9/676: truncate d5/f65 1014630 0 2024-03-14T13:02:22.960 INFO:tasks.workunit.client.0.smithi111.stdout:8/912: symlink d3/d22/d30/l12e 0 2024-03-14T13:02:22.961 INFO:tasks.workunit.client.1.smithi137.stdout:2/716: write d2/d34/fcb [462335,36543] 0 2024-03-14T13:02:22.962 INFO:tasks.workunit.client.1.smithi137.stdout:9/763: rename d6/df/d19/d3a to d6/df/d88/de5/df6 0 2024-03-14T13:02:22.964 INFO:tasks.workunit.client.0.smithi111.stdout:3/678: link d0/d2d/d35/d50/d5e/c17 d0/d2d/cfa 0 2024-03-14T13:02:22.966 INFO:tasks.workunit.client.1.smithi137.stdout:2/717: read - d2/d7/d2e/d3e/d44/d6b/d22/dbb/fc4 zero size 2024-03-14T13:02:22.966 INFO:tasks.workunit.client.1.smithi137.stdout:9/764: mknod d6/cf7 0 2024-03-14T13:02:22.967 INFO:tasks.workunit.client.0.smithi111.stdout:9/677: chown d5/d6/d8/d5b/d7c/f90 1065546 1 2024-03-14T13:02:22.967 INFO:tasks.workunit.client.0.smithi111.stdout:9/678: chown d5/c4b 290687285 1 2024-03-14T13:02:22.968 INFO:tasks.workunit.client.0.smithi111.stdout:3/679: mknod d0/d2d/d35/d50/d6c/cfb 0 2024-03-14T13:02:22.968 INFO:tasks.workunit.client.1.smithi137.stdout:1/731: dwrite d7/d9/d19/d40/d42/d9d/de5/ff8 [0,4194304] 0 2024-03-14T13:02:22.969 INFO:tasks.workunit.client.1.smithi137.stdout:2/718: stat d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 0 2024-03-14T13:02:22.970 INFO:tasks.workunit.client.1.smithi137.stdout:9/765: creat d6/d26/ff8 x:0 0 0 2024-03-14T13:02:22.972 INFO:tasks.workunit.client.1.smithi137.stdout:2/719: write d2/d7/d2e/d3e/d4e/fcf [725218,90102] 0 2024-03-14T13:02:22.972 INFO:tasks.workunit.client.1.smithi137.stdout:1/732: rename d7/f88 to d7/d9/d19/ffa 0 2024-03-14T13:02:22.973 INFO:tasks.workunit.client.1.smithi137.stdout:9/766: creat d6/d53/db7/ff9 x:0 0 0 2024-03-14T13:02:22.974 INFO:tasks.workunit.client.1.smithi137.stdout:2/720: write d2/f1e [4737534,27082] 0 2024-03-14T13:02:22.976 INFO:tasks.workunit.client.1.smithi137.stdout:9/767: truncate d6/d53/fb8 740454 0 2024-03-14T13:02:22.978 INFO:tasks.workunit.client.1.smithi137.stdout:2/721: creat d2/d7/d2e/d3e/d4e/feb x:0 0 0 2024-03-14T13:02:22.979 INFO:tasks.workunit.client.1.smithi137.stdout:9/768: creat d6/d53/db7/ffa x:0 0 0 2024-03-14T13:02:22.980 INFO:tasks.workunit.client.0.smithi111.stdout:5/831: dread d2/d16/d49/d44/f50 [0,4194304] 0 2024-03-14T13:02:22.980 INFO:tasks.workunit.client.0.smithi111.stdout:5/832: fsync d2/f7 0 2024-03-14T13:02:22.981 INFO:tasks.workunit.client.1.smithi137.stdout:7/562: dread d3/d6/d2e/f49 [0,4194304] 0 2024-03-14T13:02:22.982 INFO:tasks.workunit.client.1.smithi137.stdout:9/769: readlink d6/df/d19/la7 0 2024-03-14T13:02:22.983 INFO:tasks.workunit.client.1.smithi137.stdout:9/770: chown d6/df/d19/d33/d8f/fb2 63999 1 2024-03-14T13:02:22.984 INFO:tasks.workunit.client.1.smithi137.stdout:7/563: unlink d3/l9f 0 2024-03-14T13:02:22.986 INFO:tasks.workunit.client.0.smithi111.stdout:5/833: link d2/d16/d49/d4c/cf3 d2/d45/d79/c10b 0 2024-03-14T13:02:22.987 INFO:tasks.workunit.client.1.smithi137.stdout:7/564: mknod d3/d6/d14/cb0 0 2024-03-14T13:02:22.987 INFO:tasks.workunit.client.0.smithi111.stdout:5/834: creat d2/d13/f10c x:0 0 0 2024-03-14T13:02:22.990 INFO:tasks.workunit.client.0.smithi111.stdout:5/835: readlink d2/d16/d49/d7c/d95/d36/lba 0 2024-03-14T13:02:22.992 INFO:tasks.workunit.client.1.smithi137.stdout:9/771: dwrite d6/d53/db7/fd1 [0,4194304] 0 2024-03-14T13:02:22.992 INFO:tasks.workunit.client.0.smithi111.stdout:5/836: mkdir d2/d45/d76/d8d/d10d 0 2024-03-14T13:02:22.992 INFO:tasks.workunit.client.1.smithi137.stdout:2/722: link d2/c2a d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/cec 0 2024-03-14T13:02:22.993 INFO:tasks.workunit.client.0.smithi111.stdout:5/837: fdatasync d2/d16/d49/d7c/d95/ffc 0 2024-03-14T13:02:22.999 INFO:tasks.workunit.client.1.smithi137.stdout:2/723: rename d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f8a to d2/d34/d3c/d9d/fed 0 2024-03-14T13:02:22.999 INFO:tasks.workunit.client.1.smithi137.stdout:7/565: link d3/d5/d23/d2c/l93 d3/d5/d5a/d80/lb1 0 2024-03-14T13:02:23.000 INFO:tasks.workunit.client.0.smithi111.stdout:8/913: dread d3/d1b/d20/d35/d40/fac [0,4194304] 0 2024-03-14T13:02:23.003 INFO:tasks.workunit.client.1.smithi137.stdout:2/724: truncate d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/fd5 471156 0 2024-03-14T13:02:23.005 INFO:tasks.workunit.client.0.smithi111.stdout:7/585: dread d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:23.006 INFO:tasks.workunit.client.0.smithi111.stdout:5/838: dwrite d2/d16/d49/d7c/d8b/f91 [4194304,4194304] 0 2024-03-14T13:02:23.007 INFO:tasks.workunit.client.1.smithi137.stdout:7/566: mknod d3/cb2 0 2024-03-14T13:02:23.008 INFO:tasks.workunit.client.0.smithi111.stdout:8/914: creat d3/dc/d45/d8d/f12f x:0 0 0 2024-03-14T13:02:23.010 INFO:tasks.workunit.client.0.smithi111.stdout:3/680: write d0/d30/f3d [2778411,117543] 0 2024-03-14T13:02:23.010 INFO:tasks.workunit.client.1.smithi137.stdout:2/725: unlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f16 0 2024-03-14T13:02:23.011 INFO:tasks.workunit.client.0.smithi111.stdout:7/586: chown d0/d7e/l1d 30467 1 2024-03-14T13:02:23.013 INFO:tasks.workunit.client.0.smithi111.stdout:9/679: dread d5/d6/d8/d1a/f43 [0,4194304] 0 2024-03-14T13:02:23.015 INFO:tasks.workunit.client.0.smithi111.stdout:1/724: dwrite d20/d25/d75/f65 [0,4194304] 0 2024-03-14T13:02:23.016 INFO:tasks.workunit.client.1.smithi137.stdout:2/726: truncate d2/d7/d2e/d3e/d44/d6b/d22/dbb/fc4 312389 0 2024-03-14T13:02:23.016 INFO:tasks.workunit.client.1.smithi137.stdout:7/567: rename d3/d5/f11 to d3/d6/d59/daa/fb3 0 2024-03-14T13:02:23.023 INFO:tasks.workunit.client.0.smithi111.stdout:8/915: rename d3/d22/d30/fee to d3/dc/d45/d8d/f130 0 2024-03-14T13:02:23.023 INFO:tasks.workunit.client.0.smithi111.stdout:9/680: mkdir d5/d6/d14/d7a/dec 0 2024-03-14T13:02:23.023 INFO:tasks.workunit.client.0.smithi111.stdout:1/725: creat d20/d21/da7/ff4 x:0 0 0 2024-03-14T13:02:23.024 INFO:tasks.workunit.client.0.smithi111.stdout:1/726: readlink d20/d40/lf0 0 2024-03-14T13:02:23.024 INFO:tasks.workunit.client.0.smithi111.stdout:8/916: truncate d3/dc/d45/d11a/f5c 4808486 0 2024-03-14T13:02:23.025 INFO:tasks.workunit.client.0.smithi111.stdout:9/681: readlink d5/d6/d2a/la3 0 2024-03-14T13:02:23.027 INFO:tasks.workunit.client.1.smithi137.stdout:7/568: getdents d3/d5/d23/d2c/d5d/d6a/da1 0 2024-03-14T13:02:23.027 INFO:tasks.workunit.client.0.smithi111.stdout:9/682: readlink d5/d6/d8/d5b/d7c/l82 0 2024-03-14T13:02:23.027 INFO:tasks.workunit.client.0.smithi111.stdout:1/727: mkdir d20/d25/d31/d80/df5 0 2024-03-14T13:02:23.029 INFO:tasks.workunit.client.0.smithi111.stdout:7/587: dread d0/d1/d4a/f60 [0,4194304] 0 2024-03-14T13:02:23.030 INFO:tasks.workunit.client.0.smithi111.stdout:8/917: rename d3/d1b/dc1/de4 to d3/d5/d12d/d131 0 2024-03-14T13:02:23.030 INFO:tasks.workunit.client.1.smithi137.stdout:7/569: mkdir d3/d54/db4 0 2024-03-14T13:02:23.031 INFO:tasks.workunit.client.1.smithi137.stdout:2/727: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/f7b [0,4194304] 0 2024-03-14T13:02:23.031 INFO:tasks.workunit.client.0.smithi111.stdout:8/918: write d3/dc/f99 [2596410,101207] 0 2024-03-14T13:02:23.031 INFO:tasks.workunit.client.0.smithi111.stdout:1/728: rmdir d20/d21/d28/da0/db6 39 2024-03-14T13:02:23.032 INFO:tasks.workunit.client.0.smithi111.stdout:3/681: dread d0/d2d/d35/d50/d5e/d19/f26 [0,4194304] 0 2024-03-14T13:02:23.032 INFO:tasks.workunit.client.1.smithi137.stdout:9/772: dwrite d6/df/d19/d1f/fa4 [0,4194304] 0 2024-03-14T13:02:23.034 INFO:tasks.workunit.client.1.smithi137.stdout:2/728: chown d2/d7/d2e/d3e/daa 246 1 2024-03-14T13:02:23.034 INFO:tasks.workunit.client.0.smithi111.stdout:7/588: mknod d0/d1/d5/d38/d5c/cad 0 2024-03-14T13:02:23.035 INFO:tasks.workunit.client.1.smithi137.stdout:7/570: write d3/d54/f61 [3964511,106210] 0 2024-03-14T13:02:23.036 INFO:tasks.workunit.client.0.smithi111.stdout:8/919: mknod d3/d22/d30/c132 0 2024-03-14T13:02:23.036 INFO:tasks.workunit.client.0.smithi111.stdout:7/589: fsync d0/d1/d4a/fa7 0 2024-03-14T13:02:23.037 INFO:tasks.workunit.client.0.smithi111.stdout:9/683: link d5/d6/d2a/d2d/fa8 d5/d42/d71/dbd/de0/fed 0 2024-03-14T13:02:23.039 INFO:tasks.workunit.client.1.smithi137.stdout:2/729: mkdir d2/d34/dee 0 2024-03-14T13:02:23.039 INFO:tasks.workunit.client.0.smithi111.stdout:5/839: truncate d2/d45/d76/d8d/f77 4508069 0 2024-03-14T13:02:23.040 INFO:tasks.workunit.client.1.smithi137.stdout:1/733: truncate d7/d9/d19/d1f/d2e/f3d 1014638 0 2024-03-14T13:02:23.042 INFO:tasks.workunit.client.1.smithi137.stdout:1/734: stat d7/d9/d19/d1f 0 2024-03-14T13:02:23.044 INFO:tasks.workunit.client.0.smithi111.stdout:8/920: readlink d3/dc/l17 0 2024-03-14T13:02:23.045 INFO:tasks.workunit.client.1.smithi137.stdout:2/730: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/fd9 [766266,46566] 0 2024-03-14T13:02:23.046 INFO:tasks.workunit.client.1.smithi137.stdout:7/571: rename d3/d6/f15 to d3/d5/d23/fb5 0 2024-03-14T13:02:23.047 INFO:tasks.workunit.client.1.smithi137.stdout:2/731: stat d2/d7/d2e/d3e/d44/d6b/d22/f8f 0 2024-03-14T13:02:23.047 INFO:tasks.workunit.client.1.smithi137.stdout:7/572: chown d3/d54/f5c 22 1 2024-03-14T13:02:23.048 INFO:tasks.workunit.client.1.smithi137.stdout:2/732: read - d2/d7/d2e/d3e/d44/fab zero size 2024-03-14T13:02:23.050 INFO:tasks.workunit.client.0.smithi111.stdout:7/590: link d0/d7e/l6e d0/d1/d21/d2a/d8c/lae 0 2024-03-14T13:02:23.051 INFO:tasks.workunit.client.0.smithi111.stdout:5/840: dwrite d2/d45/fff [0,4194304] 0 2024-03-14T13:02:23.052 INFO:tasks.workunit.client.1.smithi137.stdout:1/735: fsync d7/d9/d19/fae 0 2024-03-14T13:02:23.053 INFO:tasks.workunit.client.0.smithi111.stdout:7/591: dread - d0/d1/d5/d38/d1b/fa1 zero size 2024-03-14T13:02:23.053 INFO:tasks.workunit.client.0.smithi111.stdout:1/729: getdents d20/d21/d28/da0/db6 0 2024-03-14T13:02:23.053 INFO:tasks.workunit.client.0.smithi111.stdout:3/682: getdents d0/d2d/d35/d69/de6/dd8/dde 0 2024-03-14T13:02:23.053 INFO:tasks.workunit.client.1.smithi137.stdout:2/733: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/lef 0 2024-03-14T13:02:23.054 INFO:tasks.workunit.client.1.smithi137.stdout:7/573: mknod d3/d5/d23/d2c/d5d/d6a/da1/cb6 0 2024-03-14T13:02:23.056 INFO:tasks.workunit.client.1.smithi137.stdout:1/736: rmdir d7/d9/d19/d40/d42/d9d 39 2024-03-14T13:02:23.057 INFO:tasks.workunit.client.0.smithi111.stdout:1/730: write d20/d25/d75/f65 [2518375,93147] 0 2024-03-14T13:02:23.058 INFO:tasks.workunit.client.0.smithi111.stdout:1/731: readlink d20/d40/l50 0 2024-03-14T13:02:23.059 INFO:tasks.workunit.client.0.smithi111.stdout:7/592: creat d0/d1/d22/d3e/faf x:0 0 0 2024-03-14T13:02:23.060 INFO:tasks.workunit.client.0.smithi111.stdout:5/841: symlink d2/d16/d49/d44/d72/de0/dfe/l10e 0 2024-03-14T13:02:23.060 INFO:tasks.workunit.client.0.smithi111.stdout:1/732: chown d20/d21/d28/l88 184 1 2024-03-14T13:02:23.061 INFO:tasks.workunit.client.0.smithi111.stdout:3/683: mknod d0/d30/df1/cfc 0 2024-03-14T13:02:23.063 INFO:tasks.workunit.client.0.smithi111.stdout:5/842: symlink d2/d16/d49/d44/d72/l10f 0 2024-03-14T13:02:23.065 INFO:tasks.workunit.client.0.smithi111.stdout:5/843: write d2/d45/d76/db2/df7/f106 [506076,97797] 0 2024-03-14T13:02:23.066 INFO:tasks.workunit.client.0.smithi111.stdout:5/844: write d2/d16/f70 [6909119,51486] 0 2024-03-14T13:02:23.067 INFO:tasks.workunit.client.0.smithi111.stdout:5/845: fdatasync d2/d16/d49/d7c/d95/d6a/f71 0 2024-03-14T13:02:23.067 INFO:tasks.workunit.client.0.smithi111.stdout:1/733: link d20/d25/d31/f7f d20/d25/da8/ff6 0 2024-03-14T13:02:23.068 INFO:tasks.workunit.client.0.smithi111.stdout:5/846: stat d2/d45/cc7 0 2024-03-14T13:02:23.068 INFO:tasks.workunit.client.0.smithi111.stdout:7/593: link d0/d1/d5/l69 d0/d1/d5/d38/d5c/d6d/lb0 0 2024-03-14T13:02:23.069 INFO:tasks.workunit.client.0.smithi111.stdout:3/684: link d0/d2d/d35/d50/d6c/da3/daa/fd1 d0/d2d/d35/d69/de6/df4/ffd 0 2024-03-14T13:02:23.071 INFO:tasks.workunit.client.0.smithi111.stdout:1/734: mkdir d20/d25/d75/df7 0 2024-03-14T13:02:23.072 INFO:tasks.workunit.client.0.smithi111.stdout:7/594: creat d0/d1/d21/d2a/d8c/fb1 x:0 0 0 2024-03-14T13:02:23.074 INFO:tasks.workunit.client.0.smithi111.stdout:1/735: creat d20/d25/da8/ff8 x:0 0 0 2024-03-14T13:02:23.075 INFO:tasks.workunit.client.0.smithi111.stdout:7/595: fsync d0/d1/d5/d38/d1b/fa1 0 2024-03-14T13:02:23.076 INFO:tasks.workunit.client.0.smithi111.stdout:5/847: dwrite d2/d45/d79/fc5 [0,4194304] 0 2024-03-14T13:02:23.079 INFO:tasks.workunit.client.0.smithi111.stdout:7/596: creat d0/d1/d4a/d52/fb2 x:0 0 0 2024-03-14T13:02:23.080 INFO:tasks.workunit.client.0.smithi111.stdout:7/597: write d0/d1/d21/d5d/fac [179761,74406] 0 2024-03-14T13:02:23.080 INFO:tasks.workunit.client.0.smithi111.stdout:5/848: fsync d2/d16/d49/d7c/d95/fa3 0 2024-03-14T13:02:23.083 INFO:tasks.workunit.client.0.smithi111.stdout:7/598: symlink d0/d1/d5/d56/d70/lb3 0 2024-03-14T13:02:23.085 INFO:tasks.workunit.client.0.smithi111.stdout:7/599: unlink d0/d1/d22/l77 0 2024-03-14T13:02:23.090 INFO:tasks.workunit.client.1.smithi137.stdout:5/594: sync 2024-03-14T13:02:23.100 INFO:tasks.workunit.client.0.smithi111.stdout:1/736: read d20/d21/d3f/f47 [3190379,95948] 0 2024-03-14T13:02:23.100 INFO:tasks.workunit.client.1.smithi137.stdout:9/773: dwrite d6/d56/fb3 [0,4194304] 0 2024-03-14T13:02:23.102 INFO:tasks.workunit.client.1.smithi137.stdout:7/574: creat d3/d5/d23/d2c/d5d/d6a/d75/d78/da4/fb7 x:0 0 0 2024-03-14T13:02:23.105 INFO:tasks.workunit.client.1.smithi137.stdout:1/737: readlink d7/d9/d19/d40/le3 0 2024-03-14T13:02:23.106 INFO:tasks.workunit.client.0.smithi111.stdout:1/737: dwrite d20/d25/d75/f7d [0,4194304] 0 2024-03-14T13:02:23.107 INFO:tasks.workunit.client.1.smithi137.stdout:9/774: dread - d6/df/d88/de5/df6/da9/fd6 zero size 2024-03-14T13:02:23.108 INFO:tasks.workunit.client.0.smithi111.stdout:1/738: symlink d20/dca/lf9 0 2024-03-14T13:02:23.109 INFO:tasks.workunit.client.1.smithi137.stdout:2/734: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/ddf to d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/df0 0 2024-03-14T13:02:23.114 INFO:tasks.workunit.client.1.smithi137.stdout:5/595: dwrite d7/d49/d75/fb2 [0,4194304] 0 2024-03-14T13:02:23.115 INFO:tasks.workunit.client.0.smithi111.stdout:1/739: dwrite d20/dca/fe7 [0,4194304] 0 2024-03-14T13:02:23.118 INFO:tasks.workunit.client.0.smithi111.stdout:1/740: creat d20/d25/dd0/ffa x:0 0 0 2024-03-14T13:02:23.124 INFO:tasks.workunit.client.1.smithi137.stdout:2/735: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/l90 3573 1 2024-03-14T13:02:23.124 INFO:tasks.workunit.client.1.smithi137.stdout:5/596: write d7/d16/fbc [675526,16193] 0 2024-03-14T13:02:23.126 INFO:tasks.workunit.client.1.smithi137.stdout:7/575: getdents d3/d5/d23/d2c/d5d/d6a 0 2024-03-14T13:02:23.127 INFO:tasks.workunit.client.0.smithi111.stdout:8/921: write d3/dc/fa7 [1120147,58549] 0 2024-03-14T13:02:23.128 INFO:tasks.workunit.client.1.smithi137.stdout:7/576: chown d3/fe 0 1 2024-03-14T13:02:23.129 INFO:tasks.workunit.client.1.smithi137.stdout:1/738: dwrite d7/d9/d19/d40/f46 [0,4194304] 0 2024-03-14T13:02:23.133 INFO:tasks.workunit.client.0.smithi111.stdout:8/922: link d3/d1b/d20/d35/d40/d48/d7e/fba d3/d1b/dc1/de0/f133 0 2024-03-14T13:02:23.135 INFO:tasks.workunit.client.0.smithi111.stdout:5/849: dread d2/d16/d49/d44/d4a/d9e/db7/dbb/fd2 [0,4194304] 0 2024-03-14T13:02:23.135 INFO:tasks.workunit.client.0.smithi111.stdout:8/923: fsync d3/dc/d45/d8d/f12f 0 2024-03-14T13:02:23.135 INFO:tasks.workunit.client.1.smithi137.stdout:7/577: creat d3/d6/d64/d9e/fb8 x:0 0 0 2024-03-14T13:02:23.136 INFO:tasks.workunit.client.1.smithi137.stdout:5/597: rmdir d7/d33/d65/dc2/dc8 0 2024-03-14T13:02:23.137 INFO:tasks.workunit.client.1.smithi137.stdout:1/739: mkdir d7/d31/d54/ded/dfb 0 2024-03-14T13:02:23.138 INFO:tasks.workunit.client.0.smithi111.stdout:5/850: write d2/d13/f109 [4815572,60312] 0 2024-03-14T13:02:23.138 INFO:tasks.workunit.client.0.smithi111.stdout:3/685: truncate d0/d2d/d35/d4c/dad/dee/ff6 873907 0 2024-03-14T13:02:23.139 INFO:tasks.workunit.client.1.smithi137.stdout:7/578: chown d3/d5/d5a/d80/fae 5 1 2024-03-14T13:02:23.140 INFO:tasks.workunit.client.0.smithi111.stdout:5/851: rmdir d2/d16/d49 39 2024-03-14T13:02:23.141 INFO:tasks.workunit.client.1.smithi137.stdout:2/736: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/d41/f6e [0,4194304] 0 2024-03-14T13:02:23.141 INFO:tasks.workunit.client.0.smithi111.stdout:3/686: symlink d0/d30/lfe 0 2024-03-14T13:02:23.142 INFO:tasks.workunit.client.1.smithi137.stdout:7/579: chown d3/d5/d23/d2c/d5d/d6a/d75/d78 1997 1 2024-03-14T13:02:23.143 INFO:tasks.workunit.client.0.smithi111.stdout:5/852: symlink d2/d45/d79/l110 0 2024-03-14T13:02:23.146 INFO:tasks.workunit.client.1.smithi137.stdout:1/740: symlink d7/lfc 0 2024-03-14T13:02:23.146 INFO:tasks.workunit.client.1.smithi137.stdout:5/598: creat d7/d16/d18/d25/d28/d3c/fcb x:0 0 0 2024-03-14T13:02:23.147 INFO:tasks.workunit.client.1.smithi137.stdout:9/775: dread d6/df/d19/d1f/f66 [0,4194304] 0 2024-03-14T13:02:23.147 INFO:tasks.workunit.client.1.smithi137.stdout:1/741: read - d7/d9/d19/d1f/d2e/fe7 zero size 2024-03-14T13:02:23.147 INFO:tasks.workunit.client.1.smithi137.stdout:7/580: unlink d3/d6/d14/d27/c84 0 2024-03-14T13:02:23.148 INFO:tasks.workunit.client.0.smithi111.stdout:7/600: dread d0/d1/d21/f3f [4194304,4194304] 0 2024-03-14T13:02:23.150 INFO:tasks.workunit.client.1.smithi137.stdout:9/776: rmdir d6/df/d88/de5 39 2024-03-14T13:02:23.150 INFO:tasks.workunit.client.0.smithi111.stdout:3/687: fdatasync d0/d2d/d35/d4c/dad/dee/ff6 0 2024-03-14T13:02:23.151 INFO:tasks.workunit.client.0.smithi111.stdout:1/741: write fe [166549,32686] 0 2024-03-14T13:02:23.151 INFO:tasks.workunit.client.1.smithi137.stdout:1/742: creat d7/d9/d19/d1f/d26/d6c/ffd x:0 0 0 2024-03-14T13:02:23.152 INFO:tasks.workunit.client.0.smithi111.stdout:7/601: mkdir d0/d1/d5/d38/d5c/db4 0 2024-03-14T13:02:23.154 INFO:tasks.workunit.client.0.smithi111.stdout:7/602: write d0/d1/d5/d38/d5c/d6d/fa3 [910167,117830] 0 2024-03-14T13:02:23.156 INFO:tasks.workunit.client.0.smithi111.stdout:1/742: creat d20/d21/dd7/ffb x:0 0 0 2024-03-14T13:02:23.158 INFO:tasks.workunit.client.0.smithi111.stdout:5/853: getdents d2/d16/d49/d7c/d95 0 2024-03-14T13:02:23.158 INFO:tasks.workunit.client.1.smithi137.stdout:1/743: dread d7/d9/d19/d1f/d26/d7d/fd4 [0,4194304] 0 2024-03-14T13:02:23.160 INFO:tasks.workunit.client.1.smithi137.stdout:5/599: write d7/d16/d18/f2a [209024,71985] 0 2024-03-14T13:02:23.162 INFO:tasks.workunit.client.0.smithi111.stdout:1/743: truncate d20/d40/f84 1007467 0 2024-03-14T13:02:23.163 INFO:tasks.workunit.client.1.smithi137.stdout:1/744: unlink d7/d31/d54/c68 0 2024-03-14T13:02:23.163 INFO:tasks.workunit.client.1.smithi137.stdout:5/600: dread - d7/d49/d75/fb3 zero size 2024-03-14T13:02:23.163 INFO:tasks.workunit.client.0.smithi111.stdout:3/688: write d0/d2d/d35/d50/d6c/da3/daa/fd1 [50469,91735] 0 2024-03-14T13:02:23.167 INFO:tasks.workunit.client.1.smithi137.stdout:2/737: dwrite d2/d7/d2e/d3e/d44/d6b/f20 [0,4194304] 0 2024-03-14T13:02:23.167 INFO:tasks.workunit.client.0.smithi111.stdout:5/854: creat d2/d16/d49/d44/d4a/d9e/db7/dbb/f111 x:0 0 0 2024-03-14T13:02:23.168 INFO:tasks.workunit.client.0.smithi111.stdout:7/603: dwrite d0/d1/d22/f7b [4194304,4194304] 0 2024-03-14T13:02:23.170 INFO:tasks.workunit.client.1.smithi137.stdout:2/738: rename d2/d7/d2e/d3e/daa/de7 to d2/d7/d2e/d3e/daa/de7/df1 22 2024-03-14T13:02:23.170 INFO:tasks.workunit.client.0.smithi111.stdout:3/689: unlink d0/d30/f95 0 2024-03-14T13:02:23.171 INFO:tasks.workunit.client.0.smithi111.stdout:8/924: dwrite d3/dc/d31/f4b [0,4194304] 0 2024-03-14T13:02:23.172 INFO:tasks.workunit.client.1.smithi137.stdout:5/601: dwrite d7/d16/d18/d25/d28/f43 [0,4194304] 0 2024-03-14T13:02:23.172 INFO:tasks.workunit.client.1.smithi137.stdout:7/581: dread d3/d54/f61 [0,4194304] 0 2024-03-14T13:02:23.172 INFO:tasks.workunit.client.0.smithi111.stdout:5/855: read - d2/d16/d49/d7c/d95/d6a/f103 zero size 2024-03-14T13:02:23.174 INFO:tasks.workunit.client.1.smithi137.stdout:5/602: chown d7/d16/d18/d25/f5b 2686 1 2024-03-14T13:02:23.174 INFO:tasks.workunit.client.0.smithi111.stdout:7/604: mknod d0/d1/d5/d38/d5c/db4/cb5 0 2024-03-14T13:02:23.177 INFO:tasks.workunit.client.0.smithi111.stdout:1/744: dwrite d20/d21/d28/fe1 [0,4194304] 0 2024-03-14T13:02:23.177 INFO:tasks.workunit.client.0.smithi111.stdout:5/856: symlink d2/d16/d49/d4c/l112 0 2024-03-14T13:02:23.178 INFO:tasks.workunit.client.1.smithi137.stdout:2/739: creat d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/ff2 x:0 0 0 2024-03-14T13:02:23.178 INFO:tasks.workunit.client.1.smithi137.stdout:2/740: fdatasync d2/d34/d3c/f6d 0 2024-03-14T13:02:23.179 INFO:tasks.workunit.client.1.smithi137.stdout:7/582: unlink d3/d6/d14/d27/f7b 0 2024-03-14T13:02:23.182 INFO:tasks.workunit.client.1.smithi137.stdout:5/603: creat d7/d1d/d74/d83/dc4/fcc x:0 0 0 2024-03-14T13:02:23.182 INFO:tasks.workunit.client.0.smithi111.stdout:1/745: mknod d20/d21/dc1/cfc 0 2024-03-14T13:02:23.185 INFO:tasks.workunit.client.0.smithi111.stdout:3/690: dwrite d0/d2d/d35/d50/d6c/d98/fd6 [0,4194304] 0 2024-03-14T13:02:23.185 INFO:tasks.workunit.client.0.smithi111.stdout:8/925: dwrite d3/d1b/d20/d35/d40/fdc [0,4194304] 0 2024-03-14T13:02:23.186 INFO:tasks.workunit.client.1.smithi137.stdout:7/583: chown d3/d5/d23/la5 9500 1 2024-03-14T13:02:23.188 INFO:tasks.workunit.client.0.smithi111.stdout:3/691: dread - d0/d2d/d35/d69/fc4 zero size 2024-03-14T13:02:23.189 INFO:tasks.workunit.client.1.smithi137.stdout:1/745: dread d7/d9/d19/d40/fa8 [0,4194304] 0 2024-03-14T13:02:23.190 INFO:tasks.workunit.client.0.smithi111.stdout:8/926: creat d3/d1b/d20/d35/d40/d48/d7e/f134 x:0 0 0 2024-03-14T13:02:23.192 INFO:tasks.workunit.client.0.smithi111.stdout:1/746: dread d20/d21/d3f/d59/f74 [0,4194304] 0 2024-03-14T13:02:23.193 INFO:tasks.workunit.client.1.smithi137.stdout:7/584: rename d3/d6/lc to d3/d5/d23/d2c/d5d/d6a/d75/lb9 0 2024-03-14T13:02:23.193 INFO:tasks.workunit.client.0.smithi111.stdout:3/692: unlink d0/d2d/f86 0 2024-03-14T13:02:23.194 INFO:tasks.workunit.client.0.smithi111.stdout:8/927: symlink d3/d5/l135 0 2024-03-14T13:02:23.196 INFO:tasks.workunit.client.1.smithi137.stdout:7/585: symlink d3/d5/d5a/d80/lba 0 2024-03-14T13:02:23.197 INFO:tasks.workunit.client.1.smithi137.stdout:2/741: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/f9b [0,4194304] 0 2024-03-14T13:02:23.198 INFO:tasks.workunit.client.0.smithi111.stdout:1/747: mknod d20/d40/da6/cfd 0 2024-03-14T13:02:23.198 INFO:tasks.workunit.client.0.smithi111.stdout:3/693: fdatasync d0/d2d/d35/d50/f88 0 2024-03-14T13:02:23.199 INFO:tasks.workunit.client.0.smithi111.stdout:8/928: read d3/dc/f99 [1789007,61526] 0 2024-03-14T13:02:23.199 INFO:tasks.workunit.client.1.smithi137.stdout:1/746: creat d7/d9/d19/d40/ffe x:0 0 0 2024-03-14T13:02:23.200 INFO:tasks.workunit.client.1.smithi137.stdout:2/742: mkdir d2/d7/d2e/d3e/df3 0 2024-03-14T13:02:23.202 INFO:tasks.workunit.client.1.smithi137.stdout:7/586: creat d3/d5/d23/d2c/d5d/d6a/fbb x:0 0 0 2024-03-14T13:02:23.203 INFO:tasks.workunit.client.0.smithi111.stdout:8/929: fdatasync d3/d1b/f75 0 2024-03-14T13:02:23.204 INFO:tasks.workunit.client.1.smithi137.stdout:2/743: write d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/fa7 [196158,51751] 0 2024-03-14T13:02:23.204 INFO:tasks.workunit.client.1.smithi137.stdout:7/587: write d3/d5/f37 [3982321,44022] 0 2024-03-14T13:02:23.205 INFO:tasks.workunit.client.1.smithi137.stdout:1/747: creat d7/d31/d82/fff x:0 0 0 2024-03-14T13:02:23.207 INFO:tasks.workunit.client.1.smithi137.stdout:1/748: dread - d7/f9a zero size 2024-03-14T13:02:23.208 INFO:tasks.workunit.client.1.smithi137.stdout:2/744: write d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/ff2 [448161,105517] 0 2024-03-14T13:02:23.210 INFO:tasks.workunit.client.1.smithi137.stdout:7/588: getdents d3/d5 0 2024-03-14T13:02:23.211 INFO:tasks.workunit.client.0.smithi111.stdout:8/930: fsync d3/d1b/d20/d35/d40/d48/d7e/db3/db5/ff0 0 2024-03-14T13:02:23.211 INFO:tasks.workunit.client.0.smithi111.stdout:1/748: dwrite d20/d25/d31/fc6 [0,4194304] 0 2024-03-14T13:02:23.211 INFO:tasks.workunit.client.0.smithi111.stdout:3/694: dwrite d0/d2d/d35/d50/d5e/f97 [0,4194304] 0 2024-03-14T13:02:23.211 INFO:tasks.workunit.client.1.smithi137.stdout:1/749: fsync d7/d9/d19/d40/d42/d9d/de5/ff4 0 2024-03-14T13:02:23.214 INFO:tasks.workunit.client.0.smithi111.stdout:8/931: mknod d3/d5/d12d/c136 0 2024-03-14T13:02:23.214 INFO:tasks.workunit.client.1.smithi137.stdout:2/745: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/df4 0 2024-03-14T13:02:23.215 INFO:tasks.workunit.client.0.smithi111.stdout:1/749: mkdir d20/d25/da8/de9/dfe 0 2024-03-14T13:02:23.216 INFO:tasks.workunit.client.1.smithi137.stdout:2/746: creat d2/d7/d2e/ff5 x:0 0 0 2024-03-14T13:02:23.217 INFO:tasks.workunit.client.0.smithi111.stdout:3/695: symlink d0/d2d/d35/d69/de6/dd8/lff 0 2024-03-14T13:02:23.219 INFO:tasks.workunit.client.0.smithi111.stdout:8/932: creat d3/d1b/d34/f137 x:0 0 0 2024-03-14T13:02:23.220 INFO:tasks.workunit.client.0.smithi111.stdout:1/750: rename d20/d21/d28/da0 to d20/d25/d31/dff 0 2024-03-14T13:02:23.221 INFO:tasks.workunit.client.0.smithi111.stdout:1/751: write f1a [1844702,25898] 0 2024-03-14T13:02:23.221 INFO:tasks.workunit.client.0.smithi111.stdout:3/696: mkdir d0/d30/df1/d100 0 2024-03-14T13:02:23.222 INFO:tasks.workunit.client.0.smithi111.stdout:3/697: fdatasync d0/d2d/d35/fea 0 2024-03-14T13:02:23.223 INFO:tasks.workunit.client.1.smithi137.stdout:2/747: rename d2/d7/d2e/d3e/d44/d6b to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/df6 22 2024-03-14T13:02:23.223 INFO:tasks.workunit.client.0.smithi111.stdout:8/933: creat d3/d1b/d20/d35/d40/d48/dcd/f138 x:0 0 0 2024-03-14T13:02:23.223 INFO:tasks.workunit.client.1.smithi137.stdout:1/750: dwrite d7/d9/d19/d5f/f7b [0,4194304] 0 2024-03-14T13:02:23.224 INFO:tasks.workunit.client.0.smithi111.stdout:7/605: write d0/d7e/f3d [3593544,115685] 0 2024-03-14T13:02:23.224 INFO:tasks.workunit.client.0.smithi111.stdout:1/752: rmdir d20/d40/db4/ddf 39 2024-03-14T13:02:23.226 INFO:tasks.workunit.client.0.smithi111.stdout:8/934: symlink d3/d1b/dc1/l139 0 2024-03-14T13:02:23.226 INFO:tasks.workunit.client.1.smithi137.stdout:1/751: creat d7/d31/d54/ded/dfb/f100 x:0 0 0 2024-03-14T13:02:23.227 INFO:tasks.workunit.client.1.smithi137.stdout:2/748: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dda/ff7 x:0 0 0 2024-03-14T13:02:23.227 INFO:tasks.workunit.client.1.smithi137.stdout:1/752: chown d7/d9/d19/d1f/d26/d6c/ffd 61080526 1 2024-03-14T13:02:23.229 INFO:tasks.workunit.client.0.smithi111.stdout:5/857: truncate d2/d16/d49/d7c/d8b/f91 461889 0 2024-03-14T13:02:23.231 INFO:tasks.workunit.client.0.smithi111.stdout:8/935: dread d3/d1b/d20/d35/d40/fac [0,4194304] 0 2024-03-14T13:02:23.233 INFO:tasks.workunit.client.0.smithi111.stdout:8/936: fdatasync d3/d1b/d20/d35/d40/d48/d77/fe8 0 2024-03-14T13:02:23.233 INFO:tasks.workunit.client.0.smithi111.stdout:3/698: link d0/d2d/d35/fef d0/d2d/d35/d50/da2/db3/da7/f101 0 2024-03-14T13:02:23.235 INFO:tasks.workunit.client.0.smithi111.stdout:5/858: mknod d2/d45/d76/db2/c113 0 2024-03-14T13:02:23.235 INFO:tasks.workunit.client.1.smithi137.stdout:2/749: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/fdd [4194304,4194304] 0 2024-03-14T13:02:23.235 INFO:tasks.workunit.client.1.smithi137.stdout:1/753: write d7/d9/d19/d1f/d26/f48 [121430,70911] 0 2024-03-14T13:02:23.236 INFO:tasks.workunit.client.0.smithi111.stdout:7/606: getdents d0 0 2024-03-14T13:02:23.236 INFO:tasks.workunit.client.1.smithi137.stdout:1/754: read - d7/d9/d19/d1f/d89/fb2 zero size 2024-03-14T13:02:23.238 INFO:tasks.workunit.client.0.smithi111.stdout:8/937: mknod d3/dc/d31/c13a 0 2024-03-14T13:02:23.238 INFO:tasks.workunit.client.0.smithi111.stdout:7/607: read d0/d1/d22/d3e/f64 [106167,101732] 0 2024-03-14T13:02:23.239 INFO:tasks.workunit.client.0.smithi111.stdout:5/859: chown d2/d16/d49/d4c/d9c/l101 2 1 2024-03-14T13:02:23.241 INFO:tasks.workunit.client.1.smithi137.stdout:1/755: rename f5 to d7/d31/d54/f101 0 2024-03-14T13:02:23.242 INFO:tasks.workunit.client.1.smithi137.stdout:1/756: dread d7/d4d/d6b/faa [0,4194304] 0 2024-03-14T13:02:23.243 INFO:tasks.workunit.client.0.smithi111.stdout:3/699: creat d0/f102 x:0 0 0 2024-03-14T13:02:23.244 INFO:tasks.workunit.client.0.smithi111.stdout:1/753: dread d20/d25/f46 [0,4194304] 0 2024-03-14T13:02:23.245 INFO:tasks.workunit.client.0.smithi111.stdout:5/860: rename d2/d13/l30 to d2/d45/d79/l114 0 2024-03-14T13:02:23.246 INFO:tasks.workunit.client.1.smithi137.stdout:2/750: write d2/d34/f69 [913343,18260] 0 2024-03-14T13:02:23.247 INFO:tasks.workunit.client.0.smithi111.stdout:8/938: mkdir d3/dc/d31/d9d/d110/d111/d13b 0 2024-03-14T13:02:23.250 INFO:tasks.workunit.client.0.smithi111.stdout:5/861: mknod d2/d16/d49/c115 0 2024-03-14T13:02:23.252 INFO:tasks.workunit.client.0.smithi111.stdout:5/862: symlink d2/d16/d49/d7c/d95/d36/l116 0 2024-03-14T13:02:23.253 INFO:tasks.workunit.client.0.smithi111.stdout:1/754: link d20/d25/d31/c60 d20/d21/d28/c100 0 2024-03-14T13:02:23.253 INFO:tasks.workunit.client.1.smithi137.stdout:1/757: write d7/d9/d19/d1f/f2a [1534930,47957] 0 2024-03-14T13:02:23.255 INFO:tasks.workunit.client.1.smithi137.stdout:1/758: creat d7/d9/d19/d1f/d89/d90/f102 x:0 0 0 2024-03-14T13:02:23.259 INFO:tasks.workunit.client.0.smithi111.stdout:8/939: dwrite d3/d1b/f6f [0,4194304] 0 2024-03-14T13:02:23.268 INFO:tasks.workunit.client.1.smithi137.stdout:2/751: dread d2/d7/d2e/d3e/d44/d6b/d22/f91 [0,4194304] 0 2024-03-14T13:02:23.271 INFO:tasks.workunit.client.1.smithi137.stdout:2/752: mkdir d2/d7/df8 0 2024-03-14T13:02:23.275 INFO:tasks.workunit.client.1.smithi137.stdout:2/753: rmdir d2/d7/d2e/d3e/d44/d6b/d1d/d41/de1 0 2024-03-14T13:02:23.275 INFO:tasks.workunit.client.1.smithi137.stdout:2/754: readlink d2/d7/d2e/d3e/d44/lbe 0 2024-03-14T13:02:23.276 INFO:tasks.workunit.client.0.smithi111.stdout:8/940: fdatasync d3/d22/d30/f43 0 2024-03-14T13:02:23.282 INFO:tasks.workunit.client.0.smithi111.stdout:8/941: symlink d3/dc/d31/d9d/l13c 0 2024-03-14T13:02:23.283 INFO:tasks.workunit.client.0.smithi111.stdout:3/700: dread d0/f1 [0,4194304] 0 2024-03-14T13:02:23.284 INFO:tasks.workunit.client.0.smithi111.stdout:8/942: fdatasync d3/d22/d30/fa4 0 2024-03-14T13:02:23.284 INFO:tasks.workunit.client.0.smithi111.stdout:8/943: chown d3/dc 23242 1 2024-03-14T13:02:23.290 INFO:tasks.workunit.client.1.smithi137.stdout:2/755: dread d2/d7/d2e/d3e/d44/d6b/d22/dac/faf [0,4194304] 0 2024-03-14T13:02:23.292 INFO:tasks.workunit.client.1.smithi137.stdout:2/756: truncate d2/d7/d2e/d3e/d44/d6b/f39 4910495 0 2024-03-14T13:02:23.294 INFO:tasks.workunit.client.0.smithi111.stdout:5/863: write d2/d45/d76/d8d/f77 [892950,88270] 0 2024-03-14T13:02:23.294 INFO:tasks.workunit.client.0.smithi111.stdout:3/701: creat d0/d2d/d35/d50/d5e/f103 x:0 0 0 2024-03-14T13:02:23.297 INFO:tasks.workunit.client.1.smithi137.stdout:2/757: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 [0,4194304] 0 2024-03-14T13:02:23.297 INFO:tasks.workunit.client.0.smithi111.stdout:5/864: chown d2/d45/c2e 40297733 1 2024-03-14T13:02:23.297 INFO:tasks.workunit.client.0.smithi111.stdout:8/944: link d3/d1b/d20/d35/d40/d48/d7e/db3/cb0 d3/d1b/dc1/c13d 0 2024-03-14T13:02:23.299 INFO:tasks.workunit.client.1.smithi137.stdout:2/758: unlink d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f83 0 2024-03-14T13:02:23.300 INFO:tasks.workunit.client.0.smithi111.stdout:3/702: mknod d0/d2d/d35/d4c/c104 0 2024-03-14T13:02:23.300 INFO:tasks.workunit.client.0.smithi111.stdout:1/755: dwrite d20/d25/d75/f5e [4194304,4194304] 0 2024-03-14T13:02:23.301 INFO:tasks.workunit.client.0.smithi111.stdout:1/756: readlink d20/d21/d28/d5c/l67 0 2024-03-14T13:02:23.320 INFO:tasks.workunit.client.1.smithi137.stdout:3/586: sync 2024-03-14T13:02:23.323 INFO:tasks.workunit.client.1.smithi137.stdout:2/759: symlink d2/d7/d2e/d3e/d44/lf9 0 2024-03-14T13:02:23.330 INFO:tasks.workunit.client.1.smithi137.stdout:2/760: readlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/l88 0 2024-03-14T13:02:23.330 INFO:tasks.workunit.client.0.smithi111.stdout:1/757: dread d20/d21/da7/fb1 [0,4194304] 0 2024-03-14T13:02:23.331 INFO:tasks.workunit.client.1.smithi137.stdout:3/587: dwrite d4/df/d82/fc3 [4194304,4194304] 0 2024-03-14T13:02:23.332 INFO:tasks.workunit.client.0.smithi111.stdout:1/758: dread - d20/d25/d31/d80/fd1 zero size 2024-03-14T13:02:23.334 INFO:tasks.workunit.client.1.smithi137.stdout:2/761: unlink d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/fd5 0 2024-03-14T13:02:23.337 INFO:tasks.workunit.client.1.smithi137.stdout:2/762: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/cfa 0 2024-03-14T13:02:23.339 INFO:tasks.workunit.client.1.smithi137.stdout:3/588: link d4/df/d82/fb5 d4/d22/fc6 0 2024-03-14T13:02:23.339 INFO:tasks.workunit.client.1.smithi137.stdout:2/763: readlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/lce 0 2024-03-14T13:02:23.341 INFO:tasks.workunit.client.1.smithi137.stdout:7/589: dread d3/d5/f67 [0,4194304] 0 2024-03-14T13:02:23.343 INFO:tasks.workunit.client.1.smithi137.stdout:7/590: readlink d3/d6/l10 0 2024-03-14T13:02:23.347 INFO:tasks.workunit.client.1.smithi137.stdout:7/591: rmdir d3/d52/d77 39 2024-03-14T13:02:23.349 INFO:tasks.workunit.client.0.smithi111.stdout:1/759: fsync d20/d25/d31/fc6 0 2024-03-14T13:02:23.353 INFO:tasks.workunit.client.1.smithi137.stdout:3/589: dwrite d4/df/f96 [0,4194304] 0 2024-03-14T13:02:23.353 INFO:tasks.workunit.client.1.smithi137.stdout:2/764: dwrite d2/d34/d3c/d9d/fa1 [0,4194304] 0 2024-03-14T13:02:23.354 INFO:tasks.workunit.client.1.smithi137.stdout:7/592: read d3/d5/d23/d3f/f76 [2812521,68309] 0 2024-03-14T13:02:23.356 INFO:tasks.workunit.client.1.smithi137.stdout:5/604: dread d7/f1b [0,4194304] 0 2024-03-14T13:02:23.359 INFO:tasks.workunit.client.1.smithi137.stdout:7/593: read d3/d6/d14/f1a [3345791,22278] 0 2024-03-14T13:02:23.361 INFO:tasks.workunit.client.1.smithi137.stdout:2/765: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f73 [559282,88616] 0 2024-03-14T13:02:23.361 INFO:tasks.workunit.client.1.smithi137.stdout:5/605: write d7/d1d/d74/f78 [1271470,47002] 0 2024-03-14T13:02:23.365 INFO:tasks.workunit.client.1.smithi137.stdout:5/606: chown d7/d16/d18/d25/d28/d3c/f6f 18 1 2024-03-14T13:02:23.366 INFO:tasks.workunit.client.1.smithi137.stdout:5/607: fdatasync d7/d1d/d74/d81/f89 0 2024-03-14T13:02:23.366 INFO:tasks.workunit.client.1.smithi137.stdout:2/766: rmdir d2/d7/d2e/d3e/d44/d6b/d1d/d2d 39 2024-03-14T13:02:23.369 INFO:tasks.workunit.client.1.smithi137.stdout:2/767: write d2/d34/fcb [488374,123026] 0 2024-03-14T13:02:23.369 INFO:tasks.workunit.client.1.smithi137.stdout:3/590: read d4/de/d13/f21 [2688779,36270] 0 2024-03-14T13:02:23.372 INFO:tasks.workunit.client.1.smithi137.stdout:3/591: mkdir d4/de/dc7 0 2024-03-14T13:02:23.373 INFO:tasks.workunit.client.1.smithi137.stdout:2/768: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/cfb 0 2024-03-14T13:02:23.376 INFO:tasks.workunit.client.1.smithi137.stdout:2/769: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf 0 2024-03-14T13:02:23.377 INFO:tasks.workunit.client.1.smithi137.stdout:2/770: write d2/d7/d2e/d3e/d44/d6b/fb5 [1719518,71804] 0 2024-03-14T13:02:23.380 INFO:tasks.workunit.client.1.smithi137.stdout:2/771: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/lfc 0 2024-03-14T13:02:23.382 INFO:tasks.workunit.client.1.smithi137.stdout:2/772: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/f9b 0 2024-03-14T13:02:23.392 INFO:tasks.workunit.client.1.smithi137.stdout:5/608: dread d7/d49/d75/f80 [0,4194304] 0 2024-03-14T13:02:23.393 INFO:tasks.workunit.client.1.smithi137.stdout:2/773: link d2/d7/d2e/d3e/d44/d6b/d1d/f48 d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/ffd 0 2024-03-14T13:02:23.393 INFO:tasks.workunit.client.0.smithi111.stdout:3/703: unlink d0/d2d/d35/d50/da2/db3/f47 0 2024-03-14T13:02:23.399 INFO:tasks.workunit.client.1.smithi137.stdout:2/774: link d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f93 d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/ffe 0 2024-03-14T13:02:23.399 INFO:tasks.workunit.client.1.smithi137.stdout:5/609: dwrite d7/d36/f95 [0,4194304] 0 2024-03-14T13:02:23.400 INFO:tasks.workunit.client.0.smithi111.stdout:8/945: write d3/f3e [180292,49325] 0 2024-03-14T13:02:23.401 INFO:tasks.workunit.client.1.smithi137.stdout:5/610: rmdir d7/d36 39 2024-03-14T13:02:23.402 INFO:tasks.workunit.client.0.smithi111.stdout:8/946: chown d3/d1b/cc2 506846666 1 2024-03-14T13:02:23.404 INFO:tasks.workunit.client.0.smithi111.stdout:1/760: creat d20/d25/d75/df7/f101 x:0 0 0 2024-03-14T13:02:23.404 INFO:tasks.workunit.client.1.smithi137.stdout:5/611: write d7/d16/d18/f66 [3063244,58289] 0 2024-03-14T13:02:23.405 INFO:tasks.workunit.client.1.smithi137.stdout:2/775: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/fd9 [53243,111504] 0 2024-03-14T13:02:23.405 INFO:tasks.workunit.client.0.smithi111.stdout:3/704: symlink d0/d2d/d35/l105 0 2024-03-14T13:02:23.405 INFO:tasks.workunit.client.1.smithi137.stdout:3/592: write d4/d19/f75 [548345,9330] 0 2024-03-14T13:02:23.409 INFO:tasks.workunit.client.0.smithi111.stdout:1/761: truncate d20/d25/da8/ff8 109772 0 2024-03-14T13:02:23.410 INFO:tasks.workunit.client.1.smithi137.stdout:7/594: dwrite d3/d5/f46 [0,4194304] 0 2024-03-14T13:02:23.411 INFO:tasks.workunit.client.0.smithi111.stdout:3/705: creat d0/d2d/d35/d4c/f106 x:0 0 0 2024-03-14T13:02:23.413 INFO:tasks.workunit.client.1.smithi137.stdout:7/595: readlink d3/d5/l72 0 2024-03-14T13:02:23.415 INFO:tasks.workunit.client.0.smithi111.stdout:8/947: rename d3/d22/d30/l41 to d3/dc/d45/l13e 0 2024-03-14T13:02:23.416 INFO:tasks.workunit.client.1.smithi137.stdout:7/596: readlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:23.417 INFO:tasks.workunit.client.0.smithi111.stdout:1/762: read d20/d40/f54 [1913397,42679] 0 2024-03-14T13:02:23.419 INFO:tasks.workunit.client.1.smithi137.stdout:8/720: sync 2024-03-14T13:02:23.419 INFO:tasks.workunit.client.1.smithi137.stdout:0/663: sync 2024-03-14T13:02:23.419 INFO:tasks.workunit.client.1.smithi137.stdout:2/776: dwrite d2/d34/d3c/f6d [0,4194304] 0 2024-03-14T13:02:23.420 INFO:tasks.workunit.client.1.smithi137.stdout:3/593: rename d4/de/l6e to d4/d19/lc8 0 2024-03-14T13:02:23.422 INFO:tasks.workunit.client.0.smithi111.stdout:1/763: fdatasync d20/d25/d31/fc6 0 2024-03-14T13:02:23.422 INFO:tasks.workunit.client.0.smithi111.stdout:5/865: dwrite d2/d45/d76/fab [0,4194304] 0 2024-03-14T13:02:23.422 INFO:tasks.workunit.client.1.smithi137.stdout:4/654: sync 2024-03-14T13:02:23.422 INFO:tasks.workunit.client.1.smithi137.stdout:6/674: sync 2024-03-14T13:02:23.424 INFO:tasks.workunit.client.1.smithi137.stdout:7/597: stat d3/d6/d2e/f3e 0 2024-03-14T13:02:23.426 INFO:tasks.workunit.client.0.smithi111.stdout:8/948: rmdir d3/d22 39 2024-03-14T13:02:23.427 INFO:tasks.workunit.client.1.smithi137.stdout:2/777: mkdir d2/d7/d2e/d3e/dff 0 2024-03-14T13:02:23.428 INFO:tasks.workunit.client.1.smithi137.stdout:6/675: mknod d2/dd6/ce5 0 2024-03-14T13:02:23.430 INFO:tasks.workunit.client.0.smithi111.stdout:1/764: mkdir d20/d40/da6/d102 0 2024-03-14T13:02:23.433 INFO:tasks.workunit.client.0.smithi111.stdout:3/706: truncate d0/d2d/d35/d69/de6/f9e 119728 0 2024-03-14T13:02:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/2941772526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006f9d0 msgr2=0x7ff1b006e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/2941772526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006f9d0 0x7ff1b006e460 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0004940 tx=0x7ff1a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:23.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/2941772526 shutdown_connections 2024-03-14T13:02:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/2941772526 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1b006e9a0 0x7ff1b006ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/2941772526 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006f9d0 0x7ff1b006e460 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0004940 tx=0x7ff1a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.438+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/2941772526 >> 172.21.15.111:0/2941772526 conn(0x7ff1b0069de0 msgr2=0x7ff1b006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:23.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.441+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/2941772526 shutdown_connections 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.442+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/2941772526 wait complete. 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.442+0000 7ff1b7c6e700 1 Processor -- start 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.442+0000 7ff1b7c6e700 1 -- start start 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.442+0000 7ff1b7c6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b7c6e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1b0114550 0x7ff1b0119580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b7c6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0114a10 con 0x7ff1b006e9a0 2024-03-14T13:02:23.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b7c6e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b0114b50 con 0x7ff1b0114550 2024-03-14T13:02:23.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b5a0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:23.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b5a0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43192/0 (socket says 172.21.15.111:43192) 2024-03-14T13:02:23.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.443+0000 7ff1b5a0a700 1 -- 172.21.15.111:0/3815377387 learned_addr learned my addr 172.21.15.111:0/3815377387 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:23.444 INFO:tasks.workunit.client.1.smithi137.stdout:5/612: write d7/fb [574179,57380] 0 2024-03-14T13:02:23.444 INFO:tasks.workunit.client.1.smithi137.stdout:0/664: creat d1/d5/d1b/d42/d3a/fe0 x:0 0 0 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:4/655: fsync d4/d9/d3a/d41/f21 0 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:6/676: mknod d2/d6/de/ce6 0 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:8/721: creat d1/ff2 x:0 0 0 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:9/777: sync 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:1/759: sync 2024-03-14T13:02:23.445 INFO:tasks.workunit.client.1.smithi137.stdout:0/665: write d1/f51 [2272856,2252] 0 2024-03-14T13:02:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.444+0000 7ff1b5a0a700 1 -- 172.21.15.111:0/3815377387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1b0114550 msgr2=0x7ff1b0119580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.444+0000 7ff1b5a0a700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1b0114550 0x7ff1b0119580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.444+0000 7ff1b5a0a700 1 -- 172.21.15.111:0/3815377387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a0005040 con 0x7ff1b006e9a0 2024-03-14T13:02:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.445+0000 7ff1b5a0a700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000b900 tx=0x7ff1a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:23.446 INFO:tasks.workunit.client.0.smithi111.stdout:5/866: dwrite d2/d45/d76/fab [0,4194304] 0 2024-03-14T13:02:23.446 INFO:tasks.workunit.client.1.smithi137.stdout:8/722: write d1/f31 [4388289,54169] 0 2024-03-14T13:02:23.447 INFO:tasks.workunit.client.1.smithi137.stdout:6/677: dread - d2/d6/de/d45/d59/d39/f7a zero size 2024-03-14T13:02:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.446+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a000e040 con 0x7ff1b006e9a0 2024-03-14T13:02:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.446+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1a0013420 con 0x7ff1b006e9a0 2024-03-14T13:02:23.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.446+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a001c630 con 0x7ff1b006e9a0 2024-03-14T13:02:23.447 INFO:tasks.workunit.client.1.smithi137.stdout:4/656: stat d4/d9/d3a/d41/d38/l74 0 2024-03-14T13:02:23.447 INFO:tasks.workunit.client.1.smithi137.stdout:7/598: dwrite d3/d5/d23/d2c/d5d/d6a/fbb [0,4194304] 0 2024-03-14T13:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.446+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b0119ac0 con 0x7ff1b006e9a0 2024-03-14T13:02:23.449 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.446+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b011a030 con 0x7ff1b006e9a0 2024-03-14T13:02:23.449 INFO:tasks.workunit.client.0.smithi111.stdout:5/867: dread - d2/d16/d49/d44/fa7 zero size 2024-03-14T13:02:23.449 INFO:tasks.workunit.client.0.smithi111.stdout:1/765: dwrite d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:23.449 INFO:tasks.workunit.client.1.smithi137.stdout:6/678: fsync d2/d6/de/d45/d59/f40 0 2024-03-14T13:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.448+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7ff1a0013a20 con 0x7ff1b006e9a0 2024-03-14T13:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.449+0000 7ff1a67fc700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 0x7ff19c06ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.449+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff1a008c940 con 0x7ff1b006e9a0 2024-03-14T13:02:23.451 INFO:tasks.workunit.client.1.smithi137.stdout:3/594: write d4/de/d13/f9f [247911,24545] 0 2024-03-14T13:02:23.451 INFO:tasks.workunit.client.1.smithi137.stdout:0/666: truncate d1/d5/d1b/d1e/d25/fa8 750937 0 2024-03-14T13:02:23.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.452+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b0109ad0 con 0x7ff1b006e9a0 2024-03-14T13:02:23.452 INFO:tasks.workunit.client.0.smithi111.stdout:1/766: mkdir d20/d25/d31/d80/df5/d103 0 2024-03-14T13:02:23.456 INFO:tasks.workunit.client.0.smithi111.stdout:1/767: write d20/d25/f46 [500450,114653] 0 2024-03-14T13:02:23.458 INFO:tasks.workunit.client.0.smithi111.stdout:1/768: chown d20/d40/f54 1039746388 1 2024-03-14T13:02:23.458 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.453+0000 7ff1b5209700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 0x7ff19c06ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:23.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.454+0000 7ff1b5209700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 0x7ff19c06ea20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac004d00 tx=0x7ff1ac009ce0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:23.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.459+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7ff1a005a9f0 con 0x7ff1b006e9a0 2024-03-14T13:02:23.460 INFO:tasks.workunit.client.1.smithi137.stdout:5/613: write d7/d36/f82 [2272206,43063] 0 2024-03-14T13:02:23.461 INFO:tasks.workunit.client.1.smithi137.stdout:5/614: fdatasync d7/d16/d18/d25/f6b 0 2024-03-14T13:02:23.461 INFO:tasks.workunit.client.1.smithi137.stdout:5/615: stat d7/d16/d18/d25/d28/d9e 0 2024-03-14T13:02:23.464 INFO:tasks.workunit.client.1.smithi137.stdout:0/667: fsync d1/d44/d56/f5a 0 2024-03-14T13:02:23.465 INFO:tasks.workunit.client.1.smithi137.stdout:7/599: truncate d3/d5/d23/d2c/d5d/f92 163834 0 2024-03-14T13:02:23.466 INFO:tasks.workunit.client.1.smithi137.stdout:4/657: chown d4/d4d/l60 333679482 1 2024-03-14T13:02:23.467 INFO:tasks.workunit.client.1.smithi137.stdout:7/600: chown d3/d5/d23/d2c/la0 25044609 1 2024-03-14T13:02:23.469 INFO:tasks.workunit.client.1.smithi137.stdout:3/595: truncate d4/de/d16/d1c/d5d/f7c 1010639 0 2024-03-14T13:02:23.470 INFO:tasks.workunit.client.1.smithi137.stdout:5/616: mknod d7/d1d/d74/ccd 0 2024-03-14T13:02:23.471 INFO:tasks.workunit.client.1.smithi137.stdout:1/760: creat d7/d9/d19/d40/d42/f103 x:0 0 0 2024-03-14T13:02:23.475 INFO:tasks.workunit.client.1.smithi137.stdout:4/658: mknod d4/d27/d5c/d6d/cdc 0 2024-03-14T13:02:23.476 INFO:tasks.workunit.client.1.smithi137.stdout:6/679: creat d2/d54/d55/d6d/d76/fe7 x:0 0 0 2024-03-14T13:02:23.476 INFO:tasks.workunit.client.1.smithi137.stdout:2/778: truncate d2/d7/d2e/d3e/d44/d6b/f20 715924 0 2024-03-14T13:02:23.478 INFO:tasks.workunit.client.1.smithi137.stdout:9/778: write d6/df/d71/f8b [1674217,100692] 0 2024-03-14T13:02:23.479 INFO:tasks.workunit.client.1.smithi137.stdout:5/617: creat d7/d36/fce x:0 0 0 2024-03-14T13:02:23.482 INFO:tasks.workunit.client.1.smithi137.stdout:3/596: dwrite d4/de/f3c [0,4194304] 0 2024-03-14T13:02:23.484 INFO:tasks.workunit.client.1.smithi137.stdout:4/659: mknod d4/d10/d42/d73/cdd 0 2024-03-14T13:02:23.485 INFO:tasks.workunit.client.1.smithi137.stdout:6/680: write d2/d6/da7/fd4 [143746,108150] 0 2024-03-14T13:02:23.485 INFO:tasks.workunit.client.1.smithi137.stdout:8/723: dread d1/dc/fe0 [0,4194304] 0 2024-03-14T13:02:23.487 INFO:tasks.workunit.client.1.smithi137.stdout:5/618: truncate d7/d16/d18/d25/d28/d3c/d7a/fc0 5006927 0 2024-03-14T13:02:23.488 INFO:tasks.workunit.client.1.smithi137.stdout:7/601: write d3/d5/d5a/f73 [1687587,20777] 0 2024-03-14T13:02:23.489 INFO:tasks.workunit.client.1.smithi137.stdout:2/779: creat d2/d7/d2e/d3e/d44/d6b/d22/d26/f100 x:0 0 0 2024-03-14T13:02:23.490 INFO:tasks.workunit.client.1.smithi137.stdout:8/724: chown d1/dc/d77/d9f/de4/f16 608 1 2024-03-14T13:02:23.490 INFO:tasks.workunit.client.1.smithi137.stdout:4/660: chown d4/d9/d3a/d41/c22 0 1 2024-03-14T13:02:23.492 INFO:tasks.workunit.client.1.smithi137.stdout:3/597: mknod d4/de/d93/cc9 0 2024-03-14T13:02:23.492 INFO:tasks.workunit.client.1.smithi137.stdout:9/779: mkdir d6/df/d88/df4/dfb 0 2024-03-14T13:02:23.494 INFO:tasks.workunit.client.1.smithi137.stdout:0/668: link d1/f46 d1/d5/d1b/d1e/fe1 0 2024-03-14T13:02:23.494 INFO:tasks.workunit.client.1.smithi137.stdout:2/780: readlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/l6c 0 2024-03-14T13:02:23.496 INFO:tasks.workunit.client.1.smithi137.stdout:4/661: readlink d4/d9/l1d 0 2024-03-14T13:02:23.499 INFO:tasks.workunit.client.1.smithi137.stdout:0/669: dread - d1/d44/d72/d7e/fcb zero size 2024-03-14T13:02:23.500 INFO:tasks.workunit.client.1.smithi137.stdout:7/602: creat d3/d6/d59/fbc x:0 0 0 2024-03-14T13:02:23.500 INFO:tasks.workunit.client.1.smithi137.stdout:6/681: dread d2/d54/d55/d33/fcc [0,4194304] 0 2024-03-14T13:02:23.502 INFO:tasks.workunit.client.1.smithi137.stdout:8/725: mknod d1/dc/cf3 0 2024-03-14T13:02:23.505 INFO:tasks.workunit.client.1.smithi137.stdout:5/619: creat d7/d16/d18/d25/d28/d3c/fcf x:0 0 0 2024-03-14T13:02:23.506 INFO:tasks.workunit.client.1.smithi137.stdout:8/726: fsync d1/dc/d77/d9f/fcc 0 2024-03-14T13:02:23.507 INFO:tasks.workunit.client.1.smithi137.stdout:4/662: creat d4/d9/d3a/d70/d80/db7/fde x:0 0 0 2024-03-14T13:02:23.508 INFO:tasks.workunit.client.1.smithi137.stdout:1/761: dwrite d7/d9/d19/d1f/d2e/f34 [0,4194304] 0 2024-03-14T13:02:23.511 INFO:tasks.workunit.client.1.smithi137.stdout:7/603: dread - d3/d52/d77/f8d zero size 2024-03-14T13:02:23.511 INFO:tasks.workunit.client.1.smithi137.stdout:4/663: chown d4/d9/d3a/d70 275 1 2024-03-14T13:02:23.515 INFO:tasks.workunit.client.1.smithi137.stdout:4/664: fdatasync d4/f3c 0 2024-03-14T13:02:23.515 INFO:tasks.workunit.client.1.smithi137.stdout:5/620: mkdir d7/d16/d18/d25/d28/d9e/dd0 0 2024-03-14T13:02:23.517 INFO:tasks.workunit.client.0.smithi111.stdout:1/769: dread d20/f24 [0,4194304] 0 2024-03-14T13:02:23.519 INFO:tasks.workunit.client.0.smithi111.stdout:1/770: symlink d20/d40/da6/l104 0 2024-03-14T13:02:23.520 INFO:tasks.workunit.client.0.smithi111.stdout:1/771: readlink d20/l26 0 2024-03-14T13:02:23.522 INFO:tasks.workunit.client.0.smithi111.stdout:8/949: dread d3/d1b/d20/d35/d40/d48/f50 [0,4194304] 0 2024-03-14T13:02:23.523 INFO:tasks.workunit.client.0.smithi111.stdout:1/772: creat d20/d25/d31/dff/f105 x:0 0 0 2024-03-14T13:02:23.526 INFO:tasks.workunit.client.0.smithi111.stdout:8/950: mknod d3/d1b/d20/d35/d40/d48/c13f 0 2024-03-14T13:02:23.527 INFO:tasks.workunit.client.1.smithi137.stdout:1/762: link d7/d9/d19/d1f/d26/d7d/lf3 d7/d9/d19/d1f/d2e/l104 0 2024-03-14T13:02:23.528 INFO:tasks.workunit.client.1.smithi137.stdout:4/665: write d4/d10/d42/f75 [4984253,55340] 0 2024-03-14T13:02:23.529 INFO:tasks.workunit.client.0.smithi111.stdout:8/951: creat d3/dc/d45/d11a/f140 x:0 0 0 2024-03-14T13:02:23.529 INFO:tasks.workunit.client.0.smithi111.stdout:1/773: dwrite d20/d25/f3d [0,4194304] 0 2024-03-14T13:02:23.530 INFO:tasks.workunit.client.1.smithi137.stdout:3/598: dread d4/de/d16/f31 [0,4194304] 0 2024-03-14T13:02:23.530 INFO:tasks.workunit.client.1.smithi137.stdout:9/780: dwrite d6/df/f51 [0,4194304] 0 2024-03-14T13:02:23.530 INFO:tasks.workunit.client.1.smithi137.stdout:5/621: rmdir d7/d16/d18/d25/d28/d3c/d7a 39 2024-03-14T13:02:23.531 INFO:tasks.workunit.client.1.smithi137.stdout:7/604: rename d3/d5/d23/d2c/d5d/d6a/da1/cb6 to d3/d6/cbd 0 2024-03-14T13:02:23.533 INFO:tasks.workunit.client.1.smithi137.stdout:2/781: link d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/l88 d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/dd7/l101 0 2024-03-14T13:02:23.537 INFO:tasks.workunit.client.1.smithi137.stdout:4/666: stat d4/d9/l1c 0 2024-03-14T13:02:23.540 INFO:tasks.workunit.client.1.smithi137.stdout:7/605: chown d3/d6/l10 59347 1 2024-03-14T13:02:23.540 INFO:tasks.workunit.client.0.smithi111.stdout:8/952: creat d3/d1b/d20/f141 x:0 0 0 2024-03-14T13:02:23.541 INFO:tasks.workunit.client.1.smithi137.stdout:1/763: mknod d7/d9/d19/d1f/d2e/d63/c105 0 2024-03-14T13:02:23.542 INFO:tasks.workunit.client.1.smithi137.stdout:8/727: dwrite d1/dc/d77/d9f/de4/fab [0,4194304] 0 2024-03-14T13:02:23.542 INFO:tasks.workunit.client.1.smithi137.stdout:6/682: dwrite d2/d54/d55/d6d/d76/d57/d58/f98 [4194304,4194304] 0 2024-03-14T13:02:23.543 INFO:tasks.workunit.client.0.smithi111.stdout:1/774: link d20/d21/d3f/d59/l72 d20/d21/l106 0 2024-03-14T13:02:23.544 INFO:tasks.workunit.client.0.smithi111.stdout:1/775: write f19 [3197815,116745] 0 2024-03-14T13:02:23.545 INFO:tasks.workunit.client.0.smithi111.stdout:8/953: unlink d3/d1b/d20/d35/d40/d48/d77/ca8 0 2024-03-14T13:02:23.546 INFO:tasks.workunit.client.1.smithi137.stdout:5/622: truncate d7/d16/f38 5110263 0 2024-03-14T13:02:23.549 INFO:tasks.workunit.client.1.smithi137.stdout:1/764: dread - d7/d9/d19/d1f/d89/d90/db8/fbd zero size 2024-03-14T13:02:23.549 INFO:tasks.workunit.client.1.smithi137.stdout:5/623: write d7/d33/d65/f7c [322732,126460] 0 2024-03-14T13:02:23.549 INFO:tasks.workunit.client.0.smithi111.stdout:8/954: write d3/d1b/d20/d35/d40/fed [4122656,30666] 0 2024-03-14T13:02:23.550 INFO:tasks.workunit.client.1.smithi137.stdout:0/670: dread d1/d5/d1b/d42/f32 [0,4194304] 0 2024-03-14T13:02:23.550 INFO:tasks.workunit.client.1.smithi137.stdout:4/667: write d4/d10/f3f [1042956,105386] 0 2024-03-14T13:02:23.551 INFO:tasks.workunit.client.0.smithi111.stdout:8/955: write d3/d5/fd [6508965,53627] 0 2024-03-14T13:02:23.552 INFO:tasks.workunit.client.1.smithi137.stdout:9/781: creat d6/df/d88/df4/ffc x:0 0 0 2024-03-14T13:02:23.553 INFO:tasks.workunit.client.1.smithi137.stdout:3/599: creat d4/de/d16/d1c/d5d/d85/d6b/fca x:0 0 0 2024-03-14T13:02:23.556 INFO:tasks.workunit.client.1.smithi137.stdout:7/606: rename d3/d6/d14/cb0 to d3/d5/d23/d2c/d5d/d6a/d75/cbe 0 2024-03-14T13:02:23.560 INFO:tasks.workunit.client.1.smithi137.stdout:0/671: read d1/d44/d56/f5a [101448,68535] 0 2024-03-14T13:02:23.561 INFO:tasks.workunit.client.0.smithi111.stdout:8/956: link d3/dc/f94 d3/dc/d31/d9d/f142 0 2024-03-14T13:02:23.562 INFO:tasks.workunit.client.1.smithi137.stdout:9/782: chown d6/df/d88/de5/df6/f67 1098702418 1 2024-03-14T13:02:23.562 INFO:tasks.workunit.client.1.smithi137.stdout:6/683: rename d2/d54/d55/d6d/d76/d57/lcb to d2/d6/de/d45/d59/d39/d3c/d83/le8 0 2024-03-14T13:02:23.565 INFO:tasks.workunit.client.0.smithi111.stdout:1/776: truncate d20/d25/f46 7217521 0 2024-03-14T13:02:23.567 INFO:tasks.workunit.client.0.smithi111.stdout:8/957: truncate d3/d1b/d20/d35/d40/d48/dcd/fe3 877876 0 2024-03-14T13:02:23.567 INFO:tasks.workunit.client.0.smithi111.stdout:1/777: chown d20/d25/d31/d80/l9c 153 1 2024-03-14T13:02:23.568 INFO:tasks.workunit.client.0.smithi111.stdout:1/778: stat d20/d25/da8/de9/dfe 0 2024-03-14T13:02:23.568 INFO:tasks.workunit.client.1.smithi137.stdout:6/684: symlink d2/d6/da7/le9 0 2024-03-14T13:02:23.569 INFO:tasks.workunit.client.0.smithi111.stdout:8/958: write d3/d1b/d20/d35/d40/d48/d7e/db3/db5/f119 [127083,66130] 0 2024-03-14T13:02:23.570 INFO:tasks.workunit.client.1.smithi137.stdout:8/728: write d1/dc/d4e/d6d/f8b [874219,28927] 0 2024-03-14T13:02:23.571 INFO:tasks.workunit.client.0.smithi111.stdout:0/829: sync 2024-03-14T13:02:23.571 INFO:tasks.workunit.client.0.smithi111.stdout:2/611: sync 2024-03-14T13:02:23.572 INFO:tasks.workunit.client.0.smithi111.stdout:1/779: write d20/d21/d28/d5c/f93 [1029658,28432] 0 2024-03-14T13:02:23.573 INFO:tasks.workunit.client.1.smithi137.stdout:5/624: dwrite d7/f7b [0,4194304] 0 2024-03-14T13:02:23.573 INFO:tasks.workunit.client.1.smithi137.stdout:1/765: write d7/d4d/d6b/faa [1009456,115222] 0 2024-03-14T13:02:23.577 INFO:tasks.workunit.client.1.smithi137.stdout:0/672: rmdir d1/d5/d1b/d42/d3a/db0 39 2024-03-14T13:02:23.577 INFO:tasks.workunit.client.1.smithi137.stdout:4/668: dwrite d4/d10/d5a/f71 [0,4194304] 0 2024-03-14T13:02:23.577 INFO:tasks.workunit.client.1.smithi137.stdout:5/625: readlink d7/d1d/l32 0 2024-03-14T13:02:23.578 INFO:tasks.workunit.client.0.smithi111.stdout:8/959: mknod d3/d22/c143 0 2024-03-14T13:02:23.580 INFO:tasks.workunit.client.1.smithi137.stdout:1/766: fdatasync d7/d29/fc9 0 2024-03-14T13:02:23.580 INFO:tasks.workunit.client.0.smithi111.stdout:8/960: readlink d3/d1b/d34/l44 0 2024-03-14T13:02:23.581 INFO:tasks.workunit.client.1.smithi137.stdout:6/685: chown d2/d54/l4b 0 1 2024-03-14T13:02:23.581 INFO:tasks.workunit.client.1.smithi137.stdout:7/607: dwrite d3/d6/d59/f5f [0,4194304] 0 2024-03-14T13:02:23.581 INFO:tasks.workunit.client.1.smithi137.stdout:2/782: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f5f 1612689 0 2024-03-14T13:02:23.583 INFO:tasks.workunit.client.0.smithi111.stdout:8/961: stat d3/dc/d45/d11a/f5c 0 2024-03-14T13:02:23.584 INFO:tasks.workunit.client.1.smithi137.stdout:8/729: creat d1/dc/d3c/d87/d34/d82/d94/ff4 x:0 0 0 2024-03-14T13:02:23.585 INFO:tasks.workunit.client.0.smithi111.stdout:1/780: write d20/d25/f34 [948738,61295] 0 2024-03-14T13:02:23.586 INFO:tasks.workunit.client.1.smithi137.stdout:9/783: link d6/d53/f99 d6/df/d88/de5/df6/da9/dde/ffd 0 2024-03-14T13:02:23.588 INFO:tasks.workunit.client.0.smithi111.stdout:8/962: mkdir d3/dc/d31/d9d/d110/d144 0 2024-03-14T13:02:23.590 INFO:tasks.workunit.client.1.smithi137.stdout:5/626: mkdir d7/d16/d18/d2d/dd1 0 2024-03-14T13:02:23.591 INFO:tasks.workunit.client.1.smithi137.stdout:0/673: write d1/d5/d1b/d1e/d25/d55/f60 [488458,124877] 0 2024-03-14T13:02:23.592 INFO:tasks.workunit.client.0.smithi111.stdout:1/781: creat d20/d25/d75/df7/f107 x:0 0 0 2024-03-14T13:02:23.593 INFO:tasks.workunit.client.1.smithi137.stdout:9/784: chown d6/d26/l4d 39 1 2024-03-14T13:02:23.594 INFO:tasks.workunit.client.0.smithi111.stdout:8/963: creat d3/d1b/d20/d35/d40/d48/d77/f145 x:0 0 0 2024-03-14T13:02:23.595 INFO:tasks.workunit.client.0.smithi111.stdout:8/964: fdatasync d3/d22/d30/fa4 0 2024-03-14T13:02:23.596 INFO:tasks.workunit.client.1.smithi137.stdout:1/767: truncate d7/d9/d19/d1f/f97 47535 0 2024-03-14T13:02:23.597 INFO:tasks.workunit.client.0.smithi111.stdout:2/612: dwrite dd/d11/d1a/f1b [0,4194304] 0 2024-03-14T13:02:23.598 INFO:tasks.workunit.client.1.smithi137.stdout:9/785: write d6/df/d19/d1f/f95 [3001496,78427] 0 2024-03-14T13:02:23.598 INFO:tasks.workunit.client.0.smithi111.stdout:8/965: truncate d3/d1b/dc1/ffe 832062 0 2024-03-14T13:02:23.600 INFO:tasks.workunit.client.0.smithi111.stdout:2/613: chown dd/d11/d1a/f21 373 1 2024-03-14T13:02:23.600 INFO:tasks.workunit.client.1.smithi137.stdout:6/686: rename d2/d54/d55/d6d/d76/l89 to d2/d54/d55/lea 0 2024-03-14T13:02:23.602 INFO:tasks.workunit.client.1.smithi137.stdout:0/674: mkdir d1/d5/d1b/d1e/d25/d8a/d4a/d54/de2 0 2024-03-14T13:02:23.602 INFO:tasks.workunit.client.0.smithi111.stdout:2/614: chown dd/d17/d34/d73/d75/d59 0 1 2024-03-14T13:02:23.610 INFO:tasks.workunit.client.1.smithi137.stdout:5/627: dwrite d7/d33/d65/fb0 [0,4194304] 0 2024-03-14T13:02:23.610 INFO:tasks.workunit.client.1.smithi137.stdout:6/687: truncate d2/d54/d55/d9c/dc0/dc4/fcf 1007405 0 2024-03-14T13:02:23.612 INFO:tasks.workunit.client.1.smithi137.stdout:1/768: truncate d7/d9/d19/d1f/d89/d90/db8/fbd 514538 0 2024-03-14T13:02:23.614 INFO:tasks.workunit.client.1.smithi137.stdout:8/730: rename d1/dc/d3c/d87/d34/d97/c79 to d1/d5e/cf5 0 2024-03-14T13:02:23.619 INFO:tasks.workunit.client.1.smithi137.stdout:8/731: dread d1/dc/fcd [0,4194304] 0 2024-03-14T13:02:23.621 INFO:tasks.workunit.client.1.smithi137.stdout:6/688: mkdir d2/d54/d55/d6d/dc3/deb 0 2024-03-14T13:02:23.622 INFO:tasks.workunit.client.1.smithi137.stdout:5/628: creat d7/d1d/d74/d83/fd2 x:0 0 0 2024-03-14T13:02:23.623 INFO:tasks.workunit.client.1.smithi137.stdout:2/783: rename d2/d34/d3c/d9d/fa1 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f102 0 2024-03-14T13:02:23.624 INFO:tasks.workunit.client.1.smithi137.stdout:1/769: creat d7/d9/d19/d1f/d26/d6c/f106 x:0 0 0 2024-03-14T13:02:23.625 INFO:tasks.workunit.client.1.smithi137.stdout:0/675: dwrite d1/d5/d1b/d1e/d25/d55/fb4 [0,4194304] 0 2024-03-14T13:02:23.625 INFO:tasks.workunit.client.1.smithi137.stdout:6/689: chown d2/cd2 22 1 2024-03-14T13:02:23.627 INFO:tasks.workunit.client.1.smithi137.stdout:3/600: dread d4/de/d16/d1c/d5d/d85/f5a [0,4194304] 0 2024-03-14T13:02:23.633 INFO:tasks.workunit.client.1.smithi137.stdout:7/608: truncate d3/d52/f9d 2171469 0 2024-03-14T13:02:23.634 INFO:tasks.workunit.client.1.smithi137.stdout:2/784: unlink d2/d34/fcb 0 2024-03-14T13:02:23.634 INFO:tasks.workunit.client.1.smithi137.stdout:4/669: dread d4/d27/d5c/d6d/d58/f7e [0,4194304] 0 2024-03-14T13:02:23.635 INFO:tasks.workunit.client.1.smithi137.stdout:9/786: truncate d6/df/f13 1716132 0 2024-03-14T13:02:23.635 INFO:tasks.workunit.client.1.smithi137.stdout:8/732: creat d1/dc/d10/dcf/dd2/ff6 x:0 0 0 2024-03-14T13:02:23.636 INFO:tasks.workunit.client.1.smithi137.stdout:9/787: fsync d6/d56/fb3 0 2024-03-14T13:02:23.637 INFO:tasks.workunit.client.1.smithi137.stdout:6/690: mkdir d2/d54/d55/d6d/dc3/dec 0 2024-03-14T13:02:23.637 INFO:tasks.workunit.client.1.smithi137.stdout:3/601: write d4/de/d16/f31 [3452514,119632] 0 2024-03-14T13:02:23.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:23 smithi111 ceph-mon[39688]: pgmap v26: 65 pgs: 65 active+clean; 2.3 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 75 MiB/s rd, 296 MiB/s wr, 516 op/s 2024-03-14T13:02:23.648 INFO:tasks.workunit.client.1.smithi137.stdout:0/676: rename d1/d5/d1b/d42/d3a/db0/d84/fa5 to d1/d5/d1b/d1e/d25/d26/fe3 0 2024-03-14T13:02:23.649 INFO:tasks.workunit.client.1.smithi137.stdout:6/691: rmdir d2/d54/d55/d6d/dc3/dce 39 2024-03-14T13:02:23.652 INFO:tasks.workunit.client.1.smithi137.stdout:7/609: unlink d3/d5/d23/d2c/l93 0 2024-03-14T13:02:23.655 INFO:tasks.workunit.client.1.smithi137.stdout:2/785: creat d2/d34/dee/f103 x:0 0 0 2024-03-14T13:02:23.656 INFO:tasks.workunit.client.1.smithi137.stdout:3/602: dread d4/de/d16/f1a [0,4194304] 0 2024-03-14T13:02:23.657 INFO:tasks.workunit.client.1.smithi137.stdout:8/733: symlink d1/d5e/lf7 0 2024-03-14T13:02:23.657 INFO:tasks.workunit.client.1.smithi137.stdout:1/770: link d7/d9/d19/d1f/d26/d7d/la4 d7/d31/d54/ded/l107 0 2024-03-14T13:02:23.659 INFO:tasks.workunit.client.1.smithi137.stdout:0/677: creat d1/d5/d1b/d42/d3a/dc6/fe4 x:0 0 0 2024-03-14T13:02:23.661 INFO:tasks.workunit.client.1.smithi137.stdout:5/629: truncate d7/d16/f38 3448518 0 2024-03-14T13:02:23.664 INFO:tasks.workunit.client.1.smithi137.stdout:0/678: truncate d1/d1f/d82/fb9 307145 0 2024-03-14T13:02:23.664 INFO:tasks.workunit.client.1.smithi137.stdout:7/610: write d3/d6/d14/d22/f43 [3467887,4594] 0 2024-03-14T13:02:23.665 INFO:tasks.workunit.client.1.smithi137.stdout:2/786: readlink d2/d7/d2e/d3e/d44/d6b/d1d/ld2 0 2024-03-14T13:02:23.668 INFO:tasks.workunit.client.1.smithi137.stdout:4/670: dwrite d4/d9/d3a/d41/f21 [4194304,4194304] 0 2024-03-14T13:02:23.669 INFO:tasks.workunit.client.1.smithi137.stdout:0/679: dread d1/d5/d1b/d42/fbc [0,4194304] 0 2024-03-14T13:02:23.672 INFO:tasks.workunit.client.1.smithi137.stdout:4/671: dread - d4/d27/d5c/d77/fbd zero size 2024-03-14T13:02:23.674 INFO:tasks.workunit.client.1.smithi137.stdout:8/734: creat d1/dc/d3c/d87/d34/d82/de6/ff8 x:0 0 0 2024-03-14T13:02:23.676 INFO:tasks.workunit.client.1.smithi137.stdout:8/735: chown d1/dc/d3c/d87/d34/d97 4386 1 2024-03-14T13:02:23.678 INFO:tasks.workunit.client.1.smithi137.stdout:1/771: dwrite d7/d9/d19/d1f/d26/d7d/fd4 [0,4194304] 0 2024-03-14T13:02:23.680 INFO:tasks.workunit.client.0.smithi111.stdout:2/615: creat dd/d8d/fc0 x:0 0 0 2024-03-14T13:02:23.680 INFO:tasks.workunit.client.1.smithi137.stdout:9/788: getdents d6/df/d71/d94 0 2024-03-14T13:02:23.681 INFO:tasks.workunit.client.1.smithi137.stdout:8/736: write d1/dc/d77/d9f/fe7 [925373,39784] 0 2024-03-14T13:02:23.681 INFO:tasks.workunit.client.0.smithi111.stdout:0/830: dread d6/d9/d15/d55/da0/fb9 [0,4194304] 0 2024-03-14T13:02:23.682 INFO:tasks.workunit.client.0.smithi111.stdout:0/831: chown d6/d3d/d97 20179 1 2024-03-14T13:02:23.683 INFO:tasks.workunit.client.1.smithi137.stdout:6/692: truncate d2/d54/d55/fa3 118350 0 2024-03-14T13:02:23.683 INFO:tasks.workunit.client.0.smithi111.stdout:2/616: mkdir dd/d17/d34/d74/dc1 0 2024-03-14T13:02:23.684 INFO:tasks.workunit.client.0.smithi111.stdout:8/966: link d3/d1b/d20/d35/d40/d48/d7e/db3/db5/cf3 d3/dc/d31/d9d/d110/c146 0 2024-03-14T13:02:23.686 INFO:tasks.workunit.client.0.smithi111.stdout:8/967: write d3/d1b/d34/f137 [1045068,107233] 0 2024-03-14T13:02:23.686 INFO:tasks.workunit.client.0.smithi111.stdout:0/832: creat d6/d9/de/d32/d64/f126 x:0 0 0 2024-03-14T13:02:23.688 INFO:tasks.workunit.client.0.smithi111.stdout:8/968: readlink d3/d1b/d20/d35/d40/d48/d7e/db3/db5/l11f 0 2024-03-14T13:02:23.688 INFO:tasks.workunit.client.1.smithi137.stdout:3/603: creat d4/de/fcb x:0 0 0 2024-03-14T13:02:23.689 INFO:tasks.workunit.client.1.smithi137.stdout:0/680: creat d1/d5/d1b/d42/d3a/dc6/fe5 x:0 0 0 2024-03-14T13:02:23.689 INFO:tasks.workunit.client.0.smithi111.stdout:0/833: symlink d6/d9/d15/l127 0 2024-03-14T13:02:23.692 INFO:tasks.workunit.client.1.smithi137.stdout:9/789: fdatasync d6/df/d88/de5/df6/f43 0 2024-03-14T13:02:23.692 INFO:tasks.workunit.client.1.smithi137.stdout:7/611: symlink d3/d5/d5a/d80/lbf 0 2024-03-14T13:02:23.698 INFO:tasks.workunit.client.1.smithi137.stdout:0/681: readlink d1/d1f/l2c 0 2024-03-14T13:02:23.699 INFO:tasks.workunit.client.1.smithi137.stdout:7/612: chown d3/d6/d14/f1a 3 1 2024-03-14T13:02:23.699 INFO:tasks.workunit.client.1.smithi137.stdout:3/604: write d4/de/d16/d1c/d5d/d85/d6b/fc2 [610839,61504] 0 2024-03-14T13:02:23.701 INFO:tasks.workunit.client.1.smithi137.stdout:2/787: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/fa8 [0,4194304] 0 2024-03-14T13:02:23.701 INFO:tasks.workunit.client.1.smithi137.stdout:4/672: dread d4/d10/d42/f6c [0,4194304] 0 2024-03-14T13:02:23.703 INFO:tasks.workunit.client.1.smithi137.stdout:5/630: dwrite d7/d16/d18/d25/d28/d3c/d7a/fc0 [0,4194304] 0 2024-03-14T13:02:23.707 INFO:tasks.workunit.client.1.smithi137.stdout:0/682: chown d1/d44/d72/d7e/fcb 6009 1 2024-03-14T13:02:23.707 INFO:tasks.workunit.client.1.smithi137.stdout:6/693: unlink d2/d6/de/l7b 0 2024-03-14T13:02:23.713 INFO:tasks.workunit.client.1.smithi137.stdout:1/772: getdents d7/d9/d19/d1f/d89/d90/db1 0 2024-03-14T13:02:23.717 INFO:tasks.workunit.client.1.smithi137.stdout:9/790: rename d6/df/d19/d1f/de8/lea to d6/d56/lfe 0 2024-03-14T13:02:23.718 INFO:tasks.workunit.client.1.smithi137.stdout:8/737: truncate d1/dc/d77/d9f/fe7 183928 0 2024-03-14T13:02:23.719 INFO:tasks.workunit.client.1.smithi137.stdout:5/631: symlink d7/d16/d18/d25/d28/dc5/ld3 0 2024-03-14T13:02:23.721 INFO:tasks.workunit.client.1.smithi137.stdout:2/788: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/l104 0 2024-03-14T13:02:23.724 INFO:tasks.workunit.client.0.smithi111.stdout:0/834: unlink d6/d3d/le0 0 2024-03-14T13:02:23.724 INFO:tasks.workunit.client.1.smithi137.stdout:7/613: rmdir d3/d6/d64/d74 0 2024-03-14T13:02:23.725 INFO:tasks.workunit.client.0.smithi111.stdout:8/969: rename d3/dc/d31/c80 to d3/d1b/d20/c147 0 2024-03-14T13:02:23.726 INFO:tasks.workunit.client.0.smithi111.stdout:0/835: creat d6/d9/de/d32/f128 x:0 0 0 2024-03-14T13:02:23.726 INFO:tasks.workunit.client.1.smithi137.stdout:4/673: rename d4/d27/d5c/d6d/d58/d65/f7b to d4/d9/d3a/d70/fdf 0 2024-03-14T13:02:23.727 INFO:tasks.workunit.client.1.smithi137.stdout:9/791: mkdir d6/df/d19/d1f/dff 0 2024-03-14T13:02:23.729 INFO:tasks.workunit.client.1.smithi137.stdout:0/683: creat d1/d1f/fe6 x:0 0 0 2024-03-14T13:02:23.729 INFO:tasks.workunit.client.0.smithi111.stdout:8/970: creat d3/d1b/d20/d35/d40/d48/dcd/f148 x:0 0 0 2024-03-14T13:02:23.731 INFO:tasks.workunit.client.0.smithi111.stdout:0/836: fdatasync d6/d9/de/d3a/d46/d4a/f91 0 2024-03-14T13:02:23.732 INFO:tasks.workunit.client.1.smithi137.stdout:7/614: symlink d3/d52/d77/lc0 0 2024-03-14T13:02:23.733 INFO:tasks.workunit.client.0.smithi111.stdout:0/837: mkdir d6/d3d/d97/d129 0 2024-03-14T13:02:23.734 INFO:tasks.workunit.client.1.smithi137.stdout:6/694: rename d2/d54/d55/d9c/dbf/ld1 to d2/d6/de/led 0 2024-03-14T13:02:23.735 INFO:tasks.workunit.client.0.smithi111.stdout:8/971: unlink d3/cf8 0 2024-03-14T13:02:23.735 INFO:tasks.workunit.client.0.smithi111.stdout:8/972: stat d3/dc/d45/d120/ff2 0 2024-03-14T13:02:23.736 INFO:tasks.workunit.client.0.smithi111.stdout:8/973: readlink d3/dc/d45/d120/ld3 0 2024-03-14T13:02:23.736 INFO:tasks.workunit.client.0.smithi111.stdout:0/838: fdatasync d6/d9/de/d3a/f9b 0 2024-03-14T13:02:23.739 INFO:tasks.workunit.client.1.smithi137.stdout:4/674: fdatasync d4/d10/d42/d73/d9d/dbc/fb9 0 2024-03-14T13:02:23.740 INFO:tasks.workunit.client.1.smithi137.stdout:1/773: creat d7/d4d/f108 x:0 0 0 2024-03-14T13:02:23.741 INFO:tasks.workunit.client.1.smithi137.stdout:3/605: truncate d4/df/d82/fc3 2330296 0 2024-03-14T13:02:23.741 INFO:tasks.workunit.client.0.smithi111.stdout:8/974: mkdir d3/d1b/d20/d35/d40/d48/d77/d149 0 2024-03-14T13:02:23.742 INFO:tasks.workunit.client.1.smithi137.stdout:8/738: dread d1/dc/d10/fa4 [0,4194304] 0 2024-03-14T13:02:23.743 INFO:tasks.workunit.client.0.smithi111.stdout:0/839: dread d6/d9/de/d3a/d46/f83 [0,4194304] 0 2024-03-14T13:02:23.744 INFO:tasks.workunit.client.0.smithi111.stdout:8/975: creat d3/d1b/d20/d35/d40/d48/d77/f14a x:0 0 0 2024-03-14T13:02:23.745 INFO:tasks.workunit.client.1.smithi137.stdout:6/695: dwrite d2/d6/f34 [4194304,4194304] 0 2024-03-14T13:02:23.745 INFO:tasks.workunit.client.1.smithi137.stdout:7/615: creat d3/d5/d23/d3f/d5b/fc1 x:0 0 0 2024-03-14T13:02:23.748 INFO:tasks.workunit.client.0.smithi111.stdout:0/840: unlink d6/d45/d87/fe2 0 2024-03-14T13:02:23.752 INFO:tasks.workunit.client.0.smithi111.stdout:0/841: write d6/d9/d15/d49/d56/fab [1110275,126099] 0 2024-03-14T13:02:23.754 INFO:tasks.workunit.client.0.smithi111.stdout:8/976: dwrite d3/d1b/d20/d35/d40/d48/d7e/f134 [0,4194304] 0 2024-03-14T13:02:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:23 smithi137 ceph-mon[50336]: pgmap v26: 65 pgs: 65 active+clean; 2.3 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 75 MiB/s rd, 296 MiB/s wr, 516 op/s 2024-03-14T13:02:23.756 INFO:tasks.workunit.client.1.smithi137.stdout:5/632: dread d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:23.757 INFO:tasks.workunit.client.1.smithi137.stdout:6/696: write d2/d54/d55/d33/fcc [2410765,82216] 0 2024-03-14T13:02:23.758 INFO:tasks.workunit.client.1.smithi137.stdout:7/616: mknod d3/d6/d59/daa/cc2 0 2024-03-14T13:02:23.761 INFO:tasks.workunit.client.1.smithi137.stdout:3/606: symlink d4/de/lcc 0 2024-03-14T13:02:23.763 INFO:tasks.workunit.client.0.smithi111.stdout:8/977: chown d3/d1b/d20/d35/d40/fac 1 1 2024-03-14T13:02:23.764 INFO:tasks.workunit.client.1.smithi137.stdout:8/739: fsync d1/d4a/fef 0 2024-03-14T13:02:23.765 INFO:tasks.workunit.client.0.smithi111.stdout:0/842: unlink d6/d9/d15/da9/f105 0 2024-03-14T13:02:23.765 INFO:tasks.workunit.client.1.smithi137.stdout:0/684: write d1/f1c [4063595,85441] 0 2024-03-14T13:02:23.766 INFO:tasks.workunit.client.1.smithi137.stdout:1/774: creat d7/d9/f109 x:0 0 0 2024-03-14T13:02:23.766 INFO:tasks.workunit.client.1.smithi137.stdout:2/789: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f98 [0,4194304] 0 2024-03-14T13:02:23.766 INFO:tasks.workunit.client.1.smithi137.stdout:5/633: creat d7/d16/d18/d2d/fd4 x:0 0 0 2024-03-14T13:02:23.766 INFO:tasks.workunit.client.1.smithi137.stdout:6/697: write d2/d6/d86/d8a/fbb [513031,112875] 0 2024-03-14T13:02:23.766 INFO:tasks.workunit.client.1.smithi137.stdout:7/617: fsync d3/d5/d23/d2c/d5d/d6a/d75/d78/da4/fb7 0 2024-03-14T13:02:23.767 INFO:tasks.workunit.client.1.smithi137.stdout:3/607: readlink d4/de/d16/d27/l4d 0 2024-03-14T13:02:23.770 INFO:tasks.workunit.client.0.smithi111.stdout:0/843: mkdir d6/d9/de/d3a/d46/d4a/d11b/d12a 0 2024-03-14T13:02:23.773 INFO:tasks.workunit.client.0.smithi111.stdout:0/844: truncate d6/d9/de/d3a/d46/d4a/fb4 734728 0 2024-03-14T13:02:23.773 INFO:tasks.workunit.client.1.smithi137.stdout:9/792: dread d6/fe [0,4194304] 0 2024-03-14T13:02:23.774 INFO:tasks.workunit.client.0.smithi111.stdout:0/845: stat d6/d9/de/f6a 0 2024-03-14T13:02:23.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.774+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1b005e400 con 0x7ff19c06c570 2024-03-14T13:02:23.775 INFO:tasks.workunit.client.1.smithi137.stdout:2/790: creat d2/d7/d2e/d3e/daa/de7/f105 x:0 0 0 2024-03-14T13:02:23.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.776+0000 7ff1a67fc700 1 -- 172.21.15.111:0/3815377387 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7ff1b005e400 con 0x7ff19c06c570 2024-03-14T13:02:23.776 INFO:tasks.workunit.client.0.smithi111.stdout:0/846: creat d6/d9/d15/da9/f12b x:0 0 0 2024-03-14T13:02:23.777 INFO:tasks.workunit.client.1.smithi137.stdout:1/775: write d7/f9a [246396,104280] 0 2024-03-14T13:02:23.777 INFO:tasks.workunit.client.1.smithi137.stdout:3/608: dread - d4/d19/da1/fab zero size 2024-03-14T13:02:23.779 INFO:tasks.workunit.client.1.smithi137.stdout:5/634: read - d7/d49/d75/fb3 zero size 2024-03-14T13:02:23.779 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 msgr2=0x7ff19c06ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 0x7ff19c06ea20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac004d00 tx=0x7ff1ac009ce0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 msgr2=0x7ff1b0114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff1a000b900 tx=0x7ff1a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 shutdown_connections 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7ff19c06c570 0x7ff19c06ea20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1b006e9a0 0x7ff1b0114010 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 --2- 172.21.15.111:0/3815377387 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1b0114550 0x7ff1b0119580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 >> 172.21.15.111:0/3815377387 conn(0x7ff1b0069de0 msgr2=0x7ff1b006b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 shutdown_connections 2024-03-14T13:02:23.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.779+0000 7ff1b7c6e700 1 -- 172.21.15.111:0/3815377387 wait complete. 2024-03-14T13:02:23.781 INFO:tasks.workunit.client.1.smithi137.stdout:1/776: mknod d7/d92/c10a 0 2024-03-14T13:02:23.781 INFO:tasks.workunit.client.1.smithi137.stdout:6/698: dread d2/d6/da7/fd4 [0,4194304] 0 2024-03-14T13:02:23.781 INFO:tasks.workunit.client.1.smithi137.stdout:2/791: symlink d2/d7/d2e/d3e/daa/dcd/l106 0 2024-03-14T13:02:23.781 INFO:tasks.workunit.client.0.smithi111.stdout:0/847: creat d6/d3d/f12c x:0 0 0 2024-03-14T13:02:23.783 INFO:tasks.workunit.client.1.smithi137.stdout:8/740: symlink d1/dc/d3c/d87/d34/d82/d94/lf9 0 2024-03-14T13:02:23.783 INFO:tasks.workunit.client.1.smithi137.stdout:4/675: dread d4/d9/d3a/d41/d38/f53 [0,4194304] 0 2024-03-14T13:02:23.783 INFO:tasks.workunit.client.1.smithi137.stdout:9/793: fdatasync d6/df/d19/d1f/fb4 0 2024-03-14T13:02:23.785 INFO:tasks.workunit.client.1.smithi137.stdout:0/685: unlink d1/d5/d1b/d1e/d25/d26/f2d 0 2024-03-14T13:02:23.785 INFO:tasks.workunit.client.1.smithi137.stdout:5/635: fsync d7/d16/d18/d25/d28/d3c/d76/f9f 0 2024-03-14T13:02:23.786 INFO:tasks.workunit.client.1.smithi137.stdout:0/686: stat d1/d5 0 2024-03-14T13:02:23.787 INFO:tasks.workunit.client.0.smithi111.stdout:0/848: link d6/d9/d15/d55/da0/fdf d6/d9/de/d3a/f12d 0 2024-03-14T13:02:23.787 INFO:tasks.workunit.client.1.smithi137.stdout:3/609: rename d4/d22 to d4/de/d16/d1c/d8d/dcd 0 2024-03-14T13:02:23.787 INFO:tasks.workunit.client.1.smithi137.stdout:4/676: read d4/d10/d42/f4e [3515916,43929] 0 2024-03-14T13:02:23.788 INFO:tasks.workunit.client.1.smithi137.stdout:1/777: write d7/d9/d19/fc0 [557966,39214] 0 2024-03-14T13:02:23.789 INFO:tasks.workunit.client.1.smithi137.stdout:7/618: fsync d3/d5/d23/d2c/d5d/f6b 0 2024-03-14T13:02:23.789 INFO:tasks.workunit.client.1.smithi137.stdout:6/699: mkdir d2/d6/dee 0 2024-03-14T13:02:23.789 INFO:tasks.workunit.client.0.smithi111.stdout:8/978: dwrite d3/d49/fcf [0,4194304] 0 2024-03-14T13:02:23.792 INFO:tasks.workunit.client.0.smithi111.stdout:0/849: truncate d6/d9/d15/f2e 163817 0 2024-03-14T13:02:23.793 INFO:tasks.workunit.client.0.smithi111.stdout:8/979: mknod d3/d1b/d20/d35/d40/d48/dcd/c14b 0 2024-03-14T13:02:23.794 INFO:tasks.workunit.client.0.smithi111.stdout:8/980: fdatasync d3/d1b/dc1/ffe 0 2024-03-14T13:02:23.795 INFO:tasks.workunit.client.1.smithi137.stdout:9/794: read d6/df/f13 [822786,72373] 0 2024-03-14T13:02:23.795 INFO:tasks.workunit.client.1.smithi137.stdout:0/687: fdatasync d1/d5/d1b/d1e/d53/d6d/fdf 0 2024-03-14T13:02:23.795 INFO:tasks.workunit.client.1.smithi137.stdout:7/619: write d3/d54/f5c [1180448,16251] 0 2024-03-14T13:02:23.796 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:02:23.796 INFO:tasks.workunit.client.0.smithi111.stdout:8/981: write d3/dc/d45/d8d/f12f [775220,38974] 0 2024-03-14T13:02:23.797 INFO:tasks.workunit.client.0.smithi111.stdout:8/982: dread - d3/d49/f116 zero size 2024-03-14T13:02:23.797 INFO:tasks.workunit.client.1.smithi137.stdout:8/741: rename d1/dc/d77/d9f/de4/l69 to d1/d5e/dc9/dda/lfa 0 2024-03-14T13:02:23.798 INFO:tasks.workunit.client.0.smithi111.stdout:0/850: write d6/d9/d15/d49/d56/fc9 [4422131,18066] 0 2024-03-14T13:02:23.799 INFO:tasks.workunit.client.0.smithi111.stdout:8/983: chown d3/d22/d30/f74 127006353 1 2024-03-14T13:02:23.800 INFO:tasks.workunit.client.0.smithi111.stdout:8/984: write d3/d1b/f12c [714782,45553] 0 2024-03-14T13:02:23.800 INFO:tasks.workunit.client.0.smithi111.stdout:0/851: symlink d6/d45/d114/l12e 0 2024-03-14T13:02:23.802 INFO:tasks.workunit.client.1.smithi137.stdout:1/778: write d7/d9/d19/d1f/d26/d6c/f106 [430348,78508] 0 2024-03-14T13:02:23.804 INFO:tasks.workunit.client.1.smithi137.stdout:7/620: chown d3/d5/d23/d2c/d5d/f92 348555892 1 2024-03-14T13:02:23.805 INFO:tasks.workunit.client.0.smithi111.stdout:0/852: unlink d6/d9/d15/d49/l92 0 2024-03-14T13:02:23.806 INFO:tasks.workunit.client.1.smithi137.stdout:6/700: chown d2/d54/d55/d38/l72 334 1 2024-03-14T13:02:23.807 INFO:tasks.workunit.client.0.smithi111.stdout:0/853: chown d6/d9/d15/c9a 1 1 2024-03-14T13:02:23.809 INFO:tasks.workunit.client.1.smithi137.stdout:5/636: rename d7/d16/d18/d25/f6b to d7/d16/d18/d48/d9d/fd5 0 2024-03-14T13:02:23.809 INFO:tasks.workunit.client.0.smithi111.stdout:0/854: chown d6/d9/d15/c19 3514825 1 2024-03-14T13:02:23.809 INFO:tasks.workunit.client.1.smithi137.stdout:0/688: mknod d1/d1f/d80/ce7 0 2024-03-14T13:02:23.810 INFO:tasks.workunit.client.0.smithi111.stdout:0/855: dread - d6/d9/de/d32/d64/de7/ffd zero size 2024-03-14T13:02:23.811 INFO:tasks.workunit.client.1.smithi137.stdout:1/779: stat d7/d9/d19/d1f/d2e/d63/ccc 0 2024-03-14T13:02:23.814 INFO:tasks.workunit.client.1.smithi137.stdout:9/795: symlink d6/df/d19/d1f/dff/l100 0 2024-03-14T13:02:23.814 INFO:tasks.workunit.client.1.smithi137.stdout:2/792: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 [0,4194304] 0 2024-03-14T13:02:23.818 INFO:tasks.workunit.client.1.smithi137.stdout:8/742: creat d1/dc/d3c/d87/d34/d82/ffb x:0 0 0 2024-03-14T13:02:23.819 INFO:tasks.workunit.client.0.smithi111.stdout:0/856: dwrite d6/d45/d87/f10a [0,4194304] 0 2024-03-14T13:02:23.819 INFO:tasks.workunit.client.1.smithi137.stdout:0/689: mkdir d1/d44/d72/de8 0 2024-03-14T13:02:23.822 INFO:tasks.workunit.client.1.smithi137.stdout:7/621: mknod d3/d6/d64/d97/cc3 0 2024-03-14T13:02:23.826 INFO:tasks.workunit.client.0.smithi111.stdout:0/857: rename d6/d9/de/d3a/c9d to d6/d3d/d97/d129/c12f 0 2024-03-14T13:02:23.826 INFO:tasks.workunit.client.1.smithi137.stdout:3/610: dwrite d4/de/d16/d1c/d8d/dcd/fba [0,4194304] 0 2024-03-14T13:02:23.834 INFO:tasks.workunit.client.1.smithi137.stdout:2/793: read - d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd6 zero size 2024-03-14T13:02:23.834 INFO:tasks.workunit.client.1.smithi137.stdout:4/677: dread d4/d9/f18 [0,4194304] 0 2024-03-14T13:02:23.834 INFO:tasks.workunit.client.1.smithi137.stdout:3/611: fsync d4/d19/da1/fc4 0 2024-03-14T13:02:23.834 INFO:tasks.workunit.client.1.smithi137.stdout:8/743: write d1/dc/d3c/d87/d34/d82/d94/ff4 [769031,22871] 0 2024-03-14T13:02:23.837 INFO:tasks.workunit.client.1.smithi137.stdout:1/780: rename d7/d9/d19/d5f/f7b to d7/d9/d19/d1f/d2e/f10b 0 2024-03-14T13:02:23.838 INFO:tasks.workunit.client.1.smithi137.stdout:7/622: chown d3/d6/d64/d9e/fb8 2 1 2024-03-14T13:02:23.840 INFO:tasks.workunit.client.0.smithi111.stdout:8/985: dread d3/d1b/d20/d35/d40/d48/d7e/fba [0,4194304] 0 2024-03-14T13:02:23.841 INFO:tasks.workunit.client.1.smithi137.stdout:0/690: creat d1/d5/d1b/d42/d61/fe9 x:0 0 0 2024-03-14T13:02:23.841 INFO:tasks.workunit.client.1.smithi137.stdout:8/744: read d1/dc/d77/d9f/de4/f16 [223452,63881] 0 2024-03-14T13:02:23.842 INFO:tasks.workunit.client.1.smithi137.stdout:6/701: dwrite d2/d6/de/d45/f5b [0,4194304] 0 2024-03-14T13:02:23.843 INFO:tasks.workunit.client.0.smithi111.stdout:8/986: symlink d3/d1b/dc1/de0/l14c 0 2024-03-14T13:02:23.843 INFO:tasks.workunit.client.1.smithi137.stdout:9/796: getdents d6/df/d88/dd5 0 2024-03-14T13:02:23.844 INFO:tasks.workunit.client.0.smithi111.stdout:8/987: write d3/d1b/f6f [1312522,61125] 0 2024-03-14T13:02:23.844 INFO:tasks.workunit.client.0.smithi111.stdout:8/988: dread - d3/d22/d30/f95 zero size 2024-03-14T13:02:23.845 INFO:tasks.workunit.client.1.smithi137.stdout:4/678: creat d4/d9/d3a/d41/d38/d79/fe0 x:0 0 0 2024-03-14T13:02:23.846 INFO:tasks.workunit.client.1.smithi137.stdout:1/781: dread - d7/d9/d19/d40/d42/d9d/de5/ff4 zero size 2024-03-14T13:02:23.846 INFO:tasks.workunit.client.1.smithi137.stdout:7/623: readlink d3/d5/d23/d3f/d5b/l69 0 2024-03-14T13:02:23.846 INFO:tasks.workunit.client.0.smithi111.stdout:8/989: mknod d3/d5/d12d/c14d 0 2024-03-14T13:02:23.848 INFO:tasks.workunit.client.1.smithi137.stdout:4/679: chown d4/f2e 8805665 1 2024-03-14T13:02:23.849 INFO:tasks.workunit.client.0.smithi111.stdout:8/990: mkdir d3/dc/d45/d120/d14e 0 2024-03-14T13:02:23.849 INFO:tasks.workunit.client.1.smithi137.stdout:8/745: fsync d1/d4a/f66 0 2024-03-14T13:02:23.852 INFO:tasks.workunit.client.0.smithi111.stdout:8/991: creat d3/d5/f14f x:0 0 0 2024-03-14T13:02:23.855 INFO:tasks.workunit.client.0.smithi111.stdout:8/992: creat d3/d22/da1/de6/f150 x:0 0 0 2024-03-14T13:02:23.859 INFO:tasks.workunit.client.0.smithi111.stdout:0/858: dwrite d6/f72 [0,4194304] 0 2024-03-14T13:02:23.859 INFO:tasks.workunit.client.1.smithi137.stdout:9/797: write d6/df/d88/de5/df6/d40/fb6 [826591,17160] 0 2024-03-14T13:02:23.860 INFO:tasks.workunit.client.1.smithi137.stdout:7/624: unlink d3/d5/d23/c95 0 2024-03-14T13:02:23.860 INFO:tasks.workunit.client.0.smithi111.stdout:0/859: fdatasync d6/d3d/fa7 0 2024-03-14T13:02:23.861 INFO:tasks.workunit.client.1.smithi137.stdout:2/794: truncate d2/d7/d2e/d3e/d44/d6b/d22/dbb/fc4 849611 0 2024-03-14T13:02:23.861 INFO:tasks.workunit.client.1.smithi137.stdout:1/782: write d7/d29/f2b [1165387,62451] 0 2024-03-14T13:02:23.862 INFO:tasks.workunit.client.1.smithi137.stdout:0/691: creat d1/d5/d1b/d42/d3a/dc6/fea x:0 0 0 2024-03-14T13:02:23.864 INFO:tasks.workunit.client.1.smithi137.stdout:3/612: write d4/fd [1119627,123183] 0 2024-03-14T13:02:23.864 INFO:tasks.workunit.client.0.smithi111.stdout:0/860: fsync d6/d9/d15/d55/da0/fb9 0 2024-03-14T13:02:23.865 INFO:tasks.workunit.client.0.smithi111.stdout:8/993: dwrite d3/dc/ddb/f113 [0,4194304] 0 2024-03-14T13:02:23.866 INFO:tasks.workunit.client.0.smithi111.stdout:8/994: stat d3/d49/fcf 0 2024-03-14T13:02:23.869 INFO:tasks.workunit.client.1.smithi137.stdout:9/798: fsync d6/df/d19/d1f/fb4 0 2024-03-14T13:02:23.869 INFO:tasks.workunit.client.0.smithi111.stdout:8/995: creat d3/d1b/d20/d35/d40/d48/d7e/f151 x:0 0 0 2024-03-14T13:02:23.870 INFO:tasks.workunit.client.1.smithi137.stdout:5/637: dread d7/d1d/d74/d81/f89 [0,4194304] 0 2024-03-14T13:02:23.873 INFO:tasks.workunit.client.1.smithi137.stdout:8/746: mknod d1/d4a/cfc 0 2024-03-14T13:02:23.873 INFO:tasks.workunit.client.1.smithi137.stdout:6/702: dwrite d2/d6/fc [0,4194304] 0 2024-03-14T13:02:23.874 INFO:tasks.workunit.client.0.smithi111.stdout:6/628: sync 2024-03-14T13:02:23.874 INFO:tasks.workunit.client.0.smithi111.stdout:9/684: sync 2024-03-14T13:02:23.874 INFO:tasks.workunit.client.0.smithi111.stdout:8/996: chown d3/fec 590460417 1 2024-03-14T13:02:23.875 INFO:tasks.workunit.client.1.smithi137.stdout:6/703: chown d2/d54/d55/d6d/d76/fc2 3 1 2024-03-14T13:02:23.878 INFO:tasks.workunit.client.1.smithi137.stdout:3/613: creat d4/de/d16/d27/fce x:0 0 0 2024-03-14T13:02:23.878 INFO:tasks.workunit.client.0.smithi111.stdout:0/861: rename d6/d9/de/d3a/d46/d4a/f81 to d6/d3d/da6/f130 0 2024-03-14T13:02:23.878 INFO:tasks.workunit.client.0.smithi111.stdout:8/997: truncate d3/dc/d45/d8d/f130 537621 0 2024-03-14T13:02:23.879 INFO:tasks.workunit.client.1.smithi137.stdout:2/795: read d2/f4b [10231779,78480] 0 2024-03-14T13:02:23.879 INFO:tasks.workunit.client.0.smithi111.stdout:9/685: readlink d5/d6/d2a/d2d/lce 0 2024-03-14T13:02:23.880 INFO:tasks.workunit.client.1.smithi137.stdout:4/680: creat d4/fe1 x:0 0 0 2024-03-14T13:02:23.880 INFO:tasks.workunit.client.1.smithi137.stdout:5/638: write d7/d33/d65/f84 [495412,89980] 0 2024-03-14T13:02:23.880 INFO:tasks.workunit.client.1.smithi137.stdout:1/783: dread d7/d31/faf [0,4194304] 0 2024-03-14T13:02:23.883 INFO:tasks.workunit.client.0.smithi111.stdout:8/998: dread - d3/d22/da1/de6/f150 zero size 2024-03-14T13:02:23.885 INFO:tasks.workunit.client.0.smithi111.stdout:6/629: unlink d0/d17/d21/d69/d3d/d53/d81/fa4 0 2024-03-14T13:02:23.885 INFO:tasks.workunit.client.1.smithi137.stdout:3/614: rename d4/de/d16/f6a to d4/de/dc7/fcf 0 2024-03-14T13:02:23.887 INFO:tasks.workunit.client.0.smithi111.stdout:0/862: symlink d6/d9/de/d3a/l131 0 2024-03-14T13:02:23.887 INFO:tasks.workunit.client.0.smithi111.stdout:9/686: mkdir d5/d6/d2a/d5a/dee 0 2024-03-14T13:02:23.888 INFO:tasks.workunit.client.1.smithi137.stdout:0/692: dread d1/f7b [0,4194304] 0 2024-03-14T13:02:23.888 INFO:tasks.workunit.client.1.smithi137.stdout:6/704: rename d2/d6/l2e to d2/d6/lef 0 2024-03-14T13:02:23.889 INFO:tasks.workunit.client.0.smithi111.stdout:9/687: fdatasync d5/d6/d2a/d5a/f76 0 2024-03-14T13:02:23.890 INFO:tasks.workunit.client.1.smithi137.stdout:6/705: fsync d2/d6/de/d45/d59/f5e 0 2024-03-14T13:02:23.890 INFO:tasks.workunit.client.0.smithi111.stdout:8/999: mkdir d3/dc/d31/d9d/d110/d144/d152 0 2024-03-14T13:02:23.890 INFO:tasks.workunit.client.1.smithi137.stdout:2/796: creat d2/d7/d2e/d3e/df3/f107 x:0 0 0 2024-03-14T13:02:23.891 INFO:tasks.workunit.client.0.smithi111.stdout:6/630: mknod d0/d17/d21/d69/d3d/d53/d81/da1/ceb 0 2024-03-14T13:02:23.891 INFO:tasks.workunit.client.1.smithi137.stdout:0/693: write d1/d5/d1b/d42/d3a/db0/d84/fd5 [43065,1556] 0 2024-03-14T13:02:23.892 INFO:tasks.workunit.client.0.smithi111.stdout:6/631: chown d0/d17/d21/d69/d3d/d53/d81/da1 7 1 2024-03-14T13:02:23.893 INFO:tasks.workunit.client.0.smithi111.stdout:0/863: creat d6/d3d/da6/dfe/f132 x:0 0 0 2024-03-14T13:02:23.894 INFO:tasks.workunit.client.1.smithi137.stdout:4/681: creat d4/d9/d3a/d41/d38/fe2 x:0 0 0 2024-03-14T13:02:23.895 INFO:tasks.workunit.client.0.smithi111.stdout:9/688: mkdir d5/d6/d8/def 0 2024-03-14T13:02:23.895 INFO:tasks.workunit.client.0.smithi111.stdout:6/632: read - d0/d17/d20/f48 zero size 2024-03-14T13:02:23.895 INFO:tasks.workunit.client.1.smithi137.stdout:0/694: creat d1/d44/feb x:0 0 0 2024-03-14T13:02:23.896 INFO:tasks.workunit.client.0.smithi111.stdout:0/864: symlink d6/d3d/d97/d109/l133 0 2024-03-14T13:02:23.897 INFO:tasks.workunit.client.1.smithi137.stdout:1/784: rename d7/d4d/d6b/dc5/l6e to d7/d9/d19/d1f/d26/d7d/l10c 0 2024-03-14T13:02:23.897 INFO:tasks.workunit.client.1.smithi137.stdout:2/797: rename d2/d7/d2e/d3e/d44/d6b/d22/da5 to d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/df0/d108 22 2024-03-14T13:02:23.897 INFO:tasks.workunit.client.1.smithi137.stdout:0/695: truncate d1/d5/d1b/d1e/d53/d6d/fdf 973280 0 2024-03-14T13:02:23.898 INFO:tasks.workunit.client.0.smithi111.stdout:6/633: mknod d0/d17/d20/d65/d93/cec 0 2024-03-14T13:02:23.898 INFO:tasks.workunit.client.0.smithi111.stdout:0/865: write d6/d3d/dba/f110 [987410,24936] 0 2024-03-14T13:02:23.899 INFO:tasks.workunit.client.1.smithi137.stdout:4/682: symlink d4/d9/d3a/d70/le3 0 2024-03-14T13:02:23.899 INFO:tasks.workunit.client.1.smithi137.stdout:2/798: chown d2/d7/f87 25101 1 2024-03-14T13:02:23.900 INFO:tasks.workunit.client.0.smithi111.stdout:9/689: truncate d5/d6/d8/d5b/d7c/f90 860493 0 2024-03-14T13:02:23.901 INFO:tasks.workunit.client.1.smithi137.stdout:2/799: write d2/d7/d2e/d3e/daa/fba [654827,30767] 0 2024-03-14T13:02:23.901 INFO:tasks.workunit.client.1.smithi137.stdout:6/706: creat d2/d54/d55/d9c/ff0 x:0 0 0 2024-03-14T13:02:23.902 INFO:tasks.workunit.client.0.smithi111.stdout:6/634: write d0/d17/d20/f26 [1585415,109303] 0 2024-03-14T13:02:23.903 INFO:tasks.workunit.client.0.smithi111.stdout:0/866: rename d6/d5a/d120 to d6/d3d/da6/dfe/d134 0 2024-03-14T13:02:23.905 INFO:tasks.workunit.client.0.smithi111.stdout:9/690: creat d5/d42/d71/dbd/ff0 x:0 0 0 2024-03-14T13:02:23.905 INFO:tasks.workunit.client.0.smithi111.stdout:6/635: rmdir d0/d17/d99 39 2024-03-14T13:02:23.906 INFO:tasks.workunit.client.1.smithi137.stdout:6/707: mkdir d2/d6/de/df1 0 2024-03-14T13:02:23.907 INFO:tasks.workunit.client.0.smithi111.stdout:0/867: chown f5 24 1 2024-03-14T13:02:23.910 INFO:tasks.workunit.client.0.smithi111.stdout:0/868: mkdir d6/d9/d15/d55/d89/d135 0 2024-03-14T13:02:23.911 INFO:tasks.workunit.client.0.smithi111.stdout:6/636: truncate d0/d50/f56 250985 0 2024-03-14T13:02:23.911 INFO:tasks.workunit.client.1.smithi137.stdout:2/800: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f 0 2024-03-14T13:02:23.913 INFO:tasks.workunit.client.1.smithi137.stdout:8/747: dread d1/f31 [0,4194304] 0 2024-03-14T13:02:23.915 INFO:tasks.workunit.client.0.smithi111.stdout:6/637: dread - d0/d17/d20/d5a/f9b zero size 2024-03-14T13:02:23.915 INFO:tasks.workunit.client.1.smithi137.stdout:0/696: dwrite d1/d5/d1b/d1e/d25/d8a/d4a/fb6 [0,4194304] 0 2024-03-14T13:02:23.916 INFO:tasks.workunit.client.1.smithi137.stdout:2/801: mkdir d2/d34/d3c/d9d/d109 0 2024-03-14T13:02:23.917 INFO:tasks.workunit.client.0.smithi111.stdout:6/638: creat d0/d17/d21/d42/da5/fed x:0 0 0 2024-03-14T13:02:23.920 INFO:tasks.workunit.client.1.smithi137.stdout:0/697: symlink d1/d44/d72/lec 0 2024-03-14T13:02:23.920 INFO:tasks.workunit.client.1.smithi137.stdout:3/615: dread d4/f80 [0,4194304] 0 2024-03-14T13:02:23.924 INFO:tasks.workunit.client.1.smithi137.stdout:8/748: mkdir d1/dc/d3c/d87/d34/d82/de6/dfd 0 2024-03-14T13:02:23.926 INFO:tasks.workunit.client.0.smithi111.stdout:6/639: truncate d0/d17/d21/f22 4602057 0 2024-03-14T13:02:23.926 INFO:tasks.workunit.client.1.smithi137.stdout:7/625: dwrite d3/d5/d23/d2c/d5d/f6b [4194304,4194304] 0 2024-03-14T13:02:23.928 INFO:tasks.workunit.client.1.smithi137.stdout:8/749: readlink d1/dc/l25 0 2024-03-14T13:02:23.930 INFO:tasks.workunit.client.1.smithi137.stdout:4/683: dread d4/d9/d3a/d41/d38/d55/db6/fc6 [0,4194304] 0 2024-03-14T13:02:23.935 INFO:tasks.workunit.client.0.smithi111.stdout:9/691: dwrite d5/d6/d14/f81 [0,4194304] 0 2024-03-14T13:02:23.935 INFO:tasks.workunit.client.1.smithi137.stdout:5/639: dwrite d7/d16/d18/d25/f46 [0,4194304] 0 2024-03-14T13:02:23.935 INFO:tasks.workunit.client.1.smithi137.stdout:3/616: dread d4/df/f96 [0,4194304] 0 2024-03-14T13:02:23.935 INFO:tasks.workunit.client.1.smithi137.stdout:7/626: truncate d3/d5/d5a/fa7 345195 0 2024-03-14T13:02:23.936 INFO:tasks.workunit.client.1.smithi137.stdout:4/684: mknod d4/d27/d5c/d77/ce4 0 2024-03-14T13:02:23.937 INFO:tasks.workunit.client.1.smithi137.stdout:4/685: readlink d4/ld 0 2024-03-14T13:02:23.938 INFO:tasks.workunit.client.0.smithi111.stdout:9/692: chown d5/d42/d71/d96/dd3 1297 1 2024-03-14T13:02:23.941 INFO:tasks.workunit.client.1.smithi137.stdout:7/627: write d3/d5/d5a/fa7 [628910,109717] 0 2024-03-14T13:02:23.942 INFO:tasks.workunit.client.0.smithi111.stdout:9/693: symlink d5/d6/d8/d74/lf1 0 2024-03-14T13:02:23.943 INFO:tasks.workunit.client.0.smithi111.stdout:0/869: dwrite d6/d9/de/d3a/d46/d4a/f91 [0,4194304] 0 2024-03-14T13:02:23.944 INFO:tasks.workunit.client.1.smithi137.stdout:8/750: dwrite d1/dc/d10/f1a [0,4194304] 0 2024-03-14T13:02:23.945 INFO:tasks.workunit.client.0.smithi111.stdout:0/870: chown d6/d9/de/d3a/l107 50080286 1 2024-03-14T13:02:23.948 INFO:tasks.workunit.client.0.smithi111.stdout:0/871: chown d6/d9/de/d11a/l11f 891917644 1 2024-03-14T13:02:23.951 INFO:tasks.workunit.client.0.smithi111.stdout:3/707: sync 2024-03-14T13:02:23.951 INFO:tasks.workunit.client.0.smithi111.stdout:7/608: sync 2024-03-14T13:02:23.954 INFO:tasks.workunit.client.1.smithi137.stdout:7/628: creat d3/d52/d77/fc4 x:0 0 0 2024-03-14T13:02:23.955 INFO:tasks.workunit.client.1.smithi137.stdout:3/617: getdents d4/de/d16 0 2024-03-14T13:02:23.957 INFO:tasks.workunit.client.1.smithi137.stdout:7/629: chown d3/d5 6 1 2024-03-14T13:02:23.957 INFO:tasks.workunit.client.1.smithi137.stdout:3/618: fdatasync d4/de/f7d 0 2024-03-14T13:02:23.961 INFO:tasks.workunit.client.1.smithi137.stdout:5/640: getdents d7/d1d/d74/d83/dc4 0 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.957+0000 7fd187482700 1 -- 172.21.15.111:0/556442736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 msgr2=0x7fd1800fcb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.957+0000 7fd187482700 1 --2- 172.21.15.111:0/556442736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd1800fcb60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd1700049c0 tx=0x7fd170009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.958+0000 7fd187482700 1 -- 172.21.15.111:0/556442736 shutdown_connections 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.958+0000 7fd187482700 1 --2- 172.21.15.111:0/556442736 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd1800fcb60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.958+0000 7fd187482700 1 --2- 172.21.15.111:0/556442736 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd1800fb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.958+0000 7fd187482700 1 -- 172.21.15.111:0/556442736 >> 172.21.15.111:0/556442736 conn(0x7fd1800f6ba0 msgr2=0x7fd1800f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.960+0000 7fd187482700 1 -- 172.21.15.111:0/556442736 shutdown_connections 2024-03-14T13:02:23.961 INFO:tasks.workunit.client.1.smithi137.stdout:0/698: write d1/d1f/f24 [3550966,24698] 0 2024-03-14T13:02:23.962 INFO:tasks.workunit.client.1.smithi137.stdout:2/802: truncate d2/d7/d2e/d3e/d44/d6b/f39 2907203 0 2024-03-14T13:02:23.963 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.962+0000 7fd187482700 1 -- 172.21.15.111:0/556442736 wait complete. 2024-03-14T13:02:23.963 INFO:tasks.workunit.client.1.smithi137.stdout:3/619: mknod d4/df/d82/cd0 0 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 Processor -- start 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 -- start start 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd18006e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18006ef50 con 0x7fd1800fc6f0 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd187482700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18006f090 con 0x7fd1800fb440 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd184a1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.963+0000 7fd18521e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd18006e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:23.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd184a1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43208/0 (socket says 172.21.15.111:43208) 2024-03-14T13:02:23.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd184a1d700 1 -- 172.21.15.111:0/2710795653 learned_addr learned my addr 172.21.15.111:0/2710795653 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:23.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd18521e700 1 -- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 msgr2=0x7fd18006ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:23.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd18521e700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:23.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd18521e700 1 -- 172.21.15.111:0/2710795653 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17000d040 con 0x7fd1800fb440 2024-03-14T13:02:23.965 INFO:tasks.workunit.client.0.smithi111.stdout:6/640: write d0/d17/d21/f22 [4544733,22327] 0 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd184a1d700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd18521e700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd18006e4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00b580 tx=0x7fd17c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17c00f6f0 con 0x7fd1800fb440 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd17c00fd00 con 0x7fd1800fb440 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd17c011e00 con 0x7fd1800fb440 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.964+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18006f320 con 0x7fd1800fb440 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.965+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd180119de0 con 0x7fd1800fb440 2024-03-14T13:02:23.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.966+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18005f5f0 con 0x7fd1800fb440 2024-03-14T13:02:23.967 INFO:tasks.workunit.client.1.smithi137.stdout:4/686: link d4/d9/d3a/d41/d38/l43 d4/d9/d3a/d41/d38/d55/db6/le5 0 2024-03-14T13:02:23.967 INFO:tasks.workunit.client.1.smithi137.stdout:0/699: readlink d1/d1f/l33 0 2024-03-14T13:02:23.967 INFO:tasks.workunit.client.1.smithi137.stdout:8/751: creat d1/dc/d71/db0/dca/ffe x:0 0 0 2024-03-14T13:02:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.967+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7fd17c00f850 con 0x7fd1800fb440 2024-03-14T13:02:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.967+0000 7fd175ffb700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 0x7fd16c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.968+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd17c08b380 con 0x7fd1800fb440 2024-03-14T13:02:23.968 INFO:tasks.workunit.client.0.smithi111.stdout:6/641: fdatasync d0/d17/d99/fa9 0 2024-03-14T13:02:23.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.968+0000 7fd184a1d700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 0x7fd16c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:23.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.968+0000 7fd184a1d700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 0x7fd16c070c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd170005010 tx=0x7fd170009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:23.972 INFO:tasks.workunit.client.1.smithi137.stdout:7/630: mkdir d3/d5/d5a/d86/dc5 0 2024-03-14T13:02:23.972 INFO:tasks.workunit.client.1.smithi137.stdout:8/752: unlink d1/dc/l5d 0 2024-03-14T13:02:23.973 INFO:tasks.workunit.client.0.smithi111.stdout:6/642: getdents d0/d17/d21/d69/d3d/d53/d81/da1 0 2024-03-14T13:02:23.973 INFO:tasks.workunit.client.0.smithi111.stdout:6/643: chown d0/d17/d21/d42/da5/fad 357 1 2024-03-14T13:02:23.974 INFO:tasks.workunit.client.1.smithi137.stdout:5/641: write d7/d49/d75/fa4 [3822269,125865] 0 2024-03-14T13:02:23.975 INFO:tasks.workunit.client.1.smithi137.stdout:4/687: write d4/d10/d42/f6c [4291977,48761] 0 2024-03-14T13:02:23.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:23.976+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fd17c055a10 con 0x7fd1800fb440 2024-03-14T13:02:23.977 INFO:tasks.workunit.client.0.smithi111.stdout:6/644: rename d0/d17/d20/d5a/f9b to d0/d17/d24/fee 0 2024-03-14T13:02:23.978 INFO:tasks.workunit.client.0.smithi111.stdout:9/694: write d5/d42/d71/dbd/de0/fed [306325,51019] 0 2024-03-14T13:02:23.980 INFO:tasks.workunit.client.1.smithi137.stdout:0/700: dwrite d1/d5/d1b/d1e/d25/d8a/d4a/fb6 [0,4194304] 0 2024-03-14T13:02:23.981 INFO:tasks.workunit.client.1.smithi137.stdout:7/631: rename d3/d54/f5c to d3/d5/d5a/fc6 0 2024-03-14T13:02:23.981 INFO:tasks.workunit.client.1.smithi137.stdout:5/642: symlink d7/d33/d65/ld6 0 2024-03-14T13:02:23.981 INFO:tasks.workunit.client.0.smithi111.stdout:7/609: write d0/d1/d4a/f60 [1916639,110582] 0 2024-03-14T13:02:23.982 INFO:tasks.workunit.client.0.smithi111.stdout:9/695: dread - d5/d6/d8/f49 zero size 2024-03-14T13:02:23.982 INFO:tasks.workunit.client.0.smithi111.stdout:6/645: creat d0/d17/d21/d69/d3d/d53/d81/da1/fef x:0 0 0 2024-03-14T13:02:23.985 INFO:tasks.workunit.client.0.smithi111.stdout:9/696: chown d5/fa 5 1 2024-03-14T13:02:23.986 INFO:tasks.workunit.client.1.smithi137.stdout:5/643: mknod d7/d33/dab/cd7 0 2024-03-14T13:02:23.986 INFO:tasks.workunit.client.1.smithi137.stdout:8/753: truncate d1/dc/d3c/d87/d34/fc5 599354 0 2024-03-14T13:02:23.988 INFO:tasks.workunit.client.1.smithi137.stdout:3/620: write d4/de/d16/d1c/d5d/d85/d6b/f6d [2792205,123653] 0 2024-03-14T13:02:23.990 INFO:tasks.workunit.client.0.smithi111.stdout:9/697: readlink d5/d6/d8/d74/lf1 0 2024-03-14T13:02:23.990 INFO:tasks.workunit.client.0.smithi111.stdout:6/646: symlink d0/d17/d21/d69/d3d/lf0 0 2024-03-14T13:02:23.991 INFO:tasks.workunit.client.1.smithi137.stdout:0/701: rename d1/d1f/d80/f96 to d1/d5/d1b/d1e/d53/fed 0 2024-03-14T13:02:23.991 INFO:tasks.workunit.client.1.smithi137.stdout:4/688: dwrite d4/d9/d3a/d41/d38/d55/f64 [0,4194304] 0 2024-03-14T13:02:23.995 INFO:tasks.workunit.client.1.smithi137.stdout:5/644: read d7/f15 [5599911,116221] 0 2024-03-14T13:02:23.996 INFO:tasks.workunit.client.0.smithi111.stdout:0/872: dwrite d6/d3d/f5f [0,4194304] 0 2024-03-14T13:02:23.999 INFO:tasks.workunit.client.0.smithi111.stdout:7/610: dwrite d0/d1/d21/f3f [0,4194304] 0 2024-03-14T13:02:24.001 INFO:tasks.workunit.client.1.smithi137.stdout:3/621: mkdir d4/de/d16/d1c/db8/dd1 0 2024-03-14T13:02:24.002 INFO:tasks.workunit.client.1.smithi137.stdout:4/689: write d4/f11 [1263505,53728] 0 2024-03-14T13:02:24.004 INFO:tasks.workunit.client.1.smithi137.stdout:7/632: dwrite d3/d6/d2e/f3e [0,4194304] 0 2024-03-14T13:02:24.004 INFO:tasks.workunit.client.0.smithi111.stdout:9/698: symlink d5/d6/d8/d5b/d84/lf2 0 2024-03-14T13:02:24.005 INFO:tasks.workunit.client.1.smithi137.stdout:2/803: dwrite d2/d7/d2e/f3d [4194304,4194304] 0 2024-03-14T13:02:24.006 INFO:tasks.workunit.client.0.smithi111.stdout:7/611: mkdir d0/d1/d5/d38/d5c/db6 0 2024-03-14T13:02:24.009 INFO:tasks.workunit.client.1.smithi137.stdout:0/702: write d1/d5/d1b/d1e/d53/d6d/f9a [290410,104222] 0 2024-03-14T13:02:24.013 INFO:tasks.workunit.client.1.smithi137.stdout:4/690: read - d4/d27/d5c/d6d/d58/d65/fce zero size 2024-03-14T13:02:24.013 INFO:tasks.workunit.client.1.smithi137.stdout:2/804: write d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 [8687179,120790] 0 2024-03-14T13:02:24.015 INFO:tasks.workunit.client.0.smithi111.stdout:0/873: dwrite d6/d9/d15/d49/d56/fab [0,4194304] 0 2024-03-14T13:02:24.015 INFO:tasks.workunit.client.0.smithi111.stdout:7/612: link d0/d1/d21/d2a/c43 d0/d1/d4a/cb7 0 2024-03-14T13:02:24.019 INFO:tasks.workunit.client.0.smithi111.stdout:7/613: chown d0/d1/d5/d38/d5c/d6d 97 1 2024-03-14T13:02:24.019 INFO:tasks.workunit.client.1.smithi137.stdout:8/754: truncate d1/dc/d3c/d87/d34/fb4 267327 0 2024-03-14T13:02:24.027 INFO:tasks.workunit.client.1.smithi137.stdout:2/805: rmdir d2/d7/d2e/d3e/d44 39 2024-03-14T13:02:24.028 INFO:tasks.workunit.client.1.smithi137.stdout:8/755: creat d1/dc/d4e/fff x:0 0 0 2024-03-14T13:02:24.029 INFO:tasks.workunit.client.0.smithi111.stdout:0/874: creat d6/d5a/f136 x:0 0 0 2024-03-14T13:02:24.030 INFO:tasks.workunit.client.0.smithi111.stdout:9/699: dwrite d5/d42/d71/fca [0,4194304] 0 2024-03-14T13:02:24.032 INFO:tasks.workunit.client.1.smithi137.stdout:3/622: dwrite d4/de/d16/d1c/d5d/f43 [4194304,4194304] 0 2024-03-14T13:02:24.033 INFO:tasks.workunit.client.0.smithi111.stdout:7/614: read d0/d1/d5/d38/f68 [6131987,46949] 0 2024-03-14T13:02:24.037 INFO:tasks.workunit.client.0.smithi111.stdout:7/615: write d0/d1/d21/f3f [10310599,6922] 0 2024-03-14T13:02:24.038 INFO:tasks.workunit.client.0.smithi111.stdout:0/875: creat d6/d9/f137 x:0 0 0 2024-03-14T13:02:24.039 INFO:tasks.workunit.client.1.smithi137.stdout:3/623: symlink d4/de/d16/d1c/db8/dd1/ld2 0 2024-03-14T13:02:24.040 INFO:tasks.workunit.client.0.smithi111.stdout:0/876: dread - d6/d9/de/d77/fe3 zero size 2024-03-14T13:02:24.041 INFO:tasks.workunit.client.0.smithi111.stdout:7/616: unlink d0/d1/l48 0 2024-03-14T13:02:24.043 INFO:tasks.workunit.client.1.smithi137.stdout:8/756: getdents d1/dc/d10/dcf 0 2024-03-14T13:02:24.044 INFO:tasks.workunit.client.1.smithi137.stdout:2/806: creat d2/d7/d2e/d3e/d44/d6b/d22/dbb/f10a x:0 0 0 2024-03-14T13:02:24.044 INFO:tasks.workunit.client.1.smithi137.stdout:7/633: dwrite d3/d6/d59/daa/fb3 [0,4194304] 0 2024-03-14T13:02:24.046 INFO:tasks.workunit.client.0.smithi111.stdout:9/700: dwrite d5/d6/d8/f16 [4194304,4194304] 0 2024-03-14T13:02:24.047 INFO:tasks.workunit.client.1.smithi137.stdout:3/624: write d4/de/d16/fb9 [754321,96957] 0 2024-03-14T13:02:24.048 INFO:tasks.workunit.client.1.smithi137.stdout:4/691: dwrite d4/d27/d5c/d6d/d58/d65/fce [0,4194304] 0 2024-03-14T13:02:24.050 INFO:tasks.workunit.client.0.smithi111.stdout:9/701: truncate d5/d6/d8/d5b/d7c/f94 552847 0 2024-03-14T13:02:24.052 INFO:tasks.workunit.client.0.smithi111.stdout:9/702: chown d5/d6/d14/c67 225 1 2024-03-14T13:02:24.055 INFO:tasks.workunit.client.0.smithi111.stdout:9/703: rmdir d5/d6/d2a/d2d/db8 39 2024-03-14T13:02:24.056 INFO:tasks.workunit.client.1.smithi137.stdout:8/757: creat d1/dc/d10/dcf/f100 x:0 0 0 2024-03-14T13:02:24.057 INFO:tasks.workunit.client.1.smithi137.stdout:4/692: truncate d4/d9/d3a/d70/d80/db7/fcc 66647 0 2024-03-14T13:02:24.061 INFO:tasks.workunit.client.0.smithi111.stdout:6/647: dread d0/d17/d21/d42/f58 [0,4194304] 0 2024-03-14T13:02:24.062 INFO:tasks.workunit.client.1.smithi137.stdout:8/758: rmdir d1/d5e/daf/ded 0 2024-03-14T13:02:24.063 INFO:tasks.workunit.client.0.smithi111.stdout:9/704: mknod d5/d6/d14/cf3 0 2024-03-14T13:02:24.065 INFO:tasks.workunit.client.0.smithi111.stdout:6/648: mknod d0/d17/d20/d5a/d5e/dc6/cf1 0 2024-03-14T13:02:24.066 INFO:tasks.workunit.client.1.smithi137.stdout:8/759: mknod d1/d4a/c101 0 2024-03-14T13:02:24.068 INFO:tasks.workunit.client.0.smithi111.stdout:6/649: rmdir d0/d50 39 2024-03-14T13:02:24.069 INFO:tasks.workunit.client.1.smithi137.stdout:8/760: dread - d1/dc/d71/db0/dca/ffe zero size 2024-03-14T13:02:24.070 INFO:tasks.workunit.client.0.smithi111.stdout:9/705: rename d5/d42/d71/c73 to d5/d42/dd4/cf4 0 2024-03-14T13:02:24.073 INFO:tasks.workunit.client.1.smithi137.stdout:4/693: dwrite d4/d9/d3a/d41/d6a/dac/fb0 [0,4194304] 0 2024-03-14T13:02:24.075 INFO:tasks.workunit.client.1.smithi137.stdout:5/645: truncate d7/d1d/f57 3294598 0 2024-03-14T13:02:24.075 INFO:tasks.workunit.client.1.smithi137.stdout:8/761: dread - d1/dc/d3c/d87/d34/d82/fa7 zero size 2024-03-14T13:02:24.076 INFO:tasks.workunit.client.0.smithi111.stdout:7/617: rmdir d0/d1/d21 39 2024-03-14T13:02:24.076 INFO:tasks.workunit.client.1.smithi137.stdout:0/703: dwrite d1/d5/d1b/d1e/fe1 [0,4194304] 0 2024-03-14T13:02:24.080 INFO:tasks.workunit.client.1.smithi137.stdout:3/625: dread d4/fd [0,4194304] 0 2024-03-14T13:02:24.081 INFO:tasks.workunit.client.0.smithi111.stdout:7/618: creat d0/d1/d5/d38/d5c/fb8 x:0 0 0 2024-03-14T13:02:24.083 INFO:tasks.workunit.client.0.smithi111.stdout:9/706: dwrite d5/d42/d71/d96/fac [0,4194304] 0 2024-03-14T13:02:24.083 INFO:tasks.workunit.client.1.smithi137.stdout:8/762: mkdir d1/dc/d10/d102 0 2024-03-14T13:02:24.087 INFO:tasks.workunit.client.1.smithi137.stdout:0/704: unlink d1/d5/d1b/d1e/d25/d55/d73/f8f 0 2024-03-14T13:02:24.090 INFO:tasks.workunit.client.1.smithi137.stdout:3/626: rename d4/d19/da1 to d4/de/d16/d27/dd3 0 2024-03-14T13:02:24.091 INFO:tasks.workunit.client.0.smithi111.stdout:7/619: read d0/d1/d5/d38/d1b/d9c/fa9 [337803,44148] 0 2024-03-14T13:02:24.092 INFO:tasks.workunit.client.1.smithi137.stdout:3/627: fdatasync d4/de/d16/d1c/d8d/dcd/fba 0 2024-03-14T13:02:24.092 INFO:tasks.workunit.client.1.smithi137.stdout:7/634: dread d3/d5/d23/d2c/d5d/d6a/d75/d78/f83 [0,4194304] 0 2024-03-14T13:02:24.093 INFO:tasks.workunit.client.1.smithi137.stdout:5/646: mkdir d7/d16/d18/d25/d28/d3c/dd8 0 2024-03-14T13:02:24.094 INFO:tasks.workunit.client.1.smithi137.stdout:8/763: write d1/d5e/daf/fc2 [122823,47821] 0 2024-03-14T13:02:24.094 INFO:tasks.workunit.client.1.smithi137.stdout:4/694: dread d4/d9/d3a/d70/d80/fa6 [0,4194304] 0 2024-03-14T13:02:24.095 INFO:tasks.workunit.client.0.smithi111.stdout:6/650: rename d0/d17/d20/d5a/d5e to d0/d50/df2 0 2024-03-14T13:02:24.096 INFO:tasks.workunit.client.1.smithi137.stdout:3/628: truncate d4/f68 789749 0 2024-03-14T13:02:24.098 INFO:tasks.workunit.client.1.smithi137.stdout:4/695: write d4/d10/d5a/f85 [4829013,86563] 0 2024-03-14T13:02:24.098 INFO:tasks.workunit.client.0.smithi111.stdout:0/877: dwrite d6/d9/de/d3a/f61 [0,4194304] 0 2024-03-14T13:02:24.100 INFO:tasks.workunit.client.1.smithi137.stdout:2/807: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 [0,4194304] 0 2024-03-14T13:02:24.104 INFO:tasks.workunit.client.1.smithi137.stdout:4/696: creat d4/d10/d42/d73/d9d/fe6 x:0 0 0 2024-03-14T13:02:24.104 INFO:tasks.workunit.client.0.smithi111.stdout:6/651: unlink d0/f72 0 2024-03-14T13:02:24.105 INFO:tasks.workunit.client.1.smithi137.stdout:0/705: link d1/d44/d72/d7e/cd4 d1/d5/d1b/d42/cee 0 2024-03-14T13:02:24.105 INFO:tasks.workunit.client.0.smithi111.stdout:7/620: rmdir d0/d1/d5/d38/d5c/db6 0 2024-03-14T13:02:24.106 INFO:tasks.workunit.client.1.smithi137.stdout:7/635: mkdir d3/d5/d5a/d86/dc5/dc7 0 2024-03-14T13:02:24.108 INFO:tasks.workunit.client.0.smithi111.stdout:0/878: creat d6/d3d/d97/f138 x:0 0 0 2024-03-14T13:02:24.110 INFO:tasks.workunit.client.1.smithi137.stdout:4/697: chown d4/d27/d5c/d77/f7c 8278826 1 2024-03-14T13:02:24.114 INFO:tasks.workunit.client.1.smithi137.stdout:5/647: dwrite d7/d16/fb9 [0,4194304] 0 2024-03-14T13:02:24.116 INFO:tasks.workunit.client.1.smithi137.stdout:8/764: dread d1/dc/d10/f32 [0,4194304] 0 2024-03-14T13:02:24.116 INFO:tasks.workunit.client.1.smithi137.stdout:3/629: dwrite d4/de/d16/d1c/d8d/dcd/fa5 [0,4194304] 0 2024-03-14T13:02:24.118 INFO:tasks.workunit.client.0.smithi111.stdout:9/707: dread d5/d6/d8/f30 [0,4194304] 0 2024-03-14T13:02:24.119 INFO:tasks.workunit.client.1.smithi137.stdout:5/648: chown d7/d16/ca6 5965 1 2024-03-14T13:02:24.119 INFO:tasks.workunit.client.1.smithi137.stdout:0/706: write d1/d44/d56/f7d [3538112,4572] 0 2024-03-14T13:02:24.124 INFO:tasks.workunit.client.1.smithi137.stdout:8/765: creat d1/dc/d3c/d87/f103 x:0 0 0 2024-03-14T13:02:24.124 INFO:tasks.workunit.client.1.smithi137.stdout:7/636: dread d3/d5/d5a/f73 [0,4194304] 0 2024-03-14T13:02:24.124 INFO:tasks.workunit.client.1.smithi137.stdout:0/707: write d1/d5/d1b/d1e/d53/d6d/f9a [973660,63822] 0 2024-03-14T13:02:24.125 INFO:tasks.workunit.client.1.smithi137.stdout:4/698: creat d4/d9/d3a/d41/d38/d79/dbe/fe7 x:0 0 0 2024-03-14T13:02:24.127 INFO:tasks.workunit.client.0.smithi111.stdout:1/782: sync 2024-03-14T13:02:24.127 INFO:tasks.workunit.client.0.smithi111.stdout:2/617: sync 2024-03-14T13:02:24.127 INFO:tasks.workunit.client.0.smithi111.stdout:5/868: sync 2024-03-14T13:02:24.128 INFO:tasks.workunit.client.1.smithi137.stdout:3/630: rmdir d4/de/d16/d1c 39 2024-03-14T13:02:24.130 INFO:tasks.workunit.client.1.smithi137.stdout:3/631: fdatasync d4/d19/f90 0 2024-03-14T13:02:24.135 INFO:tasks.workunit.client.1.smithi137.stdout:7/637: creat d3/d5/d23/d2c/d5d/d6a/fc8 x:0 0 0 2024-03-14T13:02:24.138 INFO:tasks.workunit.client.1.smithi137.stdout:5/649: rmdir d7/d16/d41 39 2024-03-14T13:02:24.140 INFO:tasks.workunit.client.1.smithi137.stdout:7/638: creat d3/d54/fc9 x:0 0 0 2024-03-14T13:02:24.140 INFO:tasks.workunit.client.1.smithi137.stdout:4/699: dwrite d4/d9/d3a/d41/d6a/dac/fb0 [0,4194304] 0 2024-03-14T13:02:24.141 INFO:tasks.workunit.client.1.smithi137.stdout:5/650: read d7/d16/fbc [33889,107229] 0 2024-03-14T13:02:24.145 INFO:tasks.workunit.client.1.smithi137.stdout:4/700: unlink d4/f13 0 2024-03-14T13:02:24.148 INFO:tasks.workunit.client.1.smithi137.stdout:7/639: mknod d3/d5/d5a/d86/cca 0 2024-03-14T13:02:24.149 INFO:tasks.workunit.client.0.smithi111.stdout:2/618: dread dd/d17/d34/d73/d75/d59/f65 [0,4194304] 0 2024-03-14T13:02:24.153 INFO:tasks.workunit.client.1.smithi137.stdout:5/651: symlink d7/d1d/d74/d83/dc4/ld9 0 2024-03-14T13:02:24.153 INFO:tasks.workunit.client.1.smithi137.stdout:7/640: chown d3/d5/d23/d2c/d5d/d6a/fc8 10 1 2024-03-14T13:02:24.156 INFO:tasks.workunit.client.0.smithi111.stdout:3/708: sync 2024-03-14T13:02:24.157 INFO:tasks.workunit.client.1.smithi137.stdout:5/652: creat d7/d16/d18/d48/d9d/fda x:0 0 0 2024-03-14T13:02:24.159 INFO:tasks.workunit.client.1.smithi137.stdout:5/653: read d7/d16/f98 [207887,22700] 0 2024-03-14T13:02:24.159 INFO:tasks.workunit.client.0.smithi111.stdout:7/621: dread d0/d7e/f19 [4194304,4194304] 0 2024-03-14T13:02:24.159 INFO:tasks.workunit.client.1.smithi137.stdout:8/766: rename d1/dc/d3c/d87/d34/d82/fa7 to d1/dc/d71/db0/dca/f104 0 2024-03-14T13:02:24.165 INFO:tasks.workunit.client.1.smithi137.stdout:7/641: creat d3/d5/d23/d2c/d5d/d6a/d75/fcb x:0 0 0 2024-03-14T13:02:24.166 INFO:tasks.workunit.client.1.smithi137.stdout:4/701: getdents d4/d4d 0 2024-03-14T13:02:24.166 INFO:tasks.workunit.client.0.smithi111.stdout:6/652: creat d0/d17/d21/d69/d3d/d53/d81/da1/dc8/ff3 x:0 0 0 2024-03-14T13:02:24.168 INFO:tasks.workunit.client.1.smithi137.stdout:8/767: mkdir d1/dc/d71/db0/dca/d105 0 2024-03-14T13:02:24.168 INFO:tasks.workunit.client.1.smithi137.stdout:4/702: dread - d4/d9/d3a/d41/d38/d79/fe0 zero size 2024-03-14T13:02:24.170 INFO:tasks.workunit.client.0.smithi111.stdout:0/879: symlink d6/d45/d87/l139 0 2024-03-14T13:02:24.170 INFO:tasks.workunit.client.1.smithi137.stdout:8/768: fdatasync d1/dc/d3c/d87/f103 0 2024-03-14T13:02:24.171 INFO:tasks.workunit.client.1.smithi137.stdout:5/654: unlink d7/d16/d18/d25/d28/fb6 0 2024-03-14T13:02:24.173 INFO:tasks.workunit.client.1.smithi137.stdout:5/655: write d7/d33/fb1 [2013517,89689] 0 2024-03-14T13:02:24.177 INFO:tasks.workunit.client.0.smithi111.stdout:0/880: dread d6/d3d/da6/f130 [0,4194304] 0 2024-03-14T13:02:24.179 INFO:tasks.workunit.client.1.smithi137.stdout:0/708: dread d1/d5/d1b/d1e/d25/d55/d73/f98 [0,4194304] 0 2024-03-14T13:02:24.182 INFO:tasks.workunit.client.1.smithi137.stdout:8/769: mknod d1/d4a/c106 0 2024-03-14T13:02:24.182 INFO:tasks.workunit.client.0.smithi111.stdout:2/619: readlink dd/d17/d53/da4/lb4 0 2024-03-14T13:02:24.183 INFO:tasks.workunit.client.1.smithi137.stdout:7/642: read d3/d52/f9d [1783417,12933] 0 2024-03-14T13:02:24.188 INFO:tasks.workunit.client.1.smithi137.stdout:5/656: dwrite d7/f1b [0,4194304] 0 2024-03-14T13:02:24.188 INFO:tasks.workunit.client.0.smithi111.stdout:0/881: dwrite d6/d3d/f5f [0,4194304] 0 2024-03-14T13:02:24.188 INFO:tasks.workunit.client.1.smithi137.stdout:4/703: creat d4/d9/d3a/d41/fe8 x:0 0 0 2024-03-14T13:02:24.189 INFO:tasks.workunit.client.1.smithi137.stdout:0/709: mknod d1/d5/cef 0 2024-03-14T13:02:24.191 INFO:tasks.workunit.client.0.smithi111.stdout:0/882: stat d6/d3d/d97/d109 0 2024-03-14T13:02:24.194 INFO:tasks.workunit.client.1.smithi137.stdout:8/770: rename d1/l9c to d1/dc/d3c/d87/d34/d82/de6/l107 0 2024-03-14T13:02:24.197 INFO:tasks.workunit.client.1.smithi137.stdout:5/657: chown d7/d16/d18/d2d/f53 1701 1 2024-03-14T13:02:24.198 INFO:tasks.workunit.client.1.smithi137.stdout:7/643: fsync d3/d5/d5a/d80/f88 0 2024-03-14T13:02:24.199 INFO:tasks.workunit.client.1.smithi137.stdout:5/658: write d7/d49/d75/f8c [481540,24294] 0 2024-03-14T13:02:24.201 INFO:tasks.workunit.client.1.smithi137.stdout:7/644: write d3/d54/fc9 [440276,55458] 0 2024-03-14T13:02:24.201 INFO:tasks.workunit.client.1.smithi137.stdout:5/659: fsync d7/d16/fae 0 2024-03-14T13:02:24.203 INFO:tasks.workunit.client.1.smithi137.stdout:7/645: chown d3/cb2 3146 1 2024-03-14T13:02:24.205 INFO:tasks.workunit.client.0.smithi111.stdout:6/653: link d0/ff d0/d17/d20/d4a/d59/d80/ff4 0 2024-03-14T13:02:24.205 INFO:tasks.workunit.client.0.smithi111.stdout:2/620: dread dd/d17/d53/f68 [0,4194304] 0 2024-03-14T13:02:24.207 INFO:tasks.workunit.client.1.smithi137.stdout:0/710: mkdir d1/d5/d1b/d1e/d53/df0 0 2024-03-14T13:02:24.207 INFO:tasks.workunit.client.1.smithi137.stdout:5/660: symlink d7/d16/d18/d25/d28/d3c/d7a/ldb 0 2024-03-14T13:02:24.208 INFO:tasks.workunit.client.0.smithi111.stdout:2/621: write dd/d11/d1a/fb5 [25488,35024] 0 2024-03-14T13:02:24.211 INFO:tasks.workunit.client.1.smithi137.stdout:5/661: chown d7/d16/d18/d48 40686 1 2024-03-14T13:02:24.211 INFO:tasks.workunit.client.0.smithi111.stdout:2/622: read dd/d17/d34/d73/d75/f95 [1136067,25966] 0 2024-03-14T13:02:24.215 INFO:tasks.workunit.client.1.smithi137.stdout:7/646: dwrite d3/f6f [0,4194304] 0 2024-03-14T13:02:24.217 INFO:tasks.workunit.client.1.smithi137.stdout:0/711: getdents d1/d5/d1b/d1e/d25/d8a 0 2024-03-14T13:02:24.217 INFO:tasks.workunit.client.0.smithi111.stdout:6/654: dread d0/ff [4194304,4194304] 0 2024-03-14T13:02:24.220 INFO:tasks.workunit.client.1.smithi137.stdout:7/647: symlink d3/d5/d5a/da8/lcc 0 2024-03-14T13:02:24.222 INFO:tasks.workunit.client.0.smithi111.stdout:6/655: mkdir d0/d17/d24/df5 0 2024-03-14T13:02:24.228 INFO:tasks.workunit.client.1.smithi137.stdout:5/662: dwrite d7/d16/d18/d25/f5b [4194304,4194304] 0 2024-03-14T13:02:24.229 INFO:tasks.workunit.client.1.smithi137.stdout:7/648: symlink d3/d6/d59/da9/lcd 0 2024-03-14T13:02:24.234 INFO:tasks.workunit.client.1.smithi137.stdout:5/663: chown d7/d33/c35 607 1 2024-03-14T13:02:24.236 INFO:tasks.workunit.client.1.smithi137.stdout:7/649: creat d3/d6/d59/daa/fce x:0 0 0 2024-03-14T13:02:24.241 INFO:tasks.workunit.client.1.smithi137.stdout:0/712: dwrite d1/d1f/d82/fb7 [0,4194304] 0 2024-03-14T13:02:24.246 INFO:tasks.workunit.client.1.smithi137.stdout:8/771: dwrite d1/dc/d3c/d87/d34/f40 [0,4194304] 0 2024-03-14T13:02:24.246 INFO:tasks.workunit.client.1.smithi137.stdout:7/650: truncate d3/d6/d59/fbc 889253 0 2024-03-14T13:02:24.249 INFO:tasks.workunit.client.0.smithi111.stdout:5/869: truncate d2/d16/f70 5500958 0 2024-03-14T13:02:24.252 INFO:tasks.workunit.client.0.smithi111.stdout:1/783: truncate d20/d25/f2e 335369 0 2024-03-14T13:02:24.252 INFO:tasks.workunit.client.1.smithi137.stdout:4/704: dread d4/d10/d42/d73/d9d/dbc/fb9 [0,4194304] 0 2024-03-14T13:02:24.253 INFO:tasks.workunit.client.0.smithi111.stdout:7/622: truncate d0/d7e/f81 1384601 0 2024-03-14T13:02:24.253 INFO:tasks.workunit.client.0.smithi111.stdout:5/870: read d2/d13/f4f [160828,86977] 0 2024-03-14T13:02:24.255 INFO:tasks.workunit.client.0.smithi111.stdout:3/709: symlink d0/d2d/d35/d50/da2/l107 0 2024-03-14T13:02:24.255 INFO:tasks.workunit.client.1.smithi137.stdout:5/664: dwrite d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:24.258 INFO:tasks.workunit.client.1.smithi137.stdout:0/713: dwrite d1/d5/d1b/d1e/d25/d26/f39 [0,4194304] 0 2024-03-14T13:02:24.258 INFO:tasks.workunit.client.0.smithi111.stdout:0/883: write d6/d45/d87/fb0 [96408,78353] 0 2024-03-14T13:02:24.259 INFO:tasks.workunit.client.1.smithi137.stdout:8/772: getdents d1/d5e/daf/dee 0 2024-03-14T13:02:24.260 INFO:tasks.workunit.client.1.smithi137.stdout:4/705: mknod d4/ce9 0 2024-03-14T13:02:24.266 INFO:tasks.workunit.client.1.smithi137.stdout:0/714: mknod d1/d5/d1b/d42/d61/cf1 0 2024-03-14T13:02:24.270 INFO:tasks.workunit.client.0.smithi111.stdout:0/884: creat d6/d45/f13a x:0 0 0 2024-03-14T13:02:24.272 INFO:tasks.workunit.client.0.smithi111.stdout:1/784: creat d20/d40/db4/f108 x:0 0 0 2024-03-14T13:02:24.275 INFO:tasks.workunit.client.0.smithi111.stdout:0/885: unlink d6/fad 0 2024-03-14T13:02:24.276 INFO:tasks.workunit.client.0.smithi111.stdout:7/623: link d0/d1/d5/d38/d5c/cad d0/d1/d21/d2a/cb9 0 2024-03-14T13:02:24.277 INFO:tasks.workunit.client.1.smithi137.stdout:8/773: getdents d1/d5e/dc9 0 2024-03-14T13:02:24.278 INFO:tasks.workunit.client.0.smithi111.stdout:2/623: dwrite dd/d17/d53/f92 [4194304,4194304] 0 2024-03-14T13:02:24.278 INFO:tasks.workunit.client.0.smithi111.stdout:7/624: dread - d0/d1/d4a/d52/fb2 zero size 2024-03-14T13:02:24.280 INFO:tasks.workunit.client.1.smithi137.stdout:4/706: link d4/d9/d3a/d41/d38/d55/db6/ldb d4/d9/d3a/d41/d38/d55/lea 0 2024-03-14T13:02:24.283 INFO:tasks.workunit.client.0.smithi111.stdout:9/708: creat d5/d6/ff5 x:0 0 0 2024-03-14T13:02:24.284 INFO:tasks.workunit.client.0.smithi111.stdout:7/625: rmdir d0/d1/d21/d2a/d8c 39 2024-03-14T13:02:24.285 INFO:tasks.workunit.client.0.smithi111.stdout:1/785: dread d20/d25/f39 [0,4194304] 0 2024-03-14T13:02:24.287 INFO:tasks.workunit.client.1.smithi137.stdout:8/774: dwrite d1/dc/d10/dcf/f100 [0,4194304] 0 2024-03-14T13:02:24.288 INFO:tasks.workunit.client.0.smithi111.stdout:0/886: rename d6/d3d/d97/d129/c12f to d6/d3d/d97/d109/c13b 0 2024-03-14T13:02:24.297 INFO:tasks.workunit.client.1.smithi137.stdout:8/775: getdents d1/dc/d3c 0 2024-03-14T13:02:24.298 INFO:tasks.workunit.client.1.smithi137.stdout:7/651: dread d3/d6/d14/d22/f43 [0,4194304] 0 2024-03-14T13:02:24.299 INFO:tasks.workunit.client.1.smithi137.stdout:7/652: chown d3/d54/f61 4862 1 2024-03-14T13:02:24.300 INFO:tasks.workunit.client.1.smithi137.stdout:7/653: dread - d3/d6/d59/daa/fce zero size 2024-03-14T13:02:24.300 INFO:tasks.workunit.client.0.smithi111.stdout:6/656: mkdir d0/d17/d21/d69/d3d/d53/d81/da1/df6 0 2024-03-14T13:02:24.302 INFO:tasks.workunit.client.1.smithi137.stdout:5/665: truncate d7/d1d/d74/f78 55009 0 2024-03-14T13:02:24.302 INFO:tasks.workunit.client.1.smithi137.stdout:0/715: truncate d1/d1f/d82/f8b 2147597 0 2024-03-14T13:02:24.302 INFO:tasks.workunit.client.0.smithi111.stdout:0/887: dwrite d6/d9/de/f6a [0,4194304] 0 2024-03-14T13:02:24.303 INFO:tasks.workunit.client.1.smithi137.stdout:8/776: symlink d1/dc/d71/l108 0 2024-03-14T13:02:24.304 INFO:tasks.workunit.client.1.smithi137.stdout:5/666: stat d7/d16/d18/d25/d28/f3d 0 2024-03-14T13:02:24.307 INFO:tasks.workunit.client.1.smithi137.stdout:8/777: fsync d1/dc/d3c/d87/d34/d82/ffb 0 2024-03-14T13:02:24.307 INFO:tasks.workunit.client.0.smithi111.stdout:9/709: link d5/d6/d14/l29 d5/d42/lf6 0 2024-03-14T13:02:24.307 INFO:tasks.workunit.client.0.smithi111.stdout:1/786: rmdir d20/d25/da8/de9/dfe 0 2024-03-14T13:02:24.309 INFO:tasks.workunit.client.1.smithi137.stdout:0/716: rename d1/d5/d1b/d42 to d1/d5/d1b/d42/d3a/dc6/df2 22 2024-03-14T13:02:24.309 INFO:tasks.workunit.client.1.smithi137.stdout:4/707: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:24.310 INFO:tasks.workunit.client.1.smithi137.stdout:8/778: chown d1/dc/d3c/d87/d34/fc6 490514697 1 2024-03-14T13:02:24.312 INFO:tasks.workunit.client.1.smithi137.stdout:8/779: readlink d1/dc/d77/laa 0 2024-03-14T13:02:24.312 INFO:tasks.workunit.client.0.smithi111.stdout:7/626: getdents d0/d1/d22/d3e 0 2024-03-14T13:02:24.316 INFO:tasks.workunit.client.1.smithi137.stdout:4/708: write d4/d10/d42/f52 [4643941,77059] 0 2024-03-14T13:02:24.316 INFO:tasks.workunit.client.0.smithi111.stdout:3/710: mknod d0/d2d/d35/c108 0 2024-03-14T13:02:24.318 INFO:tasks.workunit.client.1.smithi137.stdout:5/667: dwrite d7/d16/d18/f34 [0,4194304] 0 2024-03-14T13:02:24.320 INFO:tasks.workunit.client.0.smithi111.stdout:0/888: write d6/d9/de/f18 [848815,68061] 0 2024-03-14T13:02:24.321 INFO:tasks.workunit.client.1.smithi137.stdout:7/654: write d3/fe [4984822,89447] 0 2024-03-14T13:02:24.328 INFO:tasks.workunit.client.0.smithi111.stdout:3/711: mknod d0/d2d/d35/d50/da2/db3/d71/c109 0 2024-03-14T13:02:24.330 INFO:tasks.workunit.client.1.smithi137.stdout:5/668: mkdir d7/d16/d18/d2d/dd1/ddc 0 2024-03-14T13:02:24.330 INFO:tasks.workunit.client.1.smithi137.stdout:0/717: dwrite d1/d44/d72/d7e/f94 [0,4194304] 0 2024-03-14T13:02:24.331 INFO:tasks.workunit.client.1.smithi137.stdout:4/709: write d4/d27/d5c/d6d/d58/d65/fce [2447617,35388] 0 2024-03-14T13:02:24.331 INFO:tasks.workunit.client.1.smithi137.stdout:7/655: chown d3/d52/d77/lc0 53736 1 2024-03-14T13:02:24.333 INFO:tasks.workunit.client.1.smithi137.stdout:0/718: creat d1/d5/d1b/d1e/d25/d8a/ff3 x:0 0 0 2024-03-14T13:02:24.335 INFO:tasks.workunit.client.0.smithi111.stdout:5/871: dwrite d2/d16/d49/d7c/d95/d6a/fa8 [0,4194304] 0 2024-03-14T13:02:24.336 INFO:tasks.workunit.client.1.smithi137.stdout:7/656: rename d3/d5/d23/d2c/d5d to d3/d6/d59/daa/dcf 0 2024-03-14T13:02:24.337 INFO:tasks.workunit.client.1.smithi137.stdout:5/669: dwrite d7/d49/d75/f8c [0,4194304] 0 2024-03-14T13:02:24.341 INFO:tasks.workunit.client.1.smithi137.stdout:8/780: write d1/dc/d3c/d87/d34/d97/f2a [4847265,7481] 0 2024-03-14T13:02:24.342 INFO:tasks.workunit.client.1.smithi137.stdout:4/710: dwrite d4/d10/d5a/f63 [0,4194304] 0 2024-03-14T13:02:24.342 INFO:tasks.workunit.client.0.smithi111.stdout:0/889: mknod d6/d9/d15/d49/d56/dea/c13c 0 2024-03-14T13:02:24.344 INFO:tasks.workunit.client.1.smithi137.stdout:8/781: fsync d1/dc/d3c/d87/d34/fc5 0 2024-03-14T13:02:24.344 INFO:tasks.workunit.client.0.smithi111.stdout:3/712: fsync d0/d30/f3d 0 2024-03-14T13:02:24.347 INFO:tasks.workunit.client.1.smithi137.stdout:8/782: truncate d1/dc/d3c/d87/d34/d97/fbb 80340 0 2024-03-14T13:02:24.350 INFO:tasks.workunit.client.0.smithi111.stdout:5/872: mkdir d2/d16/d49/d7c/d117 0 2024-03-14T13:02:24.354 INFO:tasks.workunit.client.1.smithi137.stdout:8/783: creat d1/d5e/daf/dee/f109 x:0 0 0 2024-03-14T13:02:24.355 INFO:tasks.workunit.client.1.smithi137.stdout:5/670: mknod d7/d16/d18/d25/d28/d3c/dd8/cdd 0 2024-03-14T13:02:24.356 INFO:tasks.workunit.client.1.smithi137.stdout:4/711: symlink d4/d27/d5c/d6d/d58/leb 0 2024-03-14T13:02:24.359 INFO:tasks.workunit.client.1.smithi137.stdout:5/671: symlink d7/d16/d18/d25/d28/dc5/lde 0 2024-03-14T13:02:24.360 INFO:tasks.workunit.client.0.smithi111.stdout:9/710: dwrite d5/d42/f95 [0,4194304] 0 2024-03-14T13:02:24.361 INFO:tasks.workunit.client.1.smithi137.stdout:5/672: chown d7/d16/d18/d25/f46 7294125 1 2024-03-14T13:02:24.362 INFO:tasks.workunit.client.0.smithi111.stdout:2/624: dwrite dd/d17/d34/d73/d49/f51 [0,4194304] 0 2024-03-14T13:02:24.362 INFO:tasks.workunit.client.1.smithi137.stdout:8/784: rename d1/dc/d3c/d87/d34/d86/lde to d1/dc/d10/dcf/l10a 0 2024-03-14T13:02:24.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.362+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd180100b10 con 0x7fd16c06e7c0 2024-03-14T13:02:24.364 INFO:tasks.workunit.client.0.smithi111.stdout:6/657: dread d0/d17/d20/d4a/d59/d80/ff4 [0,4194304] 0 2024-03-14T13:02:24.366 INFO:tasks.workunit.client.0.smithi111.stdout:0/890: mknod d6/d9/d15/c13d 0 2024-03-14T13:02:24.366 INFO:tasks.workunit.client.1.smithi137.stdout:8/785: write d1/dc/fcd [541959,2718] 0 2024-03-14T13:02:24.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.367+0000 7fd175ffb700 1 -- 172.21.15.111:0/2710795653 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7fd180100b10 con 0x7fd16c06e7c0 2024-03-14T13:02:24.368 INFO:tasks.workunit.client.0.smithi111.stdout:2/625: rmdir dd/d17/d34/d4f 39 2024-03-14T13:02:24.368 INFO:tasks.workunit.client.1.smithi137.stdout:5/673: creat d7/d16/d18/d2d/dd1/fdf x:0 0 0 2024-03-14T13:02:24.369 INFO:tasks.workunit.client.0.smithi111.stdout:5/873: creat d2/d16/d49/d33/de5/f118 x:0 0 0 2024-03-14T13:02:24.369 INFO:tasks.workunit.client.1.smithi137.stdout:0/719: rmdir d1/d5/d1b 39 2024-03-14T13:02:24.370 INFO:tasks.workunit.client.1.smithi137.stdout:4/712: dwrite d4/d27/d5c/d6d/d58/fc1 [0,4194304] 0 2024-03-14T13:02:24.370 INFO:tasks.workunit.client.0.smithi111.stdout:1/787: getdents d20/d21/dd7 0 2024-03-14T13:02:24.370 INFO:tasks.workunit.client.0.smithi111.stdout:7/627: write d0/f7 [3451522,20656] 0 2024-03-14T13:02:24.372 INFO:tasks.workunit.client.0.smithi111.stdout:3/713: mkdir d0/d2d/d35/d69/d10a 0 2024-03-14T13:02:24.373 INFO:tasks.workunit.client.1.smithi137.stdout:8/786: mknod d1/dc/d3c/d87/d34/c10b 0 2024-03-14T13:02:24.377 INFO:tasks.workunit.client.1.smithi137.stdout:4/713: chown d4/d9/d3a/d41/d38 1172102 1 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 msgr2=0x7fd16c070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 0x7fd16c070c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd170005010 tx=0x7fd170009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 msgr2=0x7fd18006e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd18006e4d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00b580 tx=0x7fd17c0091a0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 shutdown_connections 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1800fb440 0x7fd18006e4d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd16c06e7c0 0x7fd16c070c70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 --2- 172.21.15.111:0/2710795653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1800fc6f0 0x7fd18006ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 >> 172.21.15.111:0/2710795653 conn(0x7fd1800f6ba0 msgr2=0x7fd1800f8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 shutdown_connections 2024-03-14T13:02:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.372+0000 7fd187482700 1 -- 172.21.15.111:0/2710795653 wait complete. 2024-03-14T13:02:24.380 INFO:tasks.workunit.client.0.smithi111.stdout:0/891: truncate d6/d9/de/f1d 1713477 0 2024-03-14T13:02:24.380 INFO:tasks.workunit.client.0.smithi111.stdout:3/714: chown d0/d30/lc1 5 1 2024-03-14T13:02:24.380 INFO:tasks.workunit.client.0.smithi111.stdout:3/715: readlink d0/d30/l53 0 2024-03-14T13:02:24.380 INFO:tasks.workunit.client.0.smithi111.stdout:2/626: dwrite dd/d11/d1a/f1c [0,4194304] 0 2024-03-14T13:02:24.380 INFO:tasks.workunit.client.0.smithi111.stdout:1/788: mknod d20/d25/d31/d80/c109 0 2024-03-14T13:02:24.381 INFO:tasks.workunit.client.1.smithi137.stdout:7/657: dread d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:24.381 INFO:tasks.workunit.client.1.smithi137.stdout:5/674: creat d7/fe0 x:0 0 0 2024-03-14T13:02:24.381 INFO:tasks.workunit.client.0.smithi111.stdout:7/628: mkdir d0/d1/d21/dba 0 2024-03-14T13:02:24.384 INFO:tasks.workunit.client.1.smithi137.stdout:8/787: link d1/dc/d10/fa4 d1/dc/da9/f10c 0 2024-03-14T13:02:24.384 INFO:tasks.workunit.client.1.smithi137.stdout:0/720: chown d1/d5/d1b/d1e/d25/d8a/f21 4423 1 2024-03-14T13:02:24.386 INFO:tasks.workunit.client.1.smithi137.stdout:4/714: mknod d4/d9/cec 0 2024-03-14T13:02:24.387 INFO:tasks.workunit.client.1.smithi137.stdout:8/788: creat d1/dc/d3c/d87/d34/d86/f10d x:0 0 0 2024-03-14T13:02:24.388 INFO:tasks.workunit.client.0.smithi111.stdout:2/627: readlink dd/d17/d34/d74/lad 0 2024-03-14T13:02:24.388 INFO:tasks.workunit.client.0.smithi111.stdout:1/789: chown d20/d25/d87/fb8 1089 1 2024-03-14T13:02:24.389 INFO:tasks.workunit.client.1.smithi137.stdout:7/658: dread d3/d5/f67 [0,4194304] 0 2024-03-14T13:02:24.390 INFO:tasks.workunit.client.1.smithi137.stdout:7/659: readlink d3/d6/d59/l62 0 2024-03-14T13:02:24.392 INFO:tasks.workunit.client.0.smithi111.stdout:3/716: mkdir d0/d2d/d35/d50/da2/db3/d6f/d10b 0 2024-03-14T13:02:24.392 INFO:tasks.workunit.client.1.smithi137.stdout:8/789: mknod d1/dc/d71/db0/dca/d105/c10e 0 2024-03-14T13:02:24.394 INFO:tasks.workunit.client.1.smithi137.stdout:7/660: creat d3/fd0 x:0 0 0 2024-03-14T13:02:24.395 INFO:tasks.workunit.client.0.smithi111.stdout:3/717: chown d0/d30/c3b 1 1 2024-03-14T13:02:24.396 INFO:tasks.workunit.client.0.smithi111.stdout:3/718: chown d0/d2d/d35/d4c/d82 4968342 1 2024-03-14T13:02:24.398 INFO:tasks.workunit.client.1.smithi137.stdout:7/661: mkdir d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1 0 2024-03-14T13:02:24.399 INFO:tasks.workunit.client.0.smithi111.stdout:1/790: read d20/d21/d3f/f83 [1660895,65992] 0 2024-03-14T13:02:24.401 INFO:tasks.workunit.client.0.smithi111.stdout:5/874: write d2/f6 [1493747,41395] 0 2024-03-14T13:02:24.405 INFO:tasks.workunit.client.1.smithi137.stdout:7/662: dwrite d3/d6/d14/d22/f43 [0,4194304] 0 2024-03-14T13:02:24.407 INFO:tasks.workunit.client.0.smithi111.stdout:5/875: unlink d2/d45/d76/d9a/d9f/cf8 0 2024-03-14T13:02:24.407 INFO:tasks.workunit.client.0.smithi111.stdout:3/719: dwrite d0/d2d/d35/d50/da2/db3/d6f/f79 [0,4194304] 0 2024-03-14T13:02:24.408 INFO:tasks.workunit.client.1.smithi137.stdout:7/663: mkdir d3/d6/d64/d97/dd2 0 2024-03-14T13:02:24.408 INFO:tasks.workunit.client.0.smithi111.stdout:1/791: dread d20/d25/da8/de9/fdb [0,4194304] 0 2024-03-14T13:02:24.410 INFO:tasks.workunit.client.1.smithi137.stdout:7/664: creat d3/d52/d77/fd3 x:0 0 0 2024-03-14T13:02:24.412 INFO:tasks.workunit.client.0.smithi111.stdout:3/720: chown d0/d30/lfe 250 1 2024-03-14T13:02:24.414 INFO:tasks.workunit.client.1.smithi137.stdout:7/665: truncate d3/d6/d2e/f49 589318 0 2024-03-14T13:02:24.416 INFO:tasks.workunit.client.1.smithi137.stdout:7/666: creat d3/d5/d23/d2c/fd4 x:0 0 0 2024-03-14T13:02:24.416 INFO:tasks.workunit.client.0.smithi111.stdout:3/721: dread - d0/d2d/d35/d50/da2/db3/da7/fe7 zero size 2024-03-14T13:02:24.417 INFO:tasks.workunit.client.0.smithi111.stdout:3/722: read - d0/d2d/d35/d69/fd5 zero size 2024-03-14T13:02:24.417 INFO:tasks.workunit.client.1.smithi137.stdout:8/790: truncate d1/dc/d10/fa4 2668752 0 2024-03-14T13:02:24.420 INFO:tasks.workunit.client.1.smithi137.stdout:8/791: creat d1/dc/d71/db0/f10f x:0 0 0 2024-03-14T13:02:24.421 INFO:tasks.workunit.client.0.smithi111.stdout:0/892: dread d6/f1f [0,4194304] 0 2024-03-14T13:02:24.421 INFO:tasks.workunit.client.1.smithi137.stdout:0/721: write d1/d5/d1b/d42/d3a/fc2 [155526,13459] 0 2024-03-14T13:02:24.422 INFO:tasks.workunit.client.0.smithi111.stdout:0/893: fsync d6/d9/de/d3a/fd9 0 2024-03-14T13:02:24.424 INFO:tasks.workunit.client.1.smithi137.stdout:0/722: rmdir d1/d5/d1b 39 2024-03-14T13:02:24.425 INFO:tasks.workunit.client.0.smithi111.stdout:3/723: write d0/d2d/d35/f92 [4707042,114604] 0 2024-03-14T13:02:24.429 INFO:tasks.workunit.client.1.smithi137.stdout:4/715: dwrite d4/d4d/fa0 [0,4194304] 0 2024-03-14T13:02:24.430 INFO:tasks.workunit.client.0.smithi111.stdout:5/876: getdents d2/d16/d49/d44/d4a 0 2024-03-14T13:02:24.435 INFO:tasks.workunit.client.1.smithi137.stdout:4/716: creat d4/d9/d3a/d41/d6a/dac/fed x:0 0 0 2024-03-14T13:02:24.437 INFO:tasks.workunit.client.1.smithi137.stdout:4/717: chown d4/d27/d5c/d6d 81 1 2024-03-14T13:02:24.440 INFO:tasks.workunit.client.0.smithi111.stdout:3/724: mknod d0/d30/c10c 0 2024-03-14T13:02:24.441 INFO:tasks.workunit.client.0.smithi111.stdout:2/628: dread dd/d17/f83 [0,4194304] 0 2024-03-14T13:02:24.441 INFO:tasks.workunit.client.0.smithi111.stdout:1/792: creat d20/d21/d3f/f10a x:0 0 0 2024-03-14T13:02:24.441 INFO:tasks.workunit.client.1.smithi137.stdout:4/718: fsync d4/d27/d5c/d77/f7c 0 2024-03-14T13:02:24.442 INFO:tasks.workunit.client.1.smithi137.stdout:0/723: dwrite d1/d5/d1b/d42/d3a/dc6/fea [0,4194304] 0 2024-03-14T13:02:24.445 INFO:tasks.workunit.client.0.smithi111.stdout:7/629: dwrite d0/d1/d4a/d52/f62 [0,4194304] 0 2024-03-14T13:02:24.446 INFO:tasks.workunit.client.0.smithi111.stdout:5/877: unlink d2/d16/d49/d44/d4a/d9e/lda 0 2024-03-14T13:02:24.447 INFO:tasks.workunit.client.1.smithi137.stdout:7/667: dread d3/d6/d14/f1a [0,4194304] 0 2024-03-14T13:02:24.447 INFO:tasks.workunit.client.1.smithi137.stdout:4/719: rename d4/d9/d3a/d70/d80/d8d/la7 to d4/d9/d3a/d70/d80/lee 0 2024-03-14T13:02:24.449 INFO:tasks.workunit.client.1.smithi137.stdout:0/724: dread d1/f9b [0,4194304] 0 2024-03-14T13:02:24.453 INFO:tasks.workunit.client.0.smithi111.stdout:0/894: dwrite d6/d9/d5b/d9e/fa8 [0,4194304] 0 2024-03-14T13:02:24.454 INFO:tasks.workunit.client.0.smithi111.stdout:5/878: read - d2/d16/d49/d44/d4a/d9e/db7/f104 zero size 2024-03-14T13:02:24.454 INFO:tasks.workunit.client.1.smithi137.stdout:7/668: mknod d3/d4a/d7d/cd5 0 2024-03-14T13:02:24.456 INFO:tasks.workunit.client.1.smithi137.stdout:8/792: dread d1/dc/d3c/d87/d34/d97/fc0 [0,4194304] 0 2024-03-14T13:02:24.456 INFO:tasks.workunit.client.0.smithi111.stdout:3/725: truncate d0/d2d/d35/d69/de6/f9e 456463 0 2024-03-14T13:02:24.456 INFO:tasks.workunit.client.1.smithi137.stdout:4/720: dwrite d4/f48 [4194304,4194304] 0 2024-03-14T13:02:24.457 INFO:tasks.workunit.client.1.smithi137.stdout:0/725: symlink d1/d5/d1b/d1e/d25/d26/lf4 0 2024-03-14T13:02:24.462 INFO:tasks.workunit.client.0.smithi111.stdout:5/879: creat d2/d16/d49/d7c/d8b/f119 x:0 0 0 2024-03-14T13:02:24.462 INFO:tasks.workunit.client.1.smithi137.stdout:0/726: write d1/fb3 [97282,6227] 0 2024-03-14T13:02:24.464 INFO:tasks.workunit.client.0.smithi111.stdout:7/630: dwrite d0/d1/d5/d38/d5c/f8d [0,4194304] 0 2024-03-14T13:02:24.465 INFO:tasks.workunit.client.1.smithi137.stdout:7/669: symlink d3/d6/d59/daa/dcf/d6a/da1/ld6 0 2024-03-14T13:02:24.465 INFO:tasks.workunit.client.1.smithi137.stdout:8/793: unlink d1/dc/d71/fba 0 2024-03-14T13:02:24.468 INFO:tasks.workunit.client.1.smithi137.stdout:0/727: mknod d1/d5/d1b/d42/d3a/dc8/cf5 0 2024-03-14T13:02:24.470 INFO:tasks.workunit.client.1.smithi137.stdout:4/721: dwrite d4/d9/d8e/fd8 [0,4194304] 0 2024-03-14T13:02:24.471 INFO:tasks.workunit.client.0.smithi111.stdout:7/631: stat d0/d1/d21/c17 0 2024-03-14T13:02:24.471 INFO:tasks.workunit.client.0.smithi111.stdout:2/629: rename dd/d17/d34/d4f to dd/d8d/dc2 0 2024-03-14T13:02:24.473 INFO:tasks.workunit.client.1.smithi137.stdout:0/728: write d1/d5/d1b/d1e/d25/d8a/fd6 [96515,19310] 0 2024-03-14T13:02:24.475 INFO:tasks.workunit.client.1.smithi137.stdout:8/794: chown d1/dc/d3c/d87/d34/d97/fbb 11531 1 2024-03-14T13:02:24.480 INFO:tasks.workunit.client.1.smithi137.stdout:0/729: rmdir d1/d44/d72 39 2024-03-14T13:02:24.481 INFO:tasks.workunit.client.1.smithi137.stdout:7/670: dwrite d3/d5/d23/fb5 [0,4194304] 0 2024-03-14T13:02:24.482 INFO:tasks.workunit.client.1.smithi137.stdout:4/722: chown d4/d9/d3a/d41/d38/d55/lea 828 1 2024-03-14T13:02:24.484 INFO:tasks.workunit.client.1.smithi137.stdout:4/723: fsync d4/d9/d3a/d41/f21 0 2024-03-14T13:02:24.486 INFO:tasks.workunit.client.1.smithi137.stdout:0/730: mkdir d1/d1f/d80/df6 0 2024-03-14T13:02:24.488 INFO:tasks.workunit.client.1.smithi137.stdout:7/671: chown d3/d6/d14/d22/f43 337149967 1 2024-03-14T13:02:24.490 INFO:tasks.workunit.client.0.smithi111.stdout:0/895: symlink d6/d9/d15/d49/l13e 0 2024-03-14T13:02:24.490 INFO:tasks.workunit.client.0.smithi111.stdout:7/632: creat d0/d1/d5/d38/d1b/d9c/fbb x:0 0 0 2024-03-14T13:02:24.493 INFO:tasks.workunit.client.1.smithi137.stdout:7/672: creat d3/d5/d5a/d86/fd7 x:0 0 0 2024-03-14T13:02:24.495 INFO:tasks.workunit.client.1.smithi137.stdout:0/731: creat d1/d44/ff7 x:0 0 0 2024-03-14T13:02:24.498 INFO:tasks.workunit.client.1.smithi137.stdout:7/673: symlink d3/d6/d64/d97/ld8 0 2024-03-14T13:02:24.500 INFO:tasks.workunit.client.1.smithi137.stdout:4/724: dwrite d4/d27/d5c/d6d/d58/f7e [4194304,4194304] 0 2024-03-14T13:02:24.501 INFO:tasks.workunit.client.1.smithi137.stdout:7/674: chown d3/d5/d23/d3f/d5b/l69 11 1 2024-03-14T13:02:24.502 INFO:tasks.workunit.client.0.smithi111.stdout:3/726: dwrite d0/d2d/d35/d50/da2/db3/da7/fe7 [0,4194304] 0 2024-03-14T13:02:24.503 INFO:tasks.workunit.client.1.smithi137.stdout:4/725: fdatasync d4/d9/d3a/d41/d38/d79/dbe/fe7 0 2024-03-14T13:02:24.504 INFO:tasks.workunit.client.1.smithi137.stdout:4/726: rmdir d4/d9/d8e/dae 39 2024-03-14T13:02:24.505 INFO:tasks.workunit.client.0.smithi111.stdout:1/793: dwrite d20/d21/d28/d5c/fae [0,4194304] 0 2024-03-14T13:02:24.507 INFO:tasks.workunit.client.0.smithi111.stdout:3/727: chown d0/d2d/d35/d50/fc2 14847192 1 2024-03-14T13:02:24.509 INFO:tasks.workunit.client.1.smithi137.stdout:4/727: unlink d4/d27/l99 0 2024-03-14T13:02:24.510 INFO:tasks.workunit.client.0.smithi111.stdout:1/794: read - d20/d21/d3f/f10a zero size 2024-03-14T13:02:24.514 INFO:tasks.workunit.client.0.smithi111.stdout:2/630: write dd/d17/d34/d73/d75/f5d [3174177,23667] 0 2024-03-14T13:02:24.517 INFO:tasks.workunit.client.1.smithi137.stdout:0/732: rename d1/d5/d1b/d42/d3a/dc8 to d1/d44/d72/df8 0 2024-03-14T13:02:24.520 INFO:tasks.workunit.client.0.smithi111.stdout:7/633: dwrite d0/d1/d21/d2a/f9b [0,4194304] 0 2024-03-14T13:02:24.520 INFO:tasks.workunit.client.0.smithi111.stdout:1/795: symlink d20/d21/d28/d5c/l10b 0 2024-03-14T13:02:24.521 INFO:tasks.workunit.client.1.smithi137.stdout:7/675: rename d3/d6/d14/d22/f43 to d3/d6/d64/d97/fd9 0 2024-03-14T13:02:24.527 INFO:tasks.workunit.client.1.smithi137.stdout:0/733: dwrite d1/d44/d72/fca [0,4194304] 0 2024-03-14T13:02:24.530 INFO:tasks.workunit.client.1.smithi137.stdout:0/734: getdents d1/d44/d72/df8 0 2024-03-14T13:02:24.532 INFO:tasks.workunit.client.1.smithi137.stdout:0/735: fdatasync d1/d5/d1b/d42/d61/dae/fd2 0 2024-03-14T13:02:24.532 INFO:tasks.workunit.client.0.smithi111.stdout:7/634: dwrite d0/d1/d21/d2a/f9b [0,4194304] 0 2024-03-14T13:02:24.535 INFO:tasks.workunit.client.1.smithi137.stdout:0/736: creat d1/d5/ff9 x:0 0 0 2024-03-14T13:02:24.536 INFO:tasks.workunit.client.0.smithi111.stdout:5/880: dread d2/d16/d49/d44/d4a/d9e/db7/dbb/fd2 [0,4194304] 0 2024-03-14T13:02:24.536 INFO:tasks.workunit.client.1.smithi137.stdout:0/737: fsync d1/d5/d1b/d1e/d25/d55/fb4 0 2024-03-14T13:02:24.539 INFO:tasks.workunit.client.1.smithi137.stdout:0/738: creat d1/d44/ffa x:0 0 0 2024-03-14T13:02:24.541 INFO:tasks.workunit.client.1.smithi137.stdout:8/795: dread d1/f44 [0,4194304] 0 2024-03-14T13:02:24.542 INFO:tasks.workunit.client.0.smithi111.stdout:5/881: truncate d2/d16/d49/d33/de5/f118 218566 0 2024-03-14T13:02:24.542 INFO:tasks.workunit.client.1.smithi137.stdout:0/739: mknod d1/d5/d1b/d42/d3a/db0/cfb 0 2024-03-14T13:02:24.542 INFO:tasks.workunit.client.1.smithi137.stdout:8/796: chown d1/l6b 440 1 2024-03-14T13:02:24.548 INFO:tasks.workunit.client.1.smithi137.stdout:0/740: link d1/d5/d1b/d42/d3a/dc6/fc7 d1/da6/ffc 0 2024-03-14T13:02:24.549 INFO:tasks.workunit.client.0.smithi111.stdout:0/896: dread d6/d9/de/d3a/d46/f66 [0,4194304] 0 2024-03-14T13:02:24.552 INFO:tasks.workunit.client.0.smithi111.stdout:1/796: dwrite d20/d21/d3f/d59/f74 [0,4194304] 0 2024-03-14T13:02:24.554 INFO:tasks.workunit.client.1.smithi137.stdout:0/741: link d1/d5/d1b/c59 d1/d5/d1b/d1e/d25/d8a/d4a/cfd 0 2024-03-14T13:02:24.556 INFO:tasks.workunit.client.1.smithi137.stdout:0/742: write d1/d5/d1b/d1e/d25/d55/f60 [4968907,12151] 0 2024-03-14T13:02:24.559 INFO:tasks.workunit.client.1.smithi137.stdout:0/743: mknod d1/d1f/cfe 0 2024-03-14T13:02:24.562 INFO:tasks.workunit.client.1.smithi137.stdout:0/744: fsync d1/d44/d56/f70 0 2024-03-14T13:02:24.564 INFO:tasks.workunit.client.0.smithi111.stdout:3/728: mknod d0/d2d/d35/c10d 0 2024-03-14T13:02:24.564 INFO:tasks.workunit.client.0.smithi111.stdout:7/635: mknod d0/d1/d22/d3e/cbc 0 2024-03-14T13:02:24.565 INFO:tasks.workunit.client.0.smithi111.stdout:7/636: chown d0/f7 27340 1 2024-03-14T13:02:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.566+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/3567416781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41046a0 msgr2=0x7f0dc4104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.566+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/3567416781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41046a0 0x7f0dc4104b10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0db4004940 tx=0x7f0db4009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/3567416781 shutdown_connections 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/3567416781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41046a0 0x7f0dc4104b10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/3567416781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41033f0 0x7f0dc4103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/3567416781 >> 172.21.15.111:0/3567416781 conn(0x7f0dc4072560 msgr2=0x7f0dc40749b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/3567416781 shutdown_connections 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/3567416781 wait complete. 2024-03-14T13:02:24.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.567+0000 7f0dc9f4f700 1 Processor -- start 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.568+0000 7f0dc9f4f700 1 -- start start 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.568+0000 7f0dc9f4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41033f0 0x7f0dc4113f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.568+0000 7f0dc9f4f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.568+0000 7f0dc9f4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4114ab0 con 0x7f0dc41033f0 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.568+0000 7f0dc9f4f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc41151a0 con 0x7f0dc41046a0 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:44150/0 (socket says 172.21.15.111:44150) 2024-03-14T13:02:24.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2028636155 learned_addr learned my addr 172.21.15.111:0/2028636155 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:24.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41033f0 0x7f0dc4113f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:24.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41033f0 msgr2=0x7f0dc4113f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41033f0 0x7f0dc4113f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.569+0000 7f0dc2ffd700 1 -- 172.21.15.111:0/2028636155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4005040 con 0x7f0dc41046a0 2024-03-14T13:02:24.571 INFO:tasks.workunit.client.0.smithi111.stdout:3/729: chown d0/d2d/d35/d69/d84 141 1 2024-03-14T13:02:24.571 INFO:tasks.workunit.client.0.smithi111.stdout:2/631: symlink dd/d17/d34/dbf/lc3 0 2024-03-14T13:02:24.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.570+0000 7f0dc2ffd700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007d40 tx=0x7f0db4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:24.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.570+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db400e030 con 0x7f0dc41046a0 2024-03-14T13:02:24.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.570+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0db40093f0 con 0x7f0dc41046a0 2024-03-14T13:02:24.571 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.570+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0db400b410 con 0x7f0dc41046a0 2024-03-14T13:02:24.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.571+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc4115420 con 0x7f0dc41046a0 2024-03-14T13:02:24.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.571+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4115960 con 0x7f0dc41046a0 2024-03-14T13:02:24.572 INFO:tasks.workunit.client.1.smithi137.stdout:4/728: dwrite d4/d9/d3a/f56 [0,4194304] 0 2024-03-14T13:02:24.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.572+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc410c5b0 con 0x7f0dc41046a0 2024-03-14T13:02:24.573 INFO:tasks.workunit.client.0.smithi111.stdout:0/897: rename d6/ca to d6/d9/de/d3a/d46/d4a/c13f 0 2024-03-14T13:02:24.574 INFO:tasks.workunit.client.0.smithi111.stdout:0/898: write d6/d45/d87/f10a [1451455,39649] 0 2024-03-14T13:02:24.575 INFO:tasks.workunit.client.1.smithi137.stdout:0/745: dwrite d1/d5/d1b/d1e/d25/fa8 [0,4194304] 0 2024-03-14T13:02:24.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.576+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f0db401c400 con 0x7f0dc41046a0 2024-03-14T13:02:24.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.576+0000 7f0dabfff700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 0x7f0dac06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:24.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.576+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0db408d5f0 con 0x7f0dc41046a0 2024-03-14T13:02:24.579 INFO:tasks.workunit.client.0.smithi111.stdout:1/797: creat d20/d25/f10c x:0 0 0 2024-03-14T13:02:24.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.580+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 0x7f0dac06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:24.581 INFO:tasks.workunit.client.0.smithi111.stdout:7/637: rename d0/d1/d4a/cb7 to d0/d7e/cbd 0 2024-03-14T13:02:24.581 INFO:tasks.workunit.client.0.smithi111.stdout:3/730: link d0/d2d/d35/d69/c83 d0/d2d/d35/d69/d84/c10e 0 2024-03-14T13:02:24.582 INFO:tasks.workunit.client.1.smithi137.stdout:0/746: read d1/d5/d1b/d42/f32 [1383015,43214] 0 2024-03-14T13:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.582+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f0db405b6e0 con 0x7f0dc41046a0 2024-03-14T13:02:24.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.582+0000 7f0dc37fe700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 0x7f0dac06ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004600 tx=0x7f0db800b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:24.585 INFO:tasks.workunit.client.0.smithi111.stdout:1/798: creat d20/d25/d75/df7/f10d x:0 0 0 2024-03-14T13:02:24.586 INFO:tasks.workunit.client.1.smithi137.stdout:0/747: write d1/d5/d1b/d1e/d25/d8a/f14 [1724496,12953] 0 2024-03-14T13:02:24.586 INFO:tasks.workunit.client.1.smithi137.stdout:4/729: rmdir d4/d9/d3a/d41/d38/d79 39 2024-03-14T13:02:24.587 INFO:tasks.workunit.client.0.smithi111.stdout:3/731: creat d0/d2d/d35/d50/d6c/d98/f10f x:0 0 0 2024-03-14T13:02:24.587 INFO:tasks.workunit.client.1.smithi137.stdout:7/676: truncate d3/d5/d5a/fc6 523253 0 2024-03-14T13:02:24.589 INFO:tasks.workunit.client.0.smithi111.stdout:1/799: write d20/d21/d3f/d59/f74 [3772307,117449] 0 2024-03-14T13:02:24.589 INFO:tasks.workunit.client.0.smithi111.stdout:3/732: fsync d0/d2d/d35/d4c/db0/fed 0 2024-03-14T13:02:24.590 INFO:tasks.workunit.client.1.smithi137.stdout:4/730: creat d4/d10/d42/fef x:0 0 0 2024-03-14T13:02:24.592 INFO:tasks.workunit.client.1.smithi137.stdout:7/677: rename d3/d6/d59/da9/lad to d3/d6/d59/lda 0 2024-03-14T13:02:24.594 INFO:tasks.workunit.client.0.smithi111.stdout:0/899: rename d6/d9/de/d3a/d46/d4a/fb4 to d6/d9/f140 0 2024-03-14T13:02:24.594 INFO:tasks.workunit.client.1.smithi137.stdout:0/748: creat d1/d5/d1b/d42/fff x:0 0 0 2024-03-14T13:02:24.595 INFO:tasks.workunit.client.1.smithi137.stdout:4/731: unlink d4/d9/d3a/d70/d80/db7/lcb 0 2024-03-14T13:02:24.595 INFO:tasks.workunit.client.1.smithi137.stdout:7/678: fsync d3/d5/d5a/d80/fae 0 2024-03-14T13:02:24.597 INFO:tasks.workunit.client.1.smithi137.stdout:4/732: fdatasync d4/d10/f3f 0 2024-03-14T13:02:24.598 INFO:tasks.workunit.client.0.smithi111.stdout:1/800: unlink d20/d25/d75/c94 0 2024-03-14T13:02:24.600 INFO:tasks.workunit.client.0.smithi111.stdout:5/882: dread d2/d45/fff [0,4194304] 0 2024-03-14T13:02:24.600 INFO:tasks.workunit.client.1.smithi137.stdout:7/679: symlink d3/d54/ldb 0 2024-03-14T13:02:24.602 INFO:tasks.workunit.client.1.smithi137.stdout:8/797: dread d1/dc/d3c/d87/d34/d82/fd5 [0,4194304] 0 2024-03-14T13:02:24.603 INFO:tasks.workunit.client.1.smithi137.stdout:7/680: write d3/d6/d59/f5f [4694360,119066] 0 2024-03-14T13:02:24.605 INFO:tasks.workunit.client.0.smithi111.stdout:5/883: creat d2/d16/d49/d7c/d95/d36/f11a x:0 0 0 2024-03-14T13:02:24.605 INFO:tasks.workunit.client.0.smithi111.stdout:1/801: getdents d20/d25/da8/ddd 0 2024-03-14T13:02:24.609 INFO:tasks.workunit.client.1.smithi137.stdout:4/733: link d4/d10/l36 d4/d9/d8e/lf0 0 2024-03-14T13:02:24.609 INFO:tasks.workunit.client.0.smithi111.stdout:2/632: write dd/d11/d1a/d88/f8a [727769,111419] 0 2024-03-14T13:02:24.609 INFO:tasks.workunit.client.1.smithi137.stdout:9/799: sync 2024-03-14T13:02:24.610 INFO:tasks.workunit.client.1.smithi137.stdout:7/681: rmdir d3/d6/d59/daa/dcf/d6a/d75 39 2024-03-14T13:02:24.611 INFO:tasks.workunit.client.0.smithi111.stdout:1/802: unlink d20/d40/f9b 0 2024-03-14T13:02:24.611 INFO:tasks.workunit.client.0.smithi111.stdout:7/638: write d0/d7e/fb [1007132,81950] 0 2024-03-14T13:02:24.613 INFO:tasks.workunit.client.0.smithi111.stdout:2/633: stat dd/d17/d53/f84 0 2024-03-14T13:02:24.614 INFO:tasks.workunit.client.0.smithi111.stdout:5/884: chown d2/d16/d49/d44/d72/lb1 81 1 2024-03-14T13:02:24.614 INFO:tasks.workunit.client.1.smithi137.stdout:8/798: dwrite d1/dc/d3c/d87/d34/fc6 [0,4194304] 0 2024-03-14T13:02:24.615 INFO:tasks.workunit.client.1.smithi137.stdout:4/734: creat d4/d9/d3a/d41/d6a/dac/ff1 x:0 0 0 2024-03-14T13:02:24.616 INFO:tasks.workunit.client.1.smithi137.stdout:4/735: chown d4/d9/d3a/d70/d80/db7/l7d 1 1 2024-03-14T13:02:24.618 INFO:tasks.workunit.client.0.smithi111.stdout:3/733: truncate d0/d2d/d35/d50/da2/db3/d6f/f79 398754 0 2024-03-14T13:02:24.618 INFO:tasks.workunit.client.1.smithi137.stdout:9/800: mknod d6/df/d19/d33/c101 0 2024-03-14T13:02:24.619 INFO:tasks.workunit.client.1.smithi137.stdout:4/736: stat d4/d10/c17 0 2024-03-14T13:02:24.621 INFO:tasks.workunit.client.1.smithi137.stdout:7/682: creat d3/d6/d59/daa/dcf/d6a/d75/d78/fdc x:0 0 0 2024-03-14T13:02:24.622 INFO:tasks.workunit.client.0.smithi111.stdout:0/900: truncate d6/d9/d15/d49/fc2 2905319 0 2024-03-14T13:02:24.623 INFO:tasks.workunit.client.0.smithi111.stdout:3/734: truncate d0/d2d/d35/d50/da2/db3/d6f/fb7 1337748 0 2024-03-14T13:02:24.624 INFO:tasks.workunit.client.1.smithi137.stdout:9/801: symlink d6/d53/db7/l102 0 2024-03-14T13:02:24.625 INFO:tasks.workunit.client.1.smithi137.stdout:9/802: write d6/fa3 [3415021,44806] 0 2024-03-14T13:02:24.627 INFO:tasks.workunit.client.0.smithi111.stdout:5/885: symlink d2/d16/d49/d44/l11b 0 2024-03-14T13:02:24.629 INFO:tasks.workunit.client.0.smithi111.stdout:2/634: rename dd/d11/d1a/f3d to dd/d8d/db7/fc4 0 2024-03-14T13:02:24.629 INFO:tasks.workunit.client.0.smithi111.stdout:0/901: rmdir d6/d9/d15 39 2024-03-14T13:02:24.633 INFO:tasks.workunit.client.1.smithi137.stdout:8/799: creat d1/dc/d3c/d87/d34/d82/f110 x:0 0 0 2024-03-14T13:02:24.634 INFO:tasks.workunit.client.1.smithi137.stdout:9/803: creat d6/df/d88/df4/dfb/f103 x:0 0 0 2024-03-14T13:02:24.635 INFO:tasks.workunit.client.1.smithi137.stdout:7/683: getdents d3/d6/d14/d22 0 2024-03-14T13:02:24.636 INFO:tasks.workunit.client.1.smithi137.stdout:8/800: dread d1/dc/d3c/d87/d34/d97/fbb [0,4194304] 0 2024-03-14T13:02:24.637 INFO:tasks.workunit.client.0.smithi111.stdout:3/735: chown d0/d2d/d35/d50/da2/db3/d71/dbc 5968605 1 2024-03-14T13:02:24.637 INFO:tasks.workunit.client.1.smithi137.stdout:9/804: creat d6/d91/dc8/f104 x:0 0 0 2024-03-14T13:02:24.637 INFO:tasks.workunit.client.0.smithi111.stdout:1/803: dwrite d20/d25/d31/dff/db6/fb9 [0,4194304] 0 2024-03-14T13:02:24.638 INFO:tasks.workunit.client.1.smithi137.stdout:7/684: truncate d3/d6/d59/daa/fce 429195 0 2024-03-14T13:02:24.639 INFO:tasks.workunit.client.1.smithi137.stdout:8/801: creat d1/d5e/dc9/f111 x:0 0 0 2024-03-14T13:02:24.640 INFO:tasks.workunit.client.1.smithi137.stdout:8/802: readlink d1/d5e/lf7 0 2024-03-14T13:02:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:24 smithi111 ceph-mon[39688]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:24.642 INFO:tasks.workunit.client.1.smithi137.stdout:7/685: unlink d3/d6/d14/f57 0 2024-03-14T13:02:24.644 INFO:tasks.workunit.client.1.smithi137.stdout:9/805: symlink d6/l105 0 2024-03-14T13:02:24.645 INFO:tasks.workunit.client.0.smithi111.stdout:0/902: creat d6/d9/de/d77/f141 x:0 0 0 2024-03-14T13:02:24.649 INFO:tasks.workunit.client.0.smithi111.stdout:1/804: unlink d20/d25/da8/lde 0 2024-03-14T13:02:24.651 INFO:tasks.workunit.client.1.smithi137.stdout:4/737: truncate d4/d9/f18 3650982 0 2024-03-14T13:02:24.652 INFO:tasks.workunit.client.1.smithi137.stdout:7/686: mkdir d3/d5/d5a/d86/ddd 0 2024-03-14T13:02:24.654 INFO:tasks.workunit.client.0.smithi111.stdout:1/805: creat d20/d40/da6/d102/f10e x:0 0 0 2024-03-14T13:02:24.655 INFO:tasks.workunit.client.0.smithi111.stdout:7/639: dread d0/d1/d21/f3f [8388608,4194304] 0 2024-03-14T13:02:24.656 INFO:tasks.workunit.client.0.smithi111.stdout:1/806: write d20/d21/d3f/f83 [2136503,94488] 0 2024-03-14T13:02:24.660 INFO:tasks.workunit.client.1.smithi137.stdout:7/687: mknod d3/d6/d64/d97/cde 0 2024-03-14T13:02:24.662 INFO:tasks.workunit.client.1.smithi137.stdout:4/738: write d4/d10/d42/d73/fb5 [4384622,9396] 0 2024-03-14T13:02:24.664 INFO:tasks.workunit.client.1.smithi137.stdout:4/739: read - d4/d27/d5c/d77/fbd zero size 2024-03-14T13:02:24.664 INFO:tasks.workunit.client.1.smithi137.stdout:7/688: mkdir d3/d6/d14/d22/ddf 0 2024-03-14T13:02:24.665 INFO:tasks.workunit.client.0.smithi111.stdout:3/736: truncate d0/d2d/d35/d50/d5e/f37 1429065 0 2024-03-14T13:02:24.666 INFO:tasks.workunit.client.0.smithi111.stdout:7/640: write d0/d1/d21/d5d/f84 [484810,102922] 0 2024-03-14T13:02:24.667 INFO:tasks.workunit.client.0.smithi111.stdout:5/886: dread d2/d13/f109 [0,4194304] 0 2024-03-14T13:02:24.667 INFO:tasks.workunit.client.1.smithi137.stdout:7/689: chown d3/d5/d68/l6d 52010422 1 2024-03-14T13:02:24.668 INFO:tasks.workunit.client.1.smithi137.stdout:4/740: symlink d4/d9/d3a/d41/d6a/lf2 0 2024-03-14T13:02:24.670 INFO:tasks.workunit.client.1.smithi137.stdout:9/806: dread d6/df/d71/fa5 [0,4194304] 0 2024-03-14T13:02:24.671 INFO:tasks.workunit.client.1.smithi137.stdout:7/690: mknod d3/d5/d5a/d80/ce0 0 2024-03-14T13:02:24.673 INFO:tasks.workunit.client.1.smithi137.stdout:9/807: creat d6/df/d19/d1f/f106 x:0 0 0 2024-03-14T13:02:24.674 INFO:tasks.workunit.client.0.smithi111.stdout:5/887: symlink d2/d16/d49/d33/l11c 0 2024-03-14T13:02:24.674 INFO:tasks.workunit.client.1.smithi137.stdout:7/691: rmdir d3/d6/d59/daa 39 2024-03-14T13:02:24.676 INFO:tasks.workunit.client.1.smithi137.stdout:7/692: write d3/d5/f46 [3188102,8063] 0 2024-03-14T13:02:24.676 INFO:tasks.workunit.client.0.smithi111.stdout:0/903: dwrite d6/d9/de/f85 [0,4194304] 0 2024-03-14T13:02:24.676 INFO:tasks.workunit.client.0.smithi111.stdout:1/807: dwrite d20/d25/d31/d80/fd1 [0,4194304] 0 2024-03-14T13:02:24.677 INFO:tasks.workunit.client.1.smithi137.stdout:8/803: dread d1/dc/d3c/d87/fc1 [0,4194304] 0 2024-03-14T13:02:24.677 INFO:tasks.workunit.client.1.smithi137.stdout:4/741: rename d4/d9/d3a/d70/d80/fa6 to d4/d9/ff3 0 2024-03-14T13:02:24.679 INFO:tasks.workunit.client.1.smithi137.stdout:9/808: chown d6/df/d88/de5/df6/lb9 240850 1 2024-03-14T13:02:24.684 INFO:tasks.workunit.client.1.smithi137.stdout:8/804: creat d1/dc/d71/db0/dca/f112 x:0 0 0 2024-03-14T13:02:24.688 INFO:tasks.workunit.client.0.smithi111.stdout:7/641: link d0/d1/d5/d38/l40 d0/d11/d98/dab/lbe 0 2024-03-14T13:02:24.688 INFO:tasks.workunit.client.0.smithi111.stdout:1/808: creat d20/d25/da8/f10f x:0 0 0 2024-03-14T13:02:24.689 INFO:tasks.workunit.client.0.smithi111.stdout:3/737: write d0/d2d/d35/d50/d5e/d19/f44 [307580,28425] 0 2024-03-14T13:02:24.689 INFO:tasks.workunit.client.1.smithi137.stdout:9/809: symlink d6/d26/l107 0 2024-03-14T13:02:24.691 INFO:tasks.workunit.client.1.smithi137.stdout:7/693: symlink d3/d6/d59/daa/dcf/d6a/le1 0 2024-03-14T13:02:24.691 INFO:tasks.workunit.client.1.smithi137.stdout:7/694: stat d3/cb2 0 2024-03-14T13:02:24.692 INFO:tasks.workunit.client.1.smithi137.stdout:9/810: chown d6/c8 12190045 1 2024-03-14T13:02:24.693 INFO:tasks.workunit.client.0.smithi111.stdout:3/738: chown d0/d30/lc1 208 1 2024-03-14T13:02:24.694 INFO:tasks.workunit.client.0.smithi111.stdout:1/809: creat d20/d25/d31/f110 x:0 0 0 2024-03-14T13:02:24.695 INFO:tasks.workunit.client.0.smithi111.stdout:7/642: mkdir d0/d1/d21/dba/dbf 0 2024-03-14T13:02:24.695 INFO:tasks.workunit.client.1.smithi137.stdout:8/805: getdents d1/dc/d3c/d87/d34/d82 0 2024-03-14T13:02:24.695 INFO:tasks.workunit.client.1.smithi137.stdout:4/742: getdents d4/d9/d3a/d41/d38/d79/dc2 0 2024-03-14T13:02:24.696 INFO:tasks.workunit.client.1.smithi137.stdout:8/806: chown d1/dc/d71/db0 2005018625 1 2024-03-14T13:02:24.698 INFO:tasks.workunit.client.1.smithi137.stdout:7/695: link d3/d5/d23/d3f/d5b/fc1 d3/d5/d23/d3f/fe2 0 2024-03-14T13:02:24.698 INFO:tasks.workunit.client.1.smithi137.stdout:9/811: link d6/df/d71/c8e d6/d53/c108 0 2024-03-14T13:02:24.700 INFO:tasks.workunit.client.1.smithi137.stdout:8/807: truncate d1/dc/d3c/d87/d34/d82/d94/ff4 1373094 0 2024-03-14T13:02:24.700 INFO:tasks.workunit.client.1.smithi137.stdout:9/812: dread - d6/df/d19/d1f/f106 zero size 2024-03-14T13:02:24.700 INFO:tasks.workunit.client.1.smithi137.stdout:4/743: read d4/d9/d3a/d41/f21 [3338688,56742] 0 2024-03-14T13:02:24.700 INFO:tasks.workunit.client.1.smithi137.stdout:7/696: mknod d3/d6/d64/d97/ce3 0 2024-03-14T13:02:24.702 INFO:tasks.workunit.client.1.smithi137.stdout:8/808: chown d1/dc/f5f 358 1 2024-03-14T13:02:24.704 INFO:tasks.workunit.client.0.smithi111.stdout:0/904: dwrite d6/d9/de/d3a/fc3 [0,4194304] 0 2024-03-14T13:02:24.704 INFO:tasks.workunit.client.0.smithi111.stdout:1/810: mkdir d20/d25/d111 0 2024-03-14T13:02:24.705 INFO:tasks.workunit.client.1.smithi137.stdout:4/744: mknod d4/d9/cf4 0 2024-03-14T13:02:24.705 INFO:tasks.workunit.client.1.smithi137.stdout:9/813: dread d6/df/d88/de5/df6/fe2 [0,4194304] 0 2024-03-14T13:02:24.708 INFO:tasks.workunit.client.0.smithi111.stdout:5/888: write d2/d13/f2f [319406,50642] 0 2024-03-14T13:02:24.709 INFO:tasks.workunit.client.1.smithi137.stdout:9/814: dread - d6/df/d88/df4/dfb/f103 zero size 2024-03-14T13:02:24.711 INFO:tasks.workunit.client.0.smithi111.stdout:7/643: truncate d0/d1/d5/d38/d5c/d6d/f9e 635318 0 2024-03-14T13:02:24.712 INFO:tasks.workunit.client.1.smithi137.stdout:8/809: dwrite d1/dc/d3c/d87/d34/d82/ffb [0,4194304] 0 2024-03-14T13:02:24.713 INFO:tasks.workunit.client.0.smithi111.stdout:7/644: fsync d0/d1/d4a/d52/f62 0 2024-03-14T13:02:24.716 INFO:tasks.workunit.client.0.smithi111.stdout:7/645: truncate d0/d1/d5/d38/d1b/d9c/fbb 968672 0 2024-03-14T13:02:24.716 INFO:tasks.workunit.client.1.smithi137.stdout:9/815: creat d6/df/d88/df4/dfb/f109 x:0 0 0 2024-03-14T13:02:24.718 INFO:tasks.workunit.client.0.smithi111.stdout:7/646: stat d0/d1/d5/d38/d1b/fa1 0 2024-03-14T13:02:24.718 INFO:tasks.workunit.client.1.smithi137.stdout:4/745: dwrite d4/d9/d3a/d70/d80/d8d/fd4 [0,4194304] 0 2024-03-14T13:02:24.719 INFO:tasks.workunit.client.0.smithi111.stdout:1/811: dwrite d20/d25/d87/f8f [0,4194304] 0 2024-03-14T13:02:24.721 INFO:tasks.workunit.client.1.smithi137.stdout:7/697: dread d3/d5/d23/d3f/d5b/f79 [0,4194304] 0 2024-03-14T13:02:24.726 INFO:tasks.workunit.client.1.smithi137.stdout:7/698: mknod d3/d6/d2e/ce4 0 2024-03-14T13:02:24.728 INFO:tasks.workunit.client.1.smithi137.stdout:8/810: getdents d1/d4a 0 2024-03-14T13:02:24.729 INFO:tasks.workunit.client.1.smithi137.stdout:9/816: dwrite d6/fc [0,4194304] 0 2024-03-14T13:02:24.729 INFO:tasks.workunit.client.1.smithi137.stdout:7/699: write d3/d5/d5a/fa7 [1306797,103364] 0 2024-03-14T13:02:24.733 INFO:tasks.workunit.client.0.smithi111.stdout:0/905: rmdir d6/d9/d15/d55/da0 39 2024-03-14T13:02:24.734 INFO:tasks.workunit.client.0.smithi111.stdout:0/906: stat d6/d9/de/d3a/f9b 0 2024-03-14T13:02:24.735 INFO:tasks.workunit.client.1.smithi137.stdout:7/700: creat d3/d5/d5a/d86/fe5 x:0 0 0 2024-03-14T13:02:24.737 INFO:tasks.workunit.client.1.smithi137.stdout:7/701: write d3/d5/d5a/d86/fe5 [307012,125015] 0 2024-03-14T13:02:24.738 INFO:tasks.workunit.client.0.smithi111.stdout:3/739: link d0/d2d/d35/d50/d5e/d19/f26 d0/d2d/d35/d69/f110 0 2024-03-14T13:02:24.740 INFO:tasks.workunit.client.0.smithi111.stdout:0/907: creat d6/d9/de/d3a/d46/d4a/d11b/d12a/f142 x:0 0 0 2024-03-14T13:02:24.741 INFO:tasks.workunit.client.1.smithi137.stdout:7/702: rename d3/d52/f9d to d3/d6/d64/d97/fe6 0 2024-03-14T13:02:24.741 INFO:tasks.workunit.client.0.smithi111.stdout:7/647: dread d0/d1/d5/d56/f7c [0,4194304] 0 2024-03-14T13:02:24.742 INFO:tasks.workunit.client.1.smithi137.stdout:7/703: stat d3/d6/d64/d9e/fb8 0 2024-03-14T13:02:24.742 INFO:tasks.workunit.client.0.smithi111.stdout:0/908: stat d6/d9/de/d32/d118 0 2024-03-14T13:02:24.742 INFO:tasks.workunit.client.0.smithi111.stdout:3/740: chown d0/d2d/d35/d50/d5e/f27 117908326 1 2024-03-14T13:02:24.744 INFO:tasks.workunit.client.1.smithi137.stdout:7/704: mknod d3/d5/d23/ce7 0 2024-03-14T13:02:24.745 INFO:tasks.workunit.client.0.smithi111.stdout:7/648: creat d0/d1/d5/d38/d5c/fc0 x:0 0 0 2024-03-14T13:02:24.746 INFO:tasks.workunit.client.0.smithi111.stdout:0/909: rmdir d6/d9/de/d32/d64/de7 39 2024-03-14T13:02:24.746 INFO:tasks.workunit.client.1.smithi137.stdout:9/817: dread d6/f69 [0,4194304] 0 2024-03-14T13:02:24.746 INFO:tasks.workunit.client.1.smithi137.stdout:7/705: mkdir d3/d52/de8 0 2024-03-14T13:02:24.747 INFO:tasks.workunit.client.0.smithi111.stdout:7/649: truncate d0/d1/d4a/f60 2370422 0 2024-03-14T13:02:24.749 INFO:tasks.workunit.client.0.smithi111.stdout:0/910: unlink d6/d9/d15/da9/lbf 0 2024-03-14T13:02:24.749 INFO:tasks.workunit.client.1.smithi137.stdout:9/818: symlink d6/df/d19/d33/d8f/l10a 0 2024-03-14T13:02:24.750 INFO:tasks.workunit.client.0.smithi111.stdout:3/741: dread d0/d2d/d35/d69/de6/df4/ffd [0,4194304] 0 2024-03-14T13:02:24.750 INFO:tasks.workunit.client.1.smithi137.stdout:7/706: mknod d3/d6/ce9 0 2024-03-14T13:02:24.750 INFO:tasks.workunit.client.0.smithi111.stdout:7/650: mkdir d0/dc1 0 2024-03-14T13:02:24.751 INFO:tasks.workunit.client.0.smithi111.stdout:5/889: dread d2/d16/f70 [4194304,4194304] 0 2024-03-14T13:02:24.753 INFO:tasks.workunit.client.1.smithi137.stdout:9/819: truncate d6/f96 42425 0 2024-03-14T13:02:24.754 INFO:tasks.workunit.client.0.smithi111.stdout:0/911: rmdir d6/d9/d15/d55/da0 39 2024-03-14T13:02:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:24 smithi137 ceph-mon[50336]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:24.755 INFO:tasks.workunit.client.1.smithi137.stdout:8/811: dwrite d1/dc/d3c/d87/d34/d97/d7f/fa0 [0,4194304] 0 2024-03-14T13:02:24.756 INFO:tasks.workunit.client.0.smithi111.stdout:5/890: symlink d2/d16/d49/d44/d4a/l11d 0 2024-03-14T13:02:24.757 INFO:tasks.workunit.client.1.smithi137.stdout:9/820: mknod d6/df/d88/df4/c10b 0 2024-03-14T13:02:24.757 INFO:tasks.workunit.client.0.smithi111.stdout:3/742: mknod d0/c111 0 2024-03-14T13:02:24.758 INFO:tasks.workunit.client.1.smithi137.stdout:7/707: link d3/fd0 d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1/fea 0 2024-03-14T13:02:24.760 INFO:tasks.workunit.client.0.smithi111.stdout:0/912: rename d6/d21/d74 to d6/d9/d15/d49/d56/d143 0 2024-03-14T13:02:24.762 INFO:tasks.workunit.client.0.smithi111.stdout:3/743: mknod d0/d2d/d35/d50/ddf/c112 0 2024-03-14T13:02:24.764 INFO:tasks.workunit.client.1.smithi137.stdout:8/812: write d1/dc/d4e/fff [302319,15828] 0 2024-03-14T13:02:24.765 INFO:tasks.workunit.client.1.smithi137.stdout:9/821: creat d6/df/d88/f10c x:0 0 0 2024-03-14T13:02:24.765 INFO:tasks.workunit.client.0.smithi111.stdout:0/913: creat d6/d9/de/d3a/d46/dde/f144 x:0 0 0 2024-03-14T13:02:24.766 INFO:tasks.workunit.client.0.smithi111.stdout:3/744: chown d0/d2d/d35/d69/c83 862891101 1 2024-03-14T13:02:24.767 INFO:tasks.workunit.client.1.smithi137.stdout:9/822: chown d6/df/d19/d33/f61 206 1 2024-03-14T13:02:24.767 INFO:tasks.workunit.client.1.smithi137.stdout:7/708: dwrite d3/d4a/f90 [0,4194304] 0 2024-03-14T13:02:24.770 INFO:tasks.workunit.client.0.smithi111.stdout:0/914: rename d6/d9/d15/d49/d56/fc9 to d6/d3d/da6/dfe/d134/f145 0 2024-03-14T13:02:24.771 INFO:tasks.workunit.client.1.smithi137.stdout:7/709: truncate d3/d5/d23/d3f/d5b/fc1 70324 0 2024-03-14T13:02:24.772 INFO:tasks.workunit.client.1.smithi137.stdout:7/710: stat d3/d5/d23/d3f/c4e 0 2024-03-14T13:02:24.772 INFO:tasks.workunit.client.1.smithi137.stdout:9/823: creat d6/df/d88/de5/df6/d40/f10d x:0 0 0 2024-03-14T13:02:24.772 INFO:tasks.workunit.client.1.smithi137.stdout:7/711: fsync d3/d54/fc9 0 2024-03-14T13:02:24.774 INFO:tasks.workunit.client.1.smithi137.stdout:7/712: write d3/d6/d59/fbc [1563994,83630] 0 2024-03-14T13:02:24.777 INFO:tasks.workunit.client.1.smithi137.stdout:9/824: link d6/cb d6/df/d19/d33/ddf/c10e 0 2024-03-14T13:02:24.778 INFO:tasks.workunit.client.1.smithi137.stdout:9/825: rmdir d6/df/d19/d33 39 2024-03-14T13:02:24.781 INFO:tasks.workunit.client.1.smithi137.stdout:9/826: creat d6/d26/f10f x:0 0 0 2024-03-14T13:02:24.782 INFO:tasks.workunit.client.1.smithi137.stdout:7/713: getdents d3/d5/d23 0 2024-03-14T13:02:24.783 INFO:tasks.workunit.client.1.smithi137.stdout:8/813: truncate d1/dc/d3c/d87/fc1 2572210 0 2024-03-14T13:02:24.784 INFO:tasks.workunit.client.1.smithi137.stdout:9/827: readlink d6/df/d19/l6b 0 2024-03-14T13:02:24.786 INFO:tasks.workunit.client.1.smithi137.stdout:7/714: creat d3/d54/db4/feb x:0 0 0 2024-03-14T13:02:24.789 INFO:tasks.workunit.client.1.smithi137.stdout:7/715: creat d3/d6/d64/d9e/fec x:0 0 0 2024-03-14T13:02:24.790 INFO:tasks.workunit.client.1.smithi137.stdout:8/814: dwrite d1/dc/d10/dcf/f100 [0,4194304] 0 2024-03-14T13:02:24.795 INFO:tasks.workunit.client.1.smithi137.stdout:7/716: rename d3/d6/d59/f5f to d3/d5/d5a/d86/fed 0 2024-03-14T13:02:24.796 INFO:tasks.workunit.client.1.smithi137.stdout:7/717: rmdir d3/d5/d23/d3f 39 2024-03-14T13:02:24.800 INFO:tasks.workunit.client.1.smithi137.stdout:8/815: dwrite d1/dc/d3c/d87/d34/d97/f2a [0,4194304] 0 2024-03-14T13:02:24.801 INFO:tasks.workunit.client.1.smithi137.stdout:7/718: creat d3/d6/d59/daa/dcf/d6a/d75/d78/fee x:0 0 0 2024-03-14T13:02:24.806 INFO:tasks.workunit.client.1.smithi137.stdout:7/719: creat d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fef x:0 0 0 2024-03-14T13:02:24.807 INFO:tasks.workunit.client.0.smithi111.stdout:9/711: sync 2024-03-14T13:02:24.808 INFO:tasks.workunit.client.1.smithi137.stdout:8/816: dwrite d1/dc/d77/d9f/de4/f4f [0,4194304] 0 2024-03-14T13:02:24.810 INFO:tasks.workunit.client.1.smithi137.stdout:8/817: fsync d1/f83 0 2024-03-14T13:02:24.811 INFO:tasks.workunit.client.0.smithi111.stdout:0/915: rename d6/d9/de/d3a/d46/d4a/d11b/d12a/f142 to d6/d9/de/d3a/d46/d124/f146 0 2024-03-14T13:02:24.813 INFO:tasks.workunit.client.0.smithi111.stdout:9/712: symlink d5/d6/d2a/lf7 0 2024-03-14T13:02:24.816 INFO:tasks.workunit.client.1.smithi137.stdout:8/818: link d1/f6 d1/dc/d3c/d87/d34/d97/d7f/f113 0 2024-03-14T13:02:24.819 INFO:tasks.workunit.client.1.smithi137.stdout:8/819: mkdir d1/dc/d3c/d87/d34/d82/d114 0 2024-03-14T13:02:24.821 INFO:tasks.workunit.client.0.smithi111.stdout:9/713: fdatasync d5/d6/d2a/d2d/db8/fc3 0 2024-03-14T13:02:24.822 INFO:tasks.workunit.client.0.smithi111.stdout:0/916: getdents d6/d9/de/d32/d64/de7 0 2024-03-14T13:02:24.823 INFO:tasks.workunit.client.0.smithi111.stdout:0/917: write d6/d9/d5b/d9e/ff5 [164343,109577] 0 2024-03-14T13:02:24.825 INFO:tasks.workunit.client.0.smithi111.stdout:1/812: write d20/d25/f38 [1505380,75026] 0 2024-03-14T13:02:24.831 INFO:tasks.workunit.client.1.smithi137.stdout:9/828: dwrite d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:24.831 INFO:tasks.workunit.client.1.smithi137.stdout:7/720: write d3/d5/f7c [3288889,94326] 0 2024-03-14T13:02:24.834 INFO:tasks.workunit.client.1.smithi137.stdout:7/721: read d3/d6/d14/d27/f6c [26643,17985] 0 2024-03-14T13:02:24.838 INFO:tasks.workunit.client.1.smithi137.stdout:9/829: symlink d6/df/d88/l110 0 2024-03-14T13:02:24.839 INFO:tasks.workunit.client.0.smithi111.stdout:0/918: creat d6/d9/de/d32/d64/f147 x:0 0 0 2024-03-14T13:02:24.839 INFO:tasks.workunit.client.0.smithi111.stdout:5/891: write d2/d16/d49/d33/d68/f8a [471621,123252] 0 2024-03-14T13:02:24.840 INFO:tasks.workunit.client.0.smithi111.stdout:9/714: getdents d5/d42/d52 0 2024-03-14T13:02:24.841 INFO:tasks.workunit.client.1.smithi137.stdout:8/820: symlink d1/dc/d3c/d87/d34/d82/d96/l115 0 2024-03-14T13:02:24.842 INFO:tasks.workunit.client.1.smithi137.stdout:7/722: symlink d3/d6/d14/d22/lf0 0 2024-03-14T13:02:24.844 INFO:tasks.workunit.client.0.smithi111.stdout:9/715: fsync d5/d6/fd5 0 2024-03-14T13:02:24.845 INFO:tasks.workunit.client.0.smithi111.stdout:0/919: creat d6/d9/de/d32/f148 x:0 0 0 2024-03-14T13:02:24.845 INFO:tasks.workunit.client.0.smithi111.stdout:1/813: dwrite d20/d25/d75/df7/f101 [0,4194304] 0 2024-03-14T13:02:24.848 INFO:tasks.workunit.client.1.smithi137.stdout:9/830: dwrite d6/d53/db7/ff9 [0,4194304] 0 2024-03-14T13:02:24.848 INFO:tasks.workunit.client.0.smithi111.stdout:5/892: creat d2/d16/d49/d7c/d107/f11e x:0 0 0 2024-03-14T13:02:24.849 INFO:tasks.workunit.client.1.smithi137.stdout:8/821: unlink d1/dc/d3c/d87/d34/d82/d96/fae 0 2024-03-14T13:02:24.850 INFO:tasks.workunit.client.1.smithi137.stdout:7/723: stat d3/d5/d23/l70 0 2024-03-14T13:02:24.853 INFO:tasks.workunit.client.1.smithi137.stdout:9/831: mkdir d6/d56/d111 0 2024-03-14T13:02:24.854 INFO:tasks.workunit.client.0.smithi111.stdout:7/651: creat d0/d1/d5/d56/fc2 x:0 0 0 2024-03-14T13:02:24.854 INFO:tasks.workunit.client.0.smithi111.stdout:3/745: dwrite d0/d2d/d35/fef [0,4194304] 0 2024-03-14T13:02:24.856 INFO:tasks.workunit.client.1.smithi137.stdout:9/832: creat d6/df/d19/d1f/f112 x:0 0 0 2024-03-14T13:02:24.857 INFO:tasks.workunit.client.0.smithi111.stdout:0/920: unlink d6/d9/de/d3a/d46/d4a/l9f 0 2024-03-14T13:02:24.861 INFO:tasks.workunit.client.0.smithi111.stdout:9/716: creat d5/d42/d71/dbd/ff8 x:0 0 0 2024-03-14T13:02:24.862 INFO:tasks.workunit.client.0.smithi111.stdout:1/814: truncate d20/d21/f5f 754998 0 2024-03-14T13:02:24.864 INFO:tasks.workunit.client.0.smithi111.stdout:7/652: write d0/f7 [1266943,116199] 0 2024-03-14T13:02:24.866 INFO:tasks.workunit.client.0.smithi111.stdout:0/921: mkdir d6/d3d/d149 0 2024-03-14T13:02:24.868 INFO:tasks.workunit.client.0.smithi111.stdout:5/893: dwrite d2/d45/d76/fab [0,4194304] 0 2024-03-14T13:02:24.870 INFO:tasks.workunit.client.0.smithi111.stdout:1/815: creat d20/d25/da8/f112 x:0 0 0 2024-03-14T13:02:24.873 INFO:tasks.workunit.client.0.smithi111.stdout:3/746: creat d0/d2d/d35/f113 x:0 0 0 2024-03-14T13:02:24.874 INFO:tasks.workunit.client.0.smithi111.stdout:1/816: dread d20/d25/d31/f7f [0,4194304] 0 2024-03-14T13:02:24.876 INFO:tasks.workunit.client.0.smithi111.stdout:9/717: rmdir d5/d42/d71/dbd/ddb 0 2024-03-14T13:02:24.877 INFO:tasks.workunit.client.0.smithi111.stdout:9/718: chown d5/d6/d8/d5b 6 1 2024-03-14T13:02:24.877 INFO:tasks.workunit.client.0.smithi111.stdout:9/719: fsync d5/d6/faa 0 2024-03-14T13:02:24.878 INFO:tasks.workunit.client.0.smithi111.stdout:3/747: readlink d0/d2d/d35/d50/d5e/d19/l29 0 2024-03-14T13:02:24.880 INFO:tasks.workunit.client.0.smithi111.stdout:0/922: creat d6/d3d/f14a x:0 0 0 2024-03-14T13:02:24.881 INFO:tasks.workunit.client.0.smithi111.stdout:7/653: mknod d0/d1/d5/d38/d1b/d99/cc3 0 2024-03-14T13:02:24.883 INFO:tasks.workunit.client.0.smithi111.stdout:1/817: creat d20/d21/d28/d5c/f113 x:0 0 0 2024-03-14T13:02:24.887 INFO:tasks.workunit.client.0.smithi111.stdout:7/654: chown d0/d1/d5/d38/d5c/l63 0 1 2024-03-14T13:02:24.890 INFO:tasks.workunit.client.0.smithi111.stdout:9/720: creat d5/d6/d8/d5b/d84/db6/de7/ff9 x:0 0 0 2024-03-14T13:02:24.894 INFO:tasks.workunit.client.0.smithi111.stdout:7/655: dwrite d0/d1/d22/f7b [8388608,4194304] 0 2024-03-14T13:02:24.894 INFO:tasks.workunit.client.0.smithi111.stdout:5/894: rename d2/d16/d49/c5 to d2/d45/d76/c11f 0 2024-03-14T13:02:24.897 INFO:tasks.workunit.client.0.smithi111.stdout:1/818: unlink d20/d25/d75/f7d 0 2024-03-14T13:02:24.898 INFO:tasks.workunit.client.0.smithi111.stdout:9/721: readlink d5/d6/d14/l1f 0 2024-03-14T13:02:24.900 INFO:tasks.workunit.client.1.smithi137.stdout:1/785: sync 2024-03-14T13:02:24.902 INFO:tasks.workunit.client.0.smithi111.stdout:3/748: link d0/d30/l53 d0/d2d/d35/d50/d6c/da3/daa/l114 0 2024-03-14T13:02:24.904 INFO:tasks.workunit.client.0.smithi111.stdout:5/895: creat d2/d16/d49/d44/d72/de0/f120 x:0 0 0 2024-03-14T13:02:24.904 INFO:tasks.workunit.client.0.smithi111.stdout:1/819: creat d20/d25/d75/df7/f114 x:0 0 0 2024-03-14T13:02:24.906 INFO:tasks.workunit.client.0.smithi111.stdout:1/820: fdatasync d20/d25/da8/f112 0 2024-03-14T13:02:24.906 INFO:tasks.workunit.client.0.smithi111.stdout:3/749: mknod d0/d2d/d35/d50/da2/db3/d67/c115 0 2024-03-14T13:02:24.906 INFO:tasks.workunit.client.1.smithi137.stdout:9/833: rename d6/df/l42 to d6/d53/db7/l113 0 2024-03-14T13:02:24.907 INFO:tasks.workunit.client.0.smithi111.stdout:5/896: write d2/d16/d49/d33/f7b [751347,9156] 0 2024-03-14T13:02:24.908 INFO:tasks.workunit.client.0.smithi111.stdout:1/821: rmdir d20/d21 39 2024-03-14T13:02:24.908 INFO:tasks.workunit.client.1.smithi137.stdout:7/724: truncate d3/d6/d59/daa/fb3 2153603 0 2024-03-14T13:02:24.909 INFO:tasks.workunit.client.0.smithi111.stdout:7/656: getdents d0/d1/d22/d3e 0 2024-03-14T13:02:24.909 INFO:tasks.workunit.client.0.smithi111.stdout:5/897: stat d2/d45/d76/d9a/d9f/fb5 0 2024-03-14T13:02:24.911 INFO:tasks.workunit.client.1.smithi137.stdout:9/834: mkdir d6/df/d19/d33/ddf/d114 0 2024-03-14T13:02:24.911 INFO:tasks.workunit.client.0.smithi111.stdout:3/750: fsync d0/d2d/d35/d69/d84/f87 0 2024-03-14T13:02:24.912 INFO:tasks.workunit.client.1.smithi137.stdout:9/835: dread - d6/df/d19/d1f/f112 zero size 2024-03-14T13:02:24.912 INFO:tasks.workunit.client.1.smithi137.stdout:1/786: symlink d7/d31/d82/l10d 0 2024-03-14T13:02:24.913 INFO:tasks.workunit.client.1.smithi137.stdout:7/725: creat d3/d5/d68/ff1 x:0 0 0 2024-03-14T13:02:24.914 INFO:tasks.workunit.client.0.smithi111.stdout:5/898: unlink d2/d16/d49/d4c/d9c/lf5 0 2024-03-14T13:02:24.914 INFO:tasks.workunit.client.1.smithi137.stdout:1/787: write d7/d9/d19/d1f/d26/f48 [728533,76791] 0 2024-03-14T13:02:24.915 INFO:tasks.workunit.client.0.smithi111.stdout:1/822: mknod d20/d25/d31/dff/c115 0 2024-03-14T13:02:24.915 INFO:tasks.workunit.client.1.smithi137.stdout:9/836: rename d6/d53/fb8 to d6/df/d88/dd5/f115 0 2024-03-14T13:02:24.917 INFO:tasks.workunit.client.0.smithi111.stdout:5/899: fsync d2/d45/d76/d9a/fc8 0 2024-03-14T13:02:24.917 INFO:tasks.workunit.client.0.smithi111.stdout:1/823: write d20/d25/d31/dff/f105 [100244,112935] 0 2024-03-14T13:02:24.918 INFO:tasks.workunit.client.1.smithi137.stdout:7/726: symlink d3/d6/d64/d97/lf2 0 2024-03-14T13:02:24.919 INFO:tasks.workunit.client.0.smithi111.stdout:1/824: chown d20/d25/f39 254708 1 2024-03-14T13:02:24.919 INFO:tasks.workunit.client.1.smithi137.stdout:9/837: mkdir d6/df/d19/d33/ddf/d116 0 2024-03-14T13:02:24.920 INFO:tasks.workunit.client.0.smithi111.stdout:7/657: dread d0/d1/d5/d38/d5c/d6d/fa3 [0,4194304] 0 2024-03-14T13:02:24.920 INFO:tasks.workunit.client.1.smithi137.stdout:9/838: truncate d6/d56/fb3 4467876 0 2024-03-14T13:02:24.920 INFO:tasks.workunit.client.1.smithi137.stdout:1/788: creat d7/d9/d19/d1f/d26/d6c/dc8/f10e x:0 0 0 2024-03-14T13:02:24.921 INFO:tasks.workunit.client.1.smithi137.stdout:7/727: fdatasync d3/d5/d5a/d86/fed 0 2024-03-14T13:02:24.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.922+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dc4115d50 con 0x7f0dac06c560 2024-03-14T13:02:24.922 INFO:tasks.workunit.client.0.smithi111.stdout:1/825: symlink d20/d25/d87/l116 0 2024-03-14T13:02:24.922 INFO:tasks.workunit.client.0.smithi111.stdout:3/751: dwrite d0/d2d/d35/d50/da2/db3/da7/f101 [0,4194304] 0 2024-03-14T13:02:24.923 INFO:tasks.workunit.client.1.smithi137.stdout:7/728: write d3/d5/d23/fb5 [3263669,87679] 0 2024-03-14T13:02:24.924 INFO:tasks.workunit.client.0.smithi111.stdout:5/900: chown d2/d16/d49/d44/d4a/d9e/db7/dbb/ff9 35 1 2024-03-14T13:02:24.925 INFO:tasks.workunit.client.0.smithi111.stdout:5/901: dread - d2/d16/d49/d7c/ffd zero size 2024-03-14T13:02:24.929 INFO:tasks.workunit.client.1.smithi137.stdout:1/789: link d7/d92/fe4 d7/d9/d19/d40/d42/f10f 0 2024-03-14T13:02:24.929 INFO:tasks.workunit.client.1.smithi137.stdout:9/839: link d6/df/c50 d6/df/d88/c117 0 2024-03-14T13:02:24.930 INFO:tasks.workunit.client.0.smithi111.stdout:3/752: creat d0/d30/f116 x:0 0 0 2024-03-14T13:02:24.931 INFO:tasks.workunit.client.1.smithi137.stdout:7/729: truncate d3/d6/d14/f2f 2622675 0 2024-03-14T13:02:24.932 INFO:tasks.workunit.client.1.smithi137.stdout:1/790: creat d7/d9/d19/d5f/f110 x:0 0 0 2024-03-14T13:02:24.934 INFO:tasks.workunit.client.0.smithi111.stdout:5/902: rename d2/d45/cc7 to d2/d16/d49/d44/d4a/d9e/db7/dbb/c121 0 2024-03-14T13:02:24.935 INFO:tasks.workunit.client.0.smithi111.stdout:5/903: truncate d2/d13/f10c 707596 0 2024-03-14T13:02:24.936 INFO:tasks.workunit.client.0.smithi111.stdout:7/658: rmdir d0/d1/d21/dba/dbf 0 2024-03-14T13:02:24.937 INFO:tasks.workunit.client.1.smithi137.stdout:7/730: mkdir d3/d6/d14/d22/df3 0 2024-03-14T13:02:24.937 INFO:tasks.workunit.client.1.smithi137.stdout:1/791: read d7/d4d/f64 [226592,84967] 0 2024-03-14T13:02:24.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.938+0000 7f0dabfff700 1 -- 172.21.15.111:0/2028636155 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f0dc4115d50 con 0x7f0dac06c560 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (93s) 41s ago 10m 17.0M - 0.25.0 c8568f914cd2 70fc8e769dee 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (10m) 41s ago 10m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (8m) 42s ago 8m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (87s) 41s ago 9m 82.5M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (4m) 41s ago 4m 57.8M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (4m) 41s ago 4m 113M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (3m) 42s ago 3m 53.8M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (4m) 42s ago 4m 81.5M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:02:24.940 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (73s) 41s ago 11m 475M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283 running (2m) 42s ago 8m 61.7M - 18.2.2-825-g2f465319 a9e65ad305ae 0487350e99e4 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (11m) 41s ago 11m 54.1M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (8m) 42s ago 8m 40.2M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (99s) 41s ago 10m 9269k - 1.5.0 0da6a335fe13 c8b729c46aaa 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (79s) 42s ago 8m 8870k - 1.5.0 0da6a335fe13 37967b629a62 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (7m) 41s ago 7m 97.1M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (7m) 41s ago 7m 107M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (6m) 41s ago 6m 89.0M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (6m) 42s ago 6m 111M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (6m) 42s ago 6m 100M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (5m) 42s ago 5m 96.7M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:02:24.941 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (81s) 41s ago 9m 41.6M - 2.43.0 a07b618ecd1d d3a9067a4910 2024-03-14T13:02:24.942 INFO:tasks.workunit.client.1.smithi137.stdout:1/792: dread d7/f9a [0,4194304] 0 2024-03-14T13:02:24.942 INFO:tasks.workunit.client.0.smithi111.stdout:1/826: rename d20/d21/da7/ff4 to d20/d25/d111/f117 0 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 msgr2=0x7f0dac06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 0x7f0dac06ea10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004600 tx=0x7f0db800b040 comp rx=0 tx=0).stop 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 msgr2=0x7f0dc41144b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007d40 tx=0x7f0db4009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 shutdown_connections 2024-03-14T13:02:24.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0dac06c560 0x7f0dac06ea10 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0db8004600 tx=0x7f0db800b040 comp rx=0 tx=0).stop 2024-03-14T13:02:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0dc41033f0 0x7f0dc4113f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 --2- 172.21.15.111:0/2028636155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0dc41046a0 0x7f0dc41144b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 >> 172.21.15.111:0/2028636155 conn(0x7f0dc4072560 msgr2=0x7f0dc4073c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.942+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 shutdown_connections 2024-03-14T13:02:24.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:24.943+0000 7f0dc9f4f700 1 -- 172.21.15.111:0/2028636155 wait complete. 2024-03-14T13:02:24.944 INFO:tasks.workunit.client.1.smithi137.stdout:1/793: chown d7/d9/d19/d1f/d89/d90/db1/fd6 1785215032 1 2024-03-14T13:02:24.945 INFO:tasks.workunit.client.0.smithi111.stdout:7/659: creat d0/dc1/fc4 x:0 0 0 2024-03-14T13:02:24.947 INFO:tasks.workunit.client.1.smithi137.stdout:7/731: dwrite d3/d5/d23/d3f/d5b/fa3 [0,4194304] 0 2024-03-14T13:02:24.947 INFO:tasks.workunit.client.0.smithi111.stdout:5/904: dwrite d2/d16/d49/d33/de5/f118 [0,4194304] 0 2024-03-14T13:02:24.950 INFO:tasks.workunit.client.0.smithi111.stdout:1/827: dread - d20/d25/da8/f112 zero size 2024-03-14T13:02:24.950 INFO:tasks.workunit.client.0.smithi111.stdout:0/923: write d6/d9/d15/d55/fee [1517929,69368] 0 2024-03-14T13:02:24.952 INFO:tasks.workunit.client.0.smithi111.stdout:5/905: chown d2/d13/ced 103214898 1 2024-03-14T13:02:24.952 INFO:tasks.workunit.client.0.smithi111.stdout:1/828: chown d20/d25/d31/d80/lb3 9 1 2024-03-14T13:02:24.955 INFO:tasks.workunit.client.1.smithi137.stdout:9/840: write d6/df/d88/de5/df6/f67 [1123785,44830] 0 2024-03-14T13:02:24.960 INFO:tasks.workunit.client.1.smithi137.stdout:1/794: dread d7/d9/d19/d1f/d26/d6c/dc8/fee [0,4194304] 0 2024-03-14T13:02:24.961 INFO:tasks.workunit.client.0.smithi111.stdout:9/722: write d5/d6/f22 [88699,88665] 0 2024-03-14T13:02:24.963 INFO:tasks.workunit.client.0.smithi111.stdout:5/906: fdatasync d2/d16/d49/d7c/d95/fa3 0 2024-03-14T13:02:24.963 INFO:tasks.workunit.client.0.smithi111.stdout:7/660: dread d0/d1/d21/d5d/f84 [0,4194304] 0 2024-03-14T13:02:24.964 INFO:tasks.workunit.client.1.smithi137.stdout:1/795: creat d7/d31/d82/f111 x:0 0 0 2024-03-14T13:02:24.966 INFO:tasks.workunit.client.0.smithi111.stdout:3/753: dread d0/d2d/d35/d50/f88 [0,4194304] 0 2024-03-14T13:02:24.967 INFO:tasks.workunit.client.0.smithi111.stdout:1/829: mkdir d20/d25/d31/d118 0 2024-03-14T13:02:24.968 INFO:tasks.workunit.client.1.smithi137.stdout:9/841: dwrite d6/df/d88/de5/df6/da9/fdc [0,4194304] 0 2024-03-14T13:02:24.968 INFO:tasks.workunit.client.1.smithi137.stdout:1/796: mkdir d7/d29/d112 0 2024-03-14T13:02:24.969 INFO:tasks.workunit.client.0.smithi111.stdout:5/907: unlink d2/d45/f66 0 2024-03-14T13:02:24.970 INFO:tasks.workunit.client.1.smithi137.stdout:1/797: fsync d7/d4d/d6b/faa 0 2024-03-14T13:02:24.971 INFO:tasks.workunit.client.1.smithi137.stdout:7/732: dread d3/f1c [0,4194304] 0 2024-03-14T13:02:24.972 INFO:tasks.workunit.client.1.smithi137.stdout:1/798: stat d7/d9/d19/d1f/d26/c6a 0 2024-03-14T13:02:24.972 INFO:tasks.workunit.client.0.smithi111.stdout:0/924: dwrite d6/d9/de/d32/f148 [0,4194304] 0 2024-03-14T13:02:24.972 INFO:tasks.workunit.client.0.smithi111.stdout:3/754: stat d0/d2d/d35/d50/f88 0 2024-03-14T13:02:24.973 INFO:tasks.workunit.client.0.smithi111.stdout:7/661: dread - d0/d1/d4a/d52/f6c zero size 2024-03-14T13:02:24.975 INFO:tasks.workunit.client.1.smithi137.stdout:1/799: unlink d7/d9/d19/fc0 0 2024-03-14T13:02:24.978 INFO:tasks.workunit.client.0.smithi111.stdout:5/908: mkdir d2/d16/d49/d33/d68/d122 0 2024-03-14T13:02:24.979 INFO:tasks.workunit.client.1.smithi137.stdout:1/800: symlink d7/d9/d19/d40/d50/l113 0 2024-03-14T13:02:24.980 INFO:tasks.workunit.client.0.smithi111.stdout:0/925: mkdir d6/d3d/da6/dfe/d134/d14b 0 2024-03-14T13:02:24.980 INFO:tasks.workunit.client.0.smithi111.stdout:1/830: dread d20/d21/f5f [0,4194304] 0 2024-03-14T13:02:24.981 INFO:tasks.workunit.client.0.smithi111.stdout:3/755: write d0/d2d/d35/d50/da2/db3/d6f/fb7 [354806,5101] 0 2024-03-14T13:02:24.982 INFO:tasks.workunit.client.0.smithi111.stdout:7/662: mknod d0/cc5 0 2024-03-14T13:02:24.983 INFO:tasks.workunit.client.1.smithi137.stdout:1/801: mknod d7/d9/d19/d1f/d22/c114 0 2024-03-14T13:02:24.985 INFO:tasks.workunit.client.0.smithi111.stdout:0/926: fdatasync d6/d21/f73 0 2024-03-14T13:02:24.986 INFO:tasks.workunit.client.0.smithi111.stdout:1/831: mknod d20/d25/d75/df7/c119 0 2024-03-14T13:02:24.986 INFO:tasks.workunit.client.0.smithi111.stdout:9/723: write d5/d6/d2a/d2d/d8b/f8e [442098,57946] 0 2024-03-14T13:02:24.987 INFO:tasks.workunit.client.1.smithi137.stdout:1/802: fsync d7/d9/d19/d40/f5b 0 2024-03-14T13:02:24.988 INFO:tasks.workunit.client.1.smithi137.stdout:1/803: chown d7/f9a 5524053 1 2024-03-14T13:02:24.989 INFO:tasks.workunit.client.1.smithi137.stdout:7/733: dwrite d3/d5/d23/d3f/fe2 [0,4194304] 0 2024-03-14T13:02:24.990 INFO:tasks.workunit.client.0.smithi111.stdout:3/756: fdatasync d0/d2d/d35/d50/d6c/da3/daa/fd1 0 2024-03-14T13:02:24.991 INFO:tasks.workunit.client.0.smithi111.stdout:7/663: truncate d0/d1/d22/f42 2993420 0 2024-03-14T13:02:24.993 INFO:tasks.workunit.client.0.smithi111.stdout:7/664: write d0/d1/d4a/d52/fb2 [84178,56151] 0 2024-03-14T13:02:24.994 INFO:tasks.workunit.client.1.smithi137.stdout:1/804: dread - d7/d9/d19/d5f/ff0 zero size 2024-03-14T13:02:24.995 INFO:tasks.workunit.client.1.smithi137.stdout:7/734: getdents d3/d6/d64 0 2024-03-14T13:02:24.996 INFO:tasks.workunit.client.1.smithi137.stdout:1/805: fdatasync d7/d9/d19/d40/d42/d9d/de5/ff4 0 2024-03-14T13:02:24.996 INFO:tasks.workunit.client.0.smithi111.stdout:9/724: unlink d5/d6/d8/d5b/d7c/f94 0 2024-03-14T13:02:24.996 INFO:tasks.workunit.client.1.smithi137.stdout:1/806: truncate d7/d29/fc9 205031 0 2024-03-14T13:02:24.997 INFO:tasks.workunit.client.0.smithi111.stdout:7/665: chown d0/d1/d22/l55 7051161 1 2024-03-14T13:02:24.998 INFO:tasks.workunit.client.1.smithi137.stdout:7/735: chown d3/d6/d14/d22/lf0 29808 1 2024-03-14T13:02:25.000 INFO:tasks.workunit.client.1.smithi137.stdout:1/807: mknod d7/d31/d54/c115 0 2024-03-14T13:02:25.000 INFO:tasks.workunit.client.0.smithi111.stdout:3/757: readlink d0/d2d/d35/d4c/l7b 0 2024-03-14T13:02:25.000 INFO:tasks.workunit.client.0.smithi111.stdout:0/927: truncate d6/d9/d15/d55/da0/ff0 1619440 0 2024-03-14T13:02:25.002 INFO:tasks.workunit.client.1.smithi137.stdout:1/808: write d7/d9/d19/d40/d42/d9d/de5/ff4 [973746,129240] 0 2024-03-14T13:02:25.003 INFO:tasks.workunit.client.1.smithi137.stdout:1/809: write d7/d4d/f108 [394334,74461] 0 2024-03-14T13:02:25.003 INFO:tasks.workunit.client.0.smithi111.stdout:1/832: rename d20/d21/d28/l99 to d20/d40/db4/ddf/l11a 0 2024-03-14T13:02:25.004 INFO:tasks.workunit.client.1.smithi137.stdout:1/810: write d7/d9/d19/d5f/f7f [569607,62148] 0 2024-03-14T13:02:25.005 INFO:tasks.workunit.client.0.smithi111.stdout:3/758: dread d0/d2d/d35/d50/da2/db3/d71/fda [0,4194304] 0 2024-03-14T13:02:25.006 INFO:tasks.workunit.client.0.smithi111.stdout:9/725: unlink d5/d42/da4/la9 0 2024-03-14T13:02:25.006 INFO:tasks.workunit.client.0.smithi111.stdout:0/928: rmdir d6/d45/d114 39 2024-03-14T13:02:25.006 INFO:tasks.workunit.client.1.smithi137.stdout:1/811: creat d7/d9/d19/d1f/d89/d90/f116 x:0 0 0 2024-03-14T13:02:25.007 INFO:tasks.workunit.client.0.smithi111.stdout:0/929: fsync d6/f72 0 2024-03-14T13:02:25.010 INFO:tasks.workunit.client.0.smithi111.stdout:1/833: mkdir d20/d25/d31/dff/db6/d11b 0 2024-03-14T13:02:25.010 INFO:tasks.workunit.client.1.smithi137.stdout:1/812: creat d7/d9/d19/d1f/d2e/f117 x:0 0 0 2024-03-14T13:02:25.011 INFO:tasks.workunit.client.0.smithi111.stdout:3/759: mkdir d0/d2d/d35/d4c/dad/d117 0 2024-03-14T13:02:25.011 INFO:tasks.workunit.client.0.smithi111.stdout:1/834: write d20/d25/d75/df7/f101 [1972150,96563] 0 2024-03-14T13:02:25.015 INFO:tasks.workunit.client.0.smithi111.stdout:9/726: mknod d5/d6/d8/d5b/d7c/cfa 0 2024-03-14T13:02:25.015 INFO:tasks.workunit.client.1.smithi137.stdout:1/813: dwrite f6 [0,4194304] 0 2024-03-14T13:02:25.016 INFO:tasks.workunit.client.0.smithi111.stdout:9/727: write d5/d42/d71/dbd/dcd/fb4 [1394988,1395] 0 2024-03-14T13:02:25.018 INFO:tasks.workunit.client.1.smithi137.stdout:1/814: truncate d7/d9/d19/d1f/d26/d6c/fb4 973497 0 2024-03-14T13:02:25.019 INFO:tasks.workunit.client.1.smithi137.stdout:1/815: write d7/d4d/d6b/faa [2089982,111144] 0 2024-03-14T13:02:25.022 INFO:tasks.workunit.client.1.smithi137.stdout:1/816: creat d7/d4d/f118 x:0 0 0 2024-03-14T13:02:25.022 INFO:tasks.workunit.client.1.smithi137.stdout:1/817: readlink d7/d4d/ldc 0 2024-03-14T13:02:25.024 INFO:tasks.workunit.client.1.smithi137.stdout:1/818: rmdir d7/d9/d19/d1f/d89/d90/db1 39 2024-03-14T13:02:25.024 INFO:tasks.workunit.client.0.smithi111.stdout:3/760: mkdir d0/d2d/d35/d50/ddf/d118 0 2024-03-14T13:02:25.025 INFO:tasks.workunit.client.0.smithi111.stdout:6/658: sync 2024-03-14T13:02:25.025 INFO:tasks.workunit.client.0.smithi111.stdout:1/835: creat d20/d25/dd0/f11c x:0 0 0 2024-03-14T13:02:25.025 INFO:tasks.workunit.client.0.smithi111.stdout:9/728: symlink d5/d6/d8/d5b/d87/lfb 0 2024-03-14T13:02:25.026 INFO:tasks.workunit.client.1.smithi137.stdout:1/819: fsync d7/d9/d19/d1f/d89/fac 0 2024-03-14T13:02:25.027 INFO:tasks.workunit.client.1.smithi137.stdout:1/820: fdatasync d7/d4d/f108 0 2024-03-14T13:02:25.029 INFO:tasks.workunit.client.0.smithi111.stdout:6/659: read d0/d17/d20/d65/d93/fe0 [2409403,92915] 0 2024-03-14T13:02:25.029 INFO:tasks.workunit.client.0.smithi111.stdout:5/909: dwrite d2/d16/d49/d44/d4a/f60 [0,4194304] 0 2024-03-14T13:02:25.030 INFO:tasks.workunit.client.0.smithi111.stdout:6/660: chown d0/d17/d21/d69/f61 44571 1 2024-03-14T13:02:25.032 INFO:tasks.workunit.client.1.smithi137.stdout:1/821: getdents d7/d31/d82 0 2024-03-14T13:02:25.035 INFO:tasks.workunit.client.1.smithi137.stdout:1/822: creat d7/d9/d19/d40/d50/f119 x:0 0 0 2024-03-14T13:02:25.037 INFO:tasks.workunit.client.0.smithi111.stdout:5/910: dread d2/d16/d49/d7c/d95/d6a/fa8 [0,4194304] 0 2024-03-14T13:02:25.037 INFO:tasks.workunit.client.0.smithi111.stdout:1/836: getdents d20/d25/d31/dff/db6/d11b 0 2024-03-14T13:02:25.039 INFO:tasks.workunit.client.1.smithi137.stdout:1/823: unlink d7/d9/d19/d1f/d26/d6c/ffd 0 2024-03-14T13:02:25.040 INFO:tasks.workunit.client.0.smithi111.stdout:6/661: rename d0/d17/d21/d69/d3d/d53/d81/da1/ld0 to d0/d17/d20/d65/d93/dd9/lf7 0 2024-03-14T13:02:25.041 INFO:tasks.workunit.client.1.smithi137.stdout:1/824: mkdir d7/d9/d19/d1f/ddf/d11a 0 2024-03-14T13:02:25.042 INFO:tasks.workunit.client.1.smithi137.stdout:1/825: write d7/d9/d19/d40/f46 [4720533,113885] 0 2024-03-14T13:02:25.043 INFO:tasks.workunit.client.0.smithi111.stdout:7/666: write d0/d7e/f81 [937754,106530] 0 2024-03-14T13:02:25.043 INFO:tasks.workunit.client.0.smithi111.stdout:0/930: write d6/d9/d5b/f6d [1546650,61470] 0 2024-03-14T13:02:25.044 INFO:tasks.workunit.client.1.smithi137.stdout:1/826: mkdir d7/d9/d19/d40/d50/d11b 0 2024-03-14T13:02:25.045 INFO:tasks.workunit.client.1.smithi137.stdout:1/827: chown d7/d9/d19/d1f/d89/fb2 5016 1 2024-03-14T13:02:25.047 INFO:tasks.workunit.client.0.smithi111.stdout:1/837: symlink d20/d25/d31/d80/df5/d103/l11d 0 2024-03-14T13:02:25.047 INFO:tasks.workunit.client.1.smithi137.stdout:1/828: creat d7/d9/d19/d1f/da0/f11c x:0 0 0 2024-03-14T13:02:25.050 INFO:tasks.workunit.client.0.smithi111.stdout:7/667: write d0/d1/d4a/d52/f62 [3070707,4362] 0 2024-03-14T13:02:25.050 INFO:tasks.workunit.client.1.smithi137.stdout:1/829: fsync d7/d9/d19/d1f/d22/f25 0 2024-03-14T13:02:25.051 INFO:tasks.workunit.client.0.smithi111.stdout:0/931: read - d6/d9/d15/d55/fda zero size 2024-03-14T13:02:25.052 INFO:tasks.workunit.client.0.smithi111.stdout:6/662: creat d0/d17/d21/d69/d3d/d53/d81/da1/df6/ff8 x:0 0 0 2024-03-14T13:02:25.053 INFO:tasks.workunit.client.1.smithi137.stdout:1/830: creat d7/d4d/db7/f11d x:0 0 0 2024-03-14T13:02:25.053 INFO:tasks.workunit.client.0.smithi111.stdout:5/911: dwrite d2/d16/d49/d44/ff0 [0,4194304] 0 2024-03-14T13:02:25.054 INFO:tasks.workunit.client.1.smithi137.stdout:1/831: symlink d7/d31/l11e 0 2024-03-14T13:02:25.059 INFO:tasks.workunit.client.0.smithi111.stdout:7/668: mknod d0/d1/d21/d3a/cc6 0 2024-03-14T13:02:25.060 INFO:tasks.workunit.client.1.smithi137.stdout:1/832: link d7/d9/d19/d1f/d2e/f34 d7/d9/d19/d1f/d26/d7d/f11f 0 2024-03-14T13:02:25.060 INFO:tasks.workunit.client.1.smithi137.stdout:1/833: write d7/d4d/d6b/fe2 [867182,30104] 0 2024-03-14T13:02:25.061 INFO:tasks.workunit.client.0.smithi111.stdout:3/761: dwrite d0/f23 [4194304,4194304] 0 2024-03-14T13:02:25.061 INFO:tasks.workunit.client.1.smithi137.stdout:1/834: chown d7/d29/fc9 0 1 2024-03-14T13:02:25.062 INFO:tasks.workunit.client.1.smithi137.stdout:1/835: fsync d7/d31/d82/fff 0 2024-03-14T13:02:25.063 INFO:tasks.workunit.client.1.smithi137.stdout:1/836: dread - d7/d92/fe4 zero size 2024-03-14T13:02:25.065 INFO:tasks.workunit.client.0.smithi111.stdout:7/669: rmdir d0/d1 39 2024-03-14T13:02:25.065 INFO:tasks.workunit.client.0.smithi111.stdout:0/932: creat d6/d45/d114/f14c x:0 0 0 2024-03-14T13:02:25.066 INFO:tasks.workunit.client.1.smithi137.stdout:1/837: mkdir d7/d31/d54/ded/dfb/d120 0 2024-03-14T13:02:25.067 INFO:tasks.workunit.client.0.smithi111.stdout:5/912: unlink d2/d16/d49/d33/lf4 0 2024-03-14T13:02:25.068 INFO:tasks.workunit.client.0.smithi111.stdout:1/838: dwrite d20/d21/d3f/d59/fc8 [0,4194304] 0 2024-03-14T13:02:25.075 INFO:tasks.workunit.client.0.smithi111.stdout:3/762: unlink d0/d2d/d35/d69/d84/c90 0 2024-03-14T13:02:25.076 INFO:tasks.workunit.client.0.smithi111.stdout:6/663: dwrite d0/d17/d20/d65/fc7 [0,4194304] 0 2024-03-14T13:02:25.077 INFO:tasks.workunit.client.0.smithi111.stdout:7/670: read d0/d1/d5/d38/f68 [555938,77774] 0 2024-03-14T13:02:25.079 INFO:tasks.workunit.client.0.smithi111.stdout:0/933: symlink d6/d9/de/d77/da4/dca/dd0/de8/l14d 0 2024-03-14T13:02:25.082 INFO:tasks.workunit.client.0.smithi111.stdout:7/671: dread - d0/d1/d21/d2a/f93 zero size 2024-03-14T13:02:25.089 INFO:tasks.workunit.client.0.smithi111.stdout:3/763: rename d0/l39 to d0/d2d/d35/d50/d6c/l119 0 2024-03-14T13:02:25.089 INFO:tasks.workunit.client.0.smithi111.stdout:0/934: truncate d6/d5a/f6e 756084 0 2024-03-14T13:02:25.093 INFO:tasks.workunit.client.0.smithi111.stdout:0/935: read d6/d9/d15/d49/d56/fab [1060690,64445] 0 2024-03-14T13:02:25.094 INFO:tasks.workunit.client.0.smithi111.stdout:5/913: dwrite d2/d16/d49/d7c/d107/f11e [0,4194304] 0 2024-03-14T13:02:25.094 INFO:tasks.workunit.client.0.smithi111.stdout:3/764: unlink d0/d2d/d35/d4c/f106 0 2024-03-14T13:02:25.099 INFO:tasks.workunit.client.0.smithi111.stdout:7/672: dwrite d0/d1/d5/f4c [0,4194304] 0 2024-03-14T13:02:25.099 INFO:tasks.workunit.client.0.smithi111.stdout:5/914: rename d2/d16/d49/d44/d4a/d9e/db7/dbb to d2/d16/d49/d44/d4a/d9e/db7/dbb/d123 22 2024-03-14T13:02:25.104 INFO:tasks.workunit.client.1.smithi137.stdout:6/708: sync 2024-03-14T13:02:25.107 INFO:tasks.workunit.client.0.smithi111.stdout:1/839: dread f19 [0,4194304] 0 2024-03-14T13:02:25.109 INFO:tasks.workunit.client.0.smithi111.stdout:5/915: symlink d2/d16/d49/d33/d68/d122/l124 0 2024-03-14T13:02:25.113 INFO:tasks.workunit.client.0.smithi111.stdout:6/664: dwrite d0/d17/d20/d65/f92 [0,4194304] 0 2024-03-14T13:02:25.115 INFO:tasks.workunit.client.0.smithi111.stdout:7/673: dread d0/d1/d5/d38/d1b/d9c/fbb [0,4194304] 0 2024-03-14T13:02:25.128 INFO:tasks.workunit.client.1.smithi137.stdout:6/709: creat d2/d6/d86/d8a/ff2 x:0 0 0 2024-03-14T13:02:25.128 INFO:tasks.workunit.client.0.smithi111.stdout:5/916: read d2/d16/d49/d7c/d95/fa3 [20536,12618] 0 2024-03-14T13:02:25.128 INFO:tasks.workunit.client.0.smithi111.stdout:3/765: dwrite d0/d2d/d35/d50/da2/fec [0,4194304] 0 2024-03-14T13:02:25.130 INFO:tasks.workunit.client.0.smithi111.stdout:1/840: rename d20/d21/dc1/cfc to d20/d25/d31/dff/c11e 0 2024-03-14T13:02:25.131 INFO:tasks.workunit.client.1.smithi137.stdout:1/838: rename d7/d9/d19/d1f/f95 to d7/f121 0 2024-03-14T13:02:25.131 INFO:tasks.workunit.client.0.smithi111.stdout:7/674: dwrite d0/d1/d21/fa4 [0,4194304] 0 2024-03-14T13:02:25.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.132+0000 7f0d8d176700 1 -- 172.21.15.111:0/1366268106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d880708f0 msgr2=0x7f0d88070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.132+0000 7f0d8d176700 1 --2- 172.21.15.111:0/1366268106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d880708f0 0x7f0d88070d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c004940 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:25.133 INFO:tasks.workunit.client.1.smithi137.stdout:1/839: write d7/d9/d19/d1f/d89/d90/fc6 [191401,47110] 0 2024-03-14T13:02:25.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.133+0000 7f0d8d176700 1 -- 172.21.15.111:0/1366268106 shutdown_connections 2024-03-14T13:02:25.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.133+0000 7f0d8d176700 1 --2- 172.21.15.111:0/1366268106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d880708f0 0x7f0d88070d60 secure :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c004940 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:25.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.133+0000 7f0d8d176700 1 --2- 172.21.15.111:0/1366268106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d880703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.133+0000 7f0d8d176700 1 -- 172.21.15.111:0/1366268106 >> 172.21.15.111:0/1366268106 conn(0x7f0d880f8540 msgr2=0x7f0d880fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:25.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.134+0000 7f0d8d176700 1 -- 172.21.15.111:0/1366268106 shutdown_connections 2024-03-14T13:02:25.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.134+0000 7f0d8d176700 1 -- 172.21.15.111:0/1366268106 wait complete. 2024-03-14T13:02:25.135 INFO:tasks.workunit.client.0.smithi111.stdout:3/766: fsync d0/d2d/d35/d50/da2/db3/da7/f101 0 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 Processor -- start 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 -- start start 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d881010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 0x7f0d881024b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88101ad0 con 0x7f0d88101610 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.135+0000 7f0d8d176700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88101c40 con 0x7f0d88071ea0 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d86d9d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d881010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d86d9d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d881010d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:44170/0 (socket says 172.21.15.111:44170) 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d86d9d700 1 -- 172.21.15.111:0/2372642969 learned_addr learned my addr 172.21.15.111:0/2372642969 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:25.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d8659c700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 0x7f0d881024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.138 INFO:tasks.workunit.client.1.smithi137.stdout:6/710: dwrite d2/d4c/fbd [0,4194304] 0 2024-03-14T13:02:25.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d8659c700 1 -- 172.21.15.111:0/2372642969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 msgr2=0x7f0d881010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d8659c700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d881010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.136+0000 7f0d8659c700 1 -- 172.21.15.111:0/2372642969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c005040 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d8659c700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 0x7f0d881024b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c008000 tx=0x7f0d7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c00e040 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d88102ab0 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d88102ff0 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d7c013420 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.137+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d7c01c610 con 0x7f0d88101610 2024-03-14T13:02:25.139 INFO:tasks.workunit.client.0.smithi111.stdout:0/936: dwrite d6/d9/de/d3a/d46/f78 [4194304,4194304] 0 2024-03-14T13:02:25.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.138+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d740052e0 con 0x7f0d88101610 2024-03-14T13:02:25.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.139+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f0d7c00bb80 con 0x7f0d88101610 2024-03-14T13:02:25.141 INFO:tasks.workunit.client.1.smithi137.stdout:1/840: creat d7/d9/d19/d40/d42/f122 x:0 0 0 2024-03-14T13:02:25.141 INFO:tasks.workunit.client.1.smithi137.stdout:6/711: write d2/d54/d55/f78 [1926817,61530] 0 2024-03-14T13:02:25.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.140+0000 7f0d6f7fe700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 0x7f0d7006ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.140+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0d7c08c8e0 con 0x7f0d88101610 2024-03-14T13:02:25.146 INFO:tasks.workunit.client.1.smithi137.stdout:6/712: unlink d2/lba 0 2024-03-14T13:02:25.148 INFO:tasks.workunit.client.0.smithi111.stdout:5/917: creat d2/d16/d49/d44/d4a/d9e/db7/dd6/f125 x:0 0 0 2024-03-14T13:02:25.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.149+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f0d7c056dc0 con 0x7f0d88101610 2024-03-14T13:02:25.150 INFO:tasks.workunit.client.1.smithi137.stdout:6/713: rename d2/d6/f65 to d2/d4c/ff3 0 2024-03-14T13:02:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.150+0000 7f0d86d9d700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 0x7f0d7006ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.151+0000 7f0d86d9d700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 0x7f0d7006ea60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d88100e60 tx=0x7f0d78004570 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:25.152 INFO:tasks.workunit.client.1.smithi137.stdout:1/841: dwrite d7/d31/d82/f111 [0,4194304] 0 2024-03-14T13:02:25.152 INFO:tasks.workunit.client.0.smithi111.stdout:6/665: dwrite d0/f98 [0,4194304] 0 2024-03-14T13:02:25.153 INFO:tasks.workunit.client.1.smithi137.stdout:1/842: chown d7/d9/l13 6028 1 2024-03-14T13:02:25.153 INFO:tasks.workunit.client.0.smithi111.stdout:1/841: link d20/d21/d3f/f66 d20/d21/d3f/f11f 0 2024-03-14T13:02:25.154 INFO:tasks.workunit.client.0.smithi111.stdout:3/767: creat d0/d2d/d35/d50/d5e/f11a x:0 0 0 2024-03-14T13:02:25.155 INFO:tasks.workunit.client.1.smithi137.stdout:6/714: mkdir d2/d6/d1d/d88/df4 0 2024-03-14T13:02:25.157 INFO:tasks.workunit.client.1.smithi137.stdout:1/843: write d7/d9/d19/d40/d42/f10f [378940,114743] 0 2024-03-14T13:02:25.159 INFO:tasks.workunit.client.1.smithi137.stdout:6/715: mkdir d2/d6/dee/df5 0 2024-03-14T13:02:25.160 INFO:tasks.workunit.client.1.smithi137.stdout:1/844: readlink d7/d9/d19/d1f/d26/d7d/lb3 0 2024-03-14T13:02:25.164 INFO:tasks.workunit.client.1.smithi137.stdout:6/716: creat d2/d54/d55/d6d/dc3/ff6 x:0 0 0 2024-03-14T13:02:25.165 INFO:tasks.workunit.client.0.smithi111.stdout:6/666: dwrite d0/d17/d21/d69/d3d/fe4 [0,4194304] 0 2024-03-14T13:02:25.166 INFO:tasks.workunit.client.1.smithi137.stdout:6/717: chown d2/d54/d55/d6d/f87 14605 1 2024-03-14T13:02:25.166 INFO:tasks.workunit.client.1.smithi137.stdout:1/845: truncate d7/d9/d19/d1f/d2e/f34 2083167 0 2024-03-14T13:02:25.166 INFO:tasks.workunit.client.1.smithi137.stdout:1/846: read - d7/d4d/f118 zero size 2024-03-14T13:02:25.168 INFO:tasks.workunit.client.0.smithi111.stdout:7/675: getdents d0/d1/d5/d56/d70 0 2024-03-14T13:02:25.169 INFO:tasks.workunit.client.1.smithi137.stdout:1/847: symlink d7/d9/d19/d40/d50/l123 0 2024-03-14T13:02:25.171 INFO:tasks.workunit.client.1.smithi137.stdout:1/848: dread - d7/d9/d19/d40/ffe zero size 2024-03-14T13:02:25.171 INFO:tasks.workunit.client.1.smithi137.stdout:6/718: dread d2/d54/d55/d9c/dc0/dc4/fcf [0,4194304] 0 2024-03-14T13:02:25.177 INFO:tasks.workunit.client.1.smithi137.stdout:6/719: getdents d2/d6/de/d45/d59/d39/d3c 0 2024-03-14T13:02:25.180 INFO:tasks.workunit.client.1.smithi137.stdout:1/849: dwrite d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:25.181 INFO:tasks.workunit.client.0.smithi111.stdout:6/667: dwrite d0/d17/d21/d69/d3d/d53/d81/da1/fef [0,4194304] 0 2024-03-14T13:02:25.182 INFO:tasks.workunit.client.1.smithi137.stdout:1/850: chown d7/d9/fad 15264 1 2024-03-14T13:02:25.182 INFO:tasks.workunit.client.0.smithi111.stdout:1/842: rename d20/d25/d87/ced to d20/d25/d31/c120 0 2024-03-14T13:02:25.187 INFO:tasks.workunit.client.1.smithi137.stdout:1/851: mknod d7/d4d/d6b/c124 0 2024-03-14T13:02:25.190 INFO:tasks.workunit.client.0.smithi111.stdout:7/676: dwrite d0/d1/d22/f7b [8388608,4194304] 0 2024-03-14T13:02:25.196 INFO:tasks.workunit.client.1.smithi137.stdout:6/720: dwrite d2/d54/d55/d6d/d76/d57/f9a [4194304,4194304] 0 2024-03-14T13:02:25.197 INFO:tasks.workunit.client.1.smithi137.stdout:6/721: chown d2/d4c/f84 20465601 1 2024-03-14T13:02:25.198 INFO:tasks.workunit.client.0.smithi111.stdout:3/768: dwrite d0/d2d/d35/d69/de6/df4/ffd [0,4194304] 0 2024-03-14T13:02:25.199 INFO:tasks.workunit.client.1.smithi137.stdout:6/722: readlink d2/l52 0 2024-03-14T13:02:25.201 INFO:tasks.workunit.client.1.smithi137.stdout:6/723: symlink d2/d6/d1d/d88/df4/lf7 0 2024-03-14T13:02:25.202 INFO:tasks.workunit.client.0.smithi111.stdout:0/937: dwrite d6/d45/d87/fdc [0,4194304] 0 2024-03-14T13:02:25.203 INFO:tasks.workunit.client.1.smithi137.stdout:6/724: chown d2/d54/d55/d6d/d76/fc2 153 1 2024-03-14T13:02:25.204 INFO:tasks.workunit.client.1.smithi137.stdout:6/725: fsync d2/d6/de/fb3 0 2024-03-14T13:02:25.205 INFO:tasks.workunit.client.0.smithi111.stdout:1/843: mknod d20/d21/da7/c121 0 2024-03-14T13:02:25.207 INFO:tasks.workunit.client.0.smithi111.stdout:6/668: fsync d0/d17/d20/f9a 0 2024-03-14T13:02:25.207 INFO:tasks.workunit.client.1.smithi137.stdout:6/726: chown d2/d6/de/l36 11 1 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:2/808: sync 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:5/675: sync 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:0/749: sync 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:3/632: sync 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:4/746: sync 2024-03-14T13:02:25.210 INFO:tasks.workunit.client.1.smithi137.stdout:8/822: sync 2024-03-14T13:02:25.211 INFO:tasks.workunit.client.1.smithi137.stdout:6/727: creat d2/d54/d55/d6d/ff8 x:0 0 0 2024-03-14T13:02:25.213 INFO:tasks.workunit.client.0.smithi111.stdout:5/918: truncate d2/d45/f5c 1419404 0 2024-03-14T13:02:25.214 INFO:tasks.workunit.client.0.smithi111.stdout:0/938: symlink d6/d9/d15/da9/l14e 0 2024-03-14T13:02:25.217 INFO:tasks.workunit.client.1.smithi137.stdout:5/676: symlink d7/d16/d41/dc1/le1 0 2024-03-14T13:02:25.221 INFO:tasks.workunit.client.0.smithi111.stdout:5/919: fdatasync d2/d16/d49/f11 0 2024-03-14T13:02:25.222 INFO:tasks.workunit.client.1.smithi137.stdout:6/728: creat d2/d54/d55/d6d/d76/d57/ff9 x:0 0 0 2024-03-14T13:02:25.222 INFO:tasks.workunit.client.1.smithi137.stdout:1/852: dread d7/d9/d19/d40/d42/d9d/de5/ff4 [0,4194304] 0 2024-03-14T13:02:25.224 INFO:tasks.workunit.client.1.smithi137.stdout:0/750: unlink d1/d1f/d82/f8b 0 2024-03-14T13:02:25.225 INFO:tasks.workunit.client.1.smithi137.stdout:8/823: unlink d1/dc/d3c/d87/c20 0 2024-03-14T13:02:25.225 INFO:tasks.workunit.client.1.smithi137.stdout:0/751: dread - d1/d1f/fe6 zero size 2024-03-14T13:02:25.228 INFO:tasks.workunit.client.0.smithi111.stdout:5/920: stat d2/d16/d49/d7c/d95/d36/lba 0 2024-03-14T13:02:25.229 INFO:tasks.workunit.client.1.smithi137.stdout:1/853: truncate d7/d9/d19/d1f/d89/d90/db8/fbd 1042699 0 2024-03-14T13:02:25.229 INFO:tasks.workunit.client.1.smithi137.stdout:6/729: creat d2/d54/d55/d6d/dc3/deb/ffa x:0 0 0 2024-03-14T13:02:25.230 INFO:tasks.workunit.client.1.smithi137.stdout:8/824: mknod d1/dc/d77/d9f/c116 0 2024-03-14T13:02:25.231 INFO:tasks.workunit.client.0.smithi111.stdout:5/921: readlink d2/d16/d49/d44/l11b 0 2024-03-14T13:02:25.231 INFO:tasks.workunit.client.0.smithi111.stdout:1/844: dwrite d20/d25/da8/f10f [0,4194304] 0 2024-03-14T13:02:25.231 INFO:tasks.workunit.client.0.smithi111.stdout:3/769: dread d0/d2d/d35/d69/de6/df4/ffd [0,4194304] 0 2024-03-14T13:02:25.232 INFO:tasks.workunit.client.1.smithi137.stdout:6/730: fdatasync d2/d6/fd9 0 2024-03-14T13:02:25.233 INFO:tasks.workunit.client.0.smithi111.stdout:6/669: link d0/d17/d21/d69/f39 d0/d17/d20/d4a/d68/ff9 0 2024-03-14T13:02:25.234 INFO:tasks.workunit.client.1.smithi137.stdout:5/677: dwrite d7/d16/d18/f2a [0,4194304] 0 2024-03-14T13:02:25.234 INFO:tasks.workunit.client.0.smithi111.stdout:1/845: fsync d20/f9d 0 2024-03-14T13:02:25.235 INFO:tasks.workunit.client.1.smithi137.stdout:0/752: unlink d1/d5/d1b/d42/cee 0 2024-03-14T13:02:25.236 INFO:tasks.workunit.client.0.smithi111.stdout:5/922: write d2/d45/d76/fe3 [1099493,69094] 0 2024-03-14T13:02:25.236 INFO:tasks.workunit.client.1.smithi137.stdout:8/825: truncate d1/dc/d4e/d6d/f74 693167 0 2024-03-14T13:02:25.237 INFO:tasks.workunit.client.0.smithi111.stdout:3/770: mknod d0/d2d/d35/d4c/d82/c11b 0 2024-03-14T13:02:25.237 INFO:tasks.workunit.client.0.smithi111.stdout:5/923: write d2/f6 [178066,56031] 0 2024-03-14T13:02:25.238 INFO:tasks.workunit.client.1.smithi137.stdout:0/753: readlink d1/d5/d1b/d42/d3a/lb2 0 2024-03-14T13:02:25.240 INFO:tasks.workunit.client.0.smithi111.stdout:3/771: chown d0/d2d/d35/d69/d84/fb2 27 1 2024-03-14T13:02:25.240 INFO:tasks.workunit.client.1.smithi137.stdout:8/826: write d1/dc/d77/d9f/de4/fab [4493839,47667] 0 2024-03-14T13:02:25.242 INFO:tasks.workunit.client.0.smithi111.stdout:3/772: readlink d0/d2d/d35/d50/d5e/d19/l29 0 2024-03-14T13:02:25.243 INFO:tasks.workunit.client.1.smithi137.stdout:1/854: getdents d7/d9/d19/d1f/d2e/d63 0 2024-03-14T13:02:25.244 INFO:tasks.workunit.client.1.smithi137.stdout:3/633: write d4/d19/f66 [3920486,66431] 0 2024-03-14T13:02:25.245 INFO:tasks.workunit.client.1.smithi137.stdout:0/754: creat d1/d1f/d80/f100 x:0 0 0 2024-03-14T13:02:25.248 INFO:tasks.workunit.client.0.smithi111.stdout:5/924: rename d2/d16/d49/d4c/d9c/d10a to d2/d16/d49/d7c/d126 0 2024-03-14T13:02:25.251 INFO:tasks.workunit.client.0.smithi111.stdout:7/677: truncate d0/d1/d4a/f60 1852979 0 2024-03-14T13:02:25.252 INFO:tasks.workunit.client.1.smithi137.stdout:3/634: read d4/df/f83 [408060,45243] 0 2024-03-14T13:02:25.252 INFO:tasks.workunit.client.1.smithi137.stdout:8/827: symlink d1/dc/d3c/d87/d34/l117 0 2024-03-14T13:02:25.253 INFO:tasks.workunit.client.0.smithi111.stdout:3/773: rmdir d0/d30/df1 39 2024-03-14T13:02:25.254 INFO:tasks.workunit.client.1.smithi137.stdout:0/755: creat d1/d44/d72/de8/f101 x:0 0 0 2024-03-14T13:02:25.254 INFO:tasks.workunit.client.0.smithi111.stdout:6/670: getdents d0/d17/d20/d4a/d59 0 2024-03-14T13:02:25.255 INFO:tasks.workunit.client.1.smithi137.stdout:1/855: symlink d7/d9/d19/d1f/d89/d90/db8/l125 0 2024-03-14T13:02:25.257 INFO:tasks.workunit.client.1.smithi137.stdout:2/809: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/f63 [0,4194304] 0 2024-03-14T13:02:25.258 INFO:tasks.workunit.client.1.smithi137.stdout:6/731: rmdir d2/d54/d55/d6d/dc3/deb 39 2024-03-14T13:02:25.258 INFO:tasks.workunit.client.0.smithi111.stdout:7/678: write d0/d1/d22/d3e/faf [14552,81383] 0 2024-03-14T13:02:25.259 INFO:tasks.workunit.client.0.smithi111.stdout:3/774: rmdir d0/d2d/d35/d50/da2 39 2024-03-14T13:02:25.260 INFO:tasks.workunit.client.1.smithi137.stdout:0/756: read d1/d5/d1b/d1e/d53/d6d/f74 [2179610,16764] 0 2024-03-14T13:02:25.262 INFO:tasks.workunit.client.1.smithi137.stdout:8/828: write d1/dc/f5c [542419,89315] 0 2024-03-14T13:02:25.263 INFO:tasks.workunit.client.0.smithi111.stdout:1/846: truncate f11 3029033 0 2024-03-14T13:02:25.269 INFO:tasks.workunit.client.1.smithi137.stdout:6/732: write d2/d6/de/d45/fa8 [2335485,24243] 0 2024-03-14T13:02:25.269 INFO:tasks.workunit.client.1.smithi137.stdout:5/678: dwrite d7/d49/d75/f85 [0,4194304] 0 2024-03-14T13:02:25.273 INFO:tasks.workunit.client.1.smithi137.stdout:1/856: mknod d7/d9/d19/d40/c126 0 2024-03-14T13:02:25.273 INFO:tasks.workunit.client.0.smithi111.stdout:1/847: unlink d20/d21/d28/d5c/l10b 0 2024-03-14T13:02:25.274 INFO:tasks.workunit.client.0.smithi111.stdout:0/939: dwrite d6/d9/d15/d55/da0/fdf [0,4194304] 0 2024-03-14T13:02:25.275 INFO:tasks.workunit.client.1.smithi137.stdout:8/829: mkdir d1/dc/d4e/d6d/db5/d118 0 2024-03-14T13:02:25.275 INFO:tasks.workunit.client.0.smithi111.stdout:7/679: readlink d0/d11/d98/dab/lbe 0 2024-03-14T13:02:25.276 INFO:tasks.workunit.client.1.smithi137.stdout:8/830: chown d1/d5e 15880401 1 2024-03-14T13:02:25.277 INFO:tasks.workunit.client.0.smithi111.stdout:3/775: mknod d0/d2d/dc7/c11c 0 2024-03-14T13:02:25.283 INFO:tasks.workunit.client.1.smithi137.stdout:1/857: mknod d7/d9/d19/d40/d42/d9d/c127 0 2024-03-14T13:02:25.285 INFO:tasks.workunit.client.1.smithi137.stdout:8/831: truncate d1/dc/d71/db0/dca/ffe 225874 0 2024-03-14T13:02:25.288 INFO:tasks.workunit.client.0.smithi111.stdout:9/729: sync 2024-03-14T13:02:25.288 INFO:tasks.workunit.client.0.smithi111.stdout:2/635: sync 2024-03-14T13:02:25.289 INFO:tasks.workunit.client.0.smithi111.stdout:3/776: symlink d0/d2d/d35/d4c/dad/dee/l11d 0 2024-03-14T13:02:25.290 INFO:tasks.workunit.client.1.smithi137.stdout:1/858: write d7/d31/d82/f111 [2305740,30771] 0 2024-03-14T13:02:25.291 INFO:tasks.workunit.client.1.smithi137.stdout:6/733: dwrite d2/d54/d55/d6d/d76/fda [0,4194304] 0 2024-03-14T13:02:25.291 INFO:tasks.workunit.client.1.smithi137.stdout:8/832: mknod d1/dc/d10/c119 0 2024-03-14T13:02:25.293 INFO:tasks.workunit.client.1.smithi137.stdout:3/635: dread d4/de/d16/f31 [0,4194304] 0 2024-03-14T13:02:25.294 INFO:tasks.workunit.client.0.smithi111.stdout:6/671: dwrite d0/d17/d21/d69/d3d/d53/d81/da1/dc8/ff3 [0,4194304] 0 2024-03-14T13:02:25.295 INFO:tasks.workunit.client.1.smithi137.stdout:6/734: chown d2/c4a 7138 1 2024-03-14T13:02:25.296 INFO:tasks.workunit.client.0.smithi111.stdout:2/636: mknod dd/d17/cc5 0 2024-03-14T13:02:25.297 INFO:tasks.workunit.client.1.smithi137.stdout:3/636: write d4/fd [1822588,21246] 0 2024-03-14T13:02:25.297 INFO:tasks.workunit.client.1.smithi137.stdout:0/757: mkdir d1/d5/d1b/d1e/d53/d6d/d102 0 2024-03-14T13:02:25.308 INFO:tasks.workunit.client.1.smithi137.stdout:8/833: rename d1/dc/d3c/d87/d34 to d1/d5e/d11a 0 2024-03-14T13:02:25.309 INFO:tasks.workunit.client.1.smithi137.stdout:5/679: dwrite d7/d1d/d74/f78 [0,4194304] 0 2024-03-14T13:02:25.310 INFO:tasks.workunit.client.1.smithi137.stdout:0/758: symlink d1/d44/d56/l103 0 2024-03-14T13:02:25.311 INFO:tasks.workunit.client.0.smithi111.stdout:1/848: dwrite d20/d25/da8/ff8 [0,4194304] 0 2024-03-14T13:02:25.311 INFO:tasks.workunit.client.0.smithi111.stdout:5/925: dread d2/d13/f6c [0,4194304] 0 2024-03-14T13:02:25.316 INFO:tasks.workunit.client.1.smithi137.stdout:0/759: stat d1/d5/d1b/d1e/d25/d55/d73/cd1 0 2024-03-14T13:02:25.316 INFO:tasks.workunit.client.1.smithi137.stdout:5/680: unlink d7/d33/dab/cd7 0 2024-03-14T13:02:25.316 INFO:tasks.workunit.client.0.smithi111.stdout:0/940: dwrite d6/d9/de/d3a/d46/f78 [8388608,4194304] 0 2024-03-14T13:02:25.319 INFO:tasks.workunit.client.0.smithi111.stdout:3/777: write d0/d2d/d35/d50/da2/db3/d6f/fdb [356247,124923] 0 2024-03-14T13:02:25.319 INFO:tasks.workunit.client.0.smithi111.stdout:2/637: mknod dd/d17/d34/d73/d49/cc6 0 2024-03-14T13:02:25.321 INFO:tasks.workunit.client.0.smithi111.stdout:3/778: chown d0/d2d/d35/d69/fa0 55068 1 2024-03-14T13:02:25.321 INFO:tasks.workunit.client.1.smithi137.stdout:8/834: truncate d1/dc/d10/fa4 2210681 0 2024-03-14T13:02:25.321 INFO:tasks.workunit.client.1.smithi137.stdout:1/859: truncate d7/d4d/d6b/faa 1236035 0 2024-03-14T13:02:25.322 INFO:tasks.workunit.client.0.smithi111.stdout:1/849: mkdir d20/d21/d122 0 2024-03-14T13:02:25.322 INFO:tasks.workunit.client.0.smithi111.stdout:5/926: write d2/d16/d49/d44/f8e [1179957,29706] 0 2024-03-14T13:02:25.323 INFO:tasks.workunit.client.0.smithi111.stdout:0/941: unlink d6/d3d/lcb 0 2024-03-14T13:02:25.326 INFO:tasks.workunit.client.0.smithi111.stdout:9/730: dwrite d5/d6/d14/f54 [0,4194304] 0 2024-03-14T13:02:25.328 INFO:tasks.workunit.client.0.smithi111.stdout:0/942: stat d6/d9/d15/d49/l13e 0 2024-03-14T13:02:25.329 INFO:tasks.workunit.client.1.smithi137.stdout:0/760: mknod d1/d5/d1b/d42/d3a/c104 0 2024-03-14T13:02:25.330 INFO:tasks.workunit.client.1.smithi137.stdout:5/681: dwrite d7/d16/fb9 [4194304,4194304] 0 2024-03-14T13:02:25.330 INFO:tasks.workunit.client.0.smithi111.stdout:1/850: symlink d20/d21/d122/l123 0 2024-03-14T13:02:25.330 INFO:tasks.workunit.client.0.smithi111.stdout:5/927: symlink d2/d45/d76/d8d/l127 0 2024-03-14T13:02:25.331 INFO:tasks.workunit.client.1.smithi137.stdout:1/860: mknod d7/d9/d19/d1f/d22/c128 0 2024-03-14T13:02:25.333 INFO:tasks.workunit.client.0.smithi111.stdout:2/638: rename dd/d17/d34/f42 to dd/d8d/dc2/fc7 0 2024-03-14T13:02:25.334 INFO:tasks.workunit.client.0.smithi111.stdout:0/943: truncate d6/d9/de/d3a/f9c 408170 0 2024-03-14T13:02:25.336 INFO:tasks.workunit.client.0.smithi111.stdout:2/639: chown dd/d17/d53/da4/l99 38369510 1 2024-03-14T13:02:25.337 INFO:tasks.workunit.client.0.smithi111.stdout:0/944: write d6/d9/de/d3a/fc3 [4845524,33138] 0 2024-03-14T13:02:25.337 INFO:tasks.workunit.client.0.smithi111.stdout:5/928: creat d2/d16/d49/d33/de5/dfa/f128 x:0 0 0 2024-03-14T13:02:25.337 INFO:tasks.workunit.client.1.smithi137.stdout:0/761: read d1/d5/d1b/d1e/d53/fed [325815,44966] 0 2024-03-14T13:02:25.337 INFO:tasks.workunit.client.1.smithi137.stdout:1/861: readlink d7/l8 0 2024-03-14T13:02:25.338 INFO:tasks.workunit.client.1.smithi137.stdout:7/736: sync 2024-03-14T13:02:25.338 INFO:tasks.workunit.client.1.smithi137.stdout:9/842: sync 2024-03-14T13:02:25.339 INFO:tasks.workunit.client.0.smithi111.stdout:9/731: rename d5/d6/d8/d5b/f6e to d5/ffc 0 2024-03-14T13:02:25.339 INFO:tasks.workunit.client.0.smithi111.stdout:5/929: dread - d2/d16/d49/d44/d72/de0/f120 zero size 2024-03-14T13:02:25.341 INFO:tasks.workunit.client.0.smithi111.stdout:0/945: fsync d6/d9/de/d32/d64/de7/ffd 0 2024-03-14T13:02:25.342 INFO:tasks.workunit.client.1.smithi137.stdout:3/637: dwrite d4/df/d82/fc3 [0,4194304] 0 2024-03-14T13:02:25.342 INFO:tasks.workunit.client.1.smithi137.stdout:1/862: write d7/d9/d19/d1f/d89/d90/f116 [739818,1416] 0 2024-03-14T13:02:25.342 INFO:tasks.workunit.client.0.smithi111.stdout:9/732: creat d5/d6/d8/d5b/d84/db6/de7/ffd x:0 0 0 2024-03-14T13:02:25.343 INFO:tasks.workunit.client.0.smithi111.stdout:3/779: dread d0/d2d/d35/d4c/d82/fab [0,4194304] 0 2024-03-14T13:02:25.346 INFO:tasks.workunit.client.1.smithi137.stdout:8/835: rename d1/dc/d71/da3/ld7 to d1/dc/d77/d9f/de4/l11b 0 2024-03-14T13:02:25.346 INFO:tasks.workunit.client.1.smithi137.stdout:1/863: stat d7/d9/d19/d1f/d22/fde 0 2024-03-14T13:02:25.349 INFO:tasks.workunit.client.0.smithi111.stdout:9/733: rmdir d5/d6/d2a/d2d/db8 39 2024-03-14T13:02:25.350 INFO:tasks.workunit.client.1.smithi137.stdout:7/737: symlink d3/d6/d59/daa/lf4 0 2024-03-14T13:02:25.350 INFO:tasks.workunit.client.1.smithi137.stdout:0/762: write d1/fb3 [767436,68617] 0 2024-03-14T13:02:25.353 INFO:tasks.workunit.client.1.smithi137.stdout:8/836: truncate d1/d5e/d11a/fa6 399515 0 2024-03-14T13:02:25.353 INFO:tasks.workunit.client.1.smithi137.stdout:5/682: creat d7/d16/d18/d25/fe2 x:0 0 0 2024-03-14T13:02:25.354 INFO:tasks.workunit.client.1.smithi137.stdout:8/837: fdatasync d1/dc/f5c 0 2024-03-14T13:02:25.355 INFO:tasks.workunit.client.1.smithi137.stdout:5/683: dread - d7/d16/d18/d2d/dd1/fdf zero size 2024-03-14T13:02:25.356 INFO:tasks.workunit.client.1.smithi137.stdout:1/864: fsync d7/d9/d19/d1f/d26/fd0 0 2024-03-14T13:02:25.357 INFO:tasks.workunit.client.1.smithi137.stdout:1/865: chown d7/d4d/l73 25248 1 2024-03-14T13:02:25.361 INFO:tasks.workunit.client.1.smithi137.stdout:1/866: mkdir d7/d9/d19/d1f/d2e/da6/d129 0 2024-03-14T13:02:25.364 INFO:tasks.workunit.client.1.smithi137.stdout:5/684: dread d7/d33/d65/f7c [0,4194304] 0 2024-03-14T13:02:25.367 INFO:tasks.workunit.client.1.smithi137.stdout:1/867: rename d7/d9/d19/d1f/d26/d6c/l7c to d7/d9/d19/d1f/da0/l12a 0 2024-03-14T13:02:25.369 INFO:tasks.workunit.client.1.smithi137.stdout:5/685: creat d7/d1d/d74/fe3 x:0 0 0 2024-03-14T13:02:25.374 INFO:tasks.workunit.client.0.smithi111.stdout:6/672: write d0/d17/d20/d4a/d68/ff9 [856077,102316] 0 2024-03-14T13:02:25.377 INFO:tasks.workunit.client.1.smithi137.stdout:9/843: write d6/df/f6a [1164065,51303] 0 2024-03-14T13:02:25.378 INFO:tasks.workunit.client.1.smithi137.stdout:5/686: dwrite d7/d36/f95 [0,4194304] 0 2024-03-14T13:02:25.378 INFO:tasks.workunit.client.0.smithi111.stdout:1/851: dread d20/d25/f3d [0,4194304] 0 2024-03-14T13:02:25.379 INFO:tasks.workunit.client.0.smithi111.stdout:2/640: write dd/d17/d34/f2e [22304,101070] 0 2024-03-14T13:02:25.381 INFO:tasks.workunit.client.1.smithi137.stdout:8/838: write d1/d5e/d11a/d97/d7f/f113 [335763,103670] 0 2024-03-14T13:02:25.383 INFO:tasks.workunit.client.0.smithi111.stdout:3/780: write d0/d2d/d35/d50/da2/db3/d6f/f79 [462727,68160] 0 2024-03-14T13:02:25.383 INFO:tasks.workunit.client.0.smithi111.stdout:9/734: write d5/d6/d2a/f4e [6424521,21783] 0 2024-03-14T13:02:25.387 INFO:tasks.workunit.client.0.smithi111.stdout:1/852: creat d20/d21/d28/f124 x:0 0 0 2024-03-14T13:02:25.387 INFO:tasks.workunit.client.0.smithi111.stdout:9/735: write d5/d6/feb [639025,32930] 0 2024-03-14T13:02:25.389 INFO:tasks.workunit.client.0.smithi111.stdout:2/641: link dd/d17/d53/c57 dd/d17/d34/d74/dc1/cc8 0 2024-03-14T13:02:25.395 INFO:tasks.workunit.client.0.smithi111.stdout:3/781: creat d0/d2d/d35/d4c/f11e x:0 0 0 2024-03-14T13:02:25.397 INFO:tasks.workunit.client.0.smithi111.stdout:0/946: dwrite d6/d45/fbb [0,4194304] 0 2024-03-14T13:02:25.397 INFO:tasks.workunit.client.0.smithi111.stdout:1/853: mknod d20/d25/d31/dff/c125 0 2024-03-14T13:02:25.398 INFO:tasks.workunit.client.0.smithi111.stdout:9/736: creat d5/d6/d8/d5b/d87/ffe x:0 0 0 2024-03-14T13:02:25.400 INFO:tasks.workunit.client.0.smithi111.stdout:3/782: fsync d0/f1c 0 2024-03-14T13:02:25.401 INFO:tasks.workunit.client.0.smithi111.stdout:0/947: symlink d6/d9/de/d77/l14f 0 2024-03-14T13:02:25.402 INFO:tasks.workunit.client.0.smithi111.stdout:9/737: creat d5/d42/dd4/fff x:0 0 0 2024-03-14T13:02:25.403 INFO:tasks.workunit.client.0.smithi111.stdout:1/854: write d20/d25/f39 [3513829,22606] 0 2024-03-14T13:02:25.404 INFO:tasks.workunit.client.0.smithi111.stdout:0/948: fdatasync d6/d9/de/f27 0 2024-03-14T13:02:25.406 INFO:tasks.workunit.client.0.smithi111.stdout:1/855: symlink d20/d25/d31/d80/l126 0 2024-03-14T13:02:25.407 INFO:tasks.workunit.client.0.smithi111.stdout:1/856: stat d20/d25/d31/f3a 0 2024-03-14T13:02:25.410 INFO:tasks.workunit.client.0.smithi111.stdout:2/642: dread dd/d11/f6d [0,4194304] 0 2024-03-14T13:02:25.410 INFO:tasks.workunit.client.0.smithi111.stdout:0/949: creat d6/d3d/d97/d129/f150 x:0 0 0 2024-03-14T13:02:25.413 INFO:tasks.workunit.client.0.smithi111.stdout:2/643: creat dd/d11/d1a/d56/fc9 x:0 0 0 2024-03-14T13:02:25.415 INFO:tasks.workunit.client.0.smithi111.stdout:1/857: getdents d20/d21/da7 0 2024-03-14T13:02:25.416 INFO:tasks.workunit.client.0.smithi111.stdout:1/858: fsync d20/d25/d31/f45 0 2024-03-14T13:02:25.417 INFO:tasks.workunit.client.0.smithi111.stdout:2/644: getdents dd/d17/d34/dbf 0 2024-03-14T13:02:25.417 INFO:tasks.workunit.client.0.smithi111.stdout:0/950: dread d6/d9/de/f85 [0,4194304] 0 2024-03-14T13:02:25.418 INFO:tasks.workunit.client.0.smithi111.stdout:0/951: dread - d6/d45/f13a zero size 2024-03-14T13:02:25.419 INFO:tasks.workunit.client.0.smithi111.stdout:2/645: fsync dd/d8d/dc2/f66 0 2024-03-14T13:02:25.420 INFO:tasks.workunit.client.0.smithi111.stdout:1/859: mkdir d20/d127 0 2024-03-14T13:02:25.421 INFO:tasks.workunit.client.0.smithi111.stdout:2/646: chown dd/d17/d34/d73/c7f 61004 1 2024-03-14T13:02:25.421 INFO:tasks.workunit.client.0.smithi111.stdout:9/738: dread d5/d6/d8/f93 [0,4194304] 0 2024-03-14T13:02:25.424 INFO:tasks.workunit.client.0.smithi111.stdout:0/952: read d6/d9/de/d77/da4/feb [1425279,101310] 0 2024-03-14T13:02:25.425 INFO:tasks.workunit.client.0.smithi111.stdout:2/647: mkdir dd/d17/d34/d74/dca 0 2024-03-14T13:02:25.425 INFO:tasks.workunit.client.1.smithi137.stdout:9/844: mkdir d6/df/d19/d1f/de8/d118 0 2024-03-14T13:02:25.428 INFO:tasks.workunit.client.0.smithi111.stdout:2/648: readlink dd/d17/d34/d73/d75/la8 0 2024-03-14T13:02:25.430 INFO:tasks.workunit.client.0.smithi111.stdout:9/739: getdents d5/d6/d14/d7a 0 2024-03-14T13:02:25.430 INFO:tasks.workunit.client.0.smithi111.stdout:1/860: dwrite d20/d21/d3f/d59/f74 [0,4194304] 0 2024-03-14T13:02:25.432 INFO:tasks.workunit.client.1.smithi137.stdout:8/839: unlink d1/dc/d10/c80 0 2024-03-14T13:02:25.433 INFO:tasks.workunit.client.0.smithi111.stdout:0/953: rmdir d6/d3d/da6/dfe/d134/d14b 0 2024-03-14T13:02:25.433 INFO:tasks.workunit.client.0.smithi111.stdout:2/649: mkdir dd/d17/d34/d74/dca/dcb 0 2024-03-14T13:02:25.434 INFO:tasks.workunit.client.0.smithi111.stdout:1/861: chown d20/c2c 498 1 2024-03-14T13:02:25.435 INFO:tasks.workunit.client.0.smithi111.stdout:1/862: fsync d20/d25/d31/dff/db6/ff2 0 2024-03-14T13:02:25.438 INFO:tasks.workunit.client.0.smithi111.stdout:0/954: mknod d6/d3d/dc1/c151 0 2024-03-14T13:02:25.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/650: symlink dd/d8d/lcc 0 2024-03-14T13:02:25.439 INFO:tasks.workunit.client.0.smithi111.stdout:6/673: dread d0/d17/d21/f79 [0,4194304] 0 2024-03-14T13:02:25.439 INFO:tasks.workunit.client.1.smithi137.stdout:3/638: dwrite d4/de/d16/d1c/d5d/f7c [0,4194304] 0 2024-03-14T13:02:25.439 INFO:tasks.workunit.client.1.smithi137.stdout:7/738: dwrite d3/d6/d2e/f49 [0,4194304] 0 2024-03-14T13:02:25.440 INFO:tasks.workunit.client.0.smithi111.stdout:0/955: stat d6/d9/l65 0 2024-03-14T13:02:25.441 INFO:tasks.workunit.client.0.smithi111.stdout:1/863: mknod d20/d21/d122/c128 0 2024-03-14T13:02:25.442 INFO:tasks.workunit.client.1.smithi137.stdout:9/845: getdents d6/df/d88/df4 0 2024-03-14T13:02:25.445 INFO:tasks.workunit.client.1.smithi137.stdout:9/846: read - d6/df/d19/d1f/fb4 zero size 2024-03-14T13:02:25.445 INFO:tasks.workunit.client.1.smithi137.stdout:8/840: fdatasync d1/d5e/d11a/d97/d7f/fe8 0 2024-03-14T13:02:25.447 INFO:tasks.workunit.client.1.smithi137.stdout:1/868: dwrite d7/d9/d19/d40/f5b [0,4194304] 0 2024-03-14T13:02:25.448 INFO:tasks.workunit.client.1.smithi137.stdout:7/739: mkdir d3/d52/d77/df5 0 2024-03-14T13:02:25.449 INFO:tasks.workunit.client.0.smithi111.stdout:0/956: dread - d6/d9/de/d3a/d46/d124/f146 zero size 2024-03-14T13:02:25.449 INFO:tasks.workunit.client.0.smithi111.stdout:2/651: dwrite dd/d8d/fc0 [0,4194304] 0 2024-03-14T13:02:25.453 INFO:tasks.workunit.client.1.smithi137.stdout:3/639: rename d4/de/d16/d1c/l46 to d4/de/d13/ld4 0 2024-03-14T13:02:25.453 INFO:tasks.workunit.client.0.smithi111.stdout:0/957: truncate d6/d9/d5b/f113 627155 0 2024-03-14T13:02:25.454 INFO:tasks.workunit.client.1.smithi137.stdout:7/740: write d3/d5/d23/fb5 [3887831,49574] 0 2024-03-14T13:02:25.458 INFO:tasks.workunit.client.0.smithi111.stdout:6/674: link d0/d17/d20/d4a/d59/c60 d0/d17/d20/d4a/d59/d82/cfa 0 2024-03-14T13:02:25.458 INFO:tasks.workunit.client.1.smithi137.stdout:9/847: truncate d6/df/d88/de5/df6/da9/fe4 152211 0 2024-03-14T13:02:25.460 INFO:tasks.workunit.client.1.smithi137.stdout:9/848: dread - d6/df/d19/d1f/f112 zero size 2024-03-14T13:02:25.461 INFO:tasks.workunit.client.0.smithi111.stdout:6/675: write d0/d17/d20/d5a/fcb [1137108,50226] 0 2024-03-14T13:02:25.463 INFO:tasks.workunit.client.1.smithi137.stdout:1/869: dread d7/d9/d19/d1f/d89/d90/db8/fbd [0,4194304] 0 2024-03-14T13:02:25.465 INFO:tasks.workunit.client.0.smithi111.stdout:0/958: creat d6/d9/d15/d55/f152 x:0 0 0 2024-03-14T13:02:25.466 INFO:tasks.workunit.client.1.smithi137.stdout:8/841: dread d1/dc/f5c [0,4194304] 0 2024-03-14T13:02:25.466 INFO:tasks.workunit.client.1.smithi137.stdout:7/741: getdents d3/d6/d59/daa/dcf/d6a 0 2024-03-14T13:02:25.466 INFO:tasks.workunit.client.1.smithi137.stdout:1/870: truncate f3 863975 0 2024-03-14T13:02:25.467 INFO:tasks.workunit.client.1.smithi137.stdout:9/849: mknod d6/df/d19/d33/dec/c119 0 2024-03-14T13:02:25.467 INFO:tasks.workunit.client.0.smithi111.stdout:6/676: rename d0/d17/d24 to d0/dfb 0 2024-03-14T13:02:25.468 INFO:tasks.workunit.client.1.smithi137.stdout:5/687: dread d7/d16/d18/d2d/f63 [0,4194304] 0 2024-03-14T13:02:25.468 INFO:tasks.workunit.client.1.smithi137.stdout:7/742: fsync d3/d6/d2e/f49 0 2024-03-14T13:02:25.469 INFO:tasks.workunit.client.1.smithi137.stdout:5/688: stat d7/l12 0 2024-03-14T13:02:25.470 INFO:tasks.workunit.client.1.smithi137.stdout:8/842: truncate d1/d5e/d11a/d97/d7f/fa0 4572123 0 2024-03-14T13:02:25.471 INFO:tasks.workunit.client.0.smithi111.stdout:0/959: rename d6/d3d/cf8 to d6/d9/d15/d55/d89/c153 0 2024-03-14T13:02:25.471 INFO:tasks.workunit.client.1.smithi137.stdout:7/743: creat d3/d5/d23/d3f/ff6 x:0 0 0 2024-03-14T13:02:25.472 INFO:tasks.workunit.client.1.smithi137.stdout:9/850: mknod d6/df/d19/d33/ddf/d116/c11a 0 2024-03-14T13:02:25.474 INFO:tasks.workunit.client.1.smithi137.stdout:5/689: fsync d7/d33/f93 0 2024-03-14T13:02:25.474 INFO:tasks.workunit.client.0.smithi111.stdout:0/960: mkdir d6/d9/de/d3a/dd1/d154 0 2024-03-14T13:02:25.474 INFO:tasks.workunit.client.1.smithi137.stdout:8/843: dread - d1/d5e/d11a/d82/f110 zero size 2024-03-14T13:02:25.476 INFO:tasks.workunit.client.1.smithi137.stdout:7/744: mknod d3/d52/d77/cf7 0 2024-03-14T13:02:25.476 INFO:tasks.workunit.client.1.smithi137.stdout:5/690: read d7/d16/d18/d2d/f8f [543175,17864] 0 2024-03-14T13:02:25.476 INFO:tasks.workunit.client.1.smithi137.stdout:9/851: mknod d6/df/d19/d33/ddf/d116/c11b 0 2024-03-14T13:02:25.477 INFO:tasks.workunit.client.1.smithi137.stdout:5/691: stat d7/d1d/d74 0 2024-03-14T13:02:25.479 INFO:tasks.workunit.client.1.smithi137.stdout:8/844: creat d1/dc/d10/f11c x:0 0 0 2024-03-14T13:02:25.479 INFO:tasks.workunit.client.1.smithi137.stdout:7/745: mknod d3/d54/db4/cf8 0 2024-03-14T13:02:25.481 INFO:tasks.workunit.client.1.smithi137.stdout:5/692: rename d7/d16/d18/d25/d28/fa0 to d7/d33/d65/dc2/fe4 0 2024-03-14T13:02:25.482 INFO:tasks.workunit.client.0.smithi111.stdout:0/961: dwrite d6/d9/de/f30 [0,4194304] 0 2024-03-14T13:02:25.482 INFO:tasks.workunit.client.1.smithi137.stdout:5/693: readlink d7/l12 0 2024-03-14T13:02:25.482 INFO:tasks.workunit.client.1.smithi137.stdout:7/746: symlink d3/d54/lf9 0 2024-03-14T13:02:25.484 INFO:tasks.workunit.client.1.smithi137.stdout:9/852: creat d6/df/d19/f11c x:0 0 0 2024-03-14T13:02:25.487 INFO:tasks.workunit.client.1.smithi137.stdout:5/694: unlink d7/d49/d75/f85 0 2024-03-14T13:02:25.488 INFO:tasks.workunit.client.1.smithi137.stdout:5/695: write d7/fe0 [331467,13917] 0 2024-03-14T13:02:25.488 INFO:tasks.workunit.client.0.smithi111.stdout:2/652: dread dd/d11/d1a/f38 [0,4194304] 0 2024-03-14T13:02:25.489 INFO:tasks.workunit.client.1.smithi137.stdout:5/696: chown d7/d16/d18/f2a 0 1 2024-03-14T13:02:25.491 INFO:tasks.workunit.client.0.smithi111.stdout:9/740: dwrite d5/d6/d2a/d5a/f6f [0,4194304] 0 2024-03-14T13:02:25.491 INFO:tasks.workunit.client.1.smithi137.stdout:5/697: write d7/d33/d65/f84 [2353444,42156] 0 2024-03-14T13:02:25.492 INFO:tasks.workunit.client.0.smithi111.stdout:2/653: stat c0 0 2024-03-14T13:02:25.494 INFO:tasks.workunit.client.0.smithi111.stdout:6/677: truncate d0/d17/d21/d69/d3d/d53/d81/da1/dc8/ff3 116294 0 2024-03-14T13:02:25.496 INFO:tasks.workunit.client.1.smithi137.stdout:8/845: rename d1/dc/d4e/d6d/db5 to d1/d5e/d11a/d97/d11d 0 2024-03-14T13:02:25.497 INFO:tasks.workunit.client.0.smithi111.stdout:0/962: dwrite d6/d9/de/d3a/fc3 [0,4194304] 0 2024-03-14T13:02:25.498 INFO:tasks.workunit.client.1.smithi137.stdout:9/853: mknod d6/df/d19/c11d 0 2024-03-14T13:02:25.500 INFO:tasks.workunit.client.1.smithi137.stdout:9/854: dread - d6/df/d88/de5/df6/d40/f10d zero size 2024-03-14T13:02:25.500 INFO:tasks.workunit.client.0.smithi111.stdout:9/741: rename d5/d42/c7b to d5/d42/d71/ddc/c100 0 2024-03-14T13:02:25.501 INFO:tasks.workunit.client.1.smithi137.stdout:1/871: truncate d7/d9/d19/d1f/d2e/f79 968172 0 2024-03-14T13:02:25.502 INFO:tasks.workunit.client.1.smithi137.stdout:9/855: truncate d6/df/d71/fa5 5285584 0 2024-03-14T13:02:25.503 INFO:tasks.workunit.client.0.smithi111.stdout:9/742: chown d5/l98 0 1 2024-03-14T13:02:25.503 INFO:tasks.workunit.client.1.smithi137.stdout:7/747: getdents d3/d6/d59/daa/dcf/d6a 0 2024-03-14T13:02:25.503 INFO:tasks.workunit.client.1.smithi137.stdout:8/846: dread - d1/dc/d77/d9f/fa1 zero size 2024-03-14T13:02:25.504 INFO:tasks.workunit.client.1.smithi137.stdout:9/856: read - d6/df/d71/fc4 zero size 2024-03-14T13:02:25.505 INFO:tasks.workunit.client.1.smithi137.stdout:9/857: readlink d6/l2d 0 2024-03-14T13:02:25.505 INFO:tasks.workunit.client.1.smithi137.stdout:7/748: stat d3/d5/d23/l44 0 2024-03-14T13:02:25.506 INFO:tasks.workunit.client.0.smithi111.stdout:6/678: dread d0/d17/d20/d4a/d68/f88 [0,4194304] 0 2024-03-14T13:02:25.506 INFO:tasks.workunit.client.1.smithi137.stdout:8/847: dread - d1/dc/d71/db0/dca/f112 zero size 2024-03-14T13:02:25.507 INFO:tasks.workunit.client.1.smithi137.stdout:9/858: dread - d6/df/d19/dcf/fe9 zero size 2024-03-14T13:02:25.507 INFO:tasks.workunit.client.0.smithi111.stdout:2/654: creat dd/d11/d1a/fcd x:0 0 0 2024-03-14T13:02:25.509 INFO:tasks.workunit.client.1.smithi137.stdout:9/859: chown d6/df/d88/de5/df6/d40/f9e 1777 1 2024-03-14T13:02:25.510 INFO:tasks.workunit.client.1.smithi137.stdout:7/749: unlink d3/f1c 0 2024-03-14T13:02:25.510 INFO:tasks.workunit.client.0.smithi111.stdout:0/963: symlink d6/d9/de/d3a/l155 0 2024-03-14T13:02:25.510 INFO:tasks.workunit.client.0.smithi111.stdout:9/743: mknod d5/d42/d71/d96/dd3/c101 0 2024-03-14T13:02:25.510 INFO:tasks.workunit.client.0.smithi111.stdout:1/864: dwrite d20/d25/d31/f69 [0,4194304] 0 2024-03-14T13:02:25.510 INFO:tasks.workunit.client.0.smithi111.stdout:6/679: chown d0/d17/d21/d42/da5 344768 1 2024-03-14T13:02:25.511 INFO:tasks.workunit.client.0.smithi111.stdout:0/964: write d6/d45/d87/fb0 [746994,8826] 0 2024-03-14T13:02:25.511 INFO:tasks.workunit.client.1.smithi137.stdout:8/848: fdatasync d1/d5e/d11a/fb2 0 2024-03-14T13:02:25.513 INFO:tasks.workunit.client.0.smithi111.stdout:1/865: chown d20/dca/fe7 13 1 2024-03-14T13:02:25.514 INFO:tasks.workunit.client.1.smithi137.stdout:1/872: rename d7/d9/d19/d1f/d22/l75 to d7/d9/d19/d40/d50/l12b 0 2024-03-14T13:02:25.514 INFO:tasks.workunit.client.0.smithi111.stdout:9/744: chown d5/d6/d8/d5b/d84/caf 6001 1 2024-03-14T13:02:25.515 INFO:tasks.workunit.client.0.smithi111.stdout:6/680: symlink d0/d17/d20/d4a/d59/d80/lfc 0 2024-03-14T13:02:25.516 INFO:tasks.workunit.client.1.smithi137.stdout:1/873: stat d7/d9/d19/d1f/d2e/d63/fd3 0 2024-03-14T13:02:25.518 INFO:tasks.workunit.client.0.smithi111.stdout:1/866: fsync d20/d21/da7/fa9 0 2024-03-14T13:02:25.519 INFO:tasks.workunit.client.1.smithi137.stdout:1/874: write d7/d9/d19/d40/d42/f103 [78205,41506] 0 2024-03-14T13:02:25.521 INFO:tasks.workunit.client.0.smithi111.stdout:0/965: rename d6/d9/de/d3a/d46/dde to d6/d9/d15/da9/d156 0 2024-03-14T13:02:25.522 INFO:tasks.workunit.client.0.smithi111.stdout:0/966: chown d6/d9/de/d32/c44 5063 1 2024-03-14T13:02:25.522 INFO:tasks.workunit.client.0.smithi111.stdout:2/655: creat dd/d17/d34/d73/fce x:0 0 0 2024-03-14T13:02:25.523 INFO:tasks.workunit.client.0.smithi111.stdout:6/681: truncate d0/d17/d20/f48 207770 0 2024-03-14T13:02:25.524 INFO:tasks.workunit.client.1.smithi137.stdout:9/860: rename d6/df/d88/df4/dfb/f109 to d6/df/d88/df4/f11e 0 2024-03-14T13:02:25.525 INFO:tasks.workunit.client.0.smithi111.stdout:1/867: rmdir d20/d21/d28/d5c 39 2024-03-14T13:02:25.525 INFO:tasks.workunit.client.1.smithi137.stdout:5/698: write d7/d16/d18/d25/d28/d3c/f6f [748429,52940] 0 2024-03-14T13:02:25.527 INFO:tasks.workunit.client.0.smithi111.stdout:2/656: unlink dd/da9/caa 0 2024-03-14T13:02:25.528 INFO:tasks.workunit.client.1.smithi137.stdout:1/875: symlink d7/d31/d54/ded/dfb/d120/l12c 0 2024-03-14T13:02:25.529 INFO:tasks.workunit.client.0.smithi111.stdout:6/682: fdatasync d0/d17/d21/f5d 0 2024-03-14T13:02:25.529 INFO:tasks.workunit.client.0.smithi111.stdout:1/868: dread d20/d25/d87/f8f [0,4194304] 0 2024-03-14T13:02:25.531 INFO:tasks.workunit.client.0.smithi111.stdout:2/657: chown dd/c91 48731 1 2024-03-14T13:02:25.531 INFO:tasks.workunit.client.1.smithi137.stdout:9/861: unlink d6/d56/f89 0 2024-03-14T13:02:25.532 INFO:tasks.workunit.client.0.smithi111.stdout:9/745: getdents d5 0 2024-03-14T13:02:25.532 INFO:tasks.workunit.client.1.smithi137.stdout:5/699: mkdir d7/d16/d18/d25/d28/d9e/de5 0 2024-03-14T13:02:25.532 INFO:tasks.workunit.client.1.smithi137.stdout:9/862: stat d6/df/fd9 0 2024-03-14T13:02:25.533 INFO:tasks.workunit.client.0.smithi111.stdout:6/683: fsync d0/d17/d20/d4a/d59/f97 0 2024-03-14T13:02:25.535 INFO:tasks.workunit.client.0.smithi111.stdout:1/869: chown d20/d21/d3f/d59/f76 0 1 2024-03-14T13:02:25.536 INFO:tasks.workunit.client.0.smithi111.stdout:0/967: getdents d6/d9/d15/d49/d56/d143 0 2024-03-14T13:02:25.536 INFO:tasks.workunit.client.1.smithi137.stdout:1/876: creat d7/d9/d19/d1f/f12d x:0 0 0 2024-03-14T13:02:25.536 INFO:tasks.workunit.client.1.smithi137.stdout:8/849: dwrite d1/d5e/d11a/d82/fd5 [4194304,4194304] 0 2024-03-14T13:02:25.538 INFO:tasks.workunit.client.0.smithi111.stdout:1/870: write d20/d21/d3f/d59/fc8 [1648582,114656] 0 2024-03-14T13:02:25.539 INFO:tasks.workunit.client.1.smithi137.stdout:8/850: write d1/dc/da9/fbe [778750,35527] 0 2024-03-14T13:02:25.541 INFO:tasks.workunit.client.0.smithi111.stdout:2/658: symlink dd/d17/d34/lcf 0 2024-03-14T13:02:25.542 INFO:tasks.workunit.client.1.smithi137.stdout:9/863: mkdir d6/df/d88/de5/df6/d11f 0 2024-03-14T13:02:25.543 INFO:tasks.workunit.client.0.smithi111.stdout:2/659: stat dd/d17/d34/l79 0 2024-03-14T13:02:25.543 INFO:tasks.workunit.client.1.smithi137.stdout:1/877: creat d7/d4d/d6b/f12e x:0 0 0 2024-03-14T13:02:25.543 INFO:tasks.workunit.client.1.smithi137.stdout:9/864: stat d6/c9 0 2024-03-14T13:02:25.545 INFO:tasks.workunit.client.1.smithi137.stdout:4/747: sync 2024-03-14T13:02:25.546 INFO:tasks.workunit.client.1.smithi137.stdout:7/750: dread d3/d5/d23/f2b [0,4194304] 0 2024-03-14T13:02:25.548 INFO:tasks.workunit.client.1.smithi137.stdout:1/878: dread d7/d4d/f108 [0,4194304] 0 2024-03-14T13:02:25.551 INFO:tasks.workunit.client.1.smithi137.stdout:4/748: link d4/f2e d4/d9/d3a/d41/d38/ff5 0 2024-03-14T13:02:25.552 INFO:tasks.workunit.client.0.smithi111.stdout:1/871: write f9 [7267332,59527] 0 2024-03-14T13:02:25.552 INFO:tasks.workunit.client.0.smithi111.stdout:0/968: link d6/d9/de/d32/c11c d6/d9/de/d3a/dd1/d154/c157 0 2024-03-14T13:02:25.552 INFO:tasks.workunit.client.1.smithi137.stdout:4/749: fsync d4/f48 0 2024-03-14T13:02:25.553 INFO:tasks.workunit.client.1.smithi137.stdout:5/700: dread d7/d16/d18/d25/f46 [4194304,4194304] 0 2024-03-14T13:02:25.553 INFO:tasks.workunit.client.1.smithi137.stdout:1/879: chown d7/d9/d19/d1f/d89/d90/db1 49416 1 2024-03-14T13:02:25.556 INFO:tasks.workunit.client.1.smithi137.stdout:1/880: readlink d7/d9/d19/d1f/da0/l12a 0 2024-03-14T13:02:25.557 INFO:tasks.workunit.client.1.smithi137.stdout:4/750: symlink d4/d9/d3a/d41/dd7/lf6 0 2024-03-14T13:02:25.559 INFO:tasks.workunit.client.1.smithi137.stdout:4/751: chown d4/d9/d3a/d70/d80/d8d/dc0 729208570 1 2024-03-14T13:02:25.560 INFO:tasks.workunit.client.0.smithi111.stdout:1/872: creat d20/d25/d31/d80/f129 x:0 0 0 2024-03-14T13:02:25.562 INFO:tasks.workunit.client.1.smithi137.stdout:4/752: truncate d4/d9/d3a/d70/d80/db7/fde 510002 0 2024-03-14T13:02:25.563 INFO:tasks.workunit.client.1.smithi137.stdout:8/851: truncate d1/d5e/d11a/d82/fd5 7968039 0 2024-03-14T13:02:25.564 INFO:tasks.workunit.client.0.smithi111.stdout:6/684: dwrite d0/dfb/fee [0,4194304] 0 2024-03-14T13:02:25.564 INFO:tasks.workunit.client.1.smithi137.stdout:1/881: dread d7/d9/d19/fae [0,4194304] 0 2024-03-14T13:02:25.565 INFO:tasks.workunit.client.0.smithi111.stdout:0/969: getdents d6/d9/d15/da9/d156 0 2024-03-14T13:02:25.566 INFO:tasks.workunit.client.0.smithi111.stdout:9/746: dwrite d5/d6/d8/f30 [0,4194304] 0 2024-03-14T13:02:25.568 INFO:tasks.workunit.client.0.smithi111.stdout:2/660: dwrite dd/d11/d1a/d56/f85 [4194304,4194304] 0 2024-03-14T13:02:25.569 INFO:tasks.workunit.client.0.smithi111.stdout:6/685: chown d0/d17/d21/d69/d3d/d53/d81 6248549 1 2024-03-14T13:02:25.569 INFO:tasks.workunit.client.0.smithi111.stdout:9/747: fsync d5/d42/d71/dbd/dcd/f88 0 2024-03-14T13:02:25.572 INFO:tasks.workunit.client.1.smithi137.stdout:4/753: creat d4/d9/ff7 x:0 0 0 2024-03-14T13:02:25.572 INFO:tasks.workunit.client.1.smithi137.stdout:8/852: fdatasync d1/dc/d71/db0/dca/f104 0 2024-03-14T13:02:25.572 INFO:tasks.workunit.client.1.smithi137.stdout:1/882: symlink d7/d9/d19/d1f/d2e/da6/ddd/l12f 0 2024-03-14T13:02:25.573 INFO:tasks.workunit.client.0.smithi111.stdout:1/873: creat d20/d25/d31/f12a x:0 0 0 2024-03-14T13:02:25.574 INFO:tasks.workunit.client.1.smithi137.stdout:9/865: dwrite d6/df/d88/de5/df6/f43 [0,4194304] 0 2024-03-14T13:02:25.574 INFO:tasks.workunit.client.1.smithi137.stdout:7/751: write d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1/fea [581607,76186] 0 2024-03-14T13:02:25.575 INFO:tasks.workunit.client.0.smithi111.stdout:6/686: creat d0/d17/d21/d69/d3d/d53/ffd x:0 0 0 2024-03-14T13:02:25.577 INFO:tasks.workunit.client.1.smithi137.stdout:8/853: fsync d1/dc/d10/dcf/dd2/ff6 0 2024-03-14T13:02:25.579 INFO:tasks.workunit.client.0.smithi111.stdout:9/748: mkdir d5/d6/d14/d7a/d102 0 2024-03-14T13:02:25.580 INFO:tasks.workunit.client.0.smithi111.stdout:1/874: chown d20/d21/lba 3 1 2024-03-14T13:02:25.580 INFO:tasks.workunit.client.1.smithi137.stdout:4/754: creat d4/d9/d3a/d41/d6a/ff8 x:0 0 0 2024-03-14T13:02:25.581 INFO:tasks.workunit.client.1.smithi137.stdout:1/883: rmdir d7/d9/d19/d1f/d22 39 2024-03-14T13:02:25.582 INFO:tasks.workunit.client.1.smithi137.stdout:8/854: dread d1/dc/d71/db0/dca/ffe [0,4194304] 0 2024-03-14T13:02:25.583 INFO:tasks.workunit.client.0.smithi111.stdout:6/687: unlink d0/d17/d21/d69/ccd 0 2024-03-14T13:02:25.584 INFO:tasks.workunit.client.0.smithi111.stdout:9/749: creat d5/d6/d2a/d5a/f103 x:0 0 0 2024-03-14T13:02:25.587 INFO:tasks.workunit.client.0.smithi111.stdout:9/750: chown d5/d42/d71/d96/dd3/c101 171854111 1 2024-03-14T13:02:25.589 INFO:tasks.workunit.client.1.smithi137.stdout:7/752: rmdir d3/d52 39 2024-03-14T13:02:25.589 INFO:tasks.workunit.client.1.smithi137.stdout:9/866: creat d6/d91/dc8/f120 x:0 0 0 2024-03-14T13:02:25.590 INFO:tasks.workunit.client.1.smithi137.stdout:8/855: fsync d1/dc/d10/f11c 0 2024-03-14T13:02:25.592 INFO:tasks.workunit.client.1.smithi137.stdout:5/701: dwrite d7/d33/d65/dc2/fe4 [0,4194304] 0 2024-03-14T13:02:25.593 INFO:tasks.workunit.client.1.smithi137.stdout:8/856: chown d1/d5e/d11a/d97/fbb 197 1 2024-03-14T13:02:25.594 INFO:tasks.workunit.client.0.smithi111.stdout:1/875: creat d20/d40/db4/f12b x:0 0 0 2024-03-14T13:02:25.597 INFO:tasks.workunit.client.0.smithi111.stdout:6/688: creat d0/d17/d21/ffe x:0 0 0 2024-03-14T13:02:25.598 INFO:tasks.workunit.client.1.smithi137.stdout:4/755: dwrite d4/d9/d3a/d70/d80/d8d/fd4 [0,4194304] 0 2024-03-14T13:02:25.599 INFO:tasks.workunit.client.1.smithi137.stdout:7/753: creat d3/d6/d59/daa/dcf/d6a/d75/d78/ffa x:0 0 0 2024-03-14T13:02:25.599 INFO:tasks.workunit.client.1.smithi137.stdout:5/702: truncate d7/d49/d75/fb3 150038 0 2024-03-14T13:02:25.602 INFO:tasks.workunit.client.1.smithi137.stdout:1/884: dwrite d7/d9/d19/d1f/ff7 [0,4194304] 0 2024-03-14T13:02:25.604 INFO:tasks.workunit.client.0.smithi111.stdout:1/876: creat d20/d25/da8/ddd/f12c x:0 0 0 2024-03-14T13:02:25.607 INFO:tasks.workunit.client.1.smithi137.stdout:1/885: chown d7/d92/fe4 923112 1 2024-03-14T13:02:25.611 INFO:tasks.workunit.client.1.smithi137.stdout:7/754: chown d3/d5/d23/d2c/fd4 0 1 2024-03-14T13:02:25.611 INFO:tasks.workunit.client.1.smithi137.stdout:8/857: truncate d1/dc/fe0 2725995 0 2024-03-14T13:02:25.612 INFO:tasks.workunit.client.0.smithi111.stdout:6/689: chown d0/d17/d20/d65/cd4 14577232 1 2024-03-14T13:02:25.613 INFO:tasks.workunit.client.1.smithi137.stdout:1/886: creat d7/d31/d54/f130 x:0 0 0 2024-03-14T13:02:25.617 INFO:tasks.workunit.client.1.smithi137.stdout:1/887: chown d7/d9/d19/d1f/da0 3324669 1 2024-03-14T13:02:25.617 INFO:tasks.workunit.client.0.smithi111.stdout:9/751: dwrite d5/d6/fd5 [0,4194304] 0 2024-03-14T13:02:25.618 INFO:tasks.workunit.client.1.smithi137.stdout:7/755: symlink d3/d6/d59/daa/dcf/d6a/da1/lfb 0 2024-03-14T13:02:25.621 INFO:tasks.workunit.client.0.smithi111.stdout:6/690: rename d0/c4c to d0/d17/d20/d4a/d68/dc2/cff 0 2024-03-14T13:02:25.622 INFO:tasks.workunit.client.1.smithi137.stdout:5/703: dwrite d7/f10 [0,4194304] 0 2024-03-14T13:02:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.622+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0d74005860 con 0x7f0d88101610 2024-03-14T13:02:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.622+0000 7f0d6f7fe700 1 -- 172.21.15.111:0/2372642969 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f0d7c05a3e0 con 0x7f0d88101610 2024-03-14T13:02:25.623 INFO:tasks.workunit.client.0.smithi111.stdout:9/752: chown d5/d6/d2a/d2d/db8/cda 396810 1 2024-03-14T13:02:25.625 INFO:tasks.workunit.client.0.smithi111.stdout:6/691: creat d0/d50/f100 x:0 0 0 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:25.625 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:02:25.626 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:02:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 msgr2=0x7f0d7006ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 0x7f0d7006ea60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0d88100e60 tx=0x7f0d78004570 comp rx=0 tx=0).stop 2024-03-14T13:02:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 msgr2=0x7f0d881024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 0x7f0d881024b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c008000 tx=0x7f0d7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:25.627 INFO:tasks.workunit.client.1.smithi137.stdout:4/756: dwrite d4/d9/d3a/d41/d38/fe2 [0,4194304] 0 2024-03-14T13:02:25.628 INFO:tasks.workunit.client.1.smithi137.stdout:5/704: chown d7/d33/c56 466 1 2024-03-14T13:02:25.628 INFO:tasks.workunit.client.0.smithi111.stdout:6/692: write d0/d17/d21/d69/d3d/fe4 [3524819,10807] 0 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 shutdown_connections 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d88071ea0 0x7f0d881010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0d7006c5b0 0x7f0d7006ea60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 --2- 172.21.15.111:0/2372642969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d88101610 0x7f0d881024b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 >> 172.21.15.111:0/2372642969 conn(0x7f0d880f8540 msgr2=0x7f0d880fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:25.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 shutdown_connections 2024-03-14T13:02:25.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.626+0000 7f0d8d176700 1 -- 172.21.15.111:0/2372642969 wait complete. 2024-03-14T13:02:25.634 INFO:tasks.workunit.client.0.smithi111.stdout:6/693: rename d0/l15 to d0/dfb/db8/de9/l101 0 2024-03-14T13:02:25.635 INFO:tasks.workunit.client.1.smithi137.stdout:7/756: creat d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1/ffc x:0 0 0 2024-03-14T13:02:25.635 INFO:tasks.workunit.client.1.smithi137.stdout:5/705: mkdir d7/d16/d18/d25/d28/d9e/dd0/de6 0 2024-03-14T13:02:25.636 INFO:tasks.workunit.client.1.smithi137.stdout:4/757: fsync d4/d10/d26/f6f 0 2024-03-14T13:02:25.637 INFO:tasks.workunit.client.0.smithi111.stdout:2/661: dread dd/d17/d34/d73/d49/f51 [0,4194304] 0 2024-03-14T13:02:25.640 INFO:tasks.workunit.client.1.smithi137.stdout:8/858: dread d1/d5e/d11a/d82/d94/fdb [0,4194304] 0 2024-03-14T13:02:25.641 INFO:tasks.workunit.client.0.smithi111.stdout:9/753: dwrite d5/d6/ff5 [0,4194304] 0 2024-03-14T13:02:25.641 INFO:tasks.workunit.client.1.smithi137.stdout:7/757: write d3/d5/d5a/d86/fed [2717400,74287] 0 2024-03-14T13:02:25.643 INFO:tasks.workunit.client.0.smithi111.stdout:9/754: write d5/d6/d2a/d2d/db8/fc3 [523180,42489] 0 2024-03-14T13:02:25.643 INFO:tasks.workunit.client.1.smithi137.stdout:9/867: dread d6/df/d19/f62 [0,4194304] 0 2024-03-14T13:02:25.644 INFO:tasks.workunit.client.0.smithi111.stdout:6/694: fsync d0/d17/d21/d69/f76 0 2024-03-14T13:02:25.646 INFO:tasks.workunit.client.1.smithi137.stdout:4/758: creat d4/d9/d3a/d70/d80/ff9 x:0 0 0 2024-03-14T13:02:25.647 INFO:tasks.workunit.client.1.smithi137.stdout:8/859: mknod d1/d5e/daf/c11e 0 2024-03-14T13:02:25.650 INFO:tasks.workunit.client.1.smithi137.stdout:9/868: creat d6/df/d19/d33/dec/f121 x:0 0 0 2024-03-14T13:02:25.650 INFO:tasks.workunit.client.1.smithi137.stdout:7/758: mkdir d3/d6/d14/d22/df3/dfd 0 2024-03-14T13:02:25.651 INFO:tasks.workunit.client.1.smithi137.stdout:8/860: symlink d1/dc/d10/dcf/l11f 0 2024-03-14T13:02:25.652 INFO:tasks.workunit.client.1.smithi137.stdout:7/759: chown d3/d6/d59/daa/dcf/la2 7 1 2024-03-14T13:02:25.652 INFO:tasks.workunit.client.1.smithi137.stdout:1/888: dread d7/d9/f4b [0,4194304] 0 2024-03-14T13:02:25.652 INFO:tasks.workunit.client.1.smithi137.stdout:4/759: mkdir d4/d10/dfa 0 2024-03-14T13:02:25.652 INFO:tasks.workunit.client.0.smithi111.stdout:2/662: mkdir dd/d17/dd0 0 2024-03-14T13:02:25.653 INFO:tasks.workunit.client.0.smithi111.stdout:1/877: write d20/d21/faa [980939,90331] 0 2024-03-14T13:02:25.653 INFO:tasks.workunit.client.1.smithi137.stdout:7/760: readlink d3/d4a/l85 0 2024-03-14T13:02:25.657 INFO:tasks.workunit.client.0.smithi111.stdout:9/755: read d5/d6/fc [6435568,2235] 0 2024-03-14T13:02:25.657 INFO:tasks.workunit.client.1.smithi137.stdout:1/889: write d7/d9/d19/d40/d42/d9d/de5/ff1 [445916,58044] 0 2024-03-14T13:02:25.658 INFO:tasks.workunit.client.1.smithi137.stdout:7/761: mkdir d3/d6/d14/d27/dfe 0 2024-03-14T13:02:25.658 INFO:tasks.workunit.client.0.smithi111.stdout:1/878: dread d20/d25/da8/f10f [0,4194304] 0 2024-03-14T13:02:25.659 INFO:tasks.workunit.client.0.smithi111.stdout:2/663: mkdir dd/d17/d34/d74/dc1/dd1 0 2024-03-14T13:02:25.660 INFO:tasks.workunit.client.1.smithi137.stdout:8/861: rename d1/f49 to d1/dc/d71/db0/dca/d105/f120 0 2024-03-14T13:02:25.661 INFO:tasks.workunit.client.0.smithi111.stdout:1/879: write d20/d40/f54 [305152,32920] 0 2024-03-14T13:02:25.662 INFO:tasks.workunit.client.1.smithi137.stdout:7/762: mknod d3/d5/d5a/da8/cff 0 2024-03-14T13:02:25.663 INFO:tasks.workunit.client.0.smithi111.stdout:1/880: chown l5 1178559643 1 2024-03-14T13:02:25.665 INFO:tasks.workunit.client.0.smithi111.stdout:2/664: creat dd/d17/d34/fd2 x:0 0 0 2024-03-14T13:02:25.665 INFO:tasks.workunit.client.1.smithi137.stdout:4/760: dwrite d4/d27/d5c/d6d/d58/f76 [0,4194304] 0 2024-03-14T13:02:25.666 INFO:tasks.workunit.client.1.smithi137.stdout:8/862: readlink d1/d5e/d11a/d82/d94/l9b 0 2024-03-14T13:02:25.666 INFO:tasks.workunit.client.0.smithi111.stdout:1/881: mknod d20/d25/da8/c12d 0 2024-03-14T13:02:25.667 INFO:tasks.workunit.client.0.smithi111.stdout:9/756: link d5/l19 d5/d6/d8/d5b/l104 0 2024-03-14T13:02:25.668 INFO:tasks.workunit.client.0.smithi111.stdout:2/665: mknod dd/d17/d34/dbf/cd3 0 2024-03-14T13:02:25.670 INFO:tasks.workunit.client.0.smithi111.stdout:1/882: mkdir d20/d25/d31/d80/d12e 0 2024-03-14T13:02:25.670 INFO:tasks.workunit.client.0.smithi111.stdout:6/695: dwrite d0/d17/f1f [0,4194304] 0 2024-03-14T13:02:25.672 INFO:tasks.workunit.client.0.smithi111.stdout:9/757: mkdir d5/d42/d71/d96/d105 0 2024-03-14T13:02:25.673 INFO:tasks.workunit.client.0.smithi111.stdout:9/758: fdatasync d5/d6/d2a/f3e 0 2024-03-14T13:02:25.674 INFO:tasks.workunit.client.1.smithi137.stdout:1/890: dwrite d7/d31/d54/ded/dfb/f100 [0,4194304] 0 2024-03-14T13:02:25.674 INFO:tasks.workunit.client.0.smithi111.stdout:1/883: creat d20/d25/d75/df7/f12f x:0 0 0 2024-03-14T13:02:25.676 INFO:tasks.workunit.client.0.smithi111.stdout:6/696: read - d0/d17/d21/d69/d3d/d53/d81/fa7 zero size 2024-03-14T13:02:25.679 INFO:tasks.workunit.client.0.smithi111.stdout:9/759: creat d5/d6/d8/d5b/d84/f106 x:0 0 0 2024-03-14T13:02:25.679 INFO:tasks.workunit.client.1.smithi137.stdout:2/810: sync 2024-03-14T13:02:25.679 INFO:tasks.workunit.client.1.smithi137.stdout:6/735: sync 2024-03-14T13:02:25.680 INFO:tasks.workunit.client.0.smithi111.stdout:1/884: creat d20/d40/db4/ddf/f130 x:0 0 0 2024-03-14T13:02:25.680 INFO:tasks.workunit.client.1.smithi137.stdout:8/863: mknod d1/d4a/c121 0 2024-03-14T13:02:25.681 INFO:tasks.workunit.client.1.smithi137.stdout:9/869: write d6/df/fa0 [852289,95207] 0 2024-03-14T13:02:25.681 INFO:tasks.workunit.client.0.smithi111.stdout:6/697: creat d0/d17/d20/d4a/d59/d82/f102 x:0 0 0 2024-03-14T13:02:25.683 INFO:tasks.workunit.client.1.smithi137.stdout:4/761: getdents d4/d10/d5a 0 2024-03-14T13:02:25.685 INFO:tasks.workunit.client.1.smithi137.stdout:1/891: write d7/d31/d82/fff [209602,125377] 0 2024-03-14T13:02:25.685 INFO:tasks.workunit.client.1.smithi137.stdout:5/706: dwrite d7/d16/d18/d2d/f8e [0,4194304] 0 2024-03-14T13:02:25.686 INFO:tasks.workunit.client.0.smithi111.stdout:9/760: mkdir d5/d42/d71/dbd/d107 0 2024-03-14T13:02:25.687 INFO:tasks.workunit.client.1.smithi137.stdout:6/736: fsync d2/f5a 0 2024-03-14T13:02:25.689 INFO:tasks.workunit.client.1.smithi137.stdout:2/811: chown d2/d7/l15 719400 1 2024-03-14T13:02:25.690 INFO:tasks.workunit.client.0.smithi111.stdout:1/885: creat d20/d21/d3f/d59/f131 x:0 0 0 2024-03-14T13:02:25.691 INFO:tasks.workunit.client.1.smithi137.stdout:4/762: read d4/d9/d3a/d70/fdf [57563,63361] 0 2024-03-14T13:02:25.693 INFO:tasks.workunit.client.1.smithi137.stdout:5/707: creat d7/d16/d18/d25/d28/dc5/fe7 x:0 0 0 2024-03-14T13:02:25.694 INFO:tasks.workunit.client.1.smithi137.stdout:4/763: stat d4/d9/d3a/f56 0 2024-03-14T13:02:25.694 INFO:tasks.workunit.client.0.smithi111.stdout:1/886: unlink f11 0 2024-03-14T13:02:25.695 INFO:tasks.workunit.client.1.smithi137.stdout:2/812: chown d2/d7/d2e/d3e/d44/d6b/d1d/d41/l76 345744 1 2024-03-14T13:02:25.696 INFO:tasks.workunit.client.1.smithi137.stdout:4/764: write d4/f48 [5754845,107300] 0 2024-03-14T13:02:25.698 INFO:tasks.workunit.client.0.smithi111.stdout:1/887: fsync d20/d21/d3f/d59/fc8 0 2024-03-14T13:02:25.699 INFO:tasks.workunit.client.0.smithi111.stdout:9/761: dwrite d5/d42/dd4/fff [0,4194304] 0 2024-03-14T13:02:25.700 INFO:tasks.workunit.client.1.smithi137.stdout:4/765: read - d4/d10/d42/d73/fb8 zero size 2024-03-14T13:02:25.701 INFO:tasks.workunit.client.1.smithi137.stdout:7/763: dwrite d3/d6/d59/daa/dcf/d6a/d75/d78/fee [0,4194304] 0 2024-03-14T13:02:25.701 INFO:tasks.workunit.client.1.smithi137.stdout:6/737: rmdir d2/d6/dee/df5 0 2024-03-14T13:02:25.701 INFO:tasks.workunit.client.0.smithi111.stdout:2/666: dwrite dd/d17/d34/fa0 [0,4194304] 0 2024-03-14T13:02:25.703 INFO:tasks.workunit.client.1.smithi137.stdout:1/892: dread d7/d9/d19/d1f/f81 [0,4194304] 0 2024-03-14T13:02:25.704 INFO:tasks.workunit.client.1.smithi137.stdout:9/870: dread d6/df/d19/d1f/f24 [0,4194304] 0 2024-03-14T13:02:25.709 INFO:tasks.workunit.client.1.smithi137.stdout:7/764: truncate d3/d5/d68/f9c 206630 0 2024-03-14T13:02:25.713 INFO:tasks.workunit.client.1.smithi137.stdout:2/813: rename d2/d34 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b 0 2024-03-14T13:02:25.713 INFO:tasks.workunit.client.1.smithi137.stdout:5/708: link d7/d16/d18/l40 d7/d16/d18/d25/d28/d9e/le8 0 2024-03-14T13:02:25.715 INFO:tasks.workunit.client.1.smithi137.stdout:6/738: rmdir d2/d6/d86/de0 0 2024-03-14T13:02:25.719 INFO:tasks.workunit.client.1.smithi137.stdout:1/893: getdents d7/d29/d112 0 2024-03-14T13:02:25.719 INFO:tasks.workunit.client.1.smithi137.stdout:9/871: creat d6/df/d19/d33/ddf/d114/f122 x:0 0 0 2024-03-14T13:02:25.720 INFO:tasks.workunit.client.1.smithi137.stdout:7/765: fsync d3/d6/d64/d97/fe6 0 2024-03-14T13:02:25.722 INFO:tasks.workunit.client.1.smithi137.stdout:9/872: fdatasync d6/df/d19/f11c 0 2024-03-14T13:02:25.722 INFO:tasks.workunit.client.1.smithi137.stdout:6/739: mknod d2/d54/d55/d9c/dc0/cfb 0 2024-03-14T13:02:25.728 INFO:tasks.workunit.client.1.smithi137.stdout:7/766: creat d3/d6/d14/d27/dfe/f100 x:0 0 0 2024-03-14T13:02:25.729 INFO:tasks.workunit.client.1.smithi137.stdout:9/873: fdatasync d6/df/d88/de5/df6/d40/f41 0 2024-03-14T13:02:25.730 INFO:tasks.workunit.client.1.smithi137.stdout:1/894: dwrite d7/d9/d19/d40/fa8 [0,4194304] 0 2024-03-14T13:02:25.738 INFO:tasks.workunit.client.1.smithi137.stdout:4/766: write d4/d9/d3a/d41/d38/d79/f98 [864972,2379] 0 2024-03-14T13:02:25.739 INFO:tasks.workunit.client.1.smithi137.stdout:1/895: mknod d7/c131 0 2024-03-14T13:02:25.741 INFO:tasks.workunit.client.0.smithi111.stdout:1/888: rename d20/d21/lc5 to d20/d25/da8/ddd/l132 0 2024-03-14T13:02:25.742 INFO:tasks.workunit.client.1.smithi137.stdout:7/767: dwrite d3/d5/d5a/fa7 [0,4194304] 0 2024-03-14T13:02:25.742 INFO:tasks.workunit.client.1.smithi137.stdout:4/767: chown d4/d4d 19 1 2024-03-14T13:02:25.742 INFO:tasks.workunit.client.0.smithi111.stdout:9/762: getdents d5 0 2024-03-14T13:02:25.742 INFO:tasks.workunit.client.0.smithi111.stdout:1/889: read - d20/d40/db4/f12b zero size 2024-03-14T13:02:25.743 INFO:tasks.workunit.client.0.smithi111.stdout:1/890: chown d20/d25/f34 14 1 2024-03-14T13:02:25.744 INFO:tasks.workunit.client.1.smithi137.stdout:2/814: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/ff [0,4194304] 0 2024-03-14T13:02:25.746 INFO:tasks.workunit.client.1.smithi137.stdout:4/768: chown d4/d9/d3a/d41/d38/d55/f64 474 1 2024-03-14T13:02:25.746 INFO:tasks.workunit.client.0.smithi111.stdout:9/763: rename d5/d6/d2a/d2d/f66 to d5/d6/d14/d7a/d102/f108 0 2024-03-14T13:02:25.748 INFO:tasks.workunit.client.0.smithi111.stdout:9/764: readlink d5/d6/d2a/d2d/db8/lc9 0 2024-03-14T13:02:25.748 INFO:tasks.workunit.client.0.smithi111.stdout:9/765: readlink d5/d6/l63 0 2024-03-14T13:02:25.749 INFO:tasks.workunit.client.0.smithi111.stdout:1/891: dwrite d20/f9d [0,4194304] 0 2024-03-14T13:02:25.750 INFO:tasks.workunit.client.1.smithi137.stdout:9/874: creat d6/df/d19/d33/f123 x:0 0 0 2024-03-14T13:02:25.751 INFO:tasks.workunit.client.1.smithi137.stdout:4/769: unlink f3 0 2024-03-14T13:02:25.752 INFO:tasks.workunit.client.1.smithi137.stdout:2/815: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/la9 59145930 1 2024-03-14T13:02:25.754 INFO:tasks.workunit.client.0.smithi111.stdout:9/766: rmdir d5/d6/d8/d5b 39 2024-03-14T13:02:25.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:25 smithi137 ceph-mon[50336]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:25.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:25 smithi137 ceph-mon[50336]: pgmap v27: 65 pgs: 65 active+clean; 3.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 120 MiB/s rd, 385 MiB/s wr, 733 op/s 2024-03-14T13:02:25.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:25 smithi137 ceph-mon[50336]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:25.756 INFO:tasks.workunit.client.1.smithi137.stdout:2/816: chown d2/f14 30997 1 2024-03-14T13:02:25.756 INFO:tasks.workunit.client.1.smithi137.stdout:1/896: rename d7/d9/d19/d1f/da0/l12a to d7/d9/d19/d40/d42/l132 0 2024-03-14T13:02:25.757 INFO:tasks.workunit.client.1.smithi137.stdout:9/875: unlink d6/df/d71/c84 0 2024-03-14T13:02:25.759 INFO:tasks.workunit.client.0.smithi111.stdout:9/767: symlink d5/d6/d8/d1a/l109 0 2024-03-14T13:02:25.760 INFO:tasks.workunit.client.1.smithi137.stdout:4/770: unlink d4/d27/d5c/d77/ce4 0 2024-03-14T13:02:25.761 INFO:tasks.workunit.client.0.smithi111.stdout:1/892: dwrite d20/d25/d31/dff/f105 [0,4194304] 0 2024-03-14T13:02:25.761 INFO:tasks.workunit.client.1.smithi137.stdout:7/768: dwrite d3/f18 [4194304,4194304] 0 2024-03-14T13:02:25.761 INFO:tasks.workunit.client.1.smithi137.stdout:5/709: dwrite d7/d16/d18/d2d/f63 [0,4194304] 0 2024-03-14T13:02:25.765 INFO:tasks.workunit.client.0.smithi111.stdout:9/768: read f4 [4922661,55422] 0 2024-03-14T13:02:25.767 INFO:tasks.workunit.client.0.smithi111.stdout:1/893: dread d20/d25/da8/ff8 [0,4194304] 0 2024-03-14T13:02:25.772 INFO:tasks.workunit.client.0.smithi111.stdout:1/894: mkdir d20/d21/d28/d133 0 2024-03-14T13:02:25.772 INFO:tasks.workunit.client.0.smithi111.stdout:9/769: link d5/d6/l63 d5/d42/d71/dbd/dcd/l10a 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:1/897: write d7/d9/d19/d1f/d26/d7d/fc2 [1390499,9113] 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:9/876: truncate d6/df/d19/dcf/ff5 857449 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:6/740: dwrite d2/d54/d55/d6d/f90 [0,4194304] 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:5/710: fsync d7/d16/d18/d25/d28/d3c/f6f 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:2/817: readlink d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/l67 0 2024-03-14T13:02:25.773 INFO:tasks.workunit.client.1.smithi137.stdout:5/711: fdatasync d7/d16/d18/d25/d28/d3c/f6f 0 2024-03-14T13:02:25.776 INFO:tasks.workunit.client.1.smithi137.stdout:2/818: dread - d2/d7/d2e/d3e/d44/d6b/fa6 zero size 2024-03-14T13:02:25.777 INFO:tasks.workunit.client.1.smithi137.stdout:1/898: chown d7/f121 63061 1 2024-03-14T13:02:25.777 INFO:tasks.workunit.client.1.smithi137.stdout:7/769: mkdir d3/d6/d64/d97/dd2/d101 0 2024-03-14T13:02:25.777 INFO:tasks.workunit.client.1.smithi137.stdout:4/771: mkdir d4/d10/d42/daf/dfb 0 2024-03-14T13:02:25.780 INFO:tasks.workunit.client.1.smithi137.stdout:7/770: unlink d3/d5/d23/d2c/l34 0 2024-03-14T13:02:25.781 INFO:tasks.workunit.client.1.smithi137.stdout:4/772: stat d4/d9/d8e 0 2024-03-14T13:02:25.782 INFO:tasks.workunit.client.1.smithi137.stdout:1/899: mknod d7/d4d/d6b/c133 0 2024-03-14T13:02:25.784 INFO:tasks.workunit.client.1.smithi137.stdout:6/741: truncate d2/d6/de/d45/d59/d39/f7a 183637 0 2024-03-14T13:02:25.784 INFO:tasks.workunit.client.0.smithi111.stdout:1/895: dwrite d20/d25/d75/fe6 [0,4194304] 0 2024-03-14T13:02:25.785 INFO:tasks.workunit.client.1.smithi137.stdout:5/712: rename d7/d1d/d74/f78 to d7/d16/d18/d25/d28/d3c/dd8/fe9 0 2024-03-14T13:02:25.785 INFO:tasks.workunit.client.1.smithi137.stdout:7/771: chown d3/d6/d64/d97/dd2/d101 38 1 2024-03-14T13:02:25.786 INFO:tasks.workunit.client.0.smithi111.stdout:9/770: dwrite d5/d6/d8/f93 [0,4194304] 0 2024-03-14T13:02:25.787 INFO:tasks.workunit.client.0.smithi111.stdout:1/896: dread - d20/d40/db4/f108 zero size 2024-03-14T13:02:25.790 INFO:tasks.workunit.client.1.smithi137.stdout:9/877: dwrite d6/df/d88/df4/ffc [0,4194304] 0 2024-03-14T13:02:25.790 INFO:tasks.workunit.client.1.smithi137.stdout:5/713: chown d7/d16/d18/d25/d28/d3c/dd8/fe9 169540147 1 2024-03-14T13:02:25.792 INFO:tasks.workunit.client.1.smithi137.stdout:5/714: chown d7/d1d/c4c 13093 1 2024-03-14T13:02:25.795 INFO:tasks.workunit.client.0.smithi111.stdout:1/897: mknod d20/d25/d31/d80/d12e/c134 0 2024-03-14T13:02:25.797 INFO:tasks.workunit.client.0.smithi111.stdout:1/898: symlink d20/d25/d31/l135 0 2024-03-14T13:02:25.799 INFO:tasks.workunit.client.1.smithi137.stdout:9/878: dread d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:25.800 INFO:tasks.workunit.client.0.smithi111.stdout:1/899: mkdir d20/d40/db4/ddf/d136 0 2024-03-14T13:02:25.800 INFO:tasks.workunit.client.1.smithi137.stdout:6/742: rmdir d2/d6/de/df1 0 2024-03-14T13:02:25.800 INFO:tasks.workunit.client.1.smithi137.stdout:7/772: truncate d3/d6/d59/daa/fb3 2315368 0 2024-03-14T13:02:25.801 INFO:tasks.workunit.client.1.smithi137.stdout:5/715: chown d7/d16/d18/d25/d28/d3c/cb7 590748 1 2024-03-14T13:02:25.802 INFO:tasks.workunit.client.0.smithi111.stdout:1/900: truncate d20/d25/f9f 94722 0 2024-03-14T13:02:25.803 INFO:tasks.workunit.client.1.smithi137.stdout:7/773: chown d3/d5/f46 1284750 1 2024-03-14T13:02:25.804 INFO:tasks.workunit.client.0.smithi111.stdout:1/901: write d20/d40/f54 [4664831,113999] 0 2024-03-14T13:02:25.804 INFO:tasks.workunit.client.1.smithi137.stdout:5/716: mknod d7/d33/d65/cea 0 2024-03-14T13:02:25.804 INFO:tasks.workunit.client.1.smithi137.stdout:5/717: fdatasync d7/d36/f95 0 2024-03-14T13:02:25.806 INFO:tasks.workunit.client.1.smithi137.stdout:9/879: read d6/fa3 [991108,22764] 0 2024-03-14T13:02:25.806 INFO:tasks.workunit.client.1.smithi137.stdout:9/880: readlink d6/df/d88/l9f 0 2024-03-14T13:02:25.807 INFO:tasks.workunit.client.1.smithi137.stdout:7/774: symlink d3/d6/d59/daa/l102 0 2024-03-14T13:02:25.808 INFO:tasks.workunit.client.0.smithi111.stdout:1/902: creat d20/d25/d31/d80/df5/f137 x:0 0 0 2024-03-14T13:02:25.808 INFO:tasks.workunit.client.1.smithi137.stdout:6/743: creat d2/d54/d55/d6d/dc3/dec/ffc x:0 0 0 2024-03-14T13:02:25.809 INFO:tasks.workunit.client.0.smithi111.stdout:1/903: chown d20/d25/fd6 10166129 1 2024-03-14T13:02:25.810 INFO:tasks.workunit.client.0.smithi111.stdout:1/904: dread - d20/d40/da6/d102/f10e zero size 2024-03-14T13:02:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.809+0000 7f14bdb9e700 1 -- 172.21.15.111:0/2561206912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 msgr2=0x7f14b80fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.809+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/2561206912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b80fd290 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f14a8004940 tx=0x7f14a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.809+0000 7f14bdb9e700 1 -- 172.21.15.111:0/2561206912 shutdown_connections 2024-03-14T13:02:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/2561206912 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 0x7f14b80fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/2561206912 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b80fd290 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 -- 172.21.15.111:0/2561206912 >> 172.21.15.111:0/2561206912 conn(0x7f14b80f8580 msgr2=0x7f14b80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 -- 172.21.15.111:0/2561206912 shutdown_connections 2024-03-14T13:02:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 -- 172.21.15.111:0/2561206912 wait complete. 2024-03-14T13:02:25.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 Processor -- start 2024-03-14T13:02:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.810+0000 7f14bdb9e700 1 -- start start 2024-03-14T13:02:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.811+0000 7f14bdb9e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.811+0000 7f14bdb9e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 0x7f14b810bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.811+0000 7f14bdb9e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b810c220 con 0x7f14b80fce80 2024-03-14T13:02:25.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.811+0000 7f14bdb9e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b810ca70 con 0x7f14b80fdff0 2024-03-14T13:02:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43274/0 (socket says 172.21.15.111:43274) 2024-03-14T13:02:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 -- 172.21.15.111:0/942091352 learned_addr learned my addr 172.21.15.111:0/942091352 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:25.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b6ffd700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 0x7f14b810bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.814 INFO:tasks.workunit.client.1.smithi137.stdout:9/881: chown d6/d53/f99 395 1 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 -- 172.21.15.111:0/942091352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 msgr2=0x7f14b810bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 0x7f14b810bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 -- 172.21.15.111:0/942091352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a8005040 con 0x7f14b80fce80 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.812+0000 7f14b77fe700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f14a80137b0 tx=0x7f14a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.813+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a8021070 con 0x7f14b80fce80 2024-03-14T13:02:25.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.813+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14a801c400 con 0x7f14b80fce80 2024-03-14T13:02:25.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.813+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14a800e070 con 0x7f14b80fce80 2024-03-14T13:02:25.815 INFO:tasks.workunit.client.1.smithi137.stdout:6/744: fdatasync d2/d54/d55/d33/f64 0 2024-03-14T13:02:25.817 INFO:tasks.workunit.client.1.smithi137.stdout:7/775: rename d3/d6/d64/d9e/caf to d3/d5/d5a/d86/dc5/dc7/c103 0 2024-03-14T13:02:25.819 INFO:tasks.workunit.client.1.smithi137.stdout:9/882: readlink d6/df/d19/la6 0 2024-03-14T13:02:25.819 INFO:tasks.workunit.client.1.smithi137.stdout:5/718: dwrite d7/d16/d18/d2d/f53 [4194304,4194304] 0 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.814+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b810cc70 con 0x7f14b80fce80 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.814+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b810d200 con 0x7f14b80fce80 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.815+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b805f5f0 con 0x7f14b80fce80 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.815+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 31) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f14a801c970 con 0x7f14b80fce80 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.816+0000 7f149ffff700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 0x7f14a006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:25.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.816+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f14a808cfd0 con 0x7f14b80fce80 2024-03-14T13:02:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.820+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f14a80575f0 con 0x7f14b80fce80 2024-03-14T13:02:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.821+0000 7f14b6ffd700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 0x7f14a006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:25.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:25.821+0000 7f14b6ffd700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 0x7f14a006ea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009b80 tx=0x7f14ac00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:25.825 INFO:tasks.workunit.client.0.smithi111.stdout:1/905: write d20/d25/d75/f77 [937058,34475] 0 2024-03-14T13:02:25.826 INFO:tasks.workunit.client.1.smithi137.stdout:4/773: write d4/d9/d3a/d41/d38/f4c [5024634,35303] 0 2024-03-14T13:02:25.826 INFO:tasks.workunit.client.1.smithi137.stdout:5/719: write d7/d16/f1a [3437123,102887] 0 2024-03-14T13:02:25.828 INFO:tasks.workunit.client.1.smithi137.stdout:6/745: mknod d2/cfd 0 2024-03-14T13:02:25.831 INFO:tasks.workunit.client.1.smithi137.stdout:2/819: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/f63 634513 0 2024-03-14T13:02:25.832 INFO:tasks.workunit.client.0.smithi111.stdout:1/906: getdents d20/d21/d122 0 2024-03-14T13:02:25.832 INFO:tasks.workunit.client.1.smithi137.stdout:9/883: dread d6/df/d88/de5/df6/d40/f5c [0,4194304] 0 2024-03-14T13:02:25.833 INFO:tasks.workunit.client.0.smithi111.stdout:9/771: dwrite d5/d6/d2a/d5a/f9f [4194304,4194304] 0 2024-03-14T13:02:25.833 INFO:tasks.workunit.client.0.smithi111.stdout:1/907: dread - d20/d21/d3f/d59/f131 zero size 2024-03-14T13:02:25.835 INFO:tasks.workunit.client.1.smithi137.stdout:9/884: readlink d6/df/d88/dd5/lee 0 2024-03-14T13:02:25.835 INFO:tasks.workunit.client.1.smithi137.stdout:6/746: mkdir d2/d6/de/d45/d59/d39/d3c/dfe 0 2024-03-14T13:02:25.837 INFO:tasks.workunit.client.1.smithi137.stdout:7/776: link d3/f32 d3/d6/d59/daa/dcf/d6a/d75/d78/da4/f104 0 2024-03-14T13:02:25.837 INFO:tasks.workunit.client.1.smithi137.stdout:1/900: dwrite d7/d9/d19/d1f/d2e/f79 [0,4194304] 0 2024-03-14T13:02:25.845 INFO:tasks.workunit.client.1.smithi137.stdout:9/885: mkdir d6/df/d19/d33/d8f/d124 0 2024-03-14T13:02:25.845 INFO:tasks.workunit.client.1.smithi137.stdout:5/720: dwrite d7/f15 [0,4194304] 0 2024-03-14T13:02:25.846 INFO:tasks.workunit.client.0.smithi111.stdout:9/772: rename d5/d6/d8/d5b/d84/db6/de7/ffd to d5/d6/d8/f10b 0 2024-03-14T13:02:25.847 INFO:tasks.workunit.client.1.smithi137.stdout:2/820: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/d10c 0 2024-03-14T13:02:25.847 INFO:tasks.workunit.client.0.smithi111.stdout:1/908: dwrite d20/d21/d28/fe1 [0,4194304] 0 2024-03-14T13:02:25.848 INFO:tasks.workunit.client.1.smithi137.stdout:2/821: readlink d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/l58 0 2024-03-14T13:02:25.848 INFO:tasks.workunit.client.1.smithi137.stdout:4/774: rename d4/d10/l23 to d4/d9/d8e/dae/lfc 0 2024-03-14T13:02:25.851 INFO:tasks.workunit.client.0.smithi111.stdout:1/909: write d20/d21/d3f/d59/f74 [3710271,59286] 0 2024-03-14T13:02:25.851 INFO:tasks.workunit.client.1.smithi137.stdout:4/775: write d4/d9/d3a/d70/d80/d8d/fd4 [4296989,79692] 0 2024-03-14T13:02:25.851 INFO:tasks.workunit.client.1.smithi137.stdout:9/886: chown d6/df/d88/cbe 1 1 2024-03-14T13:02:25.852 INFO:tasks.workunit.client.0.smithi111.stdout:9/773: rename d5/d42/d71/fca to d5/d42/d71/dbd/dcd/f10c 0 2024-03-14T13:02:25.856 INFO:tasks.workunit.client.1.smithi137.stdout:5/721: creat d7/d33/d65/dc2/feb x:0 0 0 2024-03-14T13:02:25.856 INFO:tasks.workunit.client.0.smithi111.stdout:1/910: symlink d20/d40/da6/d102/l138 0 2024-03-14T13:02:25.857 INFO:tasks.workunit.client.1.smithi137.stdout:4/776: write d4/d27/d5c/d6d/d58/f76 [4891059,63803] 0 2024-03-14T13:02:25.858 INFO:tasks.workunit.client.0.smithi111.stdout:1/911: write d20/d25/d75/f77 [1767433,91956] 0 2024-03-14T13:02:25.860 INFO:tasks.workunit.client.1.smithi137.stdout:7/777: dwrite d3/d6/f45 [0,4194304] 0 2024-03-14T13:02:25.864 INFO:tasks.workunit.client.1.smithi137.stdout:4/777: write d4/d9/d3a/d41/d6a/dac/fb0 [2092444,124285] 0 2024-03-14T13:02:25.865 INFO:tasks.workunit.client.1.smithi137.stdout:7/778: stat d3/d6/d59/daa/dcf/d6a/d75/d78/da4 0 2024-03-14T13:02:25.865 INFO:tasks.workunit.client.0.smithi111.stdout:9/774: dwrite d5/d6/d8/f3a [0,4194304] 0 2024-03-14T13:02:25.866 INFO:tasks.workunit.client.1.smithi137.stdout:6/747: write d2/d54/d55/d33/f64 [928210,122042] 0 2024-03-14T13:02:25.867 INFO:tasks.workunit.client.1.smithi137.stdout:7/779: fsync d3/d5/d23/d3f/fe2 0 2024-03-14T13:02:25.868 INFO:tasks.workunit.client.1.smithi137.stdout:9/887: mkdir d6/df/d19/d1f/de8/d125 0 2024-03-14T13:02:25.868 INFO:tasks.workunit.client.1.smithi137.stdout:7/780: dread - d3/d5/d68/ff1 zero size 2024-03-14T13:02:25.870 INFO:tasks.workunit.client.0.smithi111.stdout:9/775: creat d5/d6/d2a/d5a/dee/f10d x:0 0 0 2024-03-14T13:02:25.873 INFO:tasks.workunit.client.1.smithi137.stdout:4/778: creat d4/d9/d3a/d41/d38/d55/db6/ffd x:0 0 0 2024-03-14T13:02:25.876 INFO:tasks.workunit.client.1.smithi137.stdout:5/722: mkdir d7/d1d/d74/dec 0 2024-03-14T13:02:25.876 INFO:tasks.workunit.client.0.smithi111.stdout:9/776: symlink d5/d42/d71/dbd/de0/l10e 0 2024-03-14T13:02:25.877 INFO:tasks.workunit.client.0.smithi111.stdout:9/777: chown d5/d6/d8/d74/f77 76932 1 2024-03-14T13:02:25.878 INFO:tasks.workunit.client.0.smithi111.stdout:9/778: fdatasync d5/d6/fc 0 2024-03-14T13:02:25.880 INFO:tasks.workunit.client.1.smithi137.stdout:2/822: link d2/l3 d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/l10d 0 2024-03-14T13:02:25.880 INFO:tasks.workunit.client.1.smithi137.stdout:9/888: creat d6/df/d88/de5/df6/da9/f126 x:0 0 0 2024-03-14T13:02:25.882 INFO:tasks.workunit.client.1.smithi137.stdout:7/781: creat d3/d5/d23/d3f/f105 x:0 0 0 2024-03-14T13:02:25.883 INFO:tasks.workunit.client.0.smithi111.stdout:1/912: write d20/d25/f46 [555671,54184] 0 2024-03-14T13:02:25.884 INFO:tasks.workunit.client.1.smithi137.stdout:7/782: fdatasync d3/d6/d2e/f8b 0 2024-03-14T13:02:25.884 INFO:tasks.workunit.client.1.smithi137.stdout:0/763: sync 2024-03-14T13:02:25.884 INFO:tasks.workunit.client.1.smithi137.stdout:3/640: sync 2024-03-14T13:02:25.885 INFO:tasks.workunit.client.1.smithi137.stdout:7/783: readlink d3/d5/d5a/da8/lcc 0 2024-03-14T13:02:25.887 INFO:tasks.workunit.client.1.smithi137.stdout:7/784: write d3/d5/d5a/d86/fed [160380,633] 0 2024-03-14T13:02:25.888 INFO:tasks.workunit.client.0.smithi111.stdout:1/913: link d20/d25/d31/d80/l91 d20/d21/d3f/d59/l139 0 2024-03-14T13:02:25.889 INFO:tasks.workunit.client.1.smithi137.stdout:6/748: getdents d2/d6/da7 0 2024-03-14T13:02:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:25 smithi111 ceph-mon[39688]: from='client.24459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:25 smithi111 ceph-mon[39688]: pgmap v27: 65 pgs: 65 active+clean; 3.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 120 MiB/s rd, 385 MiB/s wr, 733 op/s 2024-03-14T13:02:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:25 smithi111 ceph-mon[39688]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:25.892 INFO:tasks.workunit.client.1.smithi137.stdout:3/641: symlink d4/de/d16/d1c/ld5 0 2024-03-14T13:02:25.893 INFO:tasks.workunit.client.0.smithi111.stdout:1/914: getdents d20/d25/d31 0 2024-03-14T13:02:25.894 INFO:tasks.workunit.client.1.smithi137.stdout:5/723: dread d7/d16/d18/d25/d28/f60 [0,4194304] 0 2024-03-14T13:02:25.896 INFO:tasks.workunit.client.0.smithi111.stdout:1/915: dread d20/d25/d31/d80/f92 [0,4194304] 0 2024-03-14T13:02:25.897 INFO:tasks.workunit.client.1.smithi137.stdout:6/749: readlink d2/d6/de/d45/d59/d39/lc9 0 2024-03-14T13:02:25.899 INFO:tasks.workunit.client.1.smithi137.stdout:5/724: chown d7/d16/d41 3806954 1 2024-03-14T13:02:25.899 INFO:tasks.workunit.client.1.smithi137.stdout:9/889: dwrite d6/df/d88/dd5/f115 [0,4194304] 0 2024-03-14T13:02:25.901 INFO:tasks.workunit.client.1.smithi137.stdout:3/642: symlink d4/de/d16/d1c/d5d/ld6 0 2024-03-14T13:02:25.902 INFO:tasks.workunit.client.1.smithi137.stdout:1/901: dread d7/d9/d19/d1f/d2e/f10b [0,4194304] 0 2024-03-14T13:02:25.903 INFO:tasks.workunit.client.0.smithi111.stdout:7/680: sync 2024-03-14T13:02:25.905 INFO:tasks.workunit.client.1.smithi137.stdout:4/779: write d4/f89 [262781,80299] 0 2024-03-14T13:02:25.905 INFO:tasks.workunit.client.1.smithi137.stdout:2/823: truncate d2/f14 298934 0 2024-03-14T13:02:25.910 INFO:tasks.workunit.client.0.smithi111.stdout:1/916: dread d20/d25/f7b [0,4194304] 0 2024-03-14T13:02:25.912 INFO:tasks.workunit.client.1.smithi137.stdout:8/864: sync 2024-03-14T13:02:25.914 INFO:tasks.workunit.client.0.smithi111.stdout:1/917: write d20/d21/da7/fbe [1473981,65450] 0 2024-03-14T13:02:25.914 INFO:tasks.workunit.client.0.smithi111.stdout:7/681: dwrite d0/d1/d4a/fa7 [0,4194304] 0 2024-03-14T13:02:25.917 INFO:tasks.workunit.client.1.smithi137.stdout:0/764: dwrite d1/d1f/fc4 [0,4194304] 0 2024-03-14T13:02:25.917 INFO:tasks.workunit.client.1.smithi137.stdout:6/750: rmdir d2/d54 39 2024-03-14T13:02:25.917 INFO:tasks.workunit.client.1.smithi137.stdout:3/643: rmdir d4/de/d13 39 2024-03-14T13:02:25.918 INFO:tasks.workunit.client.0.smithi111.stdout:1/918: mkdir d20/d13a 0 2024-03-14T13:02:25.918 INFO:tasks.workunit.client.1.smithi137.stdout:7/785: write d3/d4a/f5e [879194,49175] 0 2024-03-14T13:02:25.918 INFO:tasks.workunit.client.1.smithi137.stdout:0/765: stat d1/d1f/d80 0 2024-03-14T13:02:25.919 INFO:tasks.workunit.client.1.smithi137.stdout:1/902: chown d7/d31/d54/ded/dfb 28 1 2024-03-14T13:02:25.920 INFO:tasks.workunit.client.1.smithi137.stdout:5/725: creat d7/d36/fed x:0 0 0 2024-03-14T13:02:25.923 INFO:tasks.workunit.client.1.smithi137.stdout:7/786: write d3/d6/d14/d27/dfe/f100 [842356,57424] 0 2024-03-14T13:02:25.928 INFO:tasks.workunit.client.1.smithi137.stdout:3/644: write d4/de/d16/d1c/d5d/d85/d6b/f6d [4620093,97500] 0 2024-03-14T13:02:25.929 INFO:tasks.workunit.client.0.smithi111.stdout:1/919: dwrite d20/d25/d31/dff/f105 [0,4194304] 0 2024-03-14T13:02:25.934 INFO:tasks.workunit.client.1.smithi137.stdout:2/824: dwrite d2/d7/d2e/d3e/d44/d6b/d22/dbb/f10a [0,4194304] 0 2024-03-14T13:02:25.937 INFO:tasks.workunit.client.0.smithi111.stdout:1/920: dwrite d20/dca/fe7 [0,4194304] 0 2024-03-14T13:02:25.938 INFO:tasks.workunit.client.1.smithi137.stdout:9/890: mknod d6/df/d88/de5/df6/da9/dde/c127 0 2024-03-14T13:02:25.947 INFO:tasks.workunit.client.1.smithi137.stdout:5/726: unlink d7/d16/d18/d48/l7e 0 2024-03-14T13:02:25.948 INFO:tasks.workunit.client.1.smithi137.stdout:1/903: chown d7/d9/d19/d1f/da0/cbc 3470150 1 2024-03-14T13:02:25.950 INFO:tasks.workunit.client.1.smithi137.stdout:1/904: write d7/d9/d19/d1f/d26/d7d/fd4 [1175853,2714] 0 2024-03-14T13:02:25.951 INFO:tasks.workunit.client.1.smithi137.stdout:3/645: dread - d4/fa8 zero size 2024-03-14T13:02:25.952 INFO:tasks.workunit.client.0.smithi111.stdout:7/682: dwrite d0/d1/d5/d38/f68 [0,4194304] 0 2024-03-14T13:02:25.952 INFO:tasks.workunit.client.1.smithi137.stdout:2/825: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d9d/f10e x:0 0 0 2024-03-14T13:02:25.952 INFO:tasks.workunit.client.1.smithi137.stdout:1/905: stat d7/d9/d19/d1f/d2e/c3c 0 2024-03-14T13:02:25.953 INFO:tasks.workunit.client.0.smithi111.stdout:7/683: write d0/d1/d21/d2a/f8a [1879650,13298] 0 2024-03-14T13:02:25.955 INFO:tasks.workunit.client.1.smithi137.stdout:2/826: stat d2/d7/d2e/d3e/d44/d6b/d1d/c94 0 2024-03-14T13:02:25.956 INFO:tasks.workunit.client.0.smithi111.stdout:7/684: fdatasync d0/d1/d4a/d52/f6c 0 2024-03-14T13:02:25.958 INFO:tasks.workunit.client.1.smithi137.stdout:5/727: creat d7/d33/fee x:0 0 0 2024-03-14T13:02:25.962 INFO:tasks.workunit.client.1.smithi137.stdout:1/906: write d7/d92/fe4 [1032772,124308] 0 2024-03-14T13:02:25.962 INFO:tasks.workunit.client.0.smithi111.stdout:7/685: dwrite d0/d1/d22/f46 [0,4194304] 0 2024-03-14T13:02:25.964 INFO:tasks.workunit.client.1.smithi137.stdout:3/646: unlink d4/de/d93/cc9 0 2024-03-14T13:02:25.966 INFO:tasks.workunit.client.1.smithi137.stdout:9/891: dwrite d6/df/d88/de5/df6/da9/fdb [0,4194304] 0 2024-03-14T13:02:25.971 INFO:tasks.workunit.client.1.smithi137.stdout:6/751: rename d2/f2d to d2/d54/fff 0 2024-03-14T13:02:25.973 INFO:tasks.workunit.client.1.smithi137.stdout:2/827: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/d10f 0 2024-03-14T13:02:25.974 INFO:tasks.workunit.client.1.smithi137.stdout:2/828: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/f69 0 2024-03-14T13:02:25.980 INFO:tasks.workunit.client.0.smithi111.stdout:1/921: write d20/d25/d31/f3a [2878767,11364] 0 2024-03-14T13:02:25.985 INFO:tasks.workunit.client.1.smithi137.stdout:7/787: getdents d3/d6/d59/daa/dcf/d6a/d75/d78 0 2024-03-14T13:02:25.986 INFO:tasks.workunit.client.1.smithi137.stdout:1/907: creat d7/d9/d19/d1f/d2e/da6/d129/f134 x:0 0 0 2024-03-14T13:02:25.987 INFO:tasks.workunit.client.1.smithi137.stdout:0/766: dread d1/d5/d1b/d42/f4b [0,4194304] 0 2024-03-14T13:02:25.988 INFO:tasks.workunit.client.1.smithi137.stdout:2/829: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d110 0 2024-03-14T13:02:25.989 INFO:tasks.workunit.client.1.smithi137.stdout:0/767: readlink d1/d5/d1b/d42/d3a/db0/d84/l97 0 2024-03-14T13:02:25.989 INFO:tasks.workunit.client.1.smithi137.stdout:3/647: mknod d4/de/d16/d27/dd3/cd7 0 2024-03-14T13:02:25.991 INFO:tasks.workunit.client.1.smithi137.stdout:4/780: write d4/d9/d3a/d41/d38/ff5 [1875434,125495] 0 2024-03-14T13:02:25.992 INFO:tasks.workunit.client.1.smithi137.stdout:6/752: symlink d2/d54/d55/d6d/dc3/deb/l100 0 2024-03-14T13:02:25.992 INFO:tasks.workunit.client.1.smithi137.stdout:8/865: write d1/d5e/d11a/fb2 [916387,5048] 0 2024-03-14T13:02:25.993 INFO:tasks.workunit.client.0.smithi111.stdout:1/922: write d20/d21/da7/fa9 [554700,75216] 0 2024-03-14T13:02:25.996 INFO:tasks.workunit.client.0.smithi111.stdout:1/923: symlink d20/d25/da8/ddd/l13b 0 2024-03-14T13:02:25.999 INFO:tasks.workunit.client.1.smithi137.stdout:8/866: chown d1/dc/fcd 7 1 2024-03-14T13:02:26.002 INFO:tasks.workunit.client.0.smithi111.stdout:1/924: creat d20/d25/f13c x:0 0 0 2024-03-14T13:02:26.003 INFO:tasks.workunit.client.1.smithi137.stdout:1/908: mkdir d7/d9/d135 0 2024-03-14T13:02:26.003 INFO:tasks.workunit.client.0.smithi111.stdout:7/686: dwrite d0/d1/d5/d38/d1b/d9c/fbb [0,4194304] 0 2024-03-14T13:02:26.004 INFO:tasks.workunit.client.1.smithi137.stdout:3/648: mknod d4/df/d82/cd8 0 2024-03-14T13:02:26.006 INFO:tasks.workunit.client.0.smithi111.stdout:1/925: mkdir d20/d25/d87/d13d 0 2024-03-14T13:02:26.007 INFO:tasks.workunit.client.1.smithi137.stdout:7/788: dwrite d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fb7 [0,4194304] 0 2024-03-14T13:02:26.007 INFO:tasks.workunit.client.0.smithi111.stdout:7/687: readlink d0/d1/d21/d2a/l3c 0 2024-03-14T13:02:26.008 INFO:tasks.workunit.client.0.smithi111.stdout:7/688: chown d0/d11/l7a 0 1 2024-03-14T13:02:26.009 INFO:tasks.workunit.client.1.smithi137.stdout:0/768: dwrite d1/d1f/f24 [4194304,4194304] 0 2024-03-14T13:02:26.009 INFO:tasks.workunit.client.1.smithi137.stdout:1/909: creat d7/d4d/d6b/f136 x:0 0 0 2024-03-14T13:02:26.011 INFO:tasks.workunit.client.0.smithi111.stdout:1/926: mknod d20/d21/dc1/c13e 0 2024-03-14T13:02:26.011 INFO:tasks.workunit.client.0.smithi111.stdout:7/689: dread - d0/d1/d5/d38/d1b/fa1 zero size 2024-03-14T13:02:26.013 INFO:tasks.workunit.client.0.smithi111.stdout:7/690: mkdir d0/d1/d4a/dc7 0 2024-03-14T13:02:26.014 INFO:tasks.workunit.client.1.smithi137.stdout:9/892: write d6/df/d88/de5/df6/d40/f60 [469732,10577] 0 2024-03-14T13:02:26.017 INFO:tasks.workunit.client.1.smithi137.stdout:8/867: write d1/dc/d10/f32 [604103,12411] 0 2024-03-14T13:02:26.019 INFO:tasks.workunit.client.0.smithi111.stdout:1/927: dwrite d20/d25/d31/f110 [0,4194304] 0 2024-03-14T13:02:26.021 INFO:tasks.workunit.client.0.smithi111.stdout:7/691: mknod d0/d1/d21/d2a/cc8 0 2024-03-14T13:02:26.022 INFO:tasks.workunit.client.1.smithi137.stdout:6/753: dwrite d2/d6/de/d45/d97/d9f/fb8 [0,4194304] 0 2024-03-14T13:02:26.022 INFO:tasks.workunit.client.0.smithi111.stdout:1/928: chown d20/d25/d31/d80/l126 30 1 2024-03-14T13:02:26.023 INFO:tasks.workunit.client.1.smithi137.stdout:8/868: stat d1/dc/d77/d9f/de4/f3e 0 2024-03-14T13:02:26.024 INFO:tasks.workunit.client.1.smithi137.stdout:8/869: stat d1 0 2024-03-14T13:02:26.025 INFO:tasks.workunit.client.1.smithi137.stdout:3/649: dread d4/d19/f75 [0,4194304] 0 2024-03-14T13:02:26.025 INFO:tasks.workunit.client.1.smithi137.stdout:0/769: mknod d1/d44/d72/df8/c105 0 2024-03-14T13:02:26.025 INFO:tasks.workunit.client.0.smithi111.stdout:1/929: chown d20/d25/da8/ddd 771092984 1 2024-03-14T13:02:26.025 INFO:tasks.workunit.client.0.smithi111.stdout:7/692: mkdir d0/d1/d5/d38/d5c/d6d/dc9 0 2024-03-14T13:02:26.026 INFO:tasks.workunit.client.0.smithi111.stdout:1/930: write d20/d25/d31/dff/fcf [2963618,48429] 0 2024-03-14T13:02:26.027 INFO:tasks.workunit.client.0.smithi111.stdout:1/931: write d20/d21/faa [549720,13703] 0 2024-03-14T13:02:26.033 INFO:tasks.workunit.client.0.smithi111.stdout:1/932: mknod d20/d21/d3f/c13f 0 2024-03-14T13:02:26.037 INFO:tasks.workunit.client.1.smithi137.stdout:1/910: write d7/d9/d19/d1f/d2e/f10b [3875436,62142] 0 2024-03-14T13:02:26.037 INFO:tasks.workunit.client.1.smithi137.stdout:9/893: link d6/df/d88/de5/df6/da9/fe4 d6/df/d71/f128 0 2024-03-14T13:02:26.037 INFO:tasks.workunit.client.1.smithi137.stdout:2/830: getdents d2/d7/d2e/d3e/d44/d6b/d22 0 2024-03-14T13:02:26.038 INFO:tasks.workunit.client.0.smithi111.stdout:1/933: link d20/d40/db4/ddf/f130 d20/d40/da6/f140 0 2024-03-14T13:02:26.039 INFO:tasks.workunit.client.1.smithi137.stdout:9/894: mknod d6/d26/c129 0 2024-03-14T13:02:26.042 INFO:tasks.workunit.client.1.smithi137.stdout:1/911: symlink d7/d29/l137 0 2024-03-14T13:02:26.043 INFO:tasks.workunit.client.1.smithi137.stdout:9/895: unlink d6/f6d 0 2024-03-14T13:02:26.045 INFO:tasks.workunit.client.0.smithi111.stdout:1/934: dwrite d20/d25/d75/f65 [0,4194304] 0 2024-03-14T13:02:26.047 INFO:tasks.workunit.client.1.smithi137.stdout:4/781: dwrite d4/d9/d3a/d41/d38/f3b [0,4194304] 0 2024-03-14T13:02:26.049 INFO:tasks.workunit.client.0.smithi111.stdout:1/935: creat d20/d40/f141 x:0 0 0 2024-03-14T13:02:26.049 INFO:tasks.workunit.client.1.smithi137.stdout:6/754: dwrite d2/d6/de/d45/d59/fe1 [0,4194304] 0 2024-03-14T13:02:26.052 INFO:tasks.workunit.client.1.smithi137.stdout:8/870: dwrite d1/dc/d77/d9f/de4/f9e [4194304,4194304] 0 2024-03-14T13:02:26.054 INFO:tasks.workunit.client.1.smithi137.stdout:4/782: readlink d4/d10/l93 0 2024-03-14T13:02:26.054 INFO:tasks.workunit.client.0.smithi111.stdout:1/936: unlink d20/d40/f141 0 2024-03-14T13:02:26.054 INFO:tasks.workunit.client.0.smithi111.stdout:1/937: write fe [5039952,85419] 0 2024-03-14T13:02:26.055 INFO:tasks.workunit.client.0.smithi111.stdout:1/938: chown d20/d25/d31/d80/l9c 6633 1 2024-03-14T13:02:26.056 INFO:tasks.workunit.client.1.smithi137.stdout:3/650: dread d4/de/d16/d1c/d5d/f7c [0,4194304] 0 2024-03-14T13:02:26.056 INFO:tasks.workunit.client.0.smithi111.stdout:7/693: dread d0/d7e/fb [0,4194304] 0 2024-03-14T13:02:26.056 INFO:tasks.workunit.client.0.smithi111.stdout:1/939: read d20/d25/d75/f65 [747575,61246] 0 2024-03-14T13:02:26.057 INFO:tasks.workunit.client.1.smithi137.stdout:1/912: chown d7/d9/d19/d40/d50/l12b 2405241 1 2024-03-14T13:02:26.060 INFO:tasks.workunit.client.0.smithi111.stdout:7/694: mknod d0/d1/d5/d56/d70/cca 0 2024-03-14T13:02:26.063 INFO:tasks.workunit.client.1.smithi137.stdout:6/755: read d2/d54/d55/f8d [7599,28001] 0 2024-03-14T13:02:26.068 INFO:tasks.workunit.client.1.smithi137.stdout:1/913: dread d7/d4d/f64 [0,4194304] 0 2024-03-14T13:02:26.068 INFO:tasks.workunit.client.1.smithi137.stdout:2/831: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/l111 0 2024-03-14T13:02:26.068 INFO:tasks.workunit.client.1.smithi137.stdout:7/789: dread d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:26.069 INFO:tasks.workunit.client.1.smithi137.stdout:6/756: rename d2/d54/d55/d6d/lae to d2/d6/de/d45/l101 0 2024-03-14T13:02:26.070 INFO:tasks.workunit.client.1.smithi137.stdout:9/896: link d6/df/d88/de5/df6/d40/f5c d6/d56/f12a 0 2024-03-14T13:02:26.073 INFO:tasks.workunit.client.1.smithi137.stdout:2/832: dread - d2/d7/d2e/d3e/d44/d6b/d22/d26/f100 zero size 2024-03-14T13:02:26.073 INFO:tasks.workunit.client.1.smithi137.stdout:7/790: stat d3/d4a/l85 0 2024-03-14T13:02:26.073 INFO:tasks.workunit.client.1.smithi137.stdout:4/783: mknod d4/d9/d3a/d41/cfe 0 2024-03-14T13:02:26.074 INFO:tasks.workunit.client.1.smithi137.stdout:3/651: creat d4/df/fd9 x:0 0 0 2024-03-14T13:02:26.075 INFO:tasks.workunit.client.1.smithi137.stdout:8/871: dwrite d1/dc/d77/d9f/fcc [0,4194304] 0 2024-03-14T13:02:26.075 INFO:tasks.workunit.client.1.smithi137.stdout:1/914: chown d7/d9/d19/d40/d42/l132 1587176664 1 2024-03-14T13:02:26.077 INFO:tasks.workunit.client.1.smithi137.stdout:9/897: rmdir d6/d91/dc8 39 2024-03-14T13:02:26.079 INFO:tasks.workunit.client.1.smithi137.stdout:2/833: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/cb1 694 1 2024-03-14T13:02:26.079 INFO:tasks.workunit.client.1.smithi137.stdout:6/757: chown d2/d54/d55/ca 5 1 2024-03-14T13:02:26.081 INFO:tasks.workunit.client.1.smithi137.stdout:3/652: chown d4/de/d16/d1c/d5d/d85/f5a 1889325976 1 2024-03-14T13:02:26.081 INFO:tasks.workunit.client.1.smithi137.stdout:9/898: dread - d6/df/d19/d33/dec/f121 zero size 2024-03-14T13:02:26.083 INFO:tasks.workunit.client.1.smithi137.stdout:8/872: creat d1/d5e/d11a/d97/d11d/f122 x:0 0 0 2024-03-14T13:02:26.085 INFO:tasks.workunit.client.1.smithi137.stdout:8/873: readlink d1/d4a/l99 0 2024-03-14T13:02:26.085 INFO:tasks.workunit.client.1.smithi137.stdout:2/834: chown d2/l3 215339788 1 2024-03-14T13:02:26.085 INFO:tasks.workunit.client.1.smithi137.stdout:1/915: creat d7/d9/d19/d1f/d2e/da6/d129/f138 x:0 0 0 2024-03-14T13:02:26.086 INFO:tasks.workunit.client.1.smithi137.stdout:6/758: readlink d2/d6/de/led 0 2024-03-14T13:02:26.088 INFO:tasks.workunit.client.1.smithi137.stdout:4/784: rename d4/d9/d3a/d41/d38/d79/da1 to d4/d9/d3a/d41/d6a/dff 0 2024-03-14T13:02:26.088 INFO:tasks.workunit.client.1.smithi137.stdout:7/791: dwrite d3/d6/d59/daa/dcf/f6b [0,4194304] 0 2024-03-14T13:02:26.090 INFO:tasks.workunit.client.1.smithi137.stdout:3/653: mknod d4/de/d16/d74/cda 0 2024-03-14T13:02:26.091 INFO:tasks.workunit.client.1.smithi137.stdout:1/916: mknod d7/d9/d19/d40/d42/d9d/c139 0 2024-03-14T13:02:26.092 INFO:tasks.workunit.client.1.smithi137.stdout:3/654: readlink d4/de/d16/d1c/db8/dd1/ld2 0 2024-03-14T13:02:26.092 INFO:tasks.workunit.client.1.smithi137.stdout:6/759: mkdir d2/d6/de/d45/d59/d102 0 2024-03-14T13:02:26.093 INFO:tasks.workunit.client.1.smithi137.stdout:3/655: dread - d4/df/fbe zero size 2024-03-14T13:02:26.093 INFO:tasks.workunit.client.1.smithi137.stdout:6/760: chown d2/d54/d55/f8d 2938491 1 2024-03-14T13:02:26.095 INFO:tasks.workunit.client.1.smithi137.stdout:7/792: unlink d3/d5/d5a/d80/ce0 0 2024-03-14T13:02:26.096 INFO:tasks.workunit.client.1.smithi137.stdout:6/761: mknod d2/d6/d1d/c103 0 2024-03-14T13:02:26.097 INFO:tasks.workunit.client.1.smithi137.stdout:3/656: creat d4/de/d16/d1c/db8/fdb x:0 0 0 2024-03-14T13:02:26.098 INFO:tasks.workunit.client.1.smithi137.stdout:7/793: mkdir d3/d5/d106 0 2024-03-14T13:02:26.100 INFO:tasks.workunit.client.1.smithi137.stdout:6/762: symlink d2/d54/d55/d33/l104 0 2024-03-14T13:02:26.104 INFO:tasks.workunit.client.1.smithi137.stdout:7/794: symlink d3/d6/d59/l107 0 2024-03-14T13:02:26.105 INFO:tasks.workunit.client.1.smithi137.stdout:7/795: fsync d3/d5/d23/d3f/ff6 0 2024-03-14T13:02:26.105 INFO:tasks.workunit.client.1.smithi137.stdout:6/763: symlink d2/d54/d55/d38/l105 0 2024-03-14T13:02:26.109 INFO:tasks.workunit.client.1.smithi137.stdout:6/764: mknod d2/d6/de/d45/d97/d9f/c106 0 2024-03-14T13:02:26.113 INFO:tasks.workunit.client.1.smithi137.stdout:6/765: unlink d2/d6/de/c16 0 2024-03-14T13:02:26.114 INFO:tasks.workunit.client.1.smithi137.stdout:6/766: write d2/d6/f5c [2412506,79138] 0 2024-03-14T13:02:26.117 INFO:tasks.workunit.client.1.smithi137.stdout:1/917: dread d7/d31/f76 [0,4194304] 0 2024-03-14T13:02:26.117 INFO:tasks.workunit.client.1.smithi137.stdout:2/835: dread d2/d7/d2e/d3e/d44/d6b/d22/f91 [0,4194304] 0 2024-03-14T13:02:26.120 INFO:tasks.workunit.client.1.smithi137.stdout:1/918: mkdir d7/d9/d19/d1f/d2e/da6/ddd/d13a 0 2024-03-14T13:02:26.123 INFO:tasks.workunit.client.1.smithi137.stdout:3/657: dread d4/de/d16/d1c/d5d/d85/d6b/f6d [0,4194304] 0 2024-03-14T13:02:26.125 INFO:tasks.workunit.client.1.smithi137.stdout:3/658: write d4/de/d16/d1c/d5d/d4b/f5e [4658622,129322] 0 2024-03-14T13:02:26.125 INFO:tasks.workunit.client.0.smithi111.stdout:5/930: sync 2024-03-14T13:02:26.125 INFO:tasks.workunit.client.0.smithi111.stdout:3/783: sync 2024-03-14T13:02:26.125 INFO:tasks.workunit.client.0.smithi111.stdout:0/970: sync 2024-03-14T13:02:26.126 INFO:tasks.workunit.client.1.smithi137.stdout:1/919: symlink d7/d9/d19/d1f/l13b 0 2024-03-14T13:02:26.129 INFO:tasks.workunit.client.1.smithi137.stdout:1/920: creat d7/d4d/f13c x:0 0 0 2024-03-14T13:02:26.131 INFO:tasks.workunit.client.1.smithi137.stdout:6/767: write d2/d54/d55/f8d [732435,113243] 0 2024-03-14T13:02:26.131 INFO:tasks.workunit.client.0.smithi111.stdout:3/784: rename d0/d2d/d35/d4c/db0/lcf to d0/d2d/d35/d50/d6c/d98/l11f 0 2024-03-14T13:02:26.134 INFO:tasks.workunit.client.1.smithi137.stdout:1/921: chown d7/d9/d19/d1f/d22/f25 18870 1 2024-03-14T13:02:26.135 INFO:tasks.workunit.client.1.smithi137.stdout:7/796: dwrite d3/d6/d59/daa/fb3 [0,4194304] 0 2024-03-14T13:02:26.137 INFO:tasks.workunit.client.0.smithi111.stdout:0/971: dwrite d6/d9/d15/d49/d56/d143/f99 [0,4194304] 0 2024-03-14T13:02:26.137 INFO:tasks.workunit.client.1.smithi137.stdout:6/768: creat d2/d54/d55/f107 x:0 0 0 2024-03-14T13:02:26.138 INFO:tasks.workunit.client.0.smithi111.stdout:5/931: dwrite d2/d16/d49/d44/d4a/d9e/db7/dbb/ff9 [0,4194304] 0 2024-03-14T13:02:26.139 INFO:tasks.workunit.client.1.smithi137.stdout:7/797: rmdir d3/d6/d64 39 2024-03-14T13:02:26.141 INFO:tasks.workunit.client.0.smithi111.stdout:3/785: creat d0/d2d/d35/d50/da2/db3/d71/f120 x:0 0 0 2024-03-14T13:02:26.141 INFO:tasks.workunit.client.1.smithi137.stdout:1/922: write d7/d9/d19/d1f/d2e/fe7 [478786,91621] 0 2024-03-14T13:02:26.142 INFO:tasks.workunit.client.1.smithi137.stdout:6/769: creat d2/d6/de/d45/f108 x:0 0 0 2024-03-14T13:02:26.142 INFO:tasks.workunit.client.1.smithi137.stdout:2/836: dwrite d2/f14 [0,4194304] 0 2024-03-14T13:02:26.146 INFO:tasks.workunit.client.1.smithi137.stdout:7/798: symlink d3/d6/d14/d27/dfe/l108 0 2024-03-14T13:02:26.147 INFO:tasks.workunit.client.1.smithi137.stdout:3/659: dread d4/f80 [0,4194304] 0 2024-03-14T13:02:26.147 INFO:tasks.workunit.client.1.smithi137.stdout:7/799: chown d3/d6/d14/d22 2265236 1 2024-03-14T13:02:26.149 INFO:tasks.workunit.client.0.smithi111.stdout:3/786: symlink d0/d2d/d35/d4c/l121 0 2024-03-14T13:02:26.151 INFO:tasks.workunit.client.0.smithi111.stdout:5/932: rename d2/lf6 to d2/d45/d76/l129 0 2024-03-14T13:02:26.151 INFO:tasks.workunit.client.1.smithi137.stdout:2/837: chown d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/le8 5688128 1 2024-03-14T13:02:26.156 INFO:tasks.workunit.client.1.smithi137.stdout:7/800: symlink d3/d5/d68/l109 0 2024-03-14T13:02:26.156 INFO:tasks.workunit.client.1.smithi137.stdout:1/923: creat d7/d9/d19/d1f/da0/f13d x:0 0 0 2024-03-14T13:02:26.159 INFO:tasks.workunit.client.0.smithi111.stdout:3/787: dwrite d0/d2d/d35/d50/da2/db3/d6f/fdb [0,4194304] 0 2024-03-14T13:02:26.160 INFO:tasks.workunit.client.1.smithi137.stdout:2/838: write d2/d7/d2e/d3e/d44/d6b/d22/dbb/f10a [2750061,83974] 0 2024-03-14T13:02:26.161 INFO:tasks.workunit.client.0.smithi111.stdout:5/933: creat d2/d45/d76/f12a x:0 0 0 2024-03-14T13:02:26.162 INFO:tasks.workunit.client.1.smithi137.stdout:7/801: chown d3/d6 12705316 1 2024-03-14T13:02:26.165 INFO:tasks.workunit.client.0.smithi111.stdout:3/788: symlink d0/d2d/d35/d50/da2/l122 0 2024-03-14T13:02:26.165 INFO:tasks.workunit.client.1.smithi137.stdout:6/770: dwrite d2/d54/d55/d6d/d76/d57/f9a [0,4194304] 0 2024-03-14T13:02:26.166 INFO:tasks.workunit.client.0.smithi111.stdout:0/972: dwrite d6/d9/d15/d55/da0/fd7 [0,4194304] 0 2024-03-14T13:02:26.167 INFO:tasks.workunit.client.1.smithi137.stdout:6/771: readlink d2/d54/d55/d6d/d76/d57/lde 0 2024-03-14T13:02:26.167 INFO:tasks.workunit.client.0.smithi111.stdout:5/934: mkdir d2/d16/d49/d4c/d12b 0 2024-03-14T13:02:26.169 INFO:tasks.workunit.client.1.smithi137.stdout:7/802: mknod d3/d6/d59/daa/dcf/d6a/d75/c10a 0 2024-03-14T13:02:26.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.169+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f14b810d470 con 0x7f14a006c560 2024-03-14T13:02:26.172 INFO:tasks.workunit.client.1.smithi137.stdout:6/772: rmdir d2/d54/d55/d6d/dc3/dce 39 2024-03-14T13:02:26.172 INFO:tasks.workunit.client.1.smithi137.stdout:7/803: write d3/d5/d23/d3f/ff6 [601901,100770] 0 2024-03-14T13:02:26.174 INFO:tasks.workunit.client.1.smithi137.stdout:2/839: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f112 x:0 0 0 2024-03-14T13:02:26.175 INFO:tasks.workunit.client.0.smithi111.stdout:5/935: rename d2/d45/d79/l110 to d2/d16/d49/d7c/d95/l12c 0 2024-03-14T13:02:26.175 INFO:tasks.workunit.client.1.smithi137.stdout:6/773: creat d2/d6/d86/f109 x:0 0 0 2024-03-14T13:02:26.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.176+0000 7f149ffff700 1 -- 172.21.15.111:0/942091352 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+329 (secure 0 0 0) 0x7f14b810d470 con 0x7f14a006c560 2024-03-14T13:02:26.176 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:02:26.176 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:02:26.176 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "", 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:02:26.177 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:02:26.178 INFO:tasks.workunit.client.1.smithi137.stdout:7/804: write d3/d5/d5a/d86/fed [3987286,96768] 0 2024-03-14T13:02:26.178 INFO:tasks.workunit.client.1.smithi137.stdout:2/840: chown d2/d7/d2e/d3e/d44/d6b/d1d/d2d 0 1 2024-03-14T13:02:26.178 INFO:tasks.workunit.client.1.smithi137.stdout:6/774: chown d2/d6/de/d45/d59/d39/f73 280373 1 2024-03-14T13:02:26.179 INFO:tasks.workunit.client.0.smithi111.stdout:5/936: rename d2/d16/d49/d33/db4 to d2/d16/d49/d7c/d12d 0 2024-03-14T13:02:26.180 INFO:tasks.workunit.client.1.smithi137.stdout:7/805: symlink d3/d6/d59/daa/dcf/d6a/l10b 0 2024-03-14T13:02:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 msgr2=0x7f14a006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 0x7f14a006ea10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009b80 tx=0x7f14ac00b040 comp rx=0 tx=0).stop 2024-03-14T13:02:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 msgr2=0x7f14b810b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:26.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f14a80137b0 tx=0x7f14a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 shutdown_connections 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f14a006c560 0x7f14a006ea10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14b80fce80 0x7f14b810b690 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 --2- 172.21.15.111:0/942091352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14b80fdff0 0x7f14b810bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.179+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 >> 172.21.15.111:0/942091352 conn(0x7f14b80f8580 msgr2=0x7f14b80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.180+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 shutdown_connections 2024-03-14T13:02:26.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:26.180+0000 7f14bdb9e700 1 -- 172.21.15.111:0/942091352 wait complete. 2024-03-14T13:02:26.183 INFO:tasks.workunit.client.1.smithi137.stdout:2/841: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f113 x:0 0 0 2024-03-14T13:02:26.185 INFO:tasks.workunit.client.1.smithi137.stdout:6/775: rename d2/d6/de/led to d2/d54/d55/l10a 0 2024-03-14T13:02:26.188 INFO:tasks.workunit.client.1.smithi137.stdout:6/776: chown d2/d54/d55/d33/l3a 600 1 2024-03-14T13:02:26.189 INFO:tasks.workunit.client.1.smithi137.stdout:6/777: unlink d2/d54/d55/d6d/d76/d57/f70 0 2024-03-14T13:02:26.189 INFO:tasks.workunit.client.1.smithi137.stdout:7/806: getdents d3/d5/d5a/d86/dc5/dc7 0 2024-03-14T13:02:26.191 INFO:tasks.workunit.client.1.smithi137.stdout:7/807: write d3/fd0 [216984,82949] 0 2024-03-14T13:02:26.191 INFO:tasks.workunit.client.1.smithi137.stdout:6/778: dread - d2/d54/d55/d6d/d76/d57/fd3 zero size 2024-03-14T13:02:26.193 INFO:tasks.workunit.client.1.smithi137.stdout:7/808: unlink d3/d6/l10 0 2024-03-14T13:02:26.194 INFO:tasks.workunit.client.1.smithi137.stdout:6/779: getdents d2/d6/de/d45/d59/daa 0 2024-03-14T13:02:26.195 INFO:tasks.workunit.client.1.smithi137.stdout:7/809: symlink d3/d6/d14/d27/l10c 0 2024-03-14T13:02:26.198 INFO:tasks.workunit.client.1.smithi137.stdout:6/780: rename d2/d6/de/d45/d59/f37 to d2/d6/de/d45/d97/d9f/f10b 0 2024-03-14T13:02:26.199 INFO:tasks.workunit.client.1.smithi137.stdout:7/810: mkdir d3/d6/d64/d9e/d10d 0 2024-03-14T13:02:26.200 INFO:tasks.workunit.client.1.smithi137.stdout:6/781: chown d2/d6/d86/d8a/la1 37944272 1 2024-03-14T13:02:26.204 INFO:tasks.workunit.client.1.smithi137.stdout:3/660: write d4/fa8 [680730,99017] 0 2024-03-14T13:02:26.205 INFO:tasks.workunit.client.1.smithi137.stdout:7/811: getdents d3 0 2024-03-14T13:02:26.206 INFO:tasks.workunit.client.0.smithi111.stdout:0/973: truncate d6/d9/d5b/d9e/fa8 2006109 0 2024-03-14T13:02:26.208 INFO:tasks.workunit.client.1.smithi137.stdout:6/782: rename d2/d54/d55/d6d/fb0 to d2/d54/d55/d9c/f10c 0 2024-03-14T13:02:26.210 INFO:tasks.workunit.client.0.smithi111.stdout:0/974: creat d6/d9/de/d3a/d46/d4a/f158 x:0 0 0 2024-03-14T13:02:26.210 INFO:tasks.workunit.client.0.smithi111.stdout:3/789: dwrite d0/d2d/d35/d50/d5e/f5f [0,4194304] 0 2024-03-14T13:02:26.212 INFO:tasks.workunit.client.1.smithi137.stdout:1/924: truncate d7/d9/d19/d1f/d2e/f9b 2400564 0 2024-03-14T13:02:26.215 INFO:tasks.workunit.client.1.smithi137.stdout:1/925: write d7/d4d/d6b/fe2 [1420476,724] 0 2024-03-14T13:02:26.216 INFO:tasks.workunit.client.0.smithi111.stdout:3/790: mknod d0/d2d/d35/d50/da2/db3/d71/dbc/c123 0 2024-03-14T13:02:26.218 INFO:tasks.workunit.client.0.smithi111.stdout:3/791: dread - d0/d2d/d35/f113 zero size 2024-03-14T13:02:26.219 INFO:tasks.workunit.client.0.smithi111.stdout:3/792: write d0/d2d/d35/d50/d5e/f12 [462924,65322] 0 2024-03-14T13:02:26.221 INFO:tasks.workunit.client.0.smithi111.stdout:3/793: truncate d0/d2d/d35/d50/da2/db3/d6f/fdb 4590510 0 2024-03-14T13:02:26.222 INFO:tasks.workunit.client.1.smithi137.stdout:7/812: dwrite d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:26.222 INFO:tasks.workunit.client.1.smithi137.stdout:6/783: dwrite d2/d6/de/f13 [4194304,4194304] 0 2024-03-14T13:02:26.223 INFO:tasks.workunit.client.0.smithi111.stdout:3/794: write d0/d2d/d35/d50/d6c/da3/daa/fd1 [4180758,120703] 0 2024-03-14T13:02:26.223 INFO:tasks.workunit.client.1.smithi137.stdout:1/926: write d7/d9/d19/d1f/d26/fd0 [567746,90862] 0 2024-03-14T13:02:26.227 INFO:tasks.workunit.client.1.smithi137.stdout:7/813: chown d3/d6/d59/daa/dcf 441914277 1 2024-03-14T13:02:26.228 INFO:tasks.workunit.client.0.smithi111.stdout:0/975: rename d6/d9/de/d3a to d6/d9/de/d77/d159 0 2024-03-14T13:02:26.229 INFO:tasks.workunit.client.1.smithi137.stdout:2/842: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f96 [0,4194304] 0 2024-03-14T13:02:26.229 INFO:tasks.workunit.client.0.smithi111.stdout:0/976: chown d6/d9/d15/da9/d156/ff3 278203855 1 2024-03-14T13:02:26.230 INFO:tasks.workunit.client.1.smithi137.stdout:1/927: mkdir d7/d9/d19/d40/d50/d13e 0 2024-03-14T13:02:26.230 INFO:tasks.workunit.client.0.smithi111.stdout:3/795: getdents d0/d2d/d35/d50/d5e/d19 0 2024-03-14T13:02:26.233 INFO:tasks.workunit.client.1.smithi137.stdout:2/843: readlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/l88 0 2024-03-14T13:02:26.237 INFO:tasks.workunit.client.1.smithi137.stdout:1/928: link d7/d4d/f108 d7/d9/d19/d40/d42/f13f 0 2024-03-14T13:02:26.238 INFO:tasks.workunit.client.0.smithi111.stdout:0/977: rename d6/d9/d15/d55/d108 to d6/d9/d15/d55/d89/d135/d15a 0 2024-03-14T13:02:26.238 INFO:tasks.workunit.client.1.smithi137.stdout:2/844: readlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/l90 0 2024-03-14T13:02:26.242 INFO:tasks.workunit.client.1.smithi137.stdout:2/845: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f79 [1677675,16668] 0 2024-03-14T13:02:26.242 INFO:tasks.workunit.client.0.smithi111.stdout:3/796: dwrite d0/d2d/d35/d4c/db0/fed [0,4194304] 0 2024-03-14T13:02:26.243 INFO:tasks.workunit.client.0.smithi111.stdout:3/797: stat d0/d2d/d35/d50/d5e/d19/fb9 0 2024-03-14T13:02:26.247 INFO:tasks.workunit.client.0.smithi111.stdout:0/978: link d6/d9/de/d77/d159/d46/d4a/f91 d6/d9/de/d77/f15b 0 2024-03-14T13:02:26.251 INFO:tasks.workunit.client.1.smithi137.stdout:2/846: dread d2/d7/f87 [0,4194304] 0 2024-03-14T13:02:26.252 INFO:tasks.workunit.client.0.smithi111.stdout:3/798: dwrite d0/d2d/d35/d69/fc4 [0,4194304] 0 2024-03-14T13:02:26.255 INFO:tasks.workunit.client.1.smithi137.stdout:6/784: dread d2/d6/d86/fa9 [0,4194304] 0 2024-03-14T13:02:26.256 INFO:tasks.workunit.client.0.smithi111.stdout:3/799: symlink d0/d2d/d35/d69/l124 0 2024-03-14T13:02:26.258 INFO:tasks.workunit.client.1.smithi137.stdout:6/785: dread d2/d6/de/d45/d59/d39/f7a [0,4194304] 0 2024-03-14T13:02:26.258 INFO:tasks.workunit.client.0.smithi111.stdout:3/800: getdents d0/d2d/d35/d50/d6c/da3 0 2024-03-14T13:02:26.259 INFO:tasks.workunit.client.0.smithi111.stdout:0/979: dread d6/d9/de/d77/d159/dd1/fce [0,4194304] 0 2024-03-14T13:02:26.262 INFO:tasks.workunit.client.0.smithi111.stdout:0/980: rmdir d6/d9/de/d77/d159/dd1 39 2024-03-14T13:02:26.262 INFO:tasks.workunit.client.1.smithi137.stdout:6/786: link d2/d4c/c94 d2/d6/de/d45/d97/c10d 0 2024-03-14T13:02:26.263 INFO:tasks.workunit.client.0.smithi111.stdout:3/801: symlink d0/d2d/d35/d69/de6/df4/l125 0 2024-03-14T13:02:26.264 INFO:tasks.workunit.client.1.smithi137.stdout:6/787: chown d2/d54/d55/d6d/dc3/deb/ffa 285689843 1 2024-03-14T13:02:26.265 INFO:tasks.workunit.client.0.smithi111.stdout:0/981: mkdir d6/d9/d15/d15c 0 2024-03-14T13:02:26.266 INFO:tasks.workunit.client.1.smithi137.stdout:6/788: truncate d2/d54/d55/f1f 2524855 0 2024-03-14T13:02:26.267 INFO:tasks.workunit.client.0.smithi111.stdout:0/982: write d6/d9/de/d77/d159/f9b [1051243,112773] 0 2024-03-14T13:02:26.269 INFO:tasks.workunit.client.1.smithi137.stdout:6/789: read d2/d6/de/d45/d59/d39/f73 [2268428,87990] 0 2024-03-14T13:02:26.270 INFO:tasks.workunit.client.1.smithi137.stdout:6/790: stat d2/d6/d1d/c96 0 2024-03-14T13:02:26.270 INFO:tasks.workunit.client.1.smithi137.stdout:1/929: dread d7/d9/d19/d40/d42/d9d/de5/ff8 [0,4194304] 0 2024-03-14T13:02:26.271 INFO:tasks.workunit.client.0.smithi111.stdout:0/983: dread - d6/d3d/d97/fb7 zero size 2024-03-14T13:02:26.271 INFO:tasks.workunit.client.0.smithi111.stdout:3/802: dwrite d0/d2d/d35/d50/d5e/d19/f1a [0,4194304] 0 2024-03-14T13:02:26.272 INFO:tasks.workunit.client.1.smithi137.stdout:1/930: readlink d7/d4d/l73 0 2024-03-14T13:02:26.272 INFO:tasks.workunit.client.1.smithi137.stdout:1/931: readlink d7/d29/l44 0 2024-03-14T13:02:26.273 INFO:tasks.workunit.client.0.smithi111.stdout:0/984: readlink d6/d9/de/d32/d64/l10f 0 2024-03-14T13:02:26.274 INFO:tasks.workunit.client.1.smithi137.stdout:1/932: mknod d7/d9/d19/d1f/d26/d6c/c140 0 2024-03-14T13:02:26.275 INFO:tasks.workunit.client.0.smithi111.stdout:0/985: fdatasync f5 0 2024-03-14T13:02:26.277 INFO:tasks.workunit.client.1.smithi137.stdout:1/933: rename d7/d9/d19/d40/d42/d9d/de5/ff4 to d7/d9/d19/d1f/d2e/da6/ddd/f141 0 2024-03-14T13:02:26.278 INFO:tasks.workunit.client.0.smithi111.stdout:0/986: fdatasync d6/d9/de/d77/d159/d46/d4a/f91 0 2024-03-14T13:02:26.278 INFO:tasks.workunit.client.1.smithi137.stdout:1/934: fsync d7/d29/fc9 0 2024-03-14T13:02:26.279 INFO:tasks.workunit.client.0.smithi111.stdout:3/803: dread d0/d2d/d35/d50/d5e/f27 [0,4194304] 0 2024-03-14T13:02:26.281 INFO:tasks.workunit.client.0.smithi111.stdout:3/804: symlink d0/d2d/d35/d4c/db0/l126 0 2024-03-14T13:02:26.282 INFO:tasks.workunit.client.1.smithi137.stdout:1/935: dread d7/d9/d19/d1f/d89/d90/db8/fbd [0,4194304] 0 2024-03-14T13:02:26.283 INFO:tasks.workunit.client.0.smithi111.stdout:0/987: link d6/d9/d15/d55/da0/fd7 d6/d9/de/d77/d159/d46/f15d 0 2024-03-14T13:02:26.284 INFO:tasks.workunit.client.0.smithi111.stdout:3/805: getdents d0/d2d/d35/d69/d10a 0 2024-03-14T13:02:26.285 INFO:tasks.workunit.client.0.smithi111.stdout:3/806: dread - d0/d2d/d35/d50/da2/fd9 zero size 2024-03-14T13:02:26.286 INFO:tasks.workunit.client.0.smithi111.stdout:0/988: creat d6/d9/de/d77/d159/d46/d4a/d11b/d12a/f15e x:0 0 0 2024-03-14T13:02:26.288 INFO:tasks.workunit.client.1.smithi137.stdout:2/847: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f1c [0,4194304] 0 2024-03-14T13:02:26.288 INFO:tasks.workunit.client.0.smithi111.stdout:0/989: mkdir d6/d3d/d149/d15f 0 2024-03-14T13:02:26.291 INFO:tasks.workunit.client.0.smithi111.stdout:0/990: creat d6/d9/de/d32/d64/de7/f160 x:0 0 0 2024-03-14T13:02:26.293 INFO:tasks.workunit.client.0.smithi111.stdout:0/991: read d6/d9/d5b/d9e/fa8 [605209,70982] 0 2024-03-14T13:02:26.294 INFO:tasks.workunit.client.0.smithi111.stdout:0/992: chown d6/d9/d15/d55/da0/fb9 3 1 2024-03-14T13:02:26.294 INFO:tasks.workunit.client.0.smithi111.stdout:0/993: chown d6/d9/de/d77/d159/f12d 128398922 1 2024-03-14T13:02:26.295 INFO:tasks.workunit.client.0.smithi111.stdout:0/994: chown d6/d45/c57 126475108 1 2024-03-14T13:02:26.296 INFO:tasks.workunit.client.0.smithi111.stdout:0/995: truncate d6/d9/de/d77/d159/f95 3311444 0 2024-03-14T13:02:26.298 INFO:tasks.workunit.client.0.smithi111.stdout:0/996: creat d6/d3d/d149/d15f/f161 x:0 0 0 2024-03-14T13:02:26.299 INFO:tasks.workunit.client.1.smithi137.stdout:2/848: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/fb7 458066 0 2024-03-14T13:02:26.299 INFO:tasks.workunit.client.0.smithi111.stdout:0/997: dread - d6/d21/ff7 zero size 2024-03-14T13:02:26.300 INFO:tasks.workunit.client.1.smithi137.stdout:2/849: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/fd9 [552673,112414] 0 2024-03-14T13:02:26.302 INFO:tasks.workunit.client.0.smithi111.stdout:0/998: mkdir d6/d9/de/d77/d159/dd1/d154/d162 0 2024-03-14T13:02:26.303 INFO:tasks.workunit.client.0.smithi111.stdout:0/999: readlink d6/d9/d15/d49/d56/l125 0 2024-03-14T13:02:26.304 INFO:tasks.workunit.client.1.smithi137.stdout:2/850: rename d2/d7/d2e/d3e/d44/d6b/d1d/d41/ldb to d2/d7/d2e/d3e/d44/d6b/d22/da5/l114 0 2024-03-14T13:02:26.308 INFO:tasks.workunit.client.1.smithi137.stdout:2/851: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f73 [0,4194304] 0 2024-03-14T13:02:26.309 INFO:tasks.workunit.client.1.smithi137.stdout:2/852: write d2/d7/d2e/f3d [328312,80736] 0 2024-03-14T13:02:26.311 INFO:tasks.workunit.client.1.smithi137.stdout:2/853: mkdir d2/d7/d2e/d3e/d44/d115 0 2024-03-14T13:02:26.312 INFO:tasks.workunit.client.1.smithi137.stdout:2/854: chown d2/d7/d2e/d3e/d44/d6b/d1d/d41/f78 865974 1 2024-03-14T13:02:26.314 INFO:tasks.workunit.client.1.smithi137.stdout:2/855: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/l116 0 2024-03-14T13:02:26.319 INFO:tasks.workunit.client.1.smithi137.stdout:2/856: dwrite d2/d7/d2e/d3e/d44/fab [0,4194304] 0 2024-03-14T13:02:26.326 INFO:tasks.workunit.client.1.smithi137.stdout:2/857: link d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f73 d2/f117 0 2024-03-14T13:02:26.329 INFO:tasks.workunit.client.1.smithi137.stdout:2/858: rename d2/d7/d2e/d3e/d44/d6b/fa6 to d2/d7/d2e/d3e/d4e/f118 0 2024-03-14T13:02:26.331 INFO:tasks.workunit.client.1.smithi137.stdout:2/859: symlink d2/d7/d2e/d3e/d44/d6b/d22/dac/l119 0 2024-03-14T13:02:26.333 INFO:tasks.workunit.client.1.smithi137.stdout:2/860: creat d2/f11a x:0 0 0 2024-03-14T13:02:26.351 INFO:tasks.workunit.client.1.smithi137.stdout:2/861: dread d2/d7/d2e/d3e/d44/d6b/f39 [0,4194304] 0 2024-03-14T13:02:26.354 INFO:tasks.workunit.client.1.smithi137.stdout:2/862: dread d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/fa7 [0,4194304] 0 2024-03-14T13:02:26.356 INFO:tasks.workunit.client.0.smithi111.stdout:9/779: sync 2024-03-14T13:02:26.356 INFO:tasks.workunit.client.0.smithi111.stdout:2/667: sync 2024-03-14T13:02:26.356 INFO:tasks.workunit.client.0.smithi111.stdout:6/698: sync 2024-03-14T13:02:26.356 INFO:tasks.workunit.client.1.smithi137.stdout:4/785: sync 2024-03-14T13:02:26.356 INFO:tasks.workunit.client.1.smithi137.stdout:5/728: sync 2024-03-14T13:02:26.357 INFO:tasks.workunit.client.1.smithi137.stdout:0/770: sync 2024-03-14T13:02:26.357 INFO:tasks.workunit.client.1.smithi137.stdout:8/874: sync 2024-03-14T13:02:26.357 INFO:tasks.workunit.client.1.smithi137.stdout:9/899: sync 2024-03-14T13:02:26.357 INFO:tasks.workunit.client.0.smithi111.stdout:9/780: stat d5/d6/d8/d5b/fbe 0 2024-03-14T13:02:26.358 INFO:tasks.workunit.client.1.smithi137.stdout:4/786: stat d4/d9/d3a/d41/d38/d79/fe0 0 2024-03-14T13:02:26.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/668: unlink dd/f4b 0 2024-03-14T13:02:26.360 INFO:tasks.workunit.client.1.smithi137.stdout:4/787: write d4/d9/d3a/d41/d6a/ff8 [557628,38418] 0 2024-03-14T13:02:26.361 INFO:tasks.workunit.client.1.smithi137.stdout:8/875: creat d1/f123 x:0 0 0 2024-03-14T13:02:26.363 INFO:tasks.workunit.client.0.smithi111.stdout:6/699: dwrite d0/d50/f67 [0,4194304] 0 2024-03-14T13:02:26.365 INFO:tasks.workunit.client.1.smithi137.stdout:4/788: stat d4/d9/d3a/d41/d6a/dac/ff1 0 2024-03-14T13:02:26.366 INFO:tasks.workunit.client.1.smithi137.stdout:8/876: readlink d1/dc/d10/l43 0 2024-03-14T13:02:26.369 INFO:tasks.workunit.client.0.smithi111.stdout:6/700: mkdir d0/d17/d20/d65/d103 0 2024-03-14T13:02:26.370 INFO:tasks.workunit.client.1.smithi137.stdout:8/877: fsync d1/dc/d10/f1a 0 2024-03-14T13:02:26.370 INFO:tasks.workunit.client.1.smithi137.stdout:5/729: dwrite d7/d16/d18/d25/fe2 [0,4194304] 0 2024-03-14T13:02:26.371 INFO:tasks.workunit.client.0.smithi111.stdout:6/701: truncate d0/d17/d21/d42/da5/fb4 445532 0 2024-03-14T13:02:26.372 INFO:tasks.workunit.client.0.smithi111.stdout:6/702: write d0/d50/f100 [445171,75284] 0 2024-03-14T13:02:26.373 INFO:tasks.workunit.client.0.smithi111.stdout:6/703: truncate d0/f98 4578513 0 2024-03-14T13:02:26.378 INFO:tasks.workunit.client.1.smithi137.stdout:2/863: dwrite d2/d7/d2e/d3e/d44/d6b/d22/da5/fb2 [0,4194304] 0 2024-03-14T13:02:26.382 INFO:tasks.workunit.client.1.smithi137.stdout:2/864: fsync d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/ff2 0 2024-03-14T13:02:26.382 INFO:tasks.workunit.client.1.smithi137.stdout:9/900: dread d6/df/d19/d1f/fa4 [0,4194304] 0 2024-03-14T13:02:26.382 INFO:tasks.workunit.client.0.smithi111.stdout:9/781: dwrite d5/d42/da4/fb1 [0,4194304] 0 2024-03-14T13:02:26.383 INFO:tasks.workunit.client.1.smithi137.stdout:5/730: chown d7/d16/d18/d48/d9d/cc9 1631313 1 2024-03-14T13:02:26.384 INFO:tasks.workunit.client.0.smithi111.stdout:2/669: dwrite dd/d11/d1a/f38 [0,4194304] 0 2024-03-14T13:02:26.384 INFO:tasks.workunit.client.1.smithi137.stdout:4/789: dwrite d4/d9/d3a/d41/f49 [0,4194304] 0 2024-03-14T13:02:26.385 INFO:tasks.workunit.client.1.smithi137.stdout:8/878: mkdir d1/d5e/d11a/d82/d124 0 2024-03-14T13:02:26.386 INFO:tasks.workunit.client.0.smithi111.stdout:6/704: dwrite d0/d17/d20/f6b [0,4194304] 0 2024-03-14T13:02:26.387 INFO:tasks.workunit.client.0.smithi111.stdout:2/670: mknod dd/da9/cd4 0 2024-03-14T13:02:26.389 INFO:tasks.workunit.client.1.smithi137.stdout:2/865: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/l116 0 2024-03-14T13:02:26.394 INFO:tasks.workunit.client.1.smithi137.stdout:5/731: creat d7/d33/d65/dc2/fef x:0 0 0 2024-03-14T13:02:26.395 INFO:tasks.workunit.client.0.smithi111.stdout:6/705: chown d0/d17/d20/d4a/d68/ca8 453095771 1 2024-03-14T13:02:26.396 INFO:tasks.workunit.client.0.smithi111.stdout:2/671: fsync dd/d11/d1a/f21 0 2024-03-14T13:02:26.397 INFO:tasks.workunit.client.0.smithi111.stdout:9/782: rename d5/d6/l62 to d5/d6/d2a/l10f 0 2024-03-14T13:02:26.398 INFO:tasks.workunit.client.0.smithi111.stdout:2/672: mkdir dd/da9/dd5 0 2024-03-14T13:02:26.402 INFO:tasks.workunit.client.1.smithi137.stdout:4/790: rename d4/d9/d3a/d70/d80 to d4/d27/d100 0 2024-03-14T13:02:26.403 INFO:tasks.workunit.client.0.smithi111.stdout:6/706: link d0/d17/d21/d42/f77 d0/d17/d21/d69/d3d/d53/d81/da1/dd7/f104 0 2024-03-14T13:02:26.404 INFO:tasks.workunit.client.1.smithi137.stdout:8/879: creat d1/d5e/d11a/d82/d96/f125 x:0 0 0 2024-03-14T13:02:26.404 INFO:tasks.workunit.client.1.smithi137.stdout:5/732: mkdir d7/d49/d75/df0 0 2024-03-14T13:02:26.404 INFO:tasks.workunit.client.0.smithi111.stdout:2/673: mkdir dd/d17/d34/d74/dca/dcb/dd6 0 2024-03-14T13:02:26.404 INFO:tasks.workunit.client.1.smithi137.stdout:0/771: dwrite d1/d5/d1b/d1e/d25/d55/f7a [0,4194304] 0 2024-03-14T13:02:26.405 INFO:tasks.workunit.client.0.smithi111.stdout:2/674: symlink dd/d17/d53/ld7 0 2024-03-14T13:02:26.406 INFO:tasks.workunit.client.1.smithi137.stdout:4/791: stat d4/d9/d3a/d41/c22 0 2024-03-14T13:02:26.409 INFO:tasks.workunit.client.0.smithi111.stdout:2/675: dread dd/d8d/dc2/f66 [0,4194304] 0 2024-03-14T13:02:26.410 INFO:tasks.workunit.client.0.smithi111.stdout:6/707: dwrite d0/d17/d20/d4a/d59/d82/fe8 [0,4194304] 0 2024-03-14T13:02:26.411 INFO:tasks.workunit.client.1.smithi137.stdout:9/901: dwrite d6/df/f20 [0,4194304] 0 2024-03-14T13:02:26.411 INFO:tasks.workunit.client.1.smithi137.stdout:2/866: link d2/d7/d2e/d3e/d44/d6b/fc2 d2/d7/d2e/d3e/d4e/f11b 0 2024-03-14T13:02:26.411 INFO:tasks.workunit.client.0.smithi111.stdout:2/676: fdatasync dd/da9/fb2 0 2024-03-14T13:02:26.416 INFO:tasks.workunit.client.1.smithi137.stdout:5/733: read - d7/d16/d18/d25/d28/d3c/fcf zero size 2024-03-14T13:02:26.416 INFO:tasks.workunit.client.1.smithi137.stdout:2/867: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f113 465977 1 2024-03-14T13:02:26.417 INFO:tasks.workunit.client.0.smithi111.stdout:2/677: mknod dd/d11/cd8 0 2024-03-14T13:02:26.419 INFO:tasks.workunit.client.1.smithi137.stdout:5/734: mknod d7/d1d/d74/d83/cf1 0 2024-03-14T13:02:26.419 INFO:tasks.workunit.client.1.smithi137.stdout:2/868: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/d10c 177 1 2024-03-14T13:02:26.420 INFO:tasks.workunit.client.0.smithi111.stdout:2/678: fsync dd/d11/f6d 0 2024-03-14T13:02:26.421 INFO:tasks.workunit.client.1.smithi137.stdout:0/772: dread d1/d5/d1b/d1e/d25/d26/f41 [0,4194304] 0 2024-03-14T13:02:26.422 INFO:tasks.workunit.client.1.smithi137.stdout:8/880: rename d1/dc/d10/fa4 to d1/d5e/daf/f126 0 2024-03-14T13:02:26.423 INFO:tasks.workunit.client.0.smithi111.stdout:2/679: rename dd/c12 to dd/d17/cd9 0 2024-03-14T13:02:26.424 INFO:tasks.workunit.client.0.smithi111.stdout:2/680: write dd/d11/d1a/f1c [1854347,51712] 0 2024-03-14T13:02:26.425 INFO:tasks.workunit.client.0.smithi111.stdout:2/681: chown dd/d17/d53/d9f/cb3 0 1 2024-03-14T13:02:26.425 INFO:tasks.workunit.client.1.smithi137.stdout:5/735: creat d7/d16/d18/d2d/dd1/ddc/ff2 x:0 0 0 2024-03-14T13:02:26.426 INFO:tasks.workunit.client.0.smithi111.stdout:2/682: chown dd/d17/f3a 391998504 1 2024-03-14T13:02:26.427 INFO:tasks.workunit.client.0.smithi111.stdout:7/695: sync 2024-03-14T13:02:26.427 INFO:tasks.workunit.client.0.smithi111.stdout:1/940: sync 2024-03-14T13:02:26.428 INFO:tasks.workunit.client.1.smithi137.stdout:2/869: creat d2/d7/d2e/d3e/d44/d6b/d22/d26/f11c x:0 0 0 2024-03-14T13:02:26.428 INFO:tasks.workunit.client.0.smithi111.stdout:2/683: symlink dd/d17/d34/d73/d49/lda 0 2024-03-14T13:02:26.428 INFO:tasks.workunit.client.0.smithi111.stdout:1/941: chown d20/d25/d87 55915682 1 2024-03-14T13:02:26.429 INFO:tasks.workunit.client.1.smithi137.stdout:5/736: unlink d7/d1d/la8 0 2024-03-14T13:02:26.429 INFO:tasks.workunit.client.0.smithi111.stdout:7/696: symlink d0/d1/d5/lcb 0 2024-03-14T13:02:26.431 INFO:tasks.workunit.client.0.smithi111.stdout:1/942: mknod d20/d127/c142 0 2024-03-14T13:02:26.431 INFO:tasks.workunit.client.1.smithi137.stdout:2/870: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dda/ff7 [1032486,130112] 0 2024-03-14T13:02:26.432 INFO:tasks.workunit.client.0.smithi111.stdout:1/943: dread - d20/d25/f10c zero size 2024-03-14T13:02:26.433 INFO:tasks.workunit.client.1.smithi137.stdout:0/773: getdents d1/d5/d1b/d1e/d53/d6d 0 2024-03-14T13:02:26.433 INFO:tasks.workunit.client.0.smithi111.stdout:7/697: symlink d0/d1/d21/d5d/lcc 0 2024-03-14T13:02:26.434 INFO:tasks.workunit.client.0.smithi111.stdout:1/944: fsync d20/d25/d87/fb8 0 2024-03-14T13:02:26.435 INFO:tasks.workunit.client.0.smithi111.stdout:2/684: rename dd/d17/d34/d73/d75/d59 to dd/d17/d34/ddb 0 2024-03-14T13:02:26.435 INFO:tasks.workunit.client.1.smithi137.stdout:5/737: getdents d7/d16/d18 0 2024-03-14T13:02:26.435 INFO:tasks.workunit.client.0.smithi111.stdout:1/945: symlink d20/d21/dc1/l143 0 2024-03-14T13:02:26.436 INFO:tasks.workunit.client.0.smithi111.stdout:1/946: readlink d20/lce 0 2024-03-14T13:02:26.436 INFO:tasks.workunit.client.0.smithi111.stdout:1/947: dread - d20/d25/d31/f12a zero size 2024-03-14T13:02:26.437 INFO:tasks.workunit.client.1.smithi137.stdout:2/871: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/f11d x:0 0 0 2024-03-14T13:02:26.437 INFO:tasks.workunit.client.0.smithi111.stdout:7/698: rename d0/d1/d4a/f97 to d0/d1/d5/d38/d5c/db4/fcd 0 2024-03-14T13:02:26.439 INFO:tasks.workunit.client.0.smithi111.stdout:2/685: fsync dd/d8d/dc2/f9a 0 2024-03-14T13:02:26.440 INFO:tasks.workunit.client.1.smithi137.stdout:2/872: mkdir d2/d11e 0 2024-03-14T13:02:26.442 INFO:tasks.workunit.client.0.smithi111.stdout:7/699: dread d0/d1/d5/d38/d5c/d6d/fa3 [0,4194304] 0 2024-03-14T13:02:26.442 INFO:tasks.workunit.client.0.smithi111.stdout:1/948: rmdir d20/d40/db4/ddf/d136 0 2024-03-14T13:02:26.442 INFO:tasks.workunit.client.1.smithi137.stdout:0/774: dwrite d1/d5/d1b/d1e/d25/d8a/d4a/fd9 [0,4194304] 0 2024-03-14T13:02:26.445 INFO:tasks.workunit.client.1.smithi137.stdout:0/775: readlink d1/d1f/l3c 0 2024-03-14T13:02:26.445 INFO:tasks.workunit.client.0.smithi111.stdout:1/949: rmdir d20/d25/d75 39 2024-03-14T13:02:26.446 INFO:tasks.workunit.client.1.smithi137.stdout:0/776: chown d1/d44/d72/fca 24672417 1 2024-03-14T13:02:26.447 INFO:tasks.workunit.client.0.smithi111.stdout:1/950: fdatasync f3 0 2024-03-14T13:02:26.448 INFO:tasks.workunit.client.1.smithi137.stdout:2/873: link d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/l88 d2/d7/d2e/d3e/d44/d6b/d22/dbb/l11f 0 2024-03-14T13:02:26.449 INFO:tasks.workunit.client.0.smithi111.stdout:7/700: dwrite d0/d7e/f19 [4194304,4194304] 0 2024-03-14T13:02:26.449 INFO:tasks.workunit.client.0.smithi111.stdout:1/951: write d20/d21/dd7/ffb [78300,53833] 0 2024-03-14T13:02:26.453 INFO:tasks.workunit.client.1.smithi137.stdout:2/874: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/fa8 [2028829,19366] 0 2024-03-14T13:02:26.454 INFO:tasks.workunit.client.0.smithi111.stdout:1/952: mkdir d20/d21/d3f/d59/d144 0 2024-03-14T13:02:26.455 INFO:tasks.workunit.client.0.smithi111.stdout:7/701: mkdir d0/d1/d5/d38/d5c/d6d/dc9/dce 0 2024-03-14T13:02:26.455 INFO:tasks.workunit.client.0.smithi111.stdout:1/953: fdatasync d20/d21/da7/fbe 0 2024-03-14T13:02:26.456 INFO:tasks.workunit.client.1.smithi137.stdout:2/875: readlink d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/l58 0 2024-03-14T13:02:26.456 INFO:tasks.workunit.client.1.smithi137.stdout:0/777: dwrite d1/d5/d1b/d1e/d53/d6d/fba [4194304,4194304] 0 2024-03-14T13:02:26.457 INFO:tasks.workunit.client.0.smithi111.stdout:1/954: chown d20/d25/da8/de9/lda 3303679 1 2024-03-14T13:02:26.471 INFO:tasks.workunit.client.0.smithi111.stdout:9/783: dwrite d5/ffc [0,4194304] 0 2024-03-14T13:02:26.472 INFO:tasks.workunit.client.1.smithi137.stdout:2/876: creat d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/f120 x:0 0 0 2024-03-14T13:02:26.472 INFO:tasks.workunit.client.1.smithi137.stdout:9/902: write d6/f96 [84459,55957] 0 2024-03-14T13:02:26.473 INFO:tasks.workunit.client.1.smithi137.stdout:0/778: rename d1/d5/d1b/d1e/d53/cdb to d1/d1f/c106 0 2024-03-14T13:02:26.475 INFO:tasks.workunit.client.1.smithi137.stdout:4/792: dwrite d4/d9/f1f [0,4194304] 0 2024-03-14T13:02:26.476 INFO:tasks.workunit.client.0.smithi111.stdout:9/784: fdatasync d5/d6/d2a/f75 0 2024-03-14T13:02:26.479 INFO:tasks.workunit.client.1.smithi137.stdout:2/877: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/l121 0 2024-03-14T13:02:26.479 INFO:tasks.workunit.client.1.smithi137.stdout:0/779: write d1/d5/d1b/d1e/d25/d55/d73/f98 [527225,110246] 0 2024-03-14T13:02:26.481 INFO:tasks.workunit.client.1.smithi137.stdout:9/903: mknod d6/df/d19/d33/d8f/d124/c12b 0 2024-03-14T13:02:26.482 INFO:tasks.workunit.client.1.smithi137.stdout:4/793: write d4/f8 [861595,51252] 0 2024-03-14T13:02:26.482 INFO:tasks.workunit.client.1.smithi137.stdout:2/878: chown d2/c52 613626 1 2024-03-14T13:02:26.483 INFO:tasks.workunit.client.0.smithi111.stdout:9/785: mknod d5/d42/d71/dbd/dcd/c110 0 2024-03-14T13:02:26.483 INFO:tasks.workunit.client.0.smithi111.stdout:7/702: dread d0/d1/d22/f7b [0,4194304] 0 2024-03-14T13:02:26.485 INFO:tasks.workunit.client.1.smithi137.stdout:4/794: readlink d4/d10/l93 0 2024-03-14T13:02:26.486 INFO:tasks.workunit.client.0.smithi111.stdout:1/955: dread d20/d25/d31/f3a [0,4194304] 0 2024-03-14T13:02:26.486 INFO:tasks.workunit.client.1.smithi137.stdout:3/661: sync 2024-03-14T13:02:26.487 INFO:tasks.workunit.client.0.smithi111.stdout:2/686: dwrite dd/d17/d34/d73/d49/f5c [0,4194304] 0 2024-03-14T13:02:26.489 INFO:tasks.workunit.client.0.smithi111.stdout:9/786: mknod d5/d42/d71/dbd/dcd/c111 0 2024-03-14T13:02:26.491 INFO:tasks.workunit.client.1.smithi137.stdout:4/795: creat d4/d27/d100/d8d/dc0/f101 x:0 0 0 2024-03-14T13:02:26.494 INFO:tasks.workunit.client.1.smithi137.stdout:8/881: dwrite d1/dc/d3c/d87/f48 [4194304,4194304] 0 2024-03-14T13:02:26.495 INFO:tasks.workunit.client.1.smithi137.stdout:5/738: dwrite d7/d16/d18/d25/d28/f3d [0,4194304] 0 2024-03-14T13:02:26.495 INFO:tasks.workunit.client.1.smithi137.stdout:2/879: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/l121 57 1 2024-03-14T13:02:26.496 INFO:tasks.workunit.client.1.smithi137.stdout:0/780: creat d1/d5/d1b/d1e/d53/d6d/f107 x:0 0 0 2024-03-14T13:02:26.496 INFO:tasks.workunit.client.0.smithi111.stdout:1/956: fsync d20/d25/da8/ff6 0 2024-03-14T13:02:26.497 INFO:tasks.workunit.client.0.smithi111.stdout:9/787: creat d5/d6/d14/f112 x:0 0 0 2024-03-14T13:02:26.498 INFO:tasks.workunit.client.0.smithi111.stdout:1/957: write d20/d25/d75/df7/f114 [371546,76139] 0 2024-03-14T13:02:26.499 INFO:tasks.workunit.client.0.smithi111.stdout:7/703: dread d0/d1/d4a/f60 [0,4194304] 0 2024-03-14T13:02:26.501 INFO:tasks.workunit.client.0.smithi111.stdout:1/958: mknod d20/d25/d31/d118/c145 0 2024-03-14T13:02:26.502 INFO:tasks.workunit.client.0.smithi111.stdout:1/959: readlink d20/d25/d31/l135 0 2024-03-14T13:02:26.508 INFO:tasks.workunit.client.0.smithi111.stdout:2/687: write dd/d17/f37 [565930,121254] 0 2024-03-14T13:02:26.508 INFO:tasks.workunit.client.1.smithi137.stdout:4/796: write d4/d10/d5a/f63 [421822,26320] 0 2024-03-14T13:02:26.508 INFO:tasks.workunit.client.1.smithi137.stdout:5/739: creat d7/d1d/d74/d83/ff3 x:0 0 0 2024-03-14T13:02:26.511 INFO:tasks.workunit.client.1.smithi137.stdout:8/882: write d1/dc/d77/d9f/de4/f22 [1781022,87623] 0 2024-03-14T13:02:26.511 INFO:tasks.workunit.client.1.smithi137.stdout:4/797: dread - d4/d9/d3a/d41/fc9 zero size 2024-03-14T13:02:26.512 INFO:tasks.workunit.client.0.smithi111.stdout:7/704: dwrite d0/f72 [0,4194304] 0 2024-03-14T13:02:26.512 INFO:tasks.workunit.client.0.smithi111.stdout:2/688: symlink dd/d11/d1a/d88/ldc 0 2024-03-14T13:02:26.515 INFO:tasks.workunit.client.1.smithi137.stdout:3/662: dwrite d4/de/d16/d1c/d5d/d85/f5a [0,4194304] 0 2024-03-14T13:02:26.516 INFO:tasks.workunit.client.0.smithi111.stdout:2/689: symlink dd/d11/ldd 0 2024-03-14T13:02:26.517 INFO:tasks.workunit.client.0.smithi111.stdout:7/705: truncate d0/d1/d5/f1a 4014819 0 2024-03-14T13:02:26.519 INFO:tasks.workunit.client.1.smithi137.stdout:3/663: write d4/d19/f23 [3785156,75312] 0 2024-03-14T13:02:26.519 INFO:tasks.workunit.client.0.smithi111.stdout:7/706: rmdir d0/d11 39 2024-03-14T13:02:26.521 INFO:tasks.workunit.client.0.smithi111.stdout:7/707: chown d0/d1/d5/d38/d1b/d9c/fa9 203897922 1 2024-03-14T13:02:26.522 INFO:tasks.workunit.client.1.smithi137.stdout:2/880: rename d2/d7/d2e/d3e/d44/d6b/f20 to d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/df0/f122 0 2024-03-14T13:02:26.522 INFO:tasks.workunit.client.1.smithi137.stdout:4/798: symlink d4/d10/d42/d73/d9d/dbc/l102 0 2024-03-14T13:02:26.522 INFO:tasks.workunit.client.1.smithi137.stdout:2/881: dread - d2/f11a zero size 2024-03-14T13:02:26.523 INFO:tasks.workunit.client.1.smithi137.stdout:3/664: write d4/de/d16/d1c/d5d/f43 [1856105,68173] 0 2024-03-14T13:02:26.524 INFO:tasks.workunit.client.0.smithi111.stdout:2/690: dwrite dd/d17/d53/f98 [0,4194304] 0 2024-03-14T13:02:26.526 INFO:tasks.workunit.client.0.smithi111.stdout:7/708: truncate d0/d1/d4a/f6a 1609126 0 2024-03-14T13:02:26.526 INFO:tasks.workunit.client.1.smithi137.stdout:5/740: rename d7/d16/d18/d25/d28/dc5/fe7 to d7/d49/d75/ff4 0 2024-03-14T13:02:26.527 INFO:tasks.workunit.client.1.smithi137.stdout:5/741: chown d7/d16 1 1 2024-03-14T13:02:26.527 INFO:tasks.workunit.client.1.smithi137.stdout:7/814: sync 2024-03-14T13:02:26.527 INFO:tasks.workunit.client.1.smithi137.stdout:1/936: sync 2024-03-14T13:02:26.528 INFO:tasks.workunit.client.1.smithi137.stdout:6/791: sync 2024-03-14T13:02:26.529 INFO:tasks.workunit.client.1.smithi137.stdout:8/883: dwrite d1/d5e/d11a/d97/f2a [8388608,4194304] 0 2024-03-14T13:02:26.529 INFO:tasks.workunit.client.0.smithi111.stdout:2/691: rename c7 to dd/d11/d8b/cde 0 2024-03-14T13:02:26.532 INFO:tasks.workunit.client.1.smithi137.stdout:8/884: stat d1/dc/d4e 0 2024-03-14T13:02:26.532 INFO:tasks.workunit.client.1.smithi137.stdout:2/882: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/ffd 11669440 1 2024-03-14T13:02:26.533 INFO:tasks.workunit.client.0.smithi111.stdout:7/709: write d0/d1/d21/d5d/f84 [1307660,101441] 0 2024-03-14T13:02:26.533 INFO:tasks.workunit.client.1.smithi137.stdout:3/665: write d4/de/d16/d1c/d5d/d85/d6b/fca [225948,53539] 0 2024-03-14T13:02:26.535 INFO:tasks.workunit.client.0.smithi111.stdout:2/692: truncate dd/d17/d34/ddb/f97 2128555 0 2024-03-14T13:02:26.537 INFO:tasks.workunit.client.0.smithi111.stdout:7/710: creat d0/d1/d5/fcf x:0 0 0 2024-03-14T13:02:26.538 INFO:tasks.workunit.client.1.smithi137.stdout:9/904: write d6/d56/f12a [1096847,42001] 0 2024-03-14T13:02:26.540 INFO:tasks.workunit.client.0.smithi111.stdout:9/788: dwrite d5/d42/f5c [0,4194304] 0 2024-03-14T13:02:26.542 INFO:tasks.workunit.client.0.smithi111.stdout:7/711: read d0/d11/f73 [658430,73361] 0 2024-03-14T13:02:26.544 INFO:tasks.workunit.client.1.smithi137.stdout:4/799: rename d4/d9/d3a/d41/f97 to d4/d9/f103 0 2024-03-14T13:02:26.544 INFO:tasks.workunit.client.1.smithi137.stdout:7/815: truncate d3/d5/d23/d3f/d5b/fab 921779 0 2024-03-14T13:02:26.545 INFO:tasks.workunit.client.1.smithi137.stdout:6/792: dread - d2/d6/d1d/d88/f93 zero size 2024-03-14T13:02:26.546 INFO:tasks.workunit.client.1.smithi137.stdout:2/883: rmdir d2/d7/d2e/d3e/d4e 39 2024-03-14T13:02:26.546 INFO:tasks.workunit.client.0.smithi111.stdout:1/960: dread d20/f23 [0,4194304] 0 2024-03-14T13:02:26.547 INFO:tasks.workunit.client.1.smithi137.stdout:0/781: dwrite d1/d1f/d80/f86 [0,4194304] 0 2024-03-14T13:02:26.550 INFO:tasks.workunit.client.0.smithi111.stdout:7/712: unlink d0/d1/d5/d38/d5c/d6d/lb0 0 2024-03-14T13:02:26.551 INFO:tasks.workunit.client.0.smithi111.stdout:1/961: creat d20/d25/d31/d80/d12e/f146 x:0 0 0 2024-03-14T13:02:26.552 INFO:tasks.workunit.client.0.smithi111.stdout:7/713: mknod d0/d1/d4a/d52/cd0 0 2024-03-14T13:02:26.553 INFO:tasks.workunit.client.0.smithi111.stdout:9/789: getdents d5/d42/d71/d96 0 2024-03-14T13:02:26.556 INFO:tasks.workunit.client.0.smithi111.stdout:9/790: mknod d5/d6/d14/d7a/d102/c113 0 2024-03-14T13:02:26.560 INFO:tasks.workunit.client.0.smithi111.stdout:9/791: dread d5/d6/d8/f93 [0,4194304] 0 2024-03-14T13:02:26.562 INFO:tasks.workunit.client.0.smithi111.stdout:9/792: truncate d5/d6/fc 4043343 0 2024-03-14T13:02:26.563 INFO:tasks.workunit.client.0.smithi111.stdout:9/793: write d5/d6/d8/d5b/d84/f106 [891423,20550] 0 2024-03-14T13:02:26.564 INFO:tasks.workunit.client.1.smithi137.stdout:3/666: unlink d4/f68 0 2024-03-14T13:02:26.564 INFO:tasks.workunit.client.1.smithi137.stdout:8/885: truncate d1/d5e/daf/dee/f109 329522 0 2024-03-14T13:02:26.566 INFO:tasks.workunit.client.0.smithi111.stdout:9/794: rmdir d5/d6/d8/d5b/d84/db6/de7 39 2024-03-14T13:02:26.566 INFO:tasks.workunit.client.1.smithi137.stdout:4/800: unlink d4/d4d/d6e/f9b 0 2024-03-14T13:02:26.569 INFO:tasks.workunit.client.0.smithi111.stdout:9/795: creat d5/d6/d2a/d5a/f114 x:0 0 0 2024-03-14T13:02:26.573 INFO:tasks.workunit.client.1.smithi137.stdout:7/816: fsync d3/d5/d5a/fc6 0 2024-03-14T13:02:26.576 INFO:tasks.workunit.client.1.smithi137.stdout:8/886: chown d1/d4a/dc4 32 1 2024-03-14T13:02:26.576 INFO:tasks.workunit.client.1.smithi137.stdout:5/742: dwrite d7/d16/fae [0,4194304] 0 2024-03-14T13:02:26.577 INFO:tasks.workunit.client.1.smithi137.stdout:1/937: dread d7/d31/d54/f62 [0,4194304] 0 2024-03-14T13:02:26.577 INFO:tasks.workunit.client.0.smithi111.stdout:2/693: dread dd/d17/d34/d73/d75/f5d [0,4194304] 0 2024-03-14T13:02:26.579 INFO:tasks.workunit.client.1.smithi137.stdout:7/817: read - d3/d52/d77/fc4 zero size 2024-03-14T13:02:26.581 INFO:tasks.workunit.client.1.smithi137.stdout:4/801: mkdir d4/d27/d5c/d6d/d58/d104 0 2024-03-14T13:02:26.581 INFO:tasks.workunit.client.0.smithi111.stdout:7/714: dwrite d0/d1/d5/d38/d1b/d9c/fa9 [0,4194304] 0 2024-03-14T13:02:26.582 INFO:tasks.workunit.client.0.smithi111.stdout:2/694: creat dd/fdf x:0 0 0 2024-03-14T13:02:26.582 INFO:tasks.workunit.client.1.smithi137.stdout:6/793: dwrite d2/d4c/fbd [0,4194304] 0 2024-03-14T13:02:26.584 INFO:tasks.workunit.client.0.smithi111.stdout:7/715: chown d0/d1/d4a/c58 564348 1 2024-03-14T13:02:26.591 INFO:tasks.workunit.client.0.smithi111.stdout:1/962: dread d20/d25/d75/df7/f101 [0,4194304] 0 2024-03-14T13:02:26.592 INFO:tasks.workunit.client.1.smithi137.stdout:8/887: write d1/dc/d77/d9f/fe7 [689512,98714] 0 2024-03-14T13:02:26.592 INFO:tasks.workunit.client.1.smithi137.stdout:3/667: dwrite d4/de/dc7/fcf [4194304,4194304] 0 2024-03-14T13:02:26.593 INFO:tasks.workunit.client.1.smithi137.stdout:1/938: chown d7/d9/d19/d1f/d26/c6a 4931973 1 2024-03-14T13:02:26.596 INFO:tasks.workunit.client.0.smithi111.stdout:7/716: link d0/d1/d22/d3e/faf d0/d1/d22/d3e/fd1 0 2024-03-14T13:02:26.597 INFO:tasks.workunit.client.0.smithi111.stdout:9/796: truncate d5/d6/d8/d5b/d84/fa2 3228792 0 2024-03-14T13:02:26.597 INFO:tasks.workunit.client.0.smithi111.stdout:1/963: read d20/d21/d28/d5c/f93 [2976661,112467] 0 2024-03-14T13:02:26.598 INFO:tasks.workunit.client.0.smithi111.stdout:7/717: creat d0/d1/d4a/d52/fd2 x:0 0 0 2024-03-14T13:02:26.598 INFO:tasks.workunit.client.0.smithi111.stdout:5/937: sync 2024-03-14T13:02:26.599 INFO:tasks.workunit.client.0.smithi111.stdout:9/797: creat d5/d6/d14/f115 x:0 0 0 2024-03-14T13:02:26.600 INFO:tasks.workunit.client.0.smithi111.stdout:9/798: chown d5/fa 8629 1 2024-03-14T13:02:26.600 INFO:tasks.workunit.client.1.smithi137.stdout:0/782: dwrite d1/d5/d1b/d42/d3a/dc6/fe5 [0,4194304] 0 2024-03-14T13:02:26.602 INFO:tasks.workunit.client.1.smithi137.stdout:7/818: symlink d3/d6/d14/d22/df3/dfd/l10e 0 2024-03-14T13:02:26.602 INFO:tasks.workunit.client.1.smithi137.stdout:1/939: creat d7/d31/d82/f142 x:0 0 0 2024-03-14T13:02:26.603 INFO:tasks.workunit.client.1.smithi137.stdout:4/802: creat d4/d9/d3a/d41/d38/d55/f105 x:0 0 0 2024-03-14T13:02:26.607 INFO:tasks.workunit.client.1.smithi137.stdout:7/819: fdatasync d3/d6/d14/d27/dfe/f100 0 2024-03-14T13:02:26.608 INFO:tasks.workunit.client.1.smithi137.stdout:2/884: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/fd9 [0,4194304] 0 2024-03-14T13:02:26.608 INFO:tasks.workunit.client.1.smithi137.stdout:4/803: readlink d4/ld 0 2024-03-14T13:02:26.608 INFO:tasks.workunit.client.1.smithi137.stdout:5/743: write d7/d16/d18/f2a [5241371,112753] 0 2024-03-14T13:02:26.609 INFO:tasks.workunit.client.1.smithi137.stdout:0/783: read d1/f51 [1644976,77075] 0 2024-03-14T13:02:26.609 INFO:tasks.workunit.client.1.smithi137.stdout:8/888: rename d1/d5e/d11a/f40 to d1/dc/d10/d102/f127 0 2024-03-14T13:02:26.610 INFO:tasks.workunit.client.1.smithi137.stdout:3/668: symlink d4/de/d16/d1c/d5d/d85/ldc 0 2024-03-14T13:02:26.612 INFO:tasks.workunit.client.1.smithi137.stdout:1/940: unlink d7/d9/d19/d1f/d89/d90/db8/l125 0 2024-03-14T13:02:26.613 INFO:tasks.workunit.client.1.smithi137.stdout:0/784: mknod d1/d5/d1b/d1e/d25/d8a/d9d/c108 0 2024-03-14T13:02:26.618 INFO:tasks.workunit.client.1.smithi137.stdout:7/820: rename d3/d6/d59/daa/dcf/d6a/d75/d78/f83 to d3/d6/d59/daa/f10f 0 2024-03-14T13:02:26.619 INFO:tasks.workunit.client.1.smithi137.stdout:8/889: write d1/d5e/d11a/d82/de6/ff8 [424277,104558] 0 2024-03-14T13:02:26.620 INFO:tasks.workunit.client.1.smithi137.stdout:3/669: creat d4/de/d16/d1c/db8/dd1/fdd x:0 0 0 2024-03-14T13:02:26.622 INFO:tasks.workunit.client.0.smithi111.stdout:1/964: creat d20/d25/d31/f147 x:0 0 0 2024-03-14T13:02:26.626 INFO:tasks.workunit.client.0.smithi111.stdout:9/799: creat d5/d42/d71/dbd/de0/f116 x:0 0 0 2024-03-14T13:02:26.627 INFO:tasks.workunit.client.0.smithi111.stdout:2/695: write dd/d17/d53/f64 [232465,73239] 0 2024-03-14T13:02:26.628 INFO:tasks.workunit.client.0.smithi111.stdout:1/965: symlink d20/d25/d31/d80/l148 0 2024-03-14T13:02:26.628 INFO:tasks.workunit.client.1.smithi137.stdout:8/890: creat d1/d5e/d11a/d82/de6/f128 x:0 0 0 2024-03-14T13:02:26.629 INFO:tasks.workunit.client.1.smithi137.stdout:4/804: dwrite d4/d10/d42/fef [0,4194304] 0 2024-03-14T13:02:26.631 INFO:tasks.workunit.client.0.smithi111.stdout:1/966: rmdir d20 39 2024-03-14T13:02:26.636 INFO:tasks.workunit.client.0.smithi111.stdout:5/938: dwrite d2/d16/d49/d7c/d95/d6a/fa8 [0,4194304] 0 2024-03-14T13:02:26.636 INFO:tasks.workunit.client.0.smithi111.stdout:1/967: dread - d20/d25/f13c zero size 2024-03-14T13:02:26.638 INFO:tasks.workunit.client.1.smithi137.stdout:7/821: dread d3/f6f [0,4194304] 0 2024-03-14T13:02:26.640 INFO:tasks.workunit.client.1.smithi137.stdout:0/785: dwrite d1/d5/d1b/d42/d3a/dc6/fe5 [0,4194304] 0 2024-03-14T13:02:26.640 INFO:tasks.workunit.client.0.smithi111.stdout:9/800: creat d5/d42/d71/d96/d105/f117 x:0 0 0 2024-03-14T13:02:26.641 INFO:tasks.workunit.client.1.smithi137.stdout:2/885: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/f120 [0,4194304] 0 2024-03-14T13:02:26.643 INFO:tasks.workunit.client.1.smithi137.stdout:7/822: creat d3/d4a/f110 x:0 0 0 2024-03-14T13:02:26.643 INFO:tasks.workunit.client.1.smithi137.stdout:8/891: mknod d1/dc/d3c/d87/c129 0 2024-03-14T13:02:26.644 INFO:tasks.workunit.client.0.smithi111.stdout:5/939: write d2/d45/d76/db2/df7/f106 [234612,74210] 0 2024-03-14T13:02:26.645 INFO:tasks.workunit.client.0.smithi111.stdout:5/940: chown d2/d16/d49/d7c/d8b/ce2 17262 1 2024-03-14T13:02:26.645 INFO:tasks.workunit.client.1.smithi137.stdout:0/786: readlink d1/d5/d1b/d1e/d25/laf 0 2024-03-14T13:02:26.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/805: write d4/d9/d3a/d41/d38/d55/db6/ffd [1013438,59080] 0 2024-03-14T13:02:26.649 INFO:tasks.workunit.client.1.smithi137.stdout:1/941: write d7/d9/d19/d1f/f78 [731571,40626] 0 2024-03-14T13:02:26.652 INFO:tasks.workunit.client.1.smithi137.stdout:8/892: creat d1/dc/d4e/d6d/f12a x:0 0 0 2024-03-14T13:02:26.653 INFO:tasks.workunit.client.1.smithi137.stdout:7/823: mkdir d3/d5/d23/d3f/d5b/d111 0 2024-03-14T13:02:26.654 INFO:tasks.workunit.client.1.smithi137.stdout:6/794: dwrite d2/d6/d86/fa9 [0,4194304] 0 2024-03-14T13:02:26.660 INFO:tasks.workunit.client.0.smithi111.stdout:5/941: dread d2/d45/f5c [0,4194304] 0 2024-03-14T13:02:26.661 INFO:tasks.workunit.client.0.smithi111.stdout:7/718: truncate d0/f7 2766155 0 2024-03-14T13:02:26.664 INFO:tasks.workunit.client.1.smithi137.stdout:2/886: link d2/d7/d2e/f6a d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/f123 0 2024-03-14T13:02:26.665 INFO:tasks.workunit.client.0.smithi111.stdout:5/942: dread - d2/d16/d49/d33/d68/f108 zero size 2024-03-14T13:02:26.667 INFO:tasks.workunit.client.0.smithi111.stdout:7/719: creat d0/d1/d21/fd3 x:0 0 0 2024-03-14T13:02:26.670 INFO:tasks.workunit.client.0.smithi111.stdout:5/943: chown d2/d16/d49/d7c/d95/d36/cd7 68 1 2024-03-14T13:02:26.670 INFO:tasks.workunit.client.0.smithi111.stdout:2/696: dwrite dd/d17/d34/d73/faf [0,4194304] 0 2024-03-14T13:02:26.671 INFO:tasks.workunit.client.1.smithi137.stdout:3/670: dwrite d4/de/d16/d1c/d5d/d85/d6b/f8a [4194304,4194304] 0 2024-03-14T13:02:26.672 INFO:tasks.workunit.client.0.smithi111.stdout:5/944: write d2/d16/d49/d44/f8e [40936,58111] 0 2024-03-14T13:02:26.672 INFO:tasks.workunit.client.0.smithi111.stdout:7/720: chown d0/d1/d21/d2a/d8c/lae 6 1 2024-03-14T13:02:26.673 INFO:tasks.workunit.client.1.smithi137.stdout:0/787: dwrite d1/d5/d1b/d1e/d25/d26/f39 [4194304,4194304] 0 2024-03-14T13:02:26.675 INFO:tasks.workunit.client.0.smithi111.stdout:9/801: dread d5/d6/d2a/f3e [0,4194304] 0 2024-03-14T13:02:26.677 INFO:tasks.workunit.client.0.smithi111.stdout:2/697: mkdir dd/d17/d34/de0 0 2024-03-14T13:02:26.678 INFO:tasks.workunit.client.1.smithi137.stdout:2/887: read - d2/d7/d2e/d3e/d44/d6b/d22/d4d/f11d zero size 2024-03-14T13:02:26.678 INFO:tasks.workunit.client.0.smithi111.stdout:2/698: chown dd/d17/d34/d74/dc1/dd1 5912 1 2024-03-14T13:02:26.679 INFO:tasks.workunit.client.0.smithi111.stdout:7/721: mkdir d0/d11/da8/dd4 0 2024-03-14T13:02:26.681 INFO:tasks.workunit.client.0.smithi111.stdout:7/722: mkdir d0/d1/d5/d56/d70/dd5 0 2024-03-14T13:02:26.682 INFO:tasks.workunit.client.1.smithi137.stdout:5/744: dread d7/d16/d18/d2d/f8e [0,4194304] 0 2024-03-14T13:02:26.683 INFO:tasks.workunit.client.0.smithi111.stdout:7/723: stat d0/d1/d5/l69 0 2024-03-14T13:02:26.684 INFO:tasks.workunit.client.1.smithi137.stdout:7/824: link d3/d6/d64/d9e/fb8 d3/d6/d59/daa/dcf/d6a/da1/f112 0 2024-03-14T13:02:26.685 INFO:tasks.workunit.client.1.smithi137.stdout:3/671: symlink d4/de/d16/d27/dd3/lde 0 2024-03-14T13:02:26.686 INFO:tasks.workunit.client.0.smithi111.stdout:9/802: dread d5/d6/d2a/f75 [4194304,4194304] 0 2024-03-14T13:02:26.687 INFO:tasks.workunit.client.1.smithi137.stdout:4/806: dwrite d4/d9/d3a/d41/d38/d79/f94 [4194304,4194304] 0 2024-03-14T13:02:26.688 INFO:tasks.workunit.client.0.smithi111.stdout:9/803: write d5/d6/fe1 [591017,25723] 0 2024-03-14T13:02:26.689 INFO:tasks.workunit.client.1.smithi137.stdout:8/893: rename d1/d5e/d11a/d97/d11d/fdf to d1/dc/f12b 0 2024-03-14T13:02:26.689 INFO:tasks.workunit.client.1.smithi137.stdout:8/894: fsync d1/f83 0 2024-03-14T13:02:26.690 INFO:tasks.workunit.client.0.smithi111.stdout:7/724: dwrite d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:26.693 INFO:tasks.workunit.client.0.smithi111.stdout:7/725: chown d0/d1/d5/d38/d5c/cad 2928 1 2024-03-14T13:02:26.693 INFO:tasks.workunit.client.1.smithi137.stdout:5/745: creat d7/d16/d18/d2d/dd1/ddc/ff5 x:0 0 0 2024-03-14T13:02:26.693 INFO:tasks.workunit.client.1.smithi137.stdout:3/672: dread - d4/de/d16/d27/f73 zero size 2024-03-14T13:02:26.695 INFO:tasks.workunit.client.1.smithi137.stdout:4/807: unlink d4/d27/d5c/d6d/cdc 0 2024-03-14T13:02:26.697 INFO:tasks.workunit.client.1.smithi137.stdout:7/825: symlink d3/d52/de8/l113 0 2024-03-14T13:02:26.698 INFO:tasks.workunit.client.1.smithi137.stdout:2/888: rmdir d2/d7/d2e/d3e/d44/d6b/d22/da5/de2 0 2024-03-14T13:02:26.699 INFO:tasks.workunit.client.0.smithi111.stdout:5/945: dread d2/d45/d76/d8d/fae [0,4194304] 0 2024-03-14T13:02:26.701 INFO:tasks.workunit.client.0.smithi111.stdout:7/726: dwrite d0/d1/d5/d38/d1b/f92 [0,4194304] 0 2024-03-14T13:02:26.701 INFO:tasks.workunit.client.1.smithi137.stdout:3/673: rmdir d4/de/dc7 39 2024-03-14T13:02:26.702 INFO:tasks.workunit.client.1.smithi137.stdout:5/746: mkdir d7/d16/d18/df6 0 2024-03-14T13:02:26.703 INFO:tasks.workunit.client.1.smithi137.stdout:8/895: dread d1/d5e/d11a/d82/de6/ff8 [0,4194304] 0 2024-03-14T13:02:26.704 INFO:tasks.workunit.client.1.smithi137.stdout:4/808: rmdir d4/d4d 39 2024-03-14T13:02:26.705 INFO:tasks.workunit.client.0.smithi111.stdout:7/727: fdatasync d0/d1/d5/d38/d5c/d6d/fa3 0 2024-03-14T13:02:26.707 INFO:tasks.workunit.client.0.smithi111.stdout:1/968: dread d20/d25/f3d [0,4194304] 0 2024-03-14T13:02:26.709 INFO:tasks.workunit.client.0.smithi111.stdout:7/728: mknod d0/d11/da8/dd4/cd6 0 2024-03-14T13:02:26.710 INFO:tasks.workunit.client.1.smithi137.stdout:0/788: getdents d1/d5/d1b/d1e 0 2024-03-14T13:02:26.710 INFO:tasks.workunit.client.0.smithi111.stdout:1/969: dread d20/d21/f5f [0,4194304] 0 2024-03-14T13:02:26.711 INFO:tasks.workunit.client.0.smithi111.stdout:1/970: write fe [2954706,58093] 0 2024-03-14T13:02:26.715 INFO:tasks.workunit.client.1.smithi137.stdout:1/942: dread d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:26.716 INFO:tasks.workunit.client.1.smithi137.stdout:7/826: read d3/d6/d59/daa/dcf/d6a/d75/d78/fee [3930793,32735] 0 2024-03-14T13:02:26.716 INFO:tasks.workunit.client.1.smithi137.stdout:2/889: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d9d/f10e [0,4194304] 0 2024-03-14T13:02:26.717 INFO:tasks.workunit.client.1.smithi137.stdout:5/747: mkdir d7/d1d/d74/df7 0 2024-03-14T13:02:26.719 INFO:tasks.workunit.client.1.smithi137.stdout:0/789: rmdir d1/d5/d1b/d1e/d25/d8a/d4a/d54/de2 0 2024-03-14T13:02:26.721 INFO:tasks.workunit.client.1.smithi137.stdout:1/943: symlink d7/d9/d19/d40/d42/d9d/l143 0 2024-03-14T13:02:26.723 INFO:tasks.workunit.client.0.smithi111.stdout:7/729: creat d0/d1/d21/fd7 x:0 0 0 2024-03-14T13:02:26.723 INFO:tasks.workunit.client.0.smithi111.stdout:9/804: dread d5/d6/d8/f12 [0,4194304] 0 2024-03-14T13:02:26.723 INFO:tasks.workunit.client.1.smithi137.stdout:0/790: mknod d1/d5/d1b/d1e/d25/d8a/d4a/c109 0 2024-03-14T13:02:26.725 INFO:tasks.workunit.client.1.smithi137.stdout:7/827: dread d3/d6/d2e/f35 [0,4194304] 0 2024-03-14T13:02:26.727 INFO:tasks.workunit.client.0.smithi111.stdout:9/805: fdatasync d5/d6/d2a/f3e 0 2024-03-14T13:02:26.728 INFO:tasks.workunit.client.1.smithi137.stdout:5/748: unlink d7/d1d/d74/d83/ff3 0 2024-03-14T13:02:26.728 INFO:tasks.workunit.client.0.smithi111.stdout:1/971: dwrite d20/d40/f84 [0,4194304] 0 2024-03-14T13:02:26.729 INFO:tasks.workunit.client.1.smithi137.stdout:1/944: dwrite d7/d9/d19/d40/f5b [0,4194304] 0 2024-03-14T13:02:26.730 INFO:tasks.workunit.client.1.smithi137.stdout:7/828: write d3/d5/d23/f2b [1232874,39425] 0 2024-03-14T13:02:26.733 INFO:tasks.workunit.client.1.smithi137.stdout:0/791: link d1/d5/ff9 d1/d44/d72/de8/f10a 0 2024-03-14T13:02:26.736 INFO:tasks.workunit.client.1.smithi137.stdout:1/945: rename d7/d4d/f108 to d7/d9/d19/d40/d50/d11b/f144 0 2024-03-14T13:02:26.746 INFO:tasks.workunit.client.1.smithi137.stdout:3/674: dread d4/de/d16/d1c/d5d/d85/d6b/f8b [0,4194304] 0 2024-03-14T13:02:26.749 INFO:tasks.workunit.client.1.smithi137.stdout:7/829: dwrite d3/d5/d68/ff1 [0,4194304] 0 2024-03-14T13:02:26.749 INFO:tasks.workunit.client.1.smithi137.stdout:9/905: sync 2024-03-14T13:02:26.751 INFO:tasks.workunit.client.0.smithi111.stdout:5/946: truncate d2/d16/d49/d7c/d95/f5e 343755 0 2024-03-14T13:02:26.751 INFO:tasks.workunit.client.0.smithi111.stdout:2/699: dwrite dd/d17/d34/f36 [0,4194304] 0 2024-03-14T13:02:26.755 INFO:tasks.workunit.client.1.smithi137.stdout:0/792: getdents d1/d5/d1b/d1e/d25/d9f 0 2024-03-14T13:02:26.756 INFO:tasks.workunit.client.0.smithi111.stdout:3/807: sync 2024-03-14T13:02:26.757 INFO:tasks.workunit.client.0.smithi111.stdout:3/808: chown d0/d2d/d35/d4c/d82/fab 1 1 2024-03-14T13:02:26.759 INFO:tasks.workunit.client.1.smithi137.stdout:7/830: chown d3/d6/d59/l62 1 1 2024-03-14T13:02:26.759 INFO:tasks.workunit.client.1.smithi137.stdout:7/831: stat d3/d5/d23/d3f/d5b/l69 0 2024-03-14T13:02:26.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:26 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2372642969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:26.765 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:26 smithi137 ceph-mon[50336]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:26.765 INFO:tasks.workunit.client.1.smithi137.stdout:8/896: write d1/dc/f5f [802048,25736] 0 2024-03-14T13:02:26.765 INFO:tasks.workunit.client.1.smithi137.stdout:0/793: dread - d1/d5/d1b/d1e/d25/d9f/fce zero size 2024-03-14T13:02:26.765 INFO:tasks.workunit.client.1.smithi137.stdout:1/946: creat d7/d9/d19/d40/d42/f145 x:0 0 0 2024-03-14T13:02:26.770 INFO:tasks.workunit.client.1.smithi137.stdout:1/947: fsync d7/d29/fc9 0 2024-03-14T13:02:26.772 INFO:tasks.workunit.client.1.smithi137.stdout:8/897: write d1/d5e/daf/fc2 [1139371,116240] 0 2024-03-14T13:02:26.780 INFO:tasks.workunit.client.0.smithi111.stdout:9/806: rmdir d5/d6/d8/d1a 39 2024-03-14T13:02:26.781 INFO:tasks.workunit.client.0.smithi111.stdout:1/972: mknod d20/d40/da6/c149 0 2024-03-14T13:02:26.782 INFO:tasks.workunit.client.1.smithi137.stdout:2/890: dwrite d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/df0/f122 [0,4194304] 0 2024-03-14T13:02:26.785 INFO:tasks.workunit.client.0.smithi111.stdout:7/730: dwrite d0/d11/f73 [0,4194304] 0 2024-03-14T13:02:26.785 INFO:tasks.workunit.client.1.smithi137.stdout:4/809: dwrite d4/d10/d42/d73/d9d/dbc/fb9 [0,4194304] 0 2024-03-14T13:02:26.787 INFO:tasks.workunit.client.0.smithi111.stdout:5/947: rename d2/d16/d49/d44/d4a to d2/d45/d76/d9a/d9f/d12e 0 2024-03-14T13:02:26.787 INFO:tasks.workunit.client.1.smithi137.stdout:6/795: sync 2024-03-14T13:02:26.787 INFO:tasks.workunit.client.0.smithi111.stdout:3/809: creat d0/d2d/dc7/f127 x:0 0 0 2024-03-14T13:02:26.789 INFO:tasks.workunit.client.0.smithi111.stdout:9/807: creat d5/d6/d14/d7a/f118 x:0 0 0 2024-03-14T13:02:26.794 INFO:tasks.workunit.client.1.smithi137.stdout:5/749: dread d7/d49/d75/f80 [0,4194304] 0 2024-03-14T13:02:26.794 INFO:tasks.workunit.client.0.smithi111.stdout:1/973: creat d20/d25/da8/f14a x:0 0 0 2024-03-14T13:02:26.795 INFO:tasks.workunit.client.0.smithi111.stdout:5/948: symlink d2/d16/d49/d4c/d9c/l12f 0 2024-03-14T13:02:26.800 INFO:tasks.workunit.client.0.smithi111.stdout:9/808: creat d5/d42/f119 x:0 0 0 2024-03-14T13:02:26.802 INFO:tasks.workunit.client.0.smithi111.stdout:5/949: rename d2/d16/d49/d7c/d8b to d2/d16/d49/d7c/d117/d130 0 2024-03-14T13:02:26.802 INFO:tasks.workunit.client.1.smithi137.stdout:3/675: dwrite d4/de/dc7/fcf [4194304,4194304] 0 2024-03-14T13:02:26.803 INFO:tasks.workunit.client.0.smithi111.stdout:1/974: creat d20/d40/db4/f14b x:0 0 0 2024-03-14T13:02:26.805 INFO:tasks.workunit.client.0.smithi111.stdout:1/975: symlink d20/dca/l14c 0 2024-03-14T13:02:26.808 INFO:tasks.workunit.client.0.smithi111.stdout:1/976: write d20/d21/d3f/f10a [874802,47343] 0 2024-03-14T13:02:26.808 INFO:tasks.workunit.client.1.smithi137.stdout:4/810: dwrite d4/d10/d42/f52 [0,4194304] 0 2024-03-14T13:02:26.812 INFO:tasks.workunit.client.0.smithi111.stdout:5/950: dwrite d2/d16/d49/d44/d72/de0/f120 [0,4194304] 0 2024-03-14T13:02:26.813 INFO:tasks.workunit.client.0.smithi111.stdout:6/708: sync 2024-03-14T13:02:26.813 INFO:tasks.workunit.client.1.smithi137.stdout:9/906: dread d6/df/d19/d1f/f35 [4194304,4194304] 0 2024-03-14T13:02:26.815 INFO:tasks.workunit.client.0.smithi111.stdout:5/951: chown d2/d16/d49/d44/d72/de0/ce8 2057213 1 2024-03-14T13:02:26.816 INFO:tasks.workunit.client.0.smithi111.stdout:1/977: dread d20/d25/d75/df7/f101 [0,4194304] 0 2024-03-14T13:02:26.819 INFO:tasks.workunit.client.0.smithi111.stdout:1/978: write d20/d25/d31/dff/db6/ff2 [566051,124152] 0 2024-03-14T13:02:26.819 INFO:tasks.workunit.client.0.smithi111.stdout:5/952: unlink d2/d45/d79/c89 0 2024-03-14T13:02:26.821 INFO:tasks.workunit.client.0.smithi111.stdout:9/809: dread d5/d6/fd5 [0,4194304] 0 2024-03-14T13:02:26.822 INFO:tasks.workunit.client.1.smithi137.stdout:4/811: read d4/d9/d3a/d41/d6a/fb3 [1348812,52733] 0 2024-03-14T13:02:26.823 INFO:tasks.workunit.client.0.smithi111.stdout:6/709: dwrite d0/d50/f5f [0,4194304] 0 2024-03-14T13:02:26.824 INFO:tasks.workunit.client.0.smithi111.stdout:1/979: mkdir d20/d40/db4/ddf/d14d 0 2024-03-14T13:02:26.824 INFO:tasks.workunit.client.0.smithi111.stdout:6/710: chown d0/l27 44 1 2024-03-14T13:02:26.826 INFO:tasks.workunit.client.1.smithi137.stdout:4/812: truncate d4/f89 1222632 0 2024-03-14T13:02:26.829 INFO:tasks.workunit.client.0.smithi111.stdout:5/953: unlink d2/d16/d49/d7c/d95/fa3 0 2024-03-14T13:02:26.830 INFO:tasks.workunit.client.0.smithi111.stdout:5/954: fdatasync d2/d45/d76/f12a 0 2024-03-14T13:02:26.833 INFO:tasks.workunit.client.0.smithi111.stdout:9/810: mknod d5/d6/d8/c11a 0 2024-03-14T13:02:26.834 INFO:tasks.workunit.client.0.smithi111.stdout:6/711: creat d0/dfb/df5/f105 x:0 0 0 2024-03-14T13:02:26.836 INFO:tasks.workunit.client.0.smithi111.stdout:6/712: chown d0/dfb/cce 15 1 2024-03-14T13:02:26.837 INFO:tasks.workunit.client.1.smithi137.stdout:4/813: dwrite d4/fe1 [0,4194304] 0 2024-03-14T13:02:26.838 INFO:tasks.workunit.client.0.smithi111.stdout:9/811: chown d5/d42/d52/c61 235 1 2024-03-14T13:02:26.838 INFO:tasks.workunit.client.0.smithi111.stdout:1/980: link d20/d25/d87/la5 d20/d21/da7/l14e 0 2024-03-14T13:02:26.839 INFO:tasks.workunit.client.0.smithi111.stdout:6/713: rmdir d0/d17 39 2024-03-14T13:02:26.842 INFO:tasks.workunit.client.0.smithi111.stdout:6/714: mkdir d0/dfb/d106 0 2024-03-14T13:02:26.844 INFO:tasks.workunit.client.1.smithi137.stdout:9/907: dread d6/df/d19/d33/d8f/fb2 [0,4194304] 0 2024-03-14T13:02:26.844 INFO:tasks.workunit.client.0.smithi111.stdout:1/981: symlink d20/d25/da8/de9/l14f 0 2024-03-14T13:02:26.853 INFO:tasks.workunit.client.0.smithi111.stdout:9/812: dwrite d5/d6/d14/f115 [0,4194304] 0 2024-03-14T13:02:26.855 INFO:tasks.workunit.client.0.smithi111.stdout:1/982: truncate d20/d21/d3f/fa3 4861508 0 2024-03-14T13:02:26.856 INFO:tasks.workunit.client.1.smithi137.stdout:8/898: rmdir d1/d5e/daf/dee 39 2024-03-14T13:02:26.857 INFO:tasks.workunit.client.1.smithi137.stdout:8/899: dread - d1/d5e/d11a/d82/de6/f128 zero size 2024-03-14T13:02:26.857 INFO:tasks.workunit.client.0.smithi111.stdout:5/955: dread d2/d16/d49/d7c/d117/d130/f91 [0,4194304] 0 2024-03-14T13:02:26.859 INFO:tasks.workunit.client.1.smithi137.stdout:1/948: creat d7/d9/d19/d1f/d2e/da6/ddd/f146 x:0 0 0 2024-03-14T13:02:26.860 INFO:tasks.workunit.client.0.smithi111.stdout:2/700: dread dd/d17/d34/d73/d49/f67 [0,4194304] 0 2024-03-14T13:02:26.860 INFO:tasks.workunit.client.1.smithi137.stdout:0/794: mkdir d1/d5/d1b/d1e/d25/d26/d10b 0 2024-03-14T13:02:26.864 INFO:tasks.workunit.client.1.smithi137.stdout:6/796: mknod d2/d6/d1d/c10e 0 2024-03-14T13:02:26.866 INFO:tasks.workunit.client.0.smithi111.stdout:6/715: mknod d0/d17/d21/d69/d3d/d53/d81/da1/c107 0 2024-03-14T13:02:26.868 INFO:tasks.workunit.client.1.smithi137.stdout:2/891: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/lc7 209 1 2024-03-14T13:02:26.869 INFO:tasks.workunit.client.0.smithi111.stdout:5/956: dread - d2/d16/d49/d4c/f7d zero size 2024-03-14T13:02:26.870 INFO:tasks.workunit.client.0.smithi111.stdout:3/810: dread d0/d2d/d35/d50/d6c/d98/fd6 [0,4194304] 0 2024-03-14T13:02:26.871 INFO:tasks.workunit.client.0.smithi111.stdout:2/701: fdatasync dd/d8d/f9b 0 2024-03-14T13:02:26.874 INFO:tasks.workunit.client.0.smithi111.stdout:2/702: readlink dd/d17/d34/d73/l8f 0 2024-03-14T13:02:26.874 INFO:tasks.workunit.client.1.smithi137.stdout:9/908: chown d6/d91/dc8/f104 15 1 2024-03-14T13:02:26.876 INFO:tasks.workunit.client.0.smithi111.stdout:9/813: mknod d5/d42/d71/dbd/d107/c11b 0 2024-03-14T13:02:26.877 INFO:tasks.workunit.client.0.smithi111.stdout:7/731: write d0/d1/d5/d56/f7c [1109629,42230] 0 2024-03-14T13:02:26.877 INFO:tasks.workunit.client.0.smithi111.stdout:6/716: truncate d0/f35 393852 0 2024-03-14T13:02:26.879 INFO:tasks.workunit.client.0.smithi111.stdout:2/703: creat dd/d17/d53/fe1 x:0 0 0 2024-03-14T13:02:26.879 INFO:tasks.workunit.client.0.smithi111.stdout:9/814: chown d5/d6/d2a/d2d 286 1 2024-03-14T13:02:26.880 INFO:tasks.workunit.client.0.smithi111.stdout:5/957: creat d2/d16/d49/d7c/d95/d6a/f131 x:0 0 0 2024-03-14T13:02:26.881 INFO:tasks.workunit.client.1.smithi137.stdout:5/750: getdents d7/d1d/d74/df7 0 2024-03-14T13:02:26.885 INFO:tasks.workunit.client.1.smithi137.stdout:0/795: dwrite d1/d44/d56/f5f [0,4194304] 0 2024-03-14T13:02:26.886 INFO:tasks.workunit.client.0.smithi111.stdout:6/717: rename d0/d17/d20/d94/daa to d0/dfb/d106/d108 0 2024-03-14T13:02:26.888 INFO:tasks.workunit.client.1.smithi137.stdout:9/909: creat d6/d91/dc8/f12c x:0 0 0 2024-03-14T13:02:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:26 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2372642969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:26 smithi111 ceph-mon[39688]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:26.890 INFO:tasks.workunit.client.0.smithi111.stdout:6/718: stat d0/f4 0 2024-03-14T13:02:26.896 INFO:tasks.workunit.client.1.smithi137.stdout:0/796: creat d1/d5/d1b/d42/d3a/dc6/f10c x:0 0 0 2024-03-14T13:02:26.897 INFO:tasks.workunit.client.1.smithi137.stdout:7/832: getdents d3/d5/d5a/d86 0 2024-03-14T13:02:26.902 INFO:tasks.workunit.client.1.smithi137.stdout:1/949: getdents d7/d9/d19/d1f/d89/d90/db8 0 2024-03-14T13:02:26.904 INFO:tasks.workunit.client.1.smithi137.stdout:7/833: chown d3/d52/d77/c9a 3797 1 2024-03-14T13:02:26.904 INFO:tasks.workunit.client.0.smithi111.stdout:9/815: creat d5/d6/d8/d5b/f11c x:0 0 0 2024-03-14T13:02:26.904 INFO:tasks.workunit.client.1.smithi137.stdout:5/751: creat d7/d16/d18/d25/d28/d3c/d76/ff8 x:0 0 0 2024-03-14T13:02:26.906 INFO:tasks.workunit.client.0.smithi111.stdout:7/732: dread d0/d1/d4a/d52/f62 [0,4194304] 0 2024-03-14T13:02:26.907 INFO:tasks.workunit.client.0.smithi111.stdout:5/958: rmdir d2/d16/d49/d7c/d12d 0 2024-03-14T13:02:26.908 INFO:tasks.workunit.client.0.smithi111.stdout:1/983: write d20/f44 [280157,5777] 0 2024-03-14T13:02:26.909 INFO:tasks.workunit.client.1.smithi137.stdout:9/910: dread d6/df/d88/de5/df6/d40/f41 [0,4194304] 0 2024-03-14T13:02:26.909 INFO:tasks.workunit.client.1.smithi137.stdout:1/950: write d7/d9/d19/d1f/d2e/da6/d129/f134 [769580,103495] 0 2024-03-14T13:02:26.909 INFO:tasks.workunit.client.1.smithi137.stdout:0/797: dwrite d1/d44/f65 [0,4194304] 0 2024-03-14T13:02:26.911 INFO:tasks.workunit.client.0.smithi111.stdout:6/719: getdents d0/dfb/db8/de9 0 2024-03-14T13:02:26.913 INFO:tasks.workunit.client.0.smithi111.stdout:7/733: chown d0/d1/d21/d2a/cb9 41524 1 2024-03-14T13:02:26.913 INFO:tasks.workunit.client.1.smithi137.stdout:7/834: mknod d3/d6/d64/d9e/c114 0 2024-03-14T13:02:26.914 INFO:tasks.workunit.client.0.smithi111.stdout:2/704: dwrite dd/d17/d34/d73/d75/f95 [0,4194304] 0 2024-03-14T13:02:26.916 INFO:tasks.workunit.client.0.smithi111.stdout:7/734: write d0/d1/d5/d38/d5c/d6d/f9e [860369,126999] 0 2024-03-14T13:02:26.922 INFO:tasks.workunit.client.0.smithi111.stdout:3/811: dwrite d0/d2d/d35/d50/d6c/da3/daa/fb1 [0,4194304] 0 2024-03-14T13:02:26.925 INFO:tasks.workunit.client.0.smithi111.stdout:3/812: truncate d0/d2d/d35/d50/da2/feb 294984 0 2024-03-14T13:02:26.926 INFO:tasks.workunit.client.0.smithi111.stdout:5/959: creat d2/d45/d76/f132 x:0 0 0 2024-03-14T13:02:26.928 INFO:tasks.workunit.client.1.smithi137.stdout:2/892: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/f5d [0,4194304] 0 2024-03-14T13:02:26.928 INFO:tasks.workunit.client.0.smithi111.stdout:5/960: dread - d2/d45/d76/f12a zero size 2024-03-14T13:02:26.929 INFO:tasks.workunit.client.0.smithi111.stdout:3/813: write d0/d2d/d35/d50/d6c/da3/daa/fd1 [1748901,87314] 0 2024-03-14T13:02:26.930 INFO:tasks.workunit.client.1.smithi137.stdout:0/798: stat d1/d5/d1b/d1e/d25/d26 0 2024-03-14T13:02:26.931 INFO:tasks.workunit.client.0.smithi111.stdout:7/735: dwrite d0/f2e [0,4194304] 0 2024-03-14T13:02:26.935 INFO:tasks.workunit.client.1.smithi137.stdout:0/799: chown d1/d5/d1b/d42/f58 19149526 1 2024-03-14T13:02:26.935 INFO:tasks.workunit.client.1.smithi137.stdout:1/951: chown d7/d9/d19/d1f/d2e/f79 83 1 2024-03-14T13:02:26.937 INFO:tasks.workunit.client.1.smithi137.stdout:1/952: fsync d7/d4d/db7/f11d 0 2024-03-14T13:02:26.939 INFO:tasks.workunit.client.0.smithi111.stdout:3/814: read d0/d2d/d35/f61 [851708,6055] 0 2024-03-14T13:02:26.940 INFO:tasks.workunit.client.1.smithi137.stdout:7/835: readlink d3/d6/d2e/l99 0 2024-03-14T13:02:26.940 INFO:tasks.workunit.client.0.smithi111.stdout:7/736: mknod d0/d1/d4a/dc7/cd8 0 2024-03-14T13:02:26.943 INFO:tasks.workunit.client.1.smithi137.stdout:5/752: dwrite d7/d36/f95 [0,4194304] 0 2024-03-14T13:02:26.951 INFO:tasks.workunit.client.1.smithi137.stdout:9/911: dwrite d6/df/d88/df4/f11e [0,4194304] 0 2024-03-14T13:02:26.952 INFO:tasks.workunit.client.0.smithi111.stdout:1/984: dread d20/d21/d3f/f6f [0,4194304] 0 2024-03-14T13:02:26.955 INFO:tasks.workunit.client.0.smithi111.stdout:7/737: link d0/d1/d21/d2a/c2d d0/d1/d21/d5d/cd9 0 2024-03-14T13:02:26.956 INFO:tasks.workunit.client.0.smithi111.stdout:2/705: write dd/d17/d34/d73/d49/f67 [3520936,7878] 0 2024-03-14T13:02:26.958 INFO:tasks.workunit.client.1.smithi137.stdout:5/753: dread d7/d16/d18/d25/fe2 [0,4194304] 0 2024-03-14T13:02:26.961 INFO:tasks.workunit.client.0.smithi111.stdout:1/985: write d20/d21/d3f/fa3 [922128,84992] 0 2024-03-14T13:02:26.961 INFO:tasks.workunit.client.1.smithi137.stdout:8/900: mkdir d1/dc/d71/d12c 0 2024-03-14T13:02:26.962 INFO:tasks.workunit.client.0.smithi111.stdout:6/720: dwrite d0/d17/d20/f4b [0,4194304] 0 2024-03-14T13:02:26.963 INFO:tasks.workunit.client.0.smithi111.stdout:7/738: symlink d0/d1/d21/d5d/lda 0 2024-03-14T13:02:26.969 INFO:tasks.workunit.client.0.smithi111.stdout:2/706: creat dd/d17/d34/d73/d49/fe2 x:0 0 0 2024-03-14T13:02:26.978 INFO:tasks.workunit.client.0.smithi111.stdout:6/721: fdatasync d0/f57 0 2024-03-14T13:02:26.979 INFO:tasks.workunit.client.0.smithi111.stdout:9/816: dwrite d5/f20 [0,4194304] 0 2024-03-14T13:02:26.979 INFO:tasks.workunit.client.1.smithi137.stdout:1/953: creat d7/d9/d19/d1f/d26/d7d/f147 x:0 0 0 2024-03-14T13:02:26.979 INFO:tasks.workunit.client.0.smithi111.stdout:2/707: readlink dd/d17/d34/ddb/l81 0 2024-03-14T13:02:26.980 INFO:tasks.workunit.client.1.smithi137.stdout:7/836: creat d3/d5/d68/f115 x:0 0 0 2024-03-14T13:02:26.987 INFO:tasks.workunit.client.1.smithi137.stdout:5/754: fdatasync d7/d16/d18/d2d/f53 0 2024-03-14T13:02:26.987 INFO:tasks.workunit.client.1.smithi137.stdout:1/954: creat d7/d92/f148 x:0 0 0 2024-03-14T13:02:26.988 INFO:tasks.workunit.client.0.smithi111.stdout:5/961: dread d2/d16/d49/f27 [0,4194304] 0 2024-03-14T13:02:26.990 INFO:tasks.workunit.client.0.smithi111.stdout:2/708: symlink dd/d11/d8b/le3 0 2024-03-14T13:02:26.992 INFO:tasks.workunit.client.0.smithi111.stdout:1/986: dwrite d20/d25/d75/f5e [0,4194304] 0 2024-03-14T13:02:26.992 INFO:tasks.workunit.client.0.smithi111.stdout:7/739: dwrite d0/f72 [0,4194304] 0 2024-03-14T13:02:26.993 INFO:tasks.workunit.client.1.smithi137.stdout:7/837: dwrite d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fb7 [0,4194304] 0 2024-03-14T13:02:26.995 INFO:tasks.workunit.client.1.smithi137.stdout:3/676: dwrite d4/df/f83 [0,4194304] 0 2024-03-14T13:02:26.995 INFO:tasks.workunit.client.1.smithi137.stdout:8/901: unlink d1/d5e/d11a/fb4 0 2024-03-14T13:02:26.996 INFO:tasks.workunit.client.0.smithi111.stdout:9/817: creat d5/d6/d8/d5b/f11d x:0 0 0 2024-03-14T13:02:26.998 INFO:tasks.workunit.client.0.smithi111.stdout:5/962: symlink d2/d16/d49/d7c/d117/l133 0 2024-03-14T13:02:27.001 INFO:tasks.workunit.client.1.smithi137.stdout:3/677: readlink d4/de/lcc 0 2024-03-14T13:02:27.004 INFO:tasks.workunit.client.1.smithi137.stdout:4/814: sync 2024-03-14T13:02:27.005 INFO:tasks.workunit.client.0.smithi111.stdout:5/963: creat d2/d16/d49/d33/d68/f134 x:0 0 0 2024-03-14T13:02:27.005 INFO:tasks.workunit.client.0.smithi111.stdout:3/815: dwrite d0/d2d/d35/d50/f88 [0,4194304] 0 2024-03-14T13:02:27.006 INFO:tasks.workunit.client.1.smithi137.stdout:0/800: mkdir d1/d5/d1b/d1e/d25/d10d 0 2024-03-14T13:02:27.006 INFO:tasks.workunit.client.0.smithi111.stdout:9/818: chown d5/d6/c36 15077243 1 2024-03-14T13:02:27.010 INFO:tasks.workunit.client.0.smithi111.stdout:7/740: dread d0/f2e [0,4194304] 0 2024-03-14T13:02:27.011 INFO:tasks.workunit.client.1.smithi137.stdout:1/955: link d7/d9/d19/d1f/f81 d7/d9/d19/d40/d42/d9d/de5/f149 0 2024-03-14T13:02:27.011 INFO:tasks.workunit.client.1.smithi137.stdout:8/902: mkdir d1/d4a/d12d 0 2024-03-14T13:02:27.012 INFO:tasks.workunit.client.1.smithi137.stdout:3/678: dread - d4/de/d16/d1c/d8d/dcd/fa2 zero size 2024-03-14T13:02:27.013 INFO:tasks.workunit.client.1.smithi137.stdout:1/956: dread - d7/d4d/d6b/f12e zero size 2024-03-14T13:02:27.013 INFO:tasks.workunit.client.1.smithi137.stdout:4/815: creat d4/d27/d100/d8d/f106 x:0 0 0 2024-03-14T13:02:27.018 INFO:tasks.workunit.client.1.smithi137.stdout:7/838: dread d3/d6/d14/d27/dfe/f100 [0,4194304] 0 2024-03-14T13:02:27.018 INFO:tasks.workunit.client.0.smithi111.stdout:6/722: dwrite d0/d17/d21/f79 [4194304,4194304] 0 2024-03-14T13:02:27.019 INFO:tasks.workunit.client.1.smithi137.stdout:3/679: mkdir d4/de/dc7/ddf 0 2024-03-14T13:02:27.019 INFO:tasks.workunit.client.1.smithi137.stdout:7/839: chown d3/d5/f7c 3189932 1 2024-03-14T13:02:27.020 INFO:tasks.workunit.client.1.smithi137.stdout:0/801: truncate d1/f17 4508113 0 2024-03-14T13:02:27.020 INFO:tasks.workunit.client.1.smithi137.stdout:7/840: stat d3/d52 0 2024-03-14T13:02:27.021 INFO:tasks.workunit.client.0.smithi111.stdout:3/816: creat d0/d2d/d35/d69/d10a/f128 x:0 0 0 2024-03-14T13:02:27.022 INFO:tasks.workunit.client.1.smithi137.stdout:6/797: mknod d2/c10f 0 2024-03-14T13:02:27.024 INFO:tasks.workunit.client.1.smithi137.stdout:7/841: chown d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fef 31462442 1 2024-03-14T13:02:27.024 INFO:tasks.workunit.client.1.smithi137.stdout:8/903: creat d1/dc/d71/db0/f12e x:0 0 0 2024-03-14T13:02:27.025 INFO:tasks.workunit.client.1.smithi137.stdout:0/802: write d1/d5/d1b/d1e/d25/d55/f7a [7169921,117552] 0 2024-03-14T13:02:27.025 INFO:tasks.workunit.client.1.smithi137.stdout:3/680: unlink d4/df/d82/fb5 0 2024-03-14T13:02:27.027 INFO:tasks.workunit.client.1.smithi137.stdout:4/816: mknod d4/d10/dfa/c107 0 2024-03-14T13:02:27.028 INFO:tasks.workunit.client.0.smithi111.stdout:7/741: mkdir d0/d1/d5/d38/d5c/d6d/ddb 0 2024-03-14T13:02:27.028 INFO:tasks.workunit.client.0.smithi111.stdout:9/819: link d5/d6/d8/d74/ce3 d5/d6/d8/d74/c11e 0 2024-03-14T13:02:27.029 INFO:tasks.workunit.client.1.smithi137.stdout:8/904: mkdir d1/dc/da9/d12f 0 2024-03-14T13:02:27.030 INFO:tasks.workunit.client.0.smithi111.stdout:3/817: dwrite d0/d2d/d35/d50/da2/db3/da7/fe7 [0,4194304] 0 2024-03-14T13:02:27.032 INFO:tasks.workunit.client.0.smithi111.stdout:2/709: dread dd/d17/f3a [0,4194304] 0 2024-03-14T13:02:27.032 INFO:tasks.workunit.client.1.smithi137.stdout:0/803: write d1/d5/d1b/d1e/d53/d6d/fdf [1518033,61774] 0 2024-03-14T13:02:27.033 INFO:tasks.workunit.client.1.smithi137.stdout:4/817: creat d4/d27/d5c/d77/f108 x:0 0 0 2024-03-14T13:02:27.033 INFO:tasks.workunit.client.0.smithi111.stdout:7/742: symlink d0/d1/d5/d38/d5c/d6d/dc9/dce/ldc 0 2024-03-14T13:02:27.033 INFO:tasks.workunit.client.0.smithi111.stdout:2/710: stat dd/d17/d34/l23 0 2024-03-14T13:02:27.035 INFO:tasks.workunit.client.0.smithi111.stdout:2/711: read dd/d11/d1a/f38 [3794892,84917] 0 2024-03-14T13:02:27.036 INFO:tasks.workunit.client.0.smithi111.stdout:2/712: dread - dd/d8d/f9c zero size 2024-03-14T13:02:27.037 INFO:tasks.workunit.client.0.smithi111.stdout:9/820: link d5/d6/d8/d1a/f64 d5/f11f 0 2024-03-14T13:02:27.039 INFO:tasks.workunit.client.0.smithi111.stdout:3/818: truncate d0/d2d/d35/d4c/dad/dee/ff6 422279 0 2024-03-14T13:02:27.042 INFO:tasks.workunit.client.1.smithi137.stdout:3/681: chown d4/de/d16/d1c/d8d/dcd/fc6 0 1 2024-03-14T13:02:27.044 INFO:tasks.workunit.client.1.smithi137.stdout:3/682: readlink d4/de/d16/d1c/d5d/ld6 0 2024-03-14T13:02:27.046 INFO:tasks.workunit.client.0.smithi111.stdout:7/743: dwrite d0/d1/d5/d38/d5c/fc0 [0,4194304] 0 2024-03-14T13:02:27.046 INFO:tasks.workunit.client.0.smithi111.stdout:7/744: stat d0/d1/d21/d2a/f93 0 2024-03-14T13:02:27.047 INFO:tasks.workunit.client.0.smithi111.stdout:2/713: creat dd/fe4 x:0 0 0 2024-03-14T13:02:27.047 INFO:tasks.workunit.client.1.smithi137.stdout:0/804: creat d1/d5/d1b/d42/f10e x:0 0 0 2024-03-14T13:02:27.048 INFO:tasks.workunit.client.1.smithi137.stdout:3/683: read d4/de/d13/f21 [270745,111040] 0 2024-03-14T13:02:27.049 INFO:tasks.workunit.client.0.smithi111.stdout:3/819: mknod d0/d2d/d35/d50/da2/db3/d6f/d10b/c129 0 2024-03-14T13:02:27.049 INFO:tasks.workunit.client.1.smithi137.stdout:2/893: rename d2/d7/d2e/d3e/d44/d6b/f1b to d2/d7/d2e/d3e/d44/d6b/d22/d26/f124 0 2024-03-14T13:02:27.050 INFO:tasks.workunit.client.0.smithi111.stdout:7/745: write d0/d1/d5/d56/fc2 [672829,50308] 0 2024-03-14T13:02:27.050 INFO:tasks.workunit.client.0.smithi111.stdout:3/820: chown d0/d2d/d35/d69/f110 11444 1 2024-03-14T13:02:27.051 INFO:tasks.workunit.client.1.smithi137.stdout:0/805: write d1/d5/d1b/d42/d3a/dc6/fea [1847603,119935] 0 2024-03-14T13:02:27.053 INFO:tasks.workunit.client.0.smithi111.stdout:3/821: fsync d0/d2d/d35/d50/da2/db3/d71/dbc/fd0 0 2024-03-14T13:02:27.054 INFO:tasks.workunit.client.0.smithi111.stdout:1/987: write d20/d25/d31/f7f [1172050,86168] 0 2024-03-14T13:02:27.058 INFO:tasks.workunit.client.1.smithi137.stdout:2/894: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d9d/d109/l125 0 2024-03-14T13:02:27.061 INFO:tasks.workunit.client.0.smithi111.stdout:1/988: creat d20/d25/d87/f150 x:0 0 0 2024-03-14T13:02:27.063 INFO:tasks.workunit.client.0.smithi111.stdout:6/723: dread d0/d17/d20/fb1 [0,4194304] 0 2024-03-14T13:02:27.063 INFO:tasks.workunit.client.0.smithi111.stdout:3/822: symlink d0/d30/df1/l12a 0 2024-03-14T13:02:27.063 INFO:tasks.workunit.client.1.smithi137.stdout:2/895: getdents d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d 0 2024-03-14T13:02:27.064 INFO:tasks.workunit.client.0.smithi111.stdout:7/746: dwrite d0/d1/d21/d5d/fac [0,4194304] 0 2024-03-14T13:02:27.065 INFO:tasks.workunit.client.0.smithi111.stdout:1/989: read - d20/d40/db4/f12b zero size 2024-03-14T13:02:27.067 INFO:tasks.workunit.client.0.smithi111.stdout:7/747: write d0/dc1/fc4 [915301,13121] 0 2024-03-14T13:02:27.069 INFO:tasks.workunit.client.0.smithi111.stdout:7/748: truncate d0/f72 4306372 0 2024-03-14T13:02:27.070 INFO:tasks.workunit.client.0.smithi111.stdout:2/714: write dd/d17/f71 [265315,30467] 0 2024-03-14T13:02:27.070 INFO:tasks.workunit.client.1.smithi137.stdout:7/842: rename d3/d5/d23/d2c/fd4 to d3/d5/f116 0 2024-03-14T13:02:27.073 INFO:tasks.workunit.client.0.smithi111.stdout:7/749: mknod d0/d1/d21/d2a/cdd 0 2024-03-14T13:02:27.074 INFO:tasks.workunit.client.1.smithi137.stdout:7/843: readlink d3/d5/d23/d2c/l3d 0 2024-03-14T13:02:27.074 INFO:tasks.workunit.client.1.smithi137.stdout:2/896: dwrite d2/d7/d2e/d3e/d44/d6b/d22/da5/fea [0,4194304] 0 2024-03-14T13:02:27.076 INFO:tasks.workunit.client.0.smithi111.stdout:6/724: dwrite d0/d17/d20/d4a/d59/d82/fe8 [0,4194304] 0 2024-03-14T13:02:27.078 INFO:tasks.workunit.client.1.smithi137.stdout:5/755: write d7/d16/f98 [1727421,3350] 0 2024-03-14T13:02:27.078 INFO:tasks.workunit.client.0.smithi111.stdout:7/750: unlink d0/d1/d5/d56/fc2 0 2024-03-14T13:02:27.078 INFO:tasks.workunit.client.0.smithi111.stdout:6/725: write d0/d17/d20/d4a/d59/dc3/fd6 [1704405,42285] 0 2024-03-14T13:02:27.084 INFO:tasks.workunit.client.0.smithi111.stdout:3/823: dread d0/d2d/d35/d50/da2/db3/d6f/f79 [0,4194304] 0 2024-03-14T13:02:27.085 INFO:tasks.workunit.client.0.smithi111.stdout:3/824: write d0/f23 [2367194,55484] 0 2024-03-14T13:02:27.086 INFO:tasks.workunit.client.0.smithi111.stdout:2/715: rename dd/d17/d34/d73/d49/c4d to dd/d11/ce5 0 2024-03-14T13:02:27.088 INFO:tasks.workunit.client.1.smithi137.stdout:6/798: rename d2/d54/d55/f1f to d2/d54/d55/d6d/d76/d57/d58/f110 0 2024-03-14T13:02:27.088 INFO:tasks.workunit.client.1.smithi137.stdout:5/756: stat d7/d49/c4e 0 2024-03-14T13:02:27.088 INFO:tasks.workunit.client.1.smithi137.stdout:9/912: write d6/df/d19/d33/f61 [1239710,1268] 0 2024-03-14T13:02:27.088 INFO:tasks.workunit.client.1.smithi137.stdout:9/913: mkdir d6/df/d71/d94/d12d 0 2024-03-14T13:02:27.088 INFO:tasks.workunit.client.1.smithi137.stdout:6/799: write d2/d6/fc [8755152,36147] 0 2024-03-14T13:02:27.090 INFO:tasks.workunit.client.1.smithi137.stdout:4/818: rename d4/d9/d3a/d41/d38/f53 to d4/d9/d3a/d41/d6a/dac/f109 0 2024-03-14T13:02:27.090 INFO:tasks.workunit.client.1.smithi137.stdout:0/806: dread d1/d44/d72/fca [0,4194304] 0 2024-03-14T13:02:27.090 INFO:tasks.workunit.client.0.smithi111.stdout:2/716: mkdir dd/d17/d34/dbf/de6 0 2024-03-14T13:02:27.091 INFO:tasks.workunit.client.0.smithi111.stdout:7/751: link d0/d1/lf d0/d7e/lde 0 2024-03-14T13:02:27.092 INFO:tasks.workunit.client.0.smithi111.stdout:2/717: symlink dd/d11/d1a/d56/dab/le7 0 2024-03-14T13:02:27.093 INFO:tasks.workunit.client.0.smithi111.stdout:7/752: symlink d0/d1/d5/d38/ldf 0 2024-03-14T13:02:27.095 INFO:tasks.workunit.client.1.smithi137.stdout:6/800: chown d2/d6/de/d45/d59/d39/d3c/fc8 164 1 2024-03-14T13:02:27.095 INFO:tasks.workunit.client.0.smithi111.stdout:7/753: dread - d0/d1/d21/d2a/d8c/fb1 zero size 2024-03-14T13:02:27.095 INFO:tasks.workunit.client.0.smithi111.stdout:2/718: creat dd/d17/d34/de0/fe8 x:0 0 0 2024-03-14T13:02:27.095 INFO:tasks.workunit.client.1.smithi137.stdout:7/844: dwrite d3/d5/d5a/d86/fed [0,4194304] 0 2024-03-14T13:02:27.097 INFO:tasks.workunit.client.0.smithi111.stdout:2/719: creat dd/d17/d34/d73/d49/fe9 x:0 0 0 2024-03-14T13:02:27.097 INFO:tasks.workunit.client.1.smithi137.stdout:6/801: chown d2/d6/d1d/cbe 2466169 1 2024-03-14T13:02:27.101 INFO:tasks.workunit.client.1.smithi137.stdout:2/897: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/f112 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/f126 0 2024-03-14T13:02:27.101 INFO:tasks.workunit.client.1.smithi137.stdout:5/757: fdatasync d7/d16/d18/d25/d28/d3c/d76/f79 0 2024-03-14T13:02:27.102 INFO:tasks.workunit.client.0.smithi111.stdout:7/754: dwrite d0/d1/d4a/d52/f62 [0,4194304] 0 2024-03-14T13:02:27.103 INFO:tasks.workunit.client.1.smithi137.stdout:4/819: dwrite d4/d9/f1f [4194304,4194304] 0 2024-03-14T13:02:27.103 INFO:tasks.workunit.client.1.smithi137.stdout:5/758: dread - d7/d16/d18/d25/d28/d9e/fc7 zero size 2024-03-14T13:02:27.105 INFO:tasks.workunit.client.0.smithi111.stdout:7/755: symlink d0/d1/d5/d56/d70/dd5/le0 0 2024-03-14T13:02:27.108 INFO:tasks.workunit.client.0.smithi111.stdout:7/756: unlink d0/d1/d5/fcf 0 2024-03-14T13:02:27.109 INFO:tasks.workunit.client.1.smithi137.stdout:6/802: truncate d2/d54/d55/f107 733416 0 2024-03-14T13:02:27.109 INFO:tasks.workunit.client.0.smithi111.stdout:7/757: mknod d0/d1/d4a/d52/ce1 0 2024-03-14T13:02:27.112 INFO:tasks.workunit.client.1.smithi137.stdout:6/803: dread - d2/d6/de/d45/f108 zero size 2024-03-14T13:02:27.117 INFO:tasks.workunit.client.1.smithi137.stdout:6/804: write d2/d54/d55/d6d/d76/d57/ff9 [474884,30851] 0 2024-03-14T13:02:27.117 INFO:tasks.workunit.client.1.smithi137.stdout:0/807: rename d1/c45 to d1/d1f/d80/c10f 0 2024-03-14T13:02:27.119 INFO:tasks.workunit.client.1.smithi137.stdout:4/820: mknod d4/d9/d3a/d41/dd7/c10a 0 2024-03-14T13:02:27.120 INFO:tasks.workunit.client.1.smithi137.stdout:2/898: mkdir d2/d7/d2e/d3e/d44/d6b/d1d/d3a/de5/d127 0 2024-03-14T13:02:27.121 INFO:tasks.workunit.client.1.smithi137.stdout:8/905: dwrite d1/dc/d77/d9f/fe5 [0,4194304] 0 2024-03-14T13:02:27.126 INFO:tasks.workunit.client.1.smithi137.stdout:5/759: mknod d7/d16/cf9 0 2024-03-14T13:02:27.128 INFO:tasks.workunit.client.1.smithi137.stdout:6/805: fsync d2/d54/d55/d6d/d76/d57/f74 0 2024-03-14T13:02:27.128 INFO:tasks.workunit.client.1.smithi137.stdout:3/684: dwrite d4/de/d16/d1c/d8d/dcd/fa2 [0,4194304] 0 2024-03-14T13:02:27.129 INFO:tasks.workunit.client.1.smithi137.stdout:0/808: creat d1/da6/f110 x:0 0 0 2024-03-14T13:02:27.131 INFO:tasks.workunit.client.1.smithi137.stdout:8/906: truncate d1/d5e/d11a/f9a 5132605 0 2024-03-14T13:02:27.137 INFO:tasks.workunit.client.1.smithi137.stdout:0/809: write d1/d5/d1b/d1e/d25/d26/f39 [8501427,10725] 0 2024-03-14T13:02:27.137 INFO:tasks.workunit.client.1.smithi137.stdout:4/821: getdents d4/d27/d100 0 2024-03-14T13:02:27.143 INFO:tasks.workunit.client.1.smithi137.stdout:0/810: symlink d1/d44/d72/de8/l111 0 2024-03-14T13:02:27.145 INFO:tasks.workunit.client.1.smithi137.stdout:6/806: dwrite d2/d54/d55/d6d/d76/d57/f9a [4194304,4194304] 0 2024-03-14T13:02:27.147 INFO:tasks.workunit.client.1.smithi137.stdout:0/811: write d1/d44/d72/d7e/fcb [115612,119460] 0 2024-03-14T13:02:27.147 INFO:tasks.workunit.client.1.smithi137.stdout:4/822: getdents d4/d27/d100 0 2024-03-14T13:02:27.148 INFO:tasks.workunit.client.0.smithi111.stdout:3/825: dwrite d0/d2d/d35/d69/d84/ff7 [0,4194304] 0 2024-03-14T13:02:27.148 INFO:tasks.workunit.client.1.smithi137.stdout:8/907: truncate d1/d5e/d11a/d82/f8d 3368367 0 2024-03-14T13:02:27.149 INFO:tasks.workunit.client.1.smithi137.stdout:5/760: dwrite d7/d16/d18/d2d/fd4 [0,4194304] 0 2024-03-14T13:02:27.152 INFO:tasks.workunit.client.1.smithi137.stdout:8/908: dread - d1/dc/d10/dcf/dd2/ff6 zero size 2024-03-14T13:02:27.153 INFO:tasks.workunit.client.1.smithi137.stdout:8/909: dread - d1/dc/d4e/d6d/f12a zero size 2024-03-14T13:02:27.154 INFO:tasks.workunit.client.1.smithi137.stdout:6/807: creat d2/d6/de/d45/d97/d9f/f111 x:0 0 0 2024-03-14T13:02:27.155 INFO:tasks.workunit.client.1.smithi137.stdout:8/910: fsync d1/d5e/d11a/d97/f2a 0 2024-03-14T13:02:27.156 INFO:tasks.workunit.client.1.smithi137.stdout:0/812: mkdir d1/d5/d1b/d1e/d25/d10d/d112 0 2024-03-14T13:02:27.157 INFO:tasks.workunit.client.1.smithi137.stdout:4/823: mknod d4/d9/d3a/d41/d38/c10b 0 2024-03-14T13:02:27.158 INFO:tasks.workunit.client.0.smithi111.stdout:3/826: creat d0/d2d/d35/d69/de6/df4/f12b x:0 0 0 2024-03-14T13:02:27.160 INFO:tasks.workunit.client.1.smithi137.stdout:7/845: dread d3/d5/d23/d3f/d5b/f79 [0,4194304] 0 2024-03-14T13:02:27.160 INFO:tasks.workunit.client.0.smithi111.stdout:3/827: write d0/d2d/dc7/f127 [5271,76464] 0 2024-03-14T13:02:27.162 INFO:tasks.workunit.client.1.smithi137.stdout:5/761: mkdir d7/d16/d18/d48/dfa 0 2024-03-14T13:02:27.163 INFO:tasks.workunit.client.1.smithi137.stdout:0/813: fdatasync d1/d44/d72/fca 0 2024-03-14T13:02:27.168 INFO:tasks.workunit.client.1.smithi137.stdout:7/846: creat d3/d6/d59/daa/f117 x:0 0 0 2024-03-14T13:02:27.171 INFO:tasks.workunit.client.1.smithi137.stdout:4/824: symlink d4/d9/d3a/d41/l10c 0 2024-03-14T13:02:27.172 INFO:tasks.workunit.client.1.smithi137.stdout:8/911: truncate d1/d5e/d11a/fa6 1277830 0 2024-03-14T13:02:27.173 INFO:tasks.workunit.client.1.smithi137.stdout:4/825: fdatasync d4/d9/d3a/d41/d38/f4c 0 2024-03-14T13:02:27.175 INFO:tasks.workunit.client.1.smithi137.stdout:4/826: write d4/d9/d3a/d41/d38/fe2 [3424488,45139] 0 2024-03-14T13:02:27.176 INFO:tasks.workunit.client.1.smithi137.stdout:7/847: rename d3/d6/d14/d22/df3 to d3/d6/d64/d9e/d10d/d118 0 2024-03-14T13:02:27.180 INFO:tasks.workunit.client.1.smithi137.stdout:8/912: dwrite d1/d5e/dc9/dda/fdc [0,4194304] 0 2024-03-14T13:02:27.187 INFO:tasks.workunit.client.1.smithi137.stdout:2/899: write d2/d7/d2e/d3e/d44/d6b/d22/f5a [984071,25733] 0 2024-03-14T13:02:27.190 INFO:tasks.workunit.client.1.smithi137.stdout:7/848: symlink d3/d6/d64/d9e/d10d/d118/l119 0 2024-03-14T13:02:27.192 INFO:tasks.workunit.client.1.smithi137.stdout:7/849: dread - d3/d5/d68/f115 zero size 2024-03-14T13:02:27.194 INFO:tasks.workunit.client.1.smithi137.stdout:3/685: write d4/d19/f70 [1343823,113920] 0 2024-03-14T13:02:27.197 INFO:tasks.workunit.client.1.smithi137.stdout:4/827: dwrite d4/d10/f3f [0,4194304] 0 2024-03-14T13:02:27.199 INFO:tasks.workunit.client.1.smithi137.stdout:3/686: mkdir d4/de/d16/d1c/d5d/de0 0 2024-03-14T13:02:27.200 INFO:tasks.workunit.client.1.smithi137.stdout:4/828: truncate d4/d9/ff7 269608 0 2024-03-14T13:02:27.202 INFO:tasks.workunit.client.1.smithi137.stdout:2/900: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd0 [0,4194304] 0 2024-03-14T13:02:27.204 INFO:tasks.workunit.client.1.smithi137.stdout:4/829: symlink d4/d27/d100/db7/l10d 0 2024-03-14T13:02:27.206 INFO:tasks.workunit.client.1.smithi137.stdout:2/901: rmdir d2/d7/d2e/d3e 39 2024-03-14T13:02:27.212 INFO:tasks.workunit.client.1.smithi137.stdout:2/902: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d128 0 2024-03-14T13:02:27.213 INFO:tasks.workunit.client.1.smithi137.stdout:6/808: write d2/f25 [5148413,67524] 0 2024-03-14T13:02:27.215 INFO:tasks.workunit.client.1.smithi137.stdout:6/809: rmdir d2/d6/de/d45/d59/d39 39 2024-03-14T13:02:27.218 INFO:tasks.workunit.client.1.smithi137.stdout:0/814: write d1/f3e [1581604,69382] 0 2024-03-14T13:02:27.219 INFO:tasks.workunit.client.1.smithi137.stdout:0/815: write d1/d44/d72/d7e/f78 [1099371,124241] 0 2024-03-14T13:02:27.224 INFO:tasks.workunit.client.1.smithi137.stdout:0/816: creat d1/d5/d1b/d1e/d25/d8a/d4a/f113 x:0 0 0 2024-03-14T13:02:27.227 INFO:tasks.workunit.client.1.smithi137.stdout:0/817: mknod d1/d44/d72/c114 0 2024-03-14T13:02:27.228 INFO:tasks.workunit.client.1.smithi137.stdout:8/913: dwrite d1/f44 [0,4194304] 0 2024-03-14T13:02:27.228 INFO:tasks.workunit.client.1.smithi137.stdout:7/850: dwrite d3/d6/d64/d9e/fb8 [0,4194304] 0 2024-03-14T13:02:27.230 INFO:tasks.workunit.client.1.smithi137.stdout:7/851: dread - d3/d6/d59/daa/dcf/d6a/d75/fcb zero size 2024-03-14T13:02:27.231 INFO:tasks.workunit.client.1.smithi137.stdout:7/852: chown d3/d6/d64/d9e/fb8 51026 1 2024-03-14T13:02:27.231 INFO:tasks.workunit.client.1.smithi137.stdout:8/914: mkdir d1/dc/d4e/d6d/d130 0 2024-03-14T13:02:27.232 INFO:tasks.workunit.client.1.smithi137.stdout:0/818: mknod d1/d5/d1b/d1e/d25/c115 0 2024-03-14T13:02:27.233 INFO:tasks.workunit.client.1.smithi137.stdout:7/853: symlink d3/d6/d2e/l11a 0 2024-03-14T13:02:27.235 INFO:tasks.workunit.client.1.smithi137.stdout:8/915: mkdir d1/d5e/d131 0 2024-03-14T13:02:27.235 INFO:tasks.workunit.client.1.smithi137.stdout:8/916: truncate d1/f123 169462 0 2024-03-14T13:02:27.236 INFO:tasks.workunit.client.1.smithi137.stdout:0/819: rename d1/d5/d1b/d1e/d25/d26/lf4 to d1/d5/d1b/d1e/d25/d8a/d9d/l116 0 2024-03-14T13:02:27.238 INFO:tasks.workunit.client.1.smithi137.stdout:0/820: creat d1/d1f/d80/f117 x:0 0 0 2024-03-14T13:02:27.239 INFO:tasks.workunit.client.1.smithi137.stdout:7/854: getdents d3/d52/de8 0 2024-03-14T13:02:27.239 INFO:tasks.workunit.client.1.smithi137.stdout:7/855: dread - d3/d5/f116 zero size 2024-03-14T13:02:27.240 INFO:tasks.workunit.client.1.smithi137.stdout:0/821: readlink d1/d5/l63 0 2024-03-14T13:02:27.242 INFO:tasks.workunit.client.1.smithi137.stdout:0/822: fdatasync d1/d5/d1b/d1e/d25/d8a/d4a/d54/f6f 0 2024-03-14T13:02:27.244 INFO:tasks.workunit.client.1.smithi137.stdout:0/823: creat d1/d1f/d82/f118 x:0 0 0 2024-03-14T13:02:27.245 INFO:tasks.workunit.client.1.smithi137.stdout:0/824: mknod d1/d44/d72/c119 0 2024-03-14T13:02:27.247 INFO:tasks.workunit.client.1.smithi137.stdout:0/825: creat d1/d5/d1b/d1e/d25/d55/f11a x:0 0 0 2024-03-14T13:02:27.247 INFO:tasks.workunit.client.1.smithi137.stdout:0/826: dread - d1/da6/f110 zero size 2024-03-14T13:02:27.251 INFO:tasks.workunit.client.1.smithi137.stdout:0/827: dwrite d1/f1c [0,4194304] 0 2024-03-14T13:02:27.254 INFO:tasks.workunit.client.1.smithi137.stdout:0/828: symlink d1/d5/d1b/d1e/d25/d10d/l11b 0 2024-03-14T13:02:27.257 INFO:tasks.workunit.client.1.smithi137.stdout:0/829: read d1/d5/d1b/d1e/d53/d6d/fdf [1381075,103711] 0 2024-03-14T13:02:27.259 INFO:tasks.workunit.client.1.smithi137.stdout:0/830: mknod d1/d5/d1b/d1e/d25/d26/d10b/c11c 0 2024-03-14T13:02:27.283 INFO:tasks.workunit.client.0.smithi111.stdout:7/758: write d0/d1/d4a/f60 [1043685,115380] 0 2024-03-14T13:02:27.292 INFO:tasks.workunit.client.0.smithi111.stdout:2/720: dwrite dd/d8d/f9b [0,4194304] 0 2024-03-14T13:02:27.292 INFO:tasks.workunit.client.0.smithi111.stdout:7/759: link d0/d11/da8/dd4/cd6 d0/d1/d22/d3e/ce2 0 2024-03-14T13:02:27.292 INFO:tasks.workunit.client.0.smithi111.stdout:3/828: dwrite d0/d2d/d35/d69/f94 [0,4194304] 0 2024-03-14T13:02:27.293 INFO:tasks.workunit.client.0.smithi111.stdout:7/760: fsync d0/d1/d21/fd7 0 2024-03-14T13:02:27.294 INFO:tasks.workunit.client.0.smithi111.stdout:3/829: chown d0/d2d/d35/d4c/f11e 2917 1 2024-03-14T13:02:27.295 INFO:tasks.workunit.client.1.smithi137.stdout:0/831: truncate d1/d5/d1b/d42/f58 918085 0 2024-03-14T13:02:27.296 INFO:tasks.workunit.client.0.smithi111.stdout:2/721: mknod dd/d17/d34/d73/d49/cea 0 2024-03-14T13:02:27.297 INFO:tasks.workunit.client.1.smithi137.stdout:8/917: dwrite d1/dc/d77/d9f/de4/f73 [0,4194304] 0 2024-03-14T13:02:27.298 INFO:tasks.workunit.client.0.smithi111.stdout:3/830: rmdir d0 39 2024-03-14T13:02:27.299 INFO:tasks.workunit.client.0.smithi111.stdout:7/761: dread d0/d1/d21/d5d/fac [0,4194304] 0 2024-03-14T13:02:27.299 INFO:tasks.workunit.client.0.smithi111.stdout:2/722: chown dd/d17/d34/d74/cbc 53466 1 2024-03-14T13:02:27.301 INFO:tasks.workunit.client.0.smithi111.stdout:7/762: fsync d0/d1/d5/d38/d5c/fc0 0 2024-03-14T13:02:27.301 INFO:tasks.workunit.client.1.smithi137.stdout:0/832: write d1/d1f/d82/fb7 [444449,77260] 0 2024-03-14T13:02:27.301 INFO:tasks.workunit.client.0.smithi111.stdout:3/831: chown d0/d30/c49 2 1 2024-03-14T13:02:27.308 INFO:tasks.workunit.client.0.smithi111.stdout:2/723: link dd/d11/d1a/cb0 dd/d11/d1a/d56/dab/ceb 0 2024-03-14T13:02:27.313 INFO:tasks.workunit.client.0.smithi111.stdout:2/724: fsync dd/d8d/dc2/fc7 0 2024-03-14T13:02:27.314 INFO:tasks.workunit.client.0.smithi111.stdout:3/832: dwrite d0/d2d/d35/d50/d5e/f11a [0,4194304] 0 2024-03-14T13:02:27.314 INFO:tasks.workunit.client.1.smithi137.stdout:6/810: dread d2/d6/fc [0,4194304] 0 2024-03-14T13:02:27.319 INFO:tasks.workunit.client.0.smithi111.stdout:2/725: mkdir dd/d8d/dc2/dec 0 2024-03-14T13:02:27.320 INFO:tasks.workunit.client.1.smithi137.stdout:4/830: dread d4/d27/d5c/d6d/d58/f7e [4194304,4194304] 0 2024-03-14T13:02:27.322 INFO:tasks.workunit.client.0.smithi111.stdout:3/833: write d0/d2d/d35/d50/da2/db3/d6f/fb7 [1464618,1182] 0 2024-03-14T13:02:27.325 INFO:tasks.workunit.client.0.smithi111.stdout:3/834: creat d0/d2d/d35/d50/da2/db3/da7/f12c x:0 0 0 2024-03-14T13:02:27.325 INFO:tasks.workunit.client.0.smithi111.stdout:2/726: dread dd/d17/d34/d73/d49/f67 [0,4194304] 0 2024-03-14T13:02:27.328 INFO:tasks.workunit.client.1.smithi137.stdout:4/831: dread d4/d27/d5c/d77/f7c [4194304,4194304] 0 2024-03-14T13:02:27.328 INFO:tasks.workunit.client.0.smithi111.stdout:3/835: unlink d0/d2d/d35/d50/da2/db3/d6f/cac 0 2024-03-14T13:02:27.329 INFO:tasks.workunit.client.0.smithi111.stdout:2/727: creat dd/d11/fed x:0 0 0 2024-03-14T13:02:27.331 INFO:tasks.workunit.client.1.smithi137.stdout:4/832: rename d4/d10/d26/f6f to d4/d27/d5c/d6d/f10e 0 2024-03-14T13:02:27.333 INFO:tasks.workunit.client.0.smithi111.stdout:3/836: link d0/d30/c3b d0/d2d/d35/d50/da2/db3/c12d 0 2024-03-14T13:02:27.336 INFO:tasks.workunit.client.0.smithi111.stdout:3/837: unlink d0/d2d/d35/d50/d5e/d19/ff8 0 2024-03-14T13:02:27.339 INFO:tasks.workunit.client.0.smithi111.stdout:3/838: mknod d0/d2d/d35/d69/de6/dd8/c12e 0 2024-03-14T13:02:27.342 INFO:tasks.workunit.client.0.smithi111.stdout:3/839: getdents d0/d2d/d35/d50/d6c 0 2024-03-14T13:02:27.343 INFO:tasks.workunit.client.0.smithi111.stdout:3/840: chown d0/d2d/d35/d50/da2/db3/d71/dbc 15 1 2024-03-14T13:02:27.349 INFO:tasks.workunit.client.0.smithi111.stdout:7/763: dread d0/d1/f28 [0,4194304] 0 2024-03-14T13:02:27.351 INFO:tasks.workunit.client.0.smithi111.stdout:7/764: mknod d0/d1/d21/d2a/ce3 0 2024-03-14T13:02:27.351 INFO:tasks.workunit.client.1.smithi137.stdout:1/957: sync 2024-03-14T13:02:27.351 INFO:tasks.workunit.client.1.smithi137.stdout:6/811: dread d2/d6/de/d45/d97/d9f/fb8 [0,4194304] 0 2024-03-14T13:02:27.352 INFO:tasks.workunit.client.1.smithi137.stdout:1/958: chown d7/d4d/d6b/f12e 118 1 2024-03-14T13:02:27.354 INFO:tasks.workunit.client.0.smithi111.stdout:7/765: rename d0/d1/d21/d2a/l3c to d0/d1/d21/d2a/d8c/le4 0 2024-03-14T13:02:27.356 INFO:tasks.workunit.client.1.smithi137.stdout:6/812: link d2/d6/d1d/c10e d2/c112 0 2024-03-14T13:02:27.358 INFO:tasks.workunit.client.1.smithi137.stdout:8/918: truncate d1/dc/fcd 285197 0 2024-03-14T13:02:27.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/728: write dd/f4c [1907074,68406] 0 2024-03-14T13:02:27.361 INFO:tasks.workunit.client.1.smithi137.stdout:6/813: write d2/d54/d55/d6d/d76/d57/f9a [5786879,39060] 0 2024-03-14T13:02:27.361 INFO:tasks.workunit.client.1.smithi137.stdout:4/833: getdents d4/d27/d5c/d6d 0 2024-03-14T13:02:27.362 INFO:tasks.workunit.client.1.smithi137.stdout:1/959: dwrite d7/d29/fc9 [0,4194304] 0 2024-03-14T13:02:27.362 INFO:tasks.workunit.client.0.smithi111.stdout:2/729: creat dd/d11/d1a/d56/dab/fee x:0 0 0 2024-03-14T13:02:27.363 INFO:tasks.workunit.client.0.smithi111.stdout:3/841: write d0/d2d/d35/d4c/dad/dee/ff6 [1090671,111539] 0 2024-03-14T13:02:27.363 INFO:tasks.workunit.client.0.smithi111.stdout:2/730: chown dd/d17/d34/d74/fb6 73090 1 2024-03-14T13:02:27.364 INFO:tasks.workunit.client.1.smithi137.stdout:8/919: chown d1/dc/d77/c8e 984 1 2024-03-14T13:02:27.367 INFO:tasks.workunit.client.0.smithi111.stdout:3/842: creat d0/d2d/d35/d4c/db0/f12f x:0 0 0 2024-03-14T13:02:27.367 INFO:tasks.workunit.client.1.smithi137.stdout:1/960: dread - d7/d9/d19/d1f/d2e/da6/ddd/f146 zero size 2024-03-14T13:02:27.367 INFO:tasks.workunit.client.1.smithi137.stdout:6/814: mknod d2/d54/d55/d6d/dc3/dec/c113 0 2024-03-14T13:02:27.371 INFO:tasks.workunit.client.1.smithi137.stdout:8/920: rmdir d1/d5e/d11a/d97/d11d 39 2024-03-14T13:02:27.372 INFO:tasks.workunit.client.0.smithi111.stdout:3/843: creat d0/d2d/d35/d4c/dad/d117/f130 x:0 0 0 2024-03-14T13:02:27.372 INFO:tasks.workunit.client.0.smithi111.stdout:2/731: dwrite dd/d17/d34/d74/f9e [0,4194304] 0 2024-03-14T13:02:27.374 INFO:tasks.workunit.client.1.smithi137.stdout:4/834: creat d4/d9/d8e/f10f x:0 0 0 2024-03-14T13:02:27.375 INFO:tasks.workunit.client.1.smithi137.stdout:6/815: dread - d2/d6/de/d45/d97/d9f/fb1 zero size 2024-03-14T13:02:27.376 INFO:tasks.workunit.client.1.smithi137.stdout:8/921: creat d1/dc/d4e/d6d/f132 x:0 0 0 2024-03-14T13:02:27.379 INFO:tasks.workunit.client.1.smithi137.stdout:6/816: symlink d2/dd6/l114 0 2024-03-14T13:02:27.379 INFO:tasks.workunit.client.0.smithi111.stdout:2/732: link dd/d8d/dc2/f9a dd/d17/d34/d74/dca/dcb/fef 0 2024-03-14T13:02:27.384 INFO:tasks.workunit.client.1.smithi137.stdout:6/817: write d2/d6/de/f68 [1527367,30443] 0 2024-03-14T13:02:27.384 INFO:tasks.workunit.client.1.smithi137.stdout:1/961: dwrite d7/d9/f14 [0,4194304] 0 2024-03-14T13:02:27.385 INFO:tasks.workunit.client.1.smithi137.stdout:0/833: dread d1/d1f/d82/fb7 [0,4194304] 0 2024-03-14T13:02:27.391 INFO:tasks.workunit.client.1.smithi137.stdout:0/834: chown d1/d44/d72/de8 416832203 1 2024-03-14T13:02:27.391 INFO:tasks.workunit.client.1.smithi137.stdout:6/818: readlink d2/d6/de/l36 0 2024-03-14T13:02:27.391 INFO:tasks.workunit.client.1.smithi137.stdout:4/835: dwrite d4/d27/d5c/d77/f108 [0,4194304] 0 2024-03-14T13:02:27.392 INFO:tasks.workunit.client.0.smithi111.stdout:3/844: dread d0/d2d/d35/fef [0,4194304] 0 2024-03-14T13:02:27.394 INFO:tasks.workunit.client.1.smithi137.stdout:0/835: write d1/d44/d56/f70 [3422592,33733] 0 2024-03-14T13:02:27.395 INFO:tasks.workunit.client.0.smithi111.stdout:2/733: write dd/d17/f63 [2543383,99727] 0 2024-03-14T13:02:27.397 INFO:tasks.workunit.client.0.smithi111.stdout:7/766: dread d0/d7e/f49 [0,4194304] 0 2024-03-14T13:02:27.397 INFO:tasks.workunit.client.1.smithi137.stdout:0/836: fsync d1/d5/d1b/d42/d3a/dc6/f10c 0 2024-03-14T13:02:27.397 INFO:tasks.workunit.client.0.smithi111.stdout:2/734: dread - dd/d17/d34/fd2 zero size 2024-03-14T13:02:27.397 INFO:tasks.workunit.client.1.smithi137.stdout:1/962: creat d7/d9/d19/d1f/f14a x:0 0 0 2024-03-14T13:02:27.399 INFO:tasks.workunit.client.1.smithi137.stdout:4/836: creat d4/d9/d3a/d41/f110 x:0 0 0 2024-03-14T13:02:27.400 INFO:tasks.workunit.client.0.smithi111.stdout:3/845: dwrite d0/d2d/d35/d50/d6c/d98/f10f [0,4194304] 0 2024-03-14T13:02:27.402 INFO:tasks.workunit.client.1.smithi137.stdout:0/837: read d1/fb3 [448213,11230] 0 2024-03-14T13:02:27.402 INFO:tasks.workunit.client.1.smithi137.stdout:1/963: fsync d7/d31/d54/fd2 0 2024-03-14T13:02:27.402 INFO:tasks.workunit.client.0.smithi111.stdout:2/735: creat dd/d17/d34/d74/dc1/ff0 x:0 0 0 2024-03-14T13:02:27.402 INFO:tasks.workunit.client.0.smithi111.stdout:3/846: read d0/d2d/d35/d69/d84/ff7 [2876481,44054] 0 2024-03-14T13:02:27.403 INFO:tasks.workunit.client.1.smithi137.stdout:4/837: unlink d4/d9/ff7 0 2024-03-14T13:02:27.408 INFO:tasks.workunit.client.1.smithi137.stdout:0/838: mknod d1/d5/d1b/d1e/d25/d10d/d112/c11d 0 2024-03-14T13:02:27.410 INFO:tasks.workunit.client.1.smithi137.stdout:4/838: dread d4/d9/d3a/d41/d38/d79/f98 [0,4194304] 0 2024-03-14T13:02:27.410 INFO:tasks.workunit.client.1.smithi137.stdout:1/964: creat d7/f14b x:0 0 0 2024-03-14T13:02:27.411 INFO:tasks.workunit.client.1.smithi137.stdout:0/839: stat d1/d5/d1b/d1e/d25/d55/l62 0 2024-03-14T13:02:27.412 INFO:tasks.workunit.client.0.smithi111.stdout:3/847: dwrite d0/d2d/d35/ff2 [0,4194304] 0 2024-03-14T13:02:27.412 INFO:tasks.workunit.client.0.smithi111.stdout:2/736: dread dd/d17/d34/ddb/f65 [0,4194304] 0 2024-03-14T13:02:27.413 INFO:tasks.workunit.client.1.smithi137.stdout:1/965: fsync d7/d9/d19/ffa 0 2024-03-14T13:02:27.414 INFO:tasks.workunit.client.1.smithi137.stdout:0/840: creat d1/d5/d1b/d1e/d25/d8a/d4a/f11e x:0 0 0 2024-03-14T13:02:27.414 INFO:tasks.workunit.client.1.smithi137.stdout:4/839: read d4/d9/d3a/d41/d38/f4c [4922425,102089] 0 2024-03-14T13:02:27.414 INFO:tasks.workunit.client.1.smithi137.stdout:1/966: chown d7/d9/d19/d1f/d26/c6a 318 1 2024-03-14T13:02:27.415 INFO:tasks.workunit.client.0.smithi111.stdout:3/848: stat d0/d2d/d35/d50/d6c/d98/l11f 0 2024-03-14T13:02:27.416 INFO:tasks.workunit.client.0.smithi111.stdout:2/737: symlink dd/d11/d1a/lf1 0 2024-03-14T13:02:27.417 INFO:tasks.workunit.client.1.smithi137.stdout:4/840: symlink d4/d10/d42/l111 0 2024-03-14T13:02:27.418 INFO:tasks.workunit.client.0.smithi111.stdout:3/849: creat d0/d2d/d35/d50/ddf/f131 x:0 0 0 2024-03-14T13:02:27.418 INFO:tasks.workunit.client.1.smithi137.stdout:0/841: symlink d1/d44/d56/l11f 0 2024-03-14T13:02:27.418 INFO:tasks.workunit.client.1.smithi137.stdout:8/922: dread d1/d5e/daf/f126 [0,4194304] 0 2024-03-14T13:02:27.419 INFO:tasks.workunit.client.0.smithi111.stdout:2/738: read - dd/d17/d34/d74/dca/dcb/fef zero size 2024-03-14T13:02:27.420 INFO:tasks.workunit.client.1.smithi137.stdout:1/967: rename d7/d9/d19/d1f/d2e/fe7 to d7/d9/d135/f14c 0 2024-03-14T13:02:27.421 INFO:tasks.workunit.client.1.smithi137.stdout:0/842: mkdir d1/d44/d72/d120 0 2024-03-14T13:02:27.422 INFO:tasks.workunit.client.1.smithi137.stdout:8/923: symlink d1/dc/d71/db0/dca/d105/l133 0 2024-03-14T13:02:27.423 INFO:tasks.workunit.client.0.smithi111.stdout:3/850: link d0/d2d/d35/d50/da2/db3/d6f/fc5 d0/d2d/d35/d50/d5e/f132 0 2024-03-14T13:02:27.423 INFO:tasks.workunit.client.0.smithi111.stdout:2/739: read dd/d8d/db7/fc4 [409994,81408] 0 2024-03-14T13:02:27.423 INFO:tasks.workunit.client.1.smithi137.stdout:1/968: rmdir d7/d9/d19/d40/d42/d9d 39 2024-03-14T13:02:27.424 INFO:tasks.workunit.client.0.smithi111.stdout:2/740: stat dd/d17/f37 0 2024-03-14T13:02:27.427 INFO:tasks.workunit.client.1.smithi137.stdout:1/969: mknod d7/d9/d19/d1f/d89/c14d 0 2024-03-14T13:02:27.427 INFO:tasks.workunit.client.1.smithi137.stdout:8/924: chown d1/d5e/lb8 3106463 1 2024-03-14T13:02:27.428 INFO:tasks.workunit.client.0.smithi111.stdout:3/851: mknod d0/d2d/d35/d50/d5e/d19/c133 0 2024-03-14T13:02:27.429 INFO:tasks.workunit.client.1.smithi137.stdout:8/925: write d1/dc/d4e/d6d/f12a [744802,129083] 0 2024-03-14T13:02:27.430 INFO:tasks.workunit.client.1.smithi137.stdout:0/843: dwrite d1/d44/d56/f5f [8388608,4194304] 0 2024-03-14T13:02:27.431 INFO:tasks.workunit.client.0.smithi111.stdout:2/741: dwrite dd/d17/d34/d73/d49/fe9 [0,4194304] 0 2024-03-14T13:02:27.433 INFO:tasks.workunit.client.1.smithi137.stdout:8/926: creat d1/d5e/dc9/dda/f134 x:0 0 0 2024-03-14T13:02:27.434 INFO:tasks.workunit.client.0.smithi111.stdout:3/852: rename d0/d2d/d35/d69/d84/c10e to d0/d2d/d35/d69/c134 0 2024-03-14T13:02:27.437 INFO:tasks.workunit.client.1.smithi137.stdout:0/844: dread d1/d5/d1b/d1e/d25/d26/f39 [4194304,4194304] 0 2024-03-14T13:02:27.437 INFO:tasks.workunit.client.1.smithi137.stdout:8/927: rmdir d1/d5e/d11a/d82/d96 39 2024-03-14T13:02:27.438 INFO:tasks.workunit.client.1.smithi137.stdout:0/845: stat d1/d5/d1b/d42/d3a 0 2024-03-14T13:02:27.439 INFO:tasks.workunit.client.1.smithi137.stdout:0/846: chown d1/d5/d1b/d1e/d25/d8a/fb5 0 1 2024-03-14T13:02:27.439 INFO:tasks.workunit.client.1.smithi137.stdout:0/847: stat d1/d1f/d80 0 2024-03-14T13:02:27.440 INFO:tasks.workunit.client.0.smithi111.stdout:2/742: rmdir dd/d17/dd0 0 2024-03-14T13:02:27.442 INFO:tasks.workunit.client.1.smithi137.stdout:0/848: mkdir d1/d5/d1b/d1e/d53/df0/d121 0 2024-03-14T13:02:27.444 INFO:tasks.workunit.client.1.smithi137.stdout:0/849: dread - d1/d5/d1b/d42/d61/fe9 zero size 2024-03-14T13:02:27.447 INFO:tasks.workunit.client.0.smithi111.stdout:2/743: link dd/d17/d34/d73/d49/ca3 dd/d17/d53/d9f/cf2 0 2024-03-14T13:02:27.451 INFO:tasks.workunit.client.0.smithi111.stdout:7/767: write d0/d1/d22/d3e/fd1 [666200,119959] 0 2024-03-14T13:02:27.453 INFO:tasks.workunit.client.0.smithi111.stdout:7/768: write d0/d1/d5/d38/d5c/f8d [265830,116714] 0 2024-03-14T13:02:27.457 INFO:tasks.workunit.client.1.smithi137.stdout:6/819: dwrite d2/d6/de/f4f [4194304,4194304] 0 2024-03-14T13:02:27.459 INFO:tasks.workunit.client.0.smithi111.stdout:7/769: link d0/d1/d21/d2a/l8b d0/d1/d5/d38/d5c/d6d/dc9/dce/le5 0 2024-03-14T13:02:27.462 INFO:tasks.workunit.client.0.smithi111.stdout:7/770: mknod d0/d1/d21/d2a/d8c/ce6 0 2024-03-14T13:02:27.463 INFO:tasks.workunit.client.1.smithi137.stdout:4/841: truncate d4/f11 846558 0 2024-03-14T13:02:27.463 INFO:tasks.workunit.client.1.smithi137.stdout:1/970: write d7/d9/d19/d1f/d22/fde [639785,16782] 0 2024-03-14T13:02:27.464 INFO:tasks.workunit.client.1.smithi137.stdout:4/842: write d4/f8 [1941489,62157] 0 2024-03-14T13:02:27.466 INFO:tasks.workunit.client.1.smithi137.stdout:6/820: symlink d2/d6/d86/d8a/l115 0 2024-03-14T13:02:27.467 INFO:tasks.workunit.client.1.smithi137.stdout:6/821: write d2/dd6/fdb [125044,103617] 0 2024-03-14T13:02:27.467 INFO:tasks.workunit.client.1.smithi137.stdout:1/971: creat d7/d92/f14e x:0 0 0 2024-03-14T13:02:27.469 INFO:tasks.workunit.client.1.smithi137.stdout:8/928: dread d1/f31 [0,4194304] 0 2024-03-14T13:02:27.469 INFO:tasks.workunit.client.1.smithi137.stdout:4/843: unlink d4/d10/d26/la9 0 2024-03-14T13:02:27.472 INFO:tasks.workunit.client.1.smithi137.stdout:6/822: creat d2/d6/de/d45/f116 x:0 0 0 2024-03-14T13:02:27.473 INFO:tasks.workunit.client.1.smithi137.stdout:4/844: write d4/d10/d5a/f63 [620151,50004] 0 2024-03-14T13:02:27.474 INFO:tasks.workunit.client.0.smithi111.stdout:7/771: dwrite d0/d1/d21/d2a/f8a [0,4194304] 0 2024-03-14T13:02:27.475 INFO:tasks.workunit.client.1.smithi137.stdout:1/972: dread d7/d4d/f56 [0,4194304] 0 2024-03-14T13:02:27.476 INFO:tasks.workunit.client.0.smithi111.stdout:2/744: dwrite dd/d17/d34/f6f [0,4194304] 0 2024-03-14T13:02:27.477 INFO:tasks.workunit.client.0.smithi111.stdout:3/853: dwrite d0/d2d/d35/d50/d5e/d19/f26 [0,4194304] 0 2024-03-14T13:02:27.477 INFO:tasks.workunit.client.1.smithi137.stdout:8/929: truncate d1/dc/d71/da3/fd0 230688 0 2024-03-14T13:02:27.478 INFO:tasks.workunit.client.1.smithi137.stdout:6/823: chown d2/d54/d55/d6d/d76/d57/d58/l80 1040135 1 2024-03-14T13:02:27.478 INFO:tasks.workunit.client.0.smithi111.stdout:3/854: readlink d0/d2d/d35/l105 0 2024-03-14T13:02:27.480 INFO:tasks.workunit.client.1.smithi137.stdout:0/850: dwrite d1/f7b [0,4194304] 0 2024-03-14T13:02:27.483 INFO:tasks.workunit.client.0.smithi111.stdout:7/772: dread d0/d7e/f49 [0,4194304] 0 2024-03-14T13:02:27.483 INFO:tasks.workunit.client.0.smithi111.stdout:3/855: mkdir d0/d2d/d35/d50/da2/d135 0 2024-03-14T13:02:27.485 INFO:tasks.workunit.client.1.smithi137.stdout:0/851: read d1/f3e [3855821,61388] 0 2024-03-14T13:02:27.485 INFO:tasks.workunit.client.1.smithi137.stdout:4/845: mkdir d4/d10/dfa/d112 0 2024-03-14T13:02:27.487 INFO:tasks.workunit.client.1.smithi137.stdout:1/973: unlink d7/d9/d19/d40/c65 0 2024-03-14T13:02:27.490 INFO:tasks.workunit.client.1.smithi137.stdout:8/930: rmdir d1/d5e/d11a/d82/d94 39 2024-03-14T13:02:27.494 INFO:tasks.workunit.client.1.smithi137.stdout:8/931: chown d1/dc/d10/dcf 56381152 1 2024-03-14T13:02:27.496 INFO:tasks.workunit.client.0.smithi111.stdout:7/773: link d0/d1/d21/d2a/f7d d0/d1/d5/d38/d5c/d6d/dc9/dce/fe7 0 2024-03-14T13:02:27.496 INFO:tasks.workunit.client.0.smithi111.stdout:3/856: rename d0/d2d/d35/d50/d5e/c4b to d0/d2d/d35/d50/da2/db3/d6f/d10b/c136 0 2024-03-14T13:02:27.496 INFO:tasks.workunit.client.1.smithi137.stdout:1/974: mknod d7/d4d/db7/c14f 0 2024-03-14T13:02:27.497 INFO:tasks.workunit.client.0.smithi111.stdout:2/745: dread dd/d17/d34/fa0 [0,4194304] 0 2024-03-14T13:02:27.498 INFO:tasks.workunit.client.1.smithi137.stdout:0/852: dwrite d1/d5/d1b/d1e/d53/d6d/faa [0,4194304] 0 2024-03-14T13:02:27.498 INFO:tasks.workunit.client.1.smithi137.stdout:4/846: rename d4/d9/d3a/d41/d38/d55/f64 to d4/d27/d100/f113 0 2024-03-14T13:02:27.500 INFO:tasks.workunit.client.0.smithi111.stdout:7/774: write d0/d1/d5/d38/d5c/d6d/dc9/dce/fe7 [3614343,130545] 0 2024-03-14T13:02:27.503 INFO:tasks.workunit.client.0.smithi111.stdout:3/857: rename d0/d2d/d35/d69/fa0 to d0/d2d/d35/d50/da2/db3/da7/f137 0 2024-03-14T13:02:27.505 INFO:tasks.workunit.client.1.smithi137.stdout:8/932: mkdir d1/d5e/d11a/d97/d135 0 2024-03-14T13:02:27.505 INFO:tasks.workunit.client.1.smithi137.stdout:0/853: dread d1/d44/d56/f5f [0,4194304] 0 2024-03-14T13:02:27.506 INFO:tasks.workunit.client.1.smithi137.stdout:8/933: rmdir d1 39 2024-03-14T13:02:27.506 INFO:tasks.workunit.client.0.smithi111.stdout:2/746: dwrite dd/d17/d34/d73/d49/fe9 [0,4194304] 0 2024-03-14T13:02:27.508 INFO:tasks.workunit.client.1.smithi137.stdout:0/854: symlink d1/d5/d1b/d1e/d25/d8a/l122 0 2024-03-14T13:02:27.509 INFO:tasks.workunit.client.0.smithi111.stdout:2/747: creat dd/d17/d34/dbf/de6/ff3 x:0 0 0 2024-03-14T13:02:27.509 INFO:tasks.workunit.client.0.smithi111.stdout:7/775: dread d0/d1/d4a/d52/fb2 [0,4194304] 0 2024-03-14T13:02:27.510 INFO:tasks.workunit.client.1.smithi137.stdout:8/934: dread - d1/d5e/d11a/d82/d96/fe3 zero size 2024-03-14T13:02:27.512 INFO:tasks.workunit.client.1.smithi137.stdout:4/847: read d4/d9/d3a/d41/d38/d79/f94 [7735533,35422] 0 2024-03-14T13:02:27.513 INFO:tasks.workunit.client.0.smithi111.stdout:2/748: fsync dd/d17/d34/d73/d49/f82 0 2024-03-14T13:02:27.514 INFO:tasks.workunit.client.1.smithi137.stdout:0/855: rename d1/d5/c49 to d1/d5/d1b/d1e/d53/df0/d121/c123 0 2024-03-14T13:02:27.515 INFO:tasks.workunit.client.1.smithi137.stdout:9/914: sync 2024-03-14T13:02:27.515 INFO:tasks.workunit.client.1.smithi137.stdout:5/762: sync 2024-03-14T13:02:27.516 INFO:tasks.workunit.client.1.smithi137.stdout:9/915: stat d6/df/d19/f11c 0 2024-03-14T13:02:27.516 INFO:tasks.workunit.client.0.smithi111.stdout:2/749: creat dd/d17/d34/d73/d75/ff4 x:0 0 0 2024-03-14T13:02:27.517 INFO:tasks.workunit.client.0.smithi111.stdout:2/750: chown dd/d17/d53/f92 390 1 2024-03-14T13:02:27.517 INFO:tasks.workunit.client.1.smithi137.stdout:5/763: readlink d7/d16/d18/l40 0 2024-03-14T13:02:27.518 INFO:tasks.workunit.client.0.smithi111.stdout:2/751: mkdir dd/d17/d53/da4/df5 0 2024-03-14T13:02:27.519 INFO:tasks.workunit.client.1.smithi137.stdout:5/764: stat d7/d1d/d74/d81/f89 0 2024-03-14T13:02:27.519 INFO:tasks.workunit.client.0.smithi111.stdout:2/752: mkdir dd/d17/d53/da4/df5/df6 0 2024-03-14T13:02:27.520 INFO:tasks.workunit.client.1.smithi137.stdout:9/916: creat d6/df/d19/d33/ddf/d114/f12e x:0 0 0 2024-03-14T13:02:27.523 INFO:tasks.workunit.client.1.smithi137.stdout:0/856: dwrite d1/d5/d1b/d1e/d53/d6d/f107 [0,4194304] 0 2024-03-14T13:02:27.523 INFO:tasks.workunit.client.1.smithi137.stdout:5/765: write d7/d16/d18/d2d/f8f [591010,58385] 0 2024-03-14T13:02:27.525 INFO:tasks.workunit.client.1.smithi137.stdout:9/917: creat d6/df/d71/f12f x:0 0 0 2024-03-14T13:02:27.528 INFO:tasks.workunit.client.1.smithi137.stdout:0/857: getdents d1/d5/d1b/d1e/d53/d6d/d102 0 2024-03-14T13:02:27.532 INFO:tasks.workunit.client.1.smithi137.stdout:9/918: rename d6/df/d88/dd5 to d6/df/d19/d33/d8f/d130 0 2024-03-14T13:02:27.534 INFO:tasks.workunit.client.1.smithi137.stdout:0/858: creat d1/d44/f124 x:0 0 0 2024-03-14T13:02:27.535 INFO:tasks.workunit.client.1.smithi137.stdout:5/766: dwrite d7/d16/d18/d2d/dd1/fdf [0,4194304] 0 2024-03-14T13:02:27.536 INFO:tasks.workunit.client.1.smithi137.stdout:9/919: mkdir d6/df/d19/d33/ddf/d131 0 2024-03-14T13:02:27.539 INFO:tasks.workunit.client.1.smithi137.stdout:0/859: read d1/d44/d72/d7e/f78 [301105,119111] 0 2024-03-14T13:02:27.542 INFO:tasks.workunit.client.1.smithi137.stdout:9/920: rename d6/d26/cb5 to d6/df/d19/d1f/de8/d118/c132 0 2024-03-14T13:02:27.545 INFO:tasks.workunit.client.1.smithi137.stdout:5/767: dwrite d7/d16/d18/d2d/dd1/ddc/ff5 [0,4194304] 0 2024-03-14T13:02:27.545 INFO:tasks.workunit.client.1.smithi137.stdout:9/921: write d6/df/f20 [2388602,58164] 0 2024-03-14T13:02:27.546 INFO:tasks.workunit.client.1.smithi137.stdout:0/860: truncate d1/d5/d1b/d1e/d25/d8a/fb5 662787 0 2024-03-14T13:02:27.548 INFO:tasks.workunit.client.1.smithi137.stdout:9/922: chown d6/df/d19/ld7 25 1 2024-03-14T13:02:27.549 INFO:tasks.workunit.client.1.smithi137.stdout:5/768: mkdir d7/d16/d18/d25/d28/d3c/dfb 0 2024-03-14T13:02:27.551 INFO:tasks.workunit.client.1.smithi137.stdout:0/861: unlink d1/d5/d1b/d1e/d25/d26/fe3 0 2024-03-14T13:02:27.552 INFO:tasks.workunit.client.1.smithi137.stdout:0/862: dread - d1/d44/feb zero size 2024-03-14T13:02:27.554 INFO:tasks.workunit.client.1.smithi137.stdout:0/863: mkdir d1/d44/d72/de8/d125 0 2024-03-14T13:02:27.555 INFO:tasks.workunit.client.1.smithi137.stdout:0/864: stat d1/d5 0 2024-03-14T13:02:27.555 INFO:tasks.workunit.client.1.smithi137.stdout:5/769: getdents d7/d1d/d74/d83 0 2024-03-14T13:02:27.556 INFO:tasks.workunit.client.1.smithi137.stdout:0/865: dread - d1/d44/d72/de8/f10a zero size 2024-03-14T13:02:27.558 INFO:tasks.workunit.client.1.smithi137.stdout:5/770: truncate d7/d16/d18/d25/fe2 1643139 0 2024-03-14T13:02:27.560 INFO:tasks.workunit.client.1.smithi137.stdout:0/866: rename d1/d5/d1b/d42/f32 to d1/d5/d1b/d1e/d53/f126 0 2024-03-14T13:02:27.560 INFO:tasks.workunit.client.1.smithi137.stdout:5/771: rmdir d7/d16/d18/d25/d28/d9e/dd0 39 2024-03-14T13:02:27.580 INFO:tasks.workunit.client.0.smithi111.stdout:5/964: sync 2024-03-14T13:02:27.583 INFO:tasks.workunit.client.0.smithi111.stdout:5/965: dread - d2/d16/d49/d33/d68/f108 zero size 2024-03-14T13:02:27.584 INFO:tasks.workunit.client.0.smithi111.stdout:5/966: stat d2/d45/d76/d9a/dbd 0 2024-03-14T13:02:27.588 INFO:tasks.workunit.client.1.smithi137.stdout:6/824: write d2/d4c/f69 [209498,67052] 0 2024-03-14T13:02:27.591 INFO:tasks.workunit.client.1.smithi137.stdout:6/825: creat d2/d6/da7/f117 x:0 0 0 2024-03-14T13:02:27.593 INFO:tasks.workunit.client.1.smithi137.stdout:6/826: mknod d2/d6/de/d45/c118 0 2024-03-14T13:02:27.596 INFO:tasks.workunit.client.0.smithi111.stdout:7/776: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:27.596 INFO:tasks.workunit.client.1.smithi137.stdout:1/975: dwrite d7/d31/f76 [0,4194304] 0 2024-03-14T13:02:27.599 INFO:tasks.workunit.client.0.smithi111.stdout:2/753: dwrite dd/da9/fb2 [0,4194304] 0 2024-03-14T13:02:27.601 INFO:tasks.workunit.client.0.smithi111.stdout:7/777: dread d0/d1/d4a/d52/fb2 [0,4194304] 0 2024-03-14T13:02:27.602 INFO:tasks.workunit.client.0.smithi111.stdout:5/967: write d2/d45/d76/d9a/d9f/d12e/faf [707796,43757] 0 2024-03-14T13:02:27.603 INFO:tasks.workunit.client.0.smithi111.stdout:7/778: read - d0/d1/d4a/d52/f6c zero size 2024-03-14T13:02:27.604 INFO:tasks.workunit.client.0.smithi111.stdout:5/968: write d2/d16/d49/f11 [59695,93767] 0 2024-03-14T13:02:27.606 INFO:tasks.workunit.client.0.smithi111.stdout:7/779: write d0/d1/d5/d56/f7c [2152265,82662] 0 2024-03-14T13:02:27.608 INFO:tasks.workunit.client.0.smithi111.stdout:2/754: rename dd/d17/d34/f2e to dd/d17/d34/d74/dca/ff7 0 2024-03-14T13:02:27.609 INFO:tasks.workunit.client.0.smithi111.stdout:5/969: dread d2/d45/d76/d8d/fae [0,4194304] 0 2024-03-14T13:02:27.611 INFO:tasks.workunit.client.0.smithi111.stdout:2/755: creat dd/d11/d8b/ff8 x:0 0 0 2024-03-14T13:02:27.612 INFO:tasks.workunit.client.1.smithi137.stdout:7/856: sync 2024-03-14T13:02:27.612 INFO:tasks.workunit.client.1.smithi137.stdout:2/903: sync 2024-03-14T13:02:27.612 INFO:tasks.workunit.client.1.smithi137.stdout:3/687: sync 2024-03-14T13:02:27.612 INFO:tasks.workunit.client.1.smithi137.stdout:7/857: chown d3/d54/db4 0 1 2024-03-14T13:02:27.612 INFO:tasks.workunit.client.0.smithi111.stdout:5/970: mknod d2/d16/d49/d7c/d95/c135 0 2024-03-14T13:02:27.613 INFO:tasks.workunit.client.0.smithi111.stdout:2/756: chown dd/d17/d53/da4/df5/df6 225952520 1 2024-03-14T13:02:27.613 INFO:tasks.workunit.client.0.smithi111.stdout:7/780: mknod d0/d1/d5/d38/d5c/d6d/ddb/ce8 0 2024-03-14T13:02:27.615 INFO:tasks.workunit.client.0.smithi111.stdout:2/757: rmdir dd/d17/d34/ddb 39 2024-03-14T13:02:27.615 INFO:tasks.workunit.client.1.smithi137.stdout:9/923: write d6/df/d19/d33/d8f/fbd [3632565,21121] 0 2024-03-14T13:02:27.617 INFO:tasks.workunit.client.0.smithi111.stdout:7/781: unlink d0/d1/d22/f39 0 2024-03-14T13:02:27.617 INFO:tasks.workunit.client.0.smithi111.stdout:7/782: fdatasync d0/d1/d4a/d52/fd2 0 2024-03-14T13:02:27.618 INFO:tasks.workunit.client.1.smithi137.stdout:3/688: unlink d4/de/l7f 0 2024-03-14T13:02:27.619 INFO:tasks.workunit.client.0.smithi111.stdout:2/758: rename dd/d17/d34/fa0 to dd/d8d/dc2/ff9 0 2024-03-14T13:02:27.620 INFO:tasks.workunit.client.0.smithi111.stdout:5/971: getdents d2/d16/d49/d33 0 2024-03-14T13:02:27.621 INFO:tasks.workunit.client.1.smithi137.stdout:2/904: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/ff 401481 0 2024-03-14T13:02:27.621 INFO:tasks.workunit.client.1.smithi137.stdout:4/848: dread d4/d9/d3a/d41/f21 [4194304,4194304] 0 2024-03-14T13:02:27.622 INFO:tasks.workunit.client.1.smithi137.stdout:9/924: truncate d6/df/d88/de5/df6/da9/fe4 664680 0 2024-03-14T13:02:27.623 INFO:tasks.workunit.client.0.smithi111.stdout:2/759: creat dd/d17/d34/de0/ffa x:0 0 0 2024-03-14T13:02:27.623 INFO:tasks.workunit.client.1.smithi137.stdout:4/849: fdatasync d4/d9/d3a/d41/d38/d79/fe0 0 2024-03-14T13:02:27.625 INFO:tasks.workunit.client.0.smithi111.stdout:7/783: link d0/d1/d5/d56/d70/f95 d0/d1/d4a/fe9 0 2024-03-14T13:02:27.625 INFO:tasks.workunit.client.1.smithi137.stdout:4/850: stat d4/f3c 0 2024-03-14T13:02:27.625 INFO:tasks.workunit.client.1.smithi137.stdout:5/772: truncate d7/d33/fb1 1331021 0 2024-03-14T13:02:27.626 INFO:tasks.workunit.client.0.smithi111.stdout:7/784: rmdir d0/d1 39 2024-03-14T13:02:27.626 INFO:tasks.workunit.client.1.smithi137.stdout:7/858: mkdir d3/d5/d5a/d11b 0 2024-03-14T13:02:27.627 INFO:tasks.workunit.client.0.smithi111.stdout:2/760: getdents dd/da9 0 2024-03-14T13:02:27.628 INFO:tasks.workunit.client.0.smithi111.stdout:7/785: creat d0/d11/d98/fea x:0 0 0 2024-03-14T13:02:27.629 INFO:tasks.workunit.client.1.smithi137.stdout:9/925: read d6/df/d19/d1f/f3b [4589282,96127] 0 2024-03-14T13:02:27.629 INFO:tasks.workunit.client.1.smithi137.stdout:5/773: fdatasync d7/fa2 0 2024-03-14T13:02:27.630 INFO:tasks.workunit.client.1.smithi137.stdout:9/926: readlink d6/df/d19/d33/l39 0 2024-03-14T13:02:27.630 INFO:tasks.workunit.client.0.smithi111.stdout:7/786: symlink d0/d11/leb 0 2024-03-14T13:02:27.631 INFO:tasks.workunit.client.1.smithi137.stdout:2/905: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/d10f/c129 0 2024-03-14T13:02:27.632 INFO:tasks.workunit.client.0.smithi111.stdout:2/761: link dd/l29 dd/d11/d1a/lfb 0 2024-03-14T13:02:27.633 INFO:tasks.workunit.client.1.smithi137.stdout:4/851: dread d4/d9/d3a/d41/d6a/ff8 [0,4194304] 0 2024-03-14T13:02:27.633 INFO:tasks.workunit.client.1.smithi137.stdout:7/859: creat d3/d6/d14/d27/dfe/f11c x:0 0 0 2024-03-14T13:02:27.635 INFO:tasks.workunit.client.1.smithi137.stdout:7/860: write d3/d54/fc9 [907302,33703] 0 2024-03-14T13:02:27.635 INFO:tasks.workunit.client.1.smithi137.stdout:9/927: creat d6/d53/f133 x:0 0 0 2024-03-14T13:02:27.636 INFO:tasks.workunit.client.1.smithi137.stdout:4/852: write d4/d27/d100/d8d/f106 [231868,69572] 0 2024-03-14T13:02:27.637 INFO:tasks.workunit.client.1.smithi137.stdout:6/827: write d2/d6/de/d45/d59/d39/f7a [36033,26187] 0 2024-03-14T13:02:27.638 INFO:tasks.workunit.client.1.smithi137.stdout:2/906: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/cd8 0 2024-03-14T13:02:27.638 INFO:tasks.workunit.client.0.smithi111.stdout:5/972: dread d2/d16/d49/d7c/d95/d6a/fa8 [0,4194304] 0 2024-03-14T13:02:27.640 INFO:tasks.workunit.client.1.smithi137.stdout:4/853: read - d4/d10/d42/d73/fb8 zero size 2024-03-14T13:02:27.641 INFO:tasks.workunit.client.1.smithi137.stdout:0/867: write d1/d5/d1b/d1e/d25/d8a/fb5 [290047,102579] 0 2024-03-14T13:02:27.642 INFO:tasks.workunit.client.0.smithi111.stdout:7/787: getdents d0/d1/d5/d38/d5c/d6d/dc9 0 2024-03-14T13:02:27.642 INFO:tasks.workunit.client.1.smithi137.stdout:7/861: creat d3/f11d x:0 0 0 2024-03-14T13:02:27.644 INFO:tasks.workunit.client.0.smithi111.stdout:7/788: chown d0/d11/l8f 58776 1 2024-03-14T13:02:27.644 INFO:tasks.workunit.client.0.smithi111.stdout:7/789: stat d0/d1/d4a/d52/ca6 0 2024-03-14T13:02:27.645 INFO:tasks.workunit.client.1.smithi137.stdout:6/828: creat d2/d54/d55/d38/f119 x:0 0 0 2024-03-14T13:02:27.645 INFO:tasks.workunit.client.1.smithi137.stdout:9/928: symlink d6/df/d19/d33/d8f/l134 0 2024-03-14T13:02:27.647 INFO:tasks.workunit.client.1.smithi137.stdout:3/689: write d4/de/f1f [837765,35211] 0 2024-03-14T13:02:27.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/854: creat d4/d10/d5a/f114 x:0 0 0 2024-03-14T13:02:27.648 INFO:tasks.workunit.client.0.smithi111.stdout:7/790: truncate d0/d1/d5/d38/d1b/d9c/fbb 5097491 0 2024-03-14T13:02:27.651 INFO:tasks.workunit.client.0.smithi111.stdout:2/762: dwrite dd/d8d/dc2/fc7 [0,4194304] 0 2024-03-14T13:02:27.652 INFO:tasks.workunit.client.0.smithi111.stdout:2/763: chown dd/d17/d34/d73/fce 13 1 2024-03-14T13:02:27.653 INFO:tasks.workunit.client.0.smithi111.stdout:7/791: symlink d0/d1/d5/d38/d5c/d6d/dc9/lec 0 2024-03-14T13:02:27.654 INFO:tasks.workunit.client.0.smithi111.stdout:7/792: chown d0/d1/d22/c82 158767 1 2024-03-14T13:02:27.654 INFO:tasks.workunit.client.1.smithi137.stdout:4/855: stat d4/d9/d8e 0 2024-03-14T13:02:27.655 INFO:tasks.workunit.client.1.smithi137.stdout:3/690: stat d4/de/d16/d1c/d5d/d85/c87 0 2024-03-14T13:02:27.657 INFO:tasks.workunit.client.1.smithi137.stdout:1/976: dwrite d7/d31/d54/f60 [0,4194304] 0 2024-03-14T13:02:27.659 INFO:tasks.workunit.client.0.smithi111.stdout:7/793: truncate d0/d1/d22/d3e/f64 1557181 0 2024-03-14T13:02:27.659 INFO:tasks.workunit.client.1.smithi137.stdout:5/774: dread d7/d16/d18/d25/d28/f3d [0,4194304] 0 2024-03-14T13:02:27.659 INFO:tasks.workunit.client.0.smithi111.stdout:2/764: link dd/d17/d34/ddb/cae dd/d17/d34/dbf/cfc 0 2024-03-14T13:02:27.660 INFO:tasks.workunit.client.1.smithi137.stdout:4/856: dread - d4/d9/d3a/d70/fd0 zero size 2024-03-14T13:02:27.660 INFO:tasks.workunit.client.0.smithi111.stdout:2/765: stat dd/d17/d34/dbf/lc3 0 2024-03-14T13:02:27.661 INFO:tasks.workunit.client.0.smithi111.stdout:5/973: dread d2/d13/f4f [0,4194304] 0 2024-03-14T13:02:27.664 INFO:tasks.workunit.client.1.smithi137.stdout:6/829: creat d2/d54/d55/d9c/f11a x:0 0 0 2024-03-14T13:02:27.664 INFO:tasks.workunit.client.0.smithi111.stdout:5/974: truncate d2/d16/d49/d44/d72/de0/f120 5237186 0 2024-03-14T13:02:27.665 INFO:tasks.workunit.client.1.smithi137.stdout:5/775: chown d7/d16/d18/d25/d28/f3d 12540869 1 2024-03-14T13:02:27.665 INFO:tasks.workunit.client.0.smithi111.stdout:2/766: creat dd/d17/ffd x:0 0 0 2024-03-14T13:02:27.666 INFO:tasks.workunit.client.0.smithi111.stdout:5/975: write d2/d13/fb9 [3429339,54890] 0 2024-03-14T13:02:27.667 INFO:tasks.workunit.client.1.smithi137.stdout:9/929: rename d6/df/d19/la6 to d6/l135 0 2024-03-14T13:02:27.670 INFO:tasks.workunit.client.0.smithi111.stdout:2/767: mknod dd/cfe 0 2024-03-14T13:02:27.672 INFO:tasks.workunit.client.0.smithi111.stdout:5/976: mkdir d2/d16/d49/d136 0 2024-03-14T13:02:27.673 INFO:tasks.workunit.client.1.smithi137.stdout:2/907: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f77 [0,4194304] 0 2024-03-14T13:02:27.673 INFO:tasks.workunit.client.0.smithi111.stdout:2/768: symlink dd/d11/d8b/lff 0 2024-03-14T13:02:27.674 INFO:tasks.workunit.client.1.smithi137.stdout:5/776: truncate d7/d33/d65/f7c 1484518 0 2024-03-14T13:02:27.676 INFO:tasks.workunit.client.1.smithi137.stdout:3/691: dwrite d4/fa8 [0,4194304] 0 2024-03-14T13:02:27.677 INFO:tasks.workunit.client.0.smithi111.stdout:5/977: creat d2/d16/d49/d7c/d95/f137 x:0 0 0 2024-03-14T13:02:27.680 INFO:tasks.workunit.client.0.smithi111.stdout:2/769: getdents dd/d17/d53/da4 0 2024-03-14T13:02:27.680 INFO:tasks.workunit.client.0.smithi111.stdout:7/794: dwrite d0/d1/d4a/d52/f6c [0,4194304] 0 2024-03-14T13:02:27.681 INFO:tasks.workunit.client.1.smithi137.stdout:0/868: rename d1/d5/d1b/d1e/d25/d26/fab to d1/d44/d72/df8/f127 0 2024-03-14T13:02:27.683 INFO:tasks.workunit.client.1.smithi137.stdout:4/857: dwrite d4/d10/d42/f4e [4194304,4194304] 0 2024-03-14T13:02:27.684 INFO:tasks.workunit.client.1.smithi137.stdout:5/777: mknod d7/d1d/d74/dec/cfc 0 2024-03-14T13:02:27.685 INFO:tasks.workunit.client.1.smithi137.stdout:2/908: dread d2/f14 [0,4194304] 0 2024-03-14T13:02:27.686 INFO:tasks.workunit.client.1.smithi137.stdout:6/830: link d2/d54/d55/d38/l105 d2/d6/d86/l11b 0 2024-03-14T13:02:27.687 INFO:tasks.workunit.client.1.smithi137.stdout:7/862: dread d3/d6/d14/d27/f6c [0,4194304] 0 2024-03-14T13:02:27.688 INFO:tasks.workunit.client.0.smithi111.stdout:2/770: write dd/d17/f37 [2375674,72293] 0 2024-03-14T13:02:27.690 INFO:tasks.workunit.client.1.smithi137.stdout:4/858: dread - d4/d9/d3a/d70/fd0 zero size 2024-03-14T13:02:27.691 INFO:tasks.workunit.client.1.smithi137.stdout:4/859: dread - d4/d9/d8e/f10f zero size 2024-03-14T13:02:27.693 INFO:tasks.workunit.client.0.smithi111.stdout:7/795: mknod d0/d1/d5/d38/d5c/ced 0 2024-03-14T13:02:27.694 INFO:tasks.workunit.client.1.smithi137.stdout:2/909: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/dca/ld4 0 2024-03-14T13:02:27.694 INFO:tasks.workunit.client.1.smithi137.stdout:0/869: mkdir d1/d5/d1b/d1e/d53/d6d/d102/d128 0 2024-03-14T13:02:27.695 INFO:tasks.workunit.client.0.smithi111.stdout:2/771: mkdir dd/d8d/d100 0 2024-03-14T13:02:27.695 INFO:tasks.workunit.client.0.smithi111.stdout:5/978: dwrite d2/d16/d49/d44/fa7 [0,4194304] 0 2024-03-14T13:02:27.695 INFO:tasks.workunit.client.1.smithi137.stdout:1/977: truncate d7/d31/d54/f60 2485179 0 2024-03-14T13:02:27.697 INFO:tasks.workunit.client.1.smithi137.stdout:2/910: chown d2/d7/d2e/d3e/daa/dcd/cd3 23530921 1 2024-03-14T13:02:27.698 INFO:tasks.workunit.client.1.smithi137.stdout:4/860: mkdir d4/d10/d42/daf/dfb/d115 0 2024-03-14T13:02:27.700 INFO:tasks.workunit.client.0.smithi111.stdout:5/979: creat d2/d16/d49/d33/d68/d122/f138 x:0 0 0 2024-03-14T13:02:27.700 INFO:tasks.workunit.client.0.smithi111.stdout:2/772: mknod dd/d17/d53/da4/da6/c101 0 2024-03-14T13:02:27.700 INFO:tasks.workunit.client.1.smithi137.stdout:1/978: write d7/d4d/d6b/dc5/fe8 [386743,114838] 0 2024-03-14T13:02:27.701 INFO:tasks.workunit.client.1.smithi137.stdout:5/778: link d7/d33/c56 d7/d16/d18/d25/d28/d9e/dd0/cfd 0 2024-03-14T13:02:27.704 INFO:tasks.workunit.client.0.smithi111.stdout:2/773: write dd/f4c [470266,1224] 0 2024-03-14T13:02:27.705 INFO:tasks.workunit.client.1.smithi137.stdout:5/779: chown d7/d16/ca6 9 1 2024-03-14T13:02:27.705 INFO:tasks.workunit.client.0.smithi111.stdout:5/980: rmdir d2/d16/d49/d33/de5 39 2024-03-14T13:02:27.706 INFO:tasks.workunit.client.1.smithi137.stdout:2/911: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f96 5937 1 2024-03-14T13:02:27.706 INFO:tasks.workunit.client.0.smithi111.stdout:7/796: dwrite d0/d1/d21/d2a/f9b [0,4194304] 0 2024-03-14T13:02:27.708 INFO:tasks.workunit.client.0.smithi111.stdout:5/981: stat d2/d45/d76/f132 0 2024-03-14T13:02:27.708 INFO:tasks.workunit.client.0.smithi111.stdout:7/797: write d0/d1/d21/d5d/f84 [1675096,103636] 0 2024-03-14T13:02:27.710 INFO:tasks.workunit.client.1.smithi137.stdout:5/780: rename d7/f10 to d7/d36/ffe 0 2024-03-14T13:02:27.711 INFO:tasks.workunit.client.0.smithi111.stdout:7/798: chown d0/d1/d5/d38/d1b/l2c 7398 1 2024-03-14T13:02:27.713 INFO:tasks.workunit.client.0.smithi111.stdout:7/799: chown d0/d1/d21/c17 199 1 2024-03-14T13:02:27.713 INFO:tasks.workunit.client.1.smithi137.stdout:0/870: dwrite d1/d5/d1b/d1e/d25/fa8 [0,4194304] 0 2024-03-14T13:02:27.713 INFO:tasks.workunit.client.0.smithi111.stdout:2/774: mknod dd/d17/d53/da4/da6/c102 0 2024-03-14T13:02:27.714 INFO:tasks.workunit.client.0.smithi111.stdout:5/982: creat d2/d45/d76/db2/df7/f139 x:0 0 0 2024-03-14T13:02:27.716 INFO:tasks.workunit.client.1.smithi137.stdout:1/979: getdents d7/d9/d19/d1f/ddf 0 2024-03-14T13:02:27.717 INFO:tasks.workunit.client.1.smithi137.stdout:2/912: truncate d2/d7/d2e/d3e/d4e/f118 140013 0 2024-03-14T13:02:27.717 INFO:tasks.workunit.client.0.smithi111.stdout:7/800: chown d0/d1/d21/d2a/l8b 388721410 1 2024-03-14T13:02:27.718 INFO:tasks.workunit.client.0.smithi111.stdout:2/775: symlink dd/d17/d34/d74/dc1/dd1/l103 0 2024-03-14T13:02:27.719 INFO:tasks.workunit.client.0.smithi111.stdout:7/801: dread - d0/d1/d5/d56/d70/f95 zero size 2024-03-14T13:02:27.721 INFO:tasks.workunit.client.1.smithi137.stdout:5/781: mkdir d7/d16/dff 0 2024-03-14T13:02:27.721 INFO:tasks.workunit.client.1.smithi137.stdout:0/871: read d1/d5/d1b/d1e/d25/d55/fb4 [6196,50819] 0 2024-03-14T13:02:27.721 INFO:tasks.workunit.client.1.smithi137.stdout:2/913: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f102 17206 1 2024-03-14T13:02:27.722 INFO:tasks.workunit.client.1.smithi137.stdout:7/863: write d3/d6/d14/f2f [1665119,27353] 0 2024-03-14T13:02:27.723 INFO:tasks.workunit.client.1.smithi137.stdout:1/980: creat d7/d9/d19/d40/d50/f150 x:0 0 0 2024-03-14T13:02:27.726 INFO:tasks.workunit.client.1.smithi137.stdout:5/782: write d7/d49/d75/fb3 [467660,1714] 0 2024-03-14T13:02:27.727 INFO:tasks.workunit.client.1.smithi137.stdout:0/872: chown d1/d5/d1b/d1e/d25/d8a/f21 512 1 2024-03-14T13:02:27.729 INFO:tasks.workunit.client.0.smithi111.stdout:2/776: mknod dd/d17/d34/d74/dca/dcb/c104 0 2024-03-14T13:02:27.730 INFO:tasks.workunit.client.1.smithi137.stdout:5/783: fdatasync d7/d49/d75/fb2 0 2024-03-14T13:02:27.730 INFO:tasks.workunit.client.1.smithi137.stdout:6/831: dread d2/f5a [0,4194304] 0 2024-03-14T13:02:27.731 INFO:tasks.workunit.client.0.smithi111.stdout:5/983: getdents d2/d16/d49/d44/d72/de0/dfe 0 2024-03-14T13:02:27.731 INFO:tasks.workunit.client.0.smithi111.stdout:2/777: fdatasync dd/d11/fed 0 2024-03-14T13:02:27.731 INFO:tasks.workunit.client.1.smithi137.stdout:1/981: truncate d7/d92/f14e 975449 0 2024-03-14T13:02:27.733 INFO:tasks.workunit.client.1.smithi137.stdout:1/982: chown d7/d9/d19/d1f/d2e 61414385 1 2024-03-14T13:02:27.735 INFO:tasks.workunit.client.1.smithi137.stdout:2/914: creat d2/d7/d2e/d3e/d44/d6b/d1d/d41/f12a x:0 0 0 2024-03-14T13:02:27.735 INFO:tasks.workunit.client.1.smithi137.stdout:0/873: chown d1/d5/d1b/d42/d3a/cd8 0 1 2024-03-14T13:02:27.735 INFO:tasks.workunit.client.1.smithi137.stdout:7/864: rename d3/d6/d14/d27/dfe/f11c to d3/d6/d14/d22/ddf/f11e 0 2024-03-14T13:02:27.737 INFO:tasks.workunit.client.0.smithi111.stdout:5/984: dwrite d2/d16/d49/f11 [0,4194304] 0 2024-03-14T13:02:27.737 INFO:tasks.workunit.client.1.smithi137.stdout:6/832: fsync d2/d6/de/d45/f108 0 2024-03-14T13:02:27.737 INFO:tasks.workunit.client.1.smithi137.stdout:5/784: creat d7/d16/d18/d25/d28/dc5/f100 x:0 0 0 2024-03-14T13:02:27.738 INFO:tasks.workunit.client.1.smithi137.stdout:1/983: chown d7/d9/d19/d1f/d89/fb2 99959 1 2024-03-14T13:02:27.739 INFO:tasks.workunit.client.0.smithi111.stdout:5/985: dread - d2/d16/d49/d7c/d95/f137 zero size 2024-03-14T13:02:27.740 INFO:tasks.workunit.client.1.smithi137.stdout:3/692: dread d4/de/d16/d1c/d5d/f43 [4194304,4194304] 0 2024-03-14T13:02:27.740 INFO:tasks.workunit.client.1.smithi137.stdout:0/874: chown d1/d5/d1b/d1e/d53/d6d/fba 4 1 2024-03-14T13:02:27.741 INFO:tasks.workunit.client.0.smithi111.stdout:5/986: stat d2/d45/d76/d9a 0 2024-03-14T13:02:27.741 INFO:tasks.workunit.client.0.smithi111.stdout:1/990: sync 2024-03-14T13:02:27.741 INFO:tasks.workunit.client.0.smithi111.stdout:9/821: sync 2024-03-14T13:02:27.741 INFO:tasks.workunit.client.0.smithi111.stdout:6/726: sync 2024-03-14T13:02:27.742 INFO:tasks.workunit.client.1.smithi137.stdout:1/984: write d7/d9/d19/d40/fa8 [914710,105663] 0 2024-03-14T13:02:27.743 INFO:tasks.workunit.client.1.smithi137.stdout:0/875: readlink d1/d1f/d82/lac 0 2024-03-14T13:02:27.743 INFO:tasks.workunit.client.1.smithi137.stdout:7/865: read - d3/d4a/f110 zero size 2024-03-14T13:02:27.743 INFO:tasks.workunit.client.0.smithi111.stdout:9/822: fsync d5/d42/d71/dbd/de0/fed 0 2024-03-14T13:02:27.745 INFO:tasks.workunit.client.1.smithi137.stdout:0/876: chown d1/d44/d72 1 1 2024-03-14T13:02:27.746 INFO:tasks.workunit.client.0.smithi111.stdout:7/802: dread d0/d7e/f3d [0,4194304] 0 2024-03-14T13:02:27.747 INFO:tasks.workunit.client.1.smithi137.stdout:4/861: dread d4/d9/d3a/d70/fdf [0,4194304] 0 2024-03-14T13:02:27.748 INFO:tasks.workunit.client.0.smithi111.stdout:2/778: dread dd/d17/d34/d73/faf [0,4194304] 0 2024-03-14T13:02:27.748 INFO:tasks.workunit.client.0.smithi111.stdout:7/803: chown d0/d1/d21/d2a/d8c/lae 818 1 2024-03-14T13:02:27.749 INFO:tasks.workunit.client.1.smithi137.stdout:6/833: creat d2/d54/d55/d9c/dc0/dc4/f11c x:0 0 0 2024-03-14T13:02:27.749 INFO:tasks.workunit.client.1.smithi137.stdout:1/985: fdatasync d7/d9/f15 0 2024-03-14T13:02:27.750 INFO:tasks.workunit.client.0.smithi111.stdout:7/804: read d0/d1/d4a/d52/f62 [2925946,109170] 0 2024-03-14T13:02:27.751 INFO:tasks.workunit.client.1.smithi137.stdout:0/877: mknod d1/d5/d1b/d1e/d25/d8a/d4a/c129 0 2024-03-14T13:02:27.752 INFO:tasks.workunit.client.1.smithi137.stdout:4/862: unlink d4/d10/d5a/l78 0 2024-03-14T13:02:27.753 INFO:tasks.workunit.client.1.smithi137.stdout:4/863: readlink d4/d9/l4a 0 2024-03-14T13:02:27.753 INFO:tasks.workunit.client.1.smithi137.stdout:6/834: rmdir d2/dd6 39 2024-03-14T13:02:27.755 INFO:tasks.workunit.client.0.smithi111.stdout:2/779: dwrite dd/d17/d53/f98 [0,4194304] 0 2024-03-14T13:02:27.755 INFO:tasks.workunit.client.1.smithi137.stdout:4/864: chown d4/d10/dfa/d112 3530400 1 2024-03-14T13:02:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:27 smithi137 ceph-mon[50336]: pgmap v28: 65 pgs: 65 active+clean; 3.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 104 MiB/s rd, 272 MiB/s wr, 557 op/s 2024-03-14T13:02:27.756 INFO:tasks.workunit.client.1.smithi137.stdout:1/986: mkdir d7/d9/d19/d1f/da0/d151 0 2024-03-14T13:02:27.760 INFO:tasks.workunit.client.0.smithi111.stdout:6/727: rename d0/d17/d21/d69/d3d/fb5 to d0/d17/d20/d4a/d68/f109 0 2024-03-14T13:02:27.760 INFO:tasks.workunit.client.1.smithi137.stdout:2/915: link d2/d7/d2e/d3e/d44/d6b/d1d/d2d/f61 d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/f12b 0 2024-03-14T13:02:27.760 INFO:tasks.workunit.client.1.smithi137.stdout:7/866: dwrite d3/d5/d68/f9c [0,4194304] 0 2024-03-14T13:02:27.763 INFO:tasks.workunit.client.1.smithi137.stdout:7/867: dread - d3/f11d zero size 2024-03-14T13:02:27.764 INFO:tasks.workunit.client.1.smithi137.stdout:0/878: truncate d1/d5/d1b/d1e/d53/d6d/f74 4044486 0 2024-03-14T13:02:27.765 INFO:tasks.workunit.client.0.smithi111.stdout:2/780: mkdir dd/d17/d53/da4/d105 0 2024-03-14T13:02:27.766 INFO:tasks.workunit.client.0.smithi111.stdout:9/823: dread d5/d6/ff [0,4194304] 0 2024-03-14T13:02:27.766 INFO:tasks.workunit.client.0.smithi111.stdout:2/781: stat dd/d17/d34/lcf 0 2024-03-14T13:02:27.767 INFO:tasks.workunit.client.1.smithi137.stdout:4/865: chown d4/d9/d3a/d41/l35 383720 1 2024-03-14T13:02:27.767 INFO:tasks.workunit.client.1.smithi137.stdout:6/835: dwrite d2/d54/d55/d38/f51 [0,4194304] 0 2024-03-14T13:02:27.768 INFO:tasks.workunit.client.0.smithi111.stdout:7/805: rename d0/d1/d21/d3a to d0/d1/d5/d38/d1b/d9c/dee 0 2024-03-14T13:02:27.769 INFO:tasks.workunit.client.1.smithi137.stdout:6/836: chown d2/d4c/f84 26 1 2024-03-14T13:02:27.771 INFO:tasks.workunit.client.1.smithi137.stdout:3/693: getdents d4/de/d16/d27/dd3 0 2024-03-14T13:02:27.772 INFO:tasks.workunit.client.1.smithi137.stdout:6/837: symlink d2/d54/d55/d33/l11d 0 2024-03-14T13:02:27.772 INFO:tasks.workunit.client.1.smithi137.stdout:1/987: dread d7/d31/f76 [0,4194304] 0 2024-03-14T13:02:27.773 INFO:tasks.workunit.client.1.smithi137.stdout:7/868: creat d3/d54/f11f x:0 0 0 2024-03-14T13:02:27.775 INFO:tasks.workunit.client.1.smithi137.stdout:0/879: mknod d1/d1f/c12a 0 2024-03-14T13:02:27.776 INFO:tasks.workunit.client.1.smithi137.stdout:5/785: truncate d7/d16/d18/d25/d28/f3d 1986067 0 2024-03-14T13:02:27.776 INFO:tasks.workunit.client.0.smithi111.stdout:2/782: rmdir dd/d11/d1a/d56/dab 39 2024-03-14T13:02:27.776 INFO:tasks.workunit.client.0.smithi111.stdout:5/987: creat d2/d45/d76/f13a x:0 0 0 2024-03-14T13:02:27.776 INFO:tasks.workunit.client.1.smithi137.stdout:8/935: sync 2024-03-14T13:02:27.777 INFO:tasks.workunit.client.0.smithi111.stdout:9/824: mkdir d5/d42/d71/dbd/d120 0 2024-03-14T13:02:27.778 INFO:tasks.workunit.client.1.smithi137.stdout:4/866: symlink d4/d9/d3a/d41/d38/d79/dbe/l116 0 2024-03-14T13:02:27.778 INFO:tasks.workunit.client.1.smithi137.stdout:6/838: truncate d2/d6/d1d/fdd 65368 0 2024-03-14T13:02:27.780 INFO:tasks.workunit.client.1.smithi137.stdout:7/869: mknod d3/d6/d59/daa/dcf/d6a/d75/c120 0 2024-03-14T13:02:27.782 INFO:tasks.workunit.client.1.smithi137.stdout:7/870: chown d3/d6/d2e/l99 0 1 2024-03-14T13:02:27.783 INFO:tasks.workunit.client.1.smithi137.stdout:6/839: readlink d2/l5 0 2024-03-14T13:02:27.783 INFO:tasks.workunit.client.0.smithi111.stdout:7/806: rmdir d0/d1/d4a/d75 0 2024-03-14T13:02:27.785 INFO:tasks.workunit.client.1.smithi137.stdout:1/988: getdents d7/d9/d19/d1f/da0/d151 0 2024-03-14T13:02:27.785 INFO:tasks.workunit.client.1.smithi137.stdout:3/694: dwrite d4/df/f96 [0,4194304] 0 2024-03-14T13:02:27.786 INFO:tasks.workunit.client.0.smithi111.stdout:2/783: dwrite dd/fe4 [0,4194304] 0 2024-03-14T13:02:27.788 INFO:tasks.workunit.client.1.smithi137.stdout:7/871: creat d3/d54/db4/f121 x:0 0 0 2024-03-14T13:02:27.789 INFO:tasks.workunit.client.0.smithi111.stdout:7/807: fdatasync d0/d1/d4a/fe9 0 2024-03-14T13:02:27.792 INFO:tasks.workunit.client.0.smithi111.stdout:3/858: sync 2024-03-14T13:02:27.792 INFO:tasks.workunit.client.0.smithi111.stdout:9/825: symlink d5/d42/d71/dbd/d120/l121 0 2024-03-14T13:02:27.795 INFO:tasks.workunit.client.0.smithi111.stdout:6/728: dwrite d0/d17/d20/d65/d93/fe0 [0,4194304] 0 2024-03-14T13:02:27.796 INFO:tasks.workunit.client.0.smithi111.stdout:3/859: write d0/d2d/d35/d69/de6/df4/ffd [3586870,127469] 0 2024-03-14T13:02:27.796 INFO:tasks.workunit.client.0.smithi111.stdout:7/808: rmdir d0/d1/d5/d56 39 2024-03-14T13:02:27.796 INFO:tasks.workunit.client.1.smithi137.stdout:6/840: chown d2/dd6/fdb 23982431 1 2024-03-14T13:02:27.799 INFO:tasks.workunit.client.0.smithi111.stdout:3/860: fdatasync d0/d2d/d35/d69/d10a/f128 0 2024-03-14T13:02:27.800 INFO:tasks.workunit.client.0.smithi111.stdout:5/988: getdents d2/d16/d49/d7c/d107 0 2024-03-14T13:02:27.801 INFO:tasks.workunit.client.1.smithi137.stdout:6/841: write d2/dd6/fdb [99055,63292] 0 2024-03-14T13:02:27.801 INFO:tasks.workunit.client.0.smithi111.stdout:9/826: symlink d5/d42/da4/l122 0 2024-03-14T13:02:27.803 INFO:tasks.workunit.client.1.smithi137.stdout:4/867: truncate d4/f48 2377547 0 2024-03-14T13:02:27.804 INFO:tasks.workunit.client.1.smithi137.stdout:3/695: dwrite d4/d19/f90 [0,4194304] 0 2024-03-14T13:02:27.806 INFO:tasks.workunit.client.0.smithi111.stdout:6/729: creat d0/d17/d20/d94/dbd/f10a x:0 0 0 2024-03-14T13:02:27.808 INFO:tasks.workunit.client.1.smithi137.stdout:8/936: dwrite d1/dc/d71/db0/f10f [0,4194304] 0 2024-03-14T13:02:27.808 INFO:tasks.workunit.client.0.smithi111.stdout:9/827: read d5/d6/fe1 [127357,96312] 0 2024-03-14T13:02:27.817 INFO:tasks.workunit.client.1.smithi137.stdout:2/916: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf [0,4194304] 0 2024-03-14T13:02:27.818 INFO:tasks.workunit.client.0.smithi111.stdout:6/730: mkdir d0/d17/d21/d69/d3d/d10b 0 2024-03-14T13:02:27.818 INFO:tasks.workunit.client.0.smithi111.stdout:7/809: dwrite d0/d1/d5/d38/d5c/fb8 [0,4194304] 0 2024-03-14T13:02:27.819 INFO:tasks.workunit.client.1.smithi137.stdout:6/842: creat d2/d54/d55/d6d/dc3/dce/f11e x:0 0 0 2024-03-14T13:02:27.820 INFO:tasks.workunit.client.0.smithi111.stdout:5/989: rename d2/d16/d49/d7c/d95/f5e to d2/d45/d76/d9a/d9f/d12e/f13b 0 2024-03-14T13:02:27.822 INFO:tasks.workunit.client.1.smithi137.stdout:3/696: write d4/de/d16/d27/dd3/fab [683694,103245] 0 2024-03-14T13:02:27.822 INFO:tasks.workunit.client.0.smithi111.stdout:9/828: truncate d5/d42/f46 4296043 0 2024-03-14T13:02:27.823 INFO:tasks.workunit.client.1.smithi137.stdout:7/872: rename d3/d6/d64/d9e/d10d/d118 to d3/d6/d122 0 2024-03-14T13:02:27.823 INFO:tasks.workunit.client.1.smithi137.stdout:1/989: getdents d7/d9/d19/d40/d50 0 2024-03-14T13:02:27.827 INFO:tasks.workunit.client.1.smithi137.stdout:4/868: getdents d4/d10/d42 0 2024-03-14T13:02:27.829 INFO:tasks.workunit.client.0.smithi111.stdout:7/810: write d0/d11/d98/fea [326706,104328] 0 2024-03-14T13:02:27.830 INFO:tasks.workunit.client.1.smithi137.stdout:6/843: getdents d2/d6/de/d45/d59/d102 0 2024-03-14T13:02:27.831 INFO:tasks.workunit.client.0.smithi111.stdout:2/784: dread dd/d17/d34/d74/dca/ff7 [8388608,4194304] 0 2024-03-14T13:02:27.832 INFO:tasks.workunit.client.1.smithi137.stdout:2/917: dwrite d2/f4 [0,4194304] 0 2024-03-14T13:02:27.834 INFO:tasks.workunit.client.0.smithi111.stdout:9/829: unlink d5/d6/ff5 0 2024-03-14T13:02:27.834 INFO:tasks.workunit.client.0.smithi111.stdout:5/990: rmdir d2/d45/d76/d9a/d9f/d12e/d9e/db7 39 2024-03-14T13:02:27.834 INFO:tasks.workunit.client.0.smithi111.stdout:1/991: dwrite d20/d21/f5f [0,4194304] 0 2024-03-14T13:02:27.835 INFO:tasks.workunit.client.1.smithi137.stdout:3/697: mknod d4/de/d16/d1c/db8/dd1/ce1 0 2024-03-14T13:02:27.835 INFO:tasks.workunit.client.0.smithi111.stdout:6/731: creat d0/d17/f10c x:0 0 0 2024-03-14T13:02:27.835 INFO:tasks.workunit.client.1.smithi137.stdout:0/880: write d1/d5/d1b/d1e/d25/d55/f60 [3338308,48598] 0 2024-03-14T13:02:27.839 INFO:tasks.workunit.client.0.smithi111.stdout:7/811: rename d0/d1/c67 to d0/d11/da8/dd4/cef 0 2024-03-14T13:02:27.842 INFO:tasks.workunit.client.0.smithi111.stdout:1/992: rmdir d20/d25/da8/ddd 39 2024-03-14T13:02:27.843 INFO:tasks.workunit.client.0.smithi111.stdout:9/830: mkdir d5/d42/d71/dbd/d123 0 2024-03-14T13:02:27.844 INFO:tasks.workunit.client.0.smithi111.stdout:1/993: write f9 [850991,82341] 0 2024-03-14T13:02:27.844 INFO:tasks.workunit.client.0.smithi111.stdout:6/732: mknod d0/d17/d20/d4a/d59/c10d 0 2024-03-14T13:02:27.844 INFO:tasks.workunit.client.1.smithi137.stdout:7/873: creat d3/d6/d59/daa/dcf/d6a/da1/f123 x:0 0 0 2024-03-14T13:02:27.844 INFO:tasks.workunit.client.1.smithi137.stdout:1/990: fdatasync d7/d9/d19/d1f/f97 0 2024-03-14T13:02:27.848 INFO:tasks.workunit.client.0.smithi111.stdout:1/994: creat d20/d21/dc1/f151 x:0 0 0 2024-03-14T13:02:27.849 INFO:tasks.workunit.client.0.smithi111.stdout:3/861: dread d0/d2d/d35/d50/da2/db3/d71/dbc/fd0 [0,4194304] 0 2024-03-14T13:02:27.850 INFO:tasks.workunit.client.1.smithi137.stdout:6/844: dread d2/d54/d55/d9c/dc0/dc4/fcf [0,4194304] 0 2024-03-14T13:02:27.851 INFO:tasks.workunit.client.0.smithi111.stdout:3/862: write d0/d2d/d35/d50/da2/db3/d6f/fdb [3628164,67707] 0 2024-03-14T13:02:27.852 INFO:tasks.workunit.client.1.smithi137.stdout:0/881: symlink d1/d44/d72/de8/d125/l12b 0 2024-03-14T13:02:27.854 INFO:tasks.workunit.client.0.smithi111.stdout:9/831: creat d5/d6/d14/d7a/f124 x:0 0 0 2024-03-14T13:02:27.854 INFO:tasks.workunit.client.1.smithi137.stdout:3/698: symlink d4/df/le2 0 2024-03-14T13:02:27.854 INFO:tasks.workunit.client.0.smithi111.stdout:5/991: creat d2/d16/d49/d33/de5/f13c x:0 0 0 2024-03-14T13:02:27.855 INFO:tasks.workunit.client.0.smithi111.stdout:6/733: mkdir d0/d17/d20/d4a/d68/d10e 0 2024-03-14T13:02:27.855 INFO:tasks.workunit.client.0.smithi111.stdout:2/785: rename c0 to dd/d11/d1a/d56/c106 0 2024-03-14T13:02:27.855 INFO:tasks.workunit.client.0.smithi111.stdout:7/812: rename d0/d1/d5/d38/d1b/d9c/dee to d0/d1/d5/d38/d1b/d9c/dee/df0 22 2024-03-14T13:02:27.856 INFO:tasks.workunit.client.1.smithi137.stdout:1/991: mkdir d7/d29/d112/d152 0 2024-03-14T13:02:27.857 INFO:tasks.workunit.client.0.smithi111.stdout:6/734: readlink d0/ld8 0 2024-03-14T13:02:27.857 INFO:tasks.workunit.client.0.smithi111.stdout:2/786: chown dd/l39 73 1 2024-03-14T13:02:27.859 INFO:tasks.workunit.client.0.smithi111.stdout:7/813: fsync d0/d1/d21/d2a/f8a 0 2024-03-14T13:02:27.860 INFO:tasks.workunit.client.1.smithi137.stdout:7/874: rmdir d3/d5/d5a/d86/ddd 0 2024-03-14T13:02:27.861 INFO:tasks.workunit.client.1.smithi137.stdout:3/699: truncate d4/d19/fb7 1032859 0 2024-03-14T13:02:27.863 INFO:tasks.workunit.client.1.smithi137.stdout:6/845: creat d2/f11f x:0 0 0 2024-03-14T13:02:27.866 INFO:tasks.workunit.client.1.smithi137.stdout:1/992: rename d7/d9/d19/d1f/c96 to d7/d9/d19/d5f/c153 0 2024-03-14T13:02:27.870 INFO:tasks.workunit.client.1.smithi137.stdout:6/846: mknod d2/d54/d55/d6d/dc3/c120 0 2024-03-14T13:02:27.872 INFO:tasks.workunit.client.1.smithi137.stdout:1/993: unlink d7/d9/d19/d1f/d2e/da6/d129/f134 0 2024-03-14T13:02:27.873 INFO:tasks.workunit.client.0.smithi111.stdout:7/814: write d0/d1/d5/d38/d5c/db4/fcd [3419467,388] 0 2024-03-14T13:02:27.875 INFO:tasks.workunit.client.1.smithi137.stdout:1/994: stat d7/d9/d19/d1f/d22/l8d 0 2024-03-14T13:02:27.876 INFO:tasks.workunit.client.0.smithi111.stdout:2/787: creat dd/d17/d53/da4/df5/df6/f107 x:0 0 0 2024-03-14T13:02:27.877 INFO:tasks.workunit.client.1.smithi137.stdout:0/882: dwrite d1/d1f/d80/fbb [0,4194304] 0 2024-03-14T13:02:27.878 INFO:tasks.workunit.client.0.smithi111.stdout:3/863: getdents d0/d2d/d35/d50/ddf 0 2024-03-14T13:02:27.879 INFO:tasks.workunit.client.0.smithi111.stdout:6/735: rename d0/d17/l8e to d0/d17/d20/l10f 0 2024-03-14T13:02:27.879 INFO:tasks.workunit.client.0.smithi111.stdout:2/788: fdatasync dd/d17/d53/f92 0 2024-03-14T13:02:27.880 INFO:tasks.workunit.client.1.smithi137.stdout:6/847: mkdir d2/d6/de/d45/d59/d39/d3c/d83/d121 0 2024-03-14T13:02:27.882 INFO:tasks.workunit.client.1.smithi137.stdout:0/883: rename d1/d5/d1b/d42/f4f to d1/d5/d1b/d1e/d25/d8a/f12c 0 2024-03-14T13:02:27.883 INFO:tasks.workunit.client.1.smithi137.stdout:0/884: fsync d1/d44/d72/d7e/fcb 0 2024-03-14T13:02:27.884 INFO:tasks.workunit.client.1.smithi137.stdout:0/885: dread - d1/d44/d72/de8/f101 zero size 2024-03-14T13:02:27.885 INFO:tasks.workunit.client.0.smithi111.stdout:9/832: rename d5/d6/d8/d74/l9e to d5/d6/d8/def/l125 0 2024-03-14T13:02:27.886 INFO:tasks.workunit.client.1.smithi137.stdout:3/700: write d4/de/d16/fbc [857615,18454] 0 2024-03-14T13:02:27.888 INFO:tasks.workunit.client.0.smithi111.stdout:5/992: dwrite d2/d45/d76/d9a/d9f/d12e/d9e/db7/dbb/fcf [0,4194304] 0 2024-03-14T13:02:27.890 INFO:tasks.workunit.client.1.smithi137.stdout:0/886: unlink d1/d5/d1b/lad 0 2024-03-14T13:02:27.892 INFO:tasks.workunit.client.0.smithi111.stdout:2/789: truncate dd/f61 527813 0 2024-03-14T13:02:27.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:27 smithi111 ceph-mon[39688]: pgmap v28: 65 pgs: 65 active+clean; 3.0 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 104 MiB/s rd, 272 MiB/s wr, 557 op/s 2024-03-14T13:02:27.895 INFO:tasks.workunit.client.0.smithi111.stdout:7/815: dread d0/d1/d4a/d52/f6c [0,4194304] 0 2024-03-14T13:02:27.895 INFO:tasks.workunit.client.0.smithi111.stdout:1/995: dwrite d20/f23 [0,4194304] 0 2024-03-14T13:02:27.896 INFO:tasks.workunit.client.1.smithi137.stdout:6/848: dwrite d2/d6/da7/fd4 [0,4194304] 0 2024-03-14T13:02:27.900 INFO:tasks.workunit.client.1.smithi137.stdout:6/849: truncate d2/d54/d55/d6d/ff8 719789 0 2024-03-14T13:02:27.902 INFO:tasks.workunit.client.1.smithi137.stdout:1/995: dwrite d7/d9/d19/d40/d50/f80 [0,4194304] 0 2024-03-14T13:02:27.903 INFO:tasks.workunit.client.1.smithi137.stdout:6/850: chown d2/d6/d1d/d88/df4 6256 1 2024-03-14T13:02:27.904 INFO:tasks.workunit.client.0.smithi111.stdout:2/790: fdatasync dd/d17/d34/d74/dca/ff7 0 2024-03-14T13:02:27.907 INFO:tasks.workunit.client.0.smithi111.stdout:9/833: truncate d5/d42/d71/d96/fde 841556 0 2024-03-14T13:02:27.908 INFO:tasks.workunit.client.0.smithi111.stdout:3/864: truncate d0/d2d/d35/d50/da2/db3/d6f/fc5 801215 0 2024-03-14T13:02:27.909 INFO:tasks.workunit.client.0.smithi111.stdout:1/996: truncate d20/d25/d31/d80/f92 1104692 0 2024-03-14T13:02:27.910 INFO:tasks.workunit.client.0.smithi111.stdout:7/816: dwrite d0/d1/d5/d38/d1b/fa1 [0,4194304] 0 2024-03-14T13:02:27.911 INFO:tasks.workunit.client.1.smithi137.stdout:1/996: dread d7/d4d/fca [0,4194304] 0 2024-03-14T13:02:27.913 INFO:tasks.workunit.client.0.smithi111.stdout:9/834: write d5/d6/d2a/d2d/fa8 [951690,55919] 0 2024-03-14T13:02:27.915 INFO:tasks.workunit.client.1.smithi137.stdout:6/851: link d2/d6/f5c d2/d6/de/d45/d59/d39/f122 0 2024-03-14T13:02:27.918 INFO:tasks.workunit.client.0.smithi111.stdout:2/791: rmdir dd/d17/d34/dbf 39 2024-03-14T13:02:27.921 INFO:tasks.workunit.client.1.smithi137.stdout:0/887: write d1/f9b [616481,64145] 0 2024-03-14T13:02:27.926 INFO:tasks.workunit.client.1.smithi137.stdout:0/888: creat d1/d1f/f12d x:0 0 0 2024-03-14T13:02:27.927 INFO:tasks.workunit.client.1.smithi137.stdout:6/852: dwrite d2/d6/de/f68 [0,4194304] 0 2024-03-14T13:02:27.931 INFO:tasks.workunit.client.0.smithi111.stdout:3/865: creat d0/d2d/d35/d69/de6/df4/f138 x:0 0 0 2024-03-14T13:02:27.931 INFO:tasks.workunit.client.0.smithi111.stdout:6/736: dwrite d0/d17/d20/d4a/d59/d82/f102 [0,4194304] 0 2024-03-14T13:02:27.931 INFO:tasks.workunit.client.0.smithi111.stdout:1/997: dread d20/f44 [0,4194304] 0 2024-03-14T13:02:27.931 INFO:tasks.workunit.client.0.smithi111.stdout:7/817: rmdir d0/d1/d5/d56/d70/dd5 39 2024-03-14T13:02:27.931 INFO:tasks.workunit.client.0.smithi111.stdout:6/737: unlink d0/d17/d21/d69/d3d/d53/d81/da1/ca6 0 2024-03-14T13:02:27.937 INFO:tasks.workunit.client.1.smithi137.stdout:0/889: readlink d1/d5/d1b/d1e/d25/d8a/l75 0 2024-03-14T13:02:27.944 INFO:tasks.workunit.client.0.smithi111.stdout:3/866: symlink d0/d2d/d35/d50/da2/db3/d71/dbc/l139 0 2024-03-14T13:02:27.946 INFO:tasks.workunit.client.0.smithi111.stdout:6/738: mkdir d0/d17/d21/d69/d3d/d10b/d110 0 2024-03-14T13:02:27.947 INFO:tasks.workunit.client.1.smithi137.stdout:3/701: truncate d4/de/d13/f36 6864460 0 2024-03-14T13:02:27.948 INFO:tasks.workunit.client.1.smithi137.stdout:3/702: chown d4/de/d16/d27/dd3/c91 90 1 2024-03-14T13:02:27.948 INFO:tasks.workunit.client.0.smithi111.stdout:7/818: getdents d0/d1/d5/d38/d1b/d99 0 2024-03-14T13:02:27.951 INFO:tasks.workunit.client.1.smithi137.stdout:1/997: truncate d7/d31/d54/f101 4640146 0 2024-03-14T13:02:27.953 INFO:tasks.workunit.client.1.smithi137.stdout:6/853: fsync d2/d6/de/d45/d97/d9f/fb1 0 2024-03-14T13:02:27.956 INFO:tasks.workunit.client.1.smithi137.stdout:1/998: rename d7/d9/d19/d1f/d26/d7d/la4 to d7/d9/d19/d1f/d26/d7d/d91/l154 0 2024-03-14T13:02:27.957 INFO:tasks.workunit.client.1.smithi137.stdout:6/854: dread - d2/d6/de/f8c zero size 2024-03-14T13:02:27.960 INFO:tasks.workunit.client.1.smithi137.stdout:0/890: getdents d1/d5/d1b/d1e/d25/d26/d10b 0 2024-03-14T13:02:27.962 INFO:tasks.workunit.client.1.smithi137.stdout:0/891: chown d1/d5/d1b/d1e/d25/d8a/d4a/d54/f6f 2 1 2024-03-14T13:02:27.962 INFO:tasks.workunit.client.0.smithi111.stdout:3/867: dwrite d0/d2d/d35/d69/de6/df4/ffd [0,4194304] 0 2024-03-14T13:02:27.963 INFO:tasks.workunit.client.1.smithi137.stdout:1/999: rename d7/d9/d19/d40/d42/f13f to d7/d31/d54/ded/dfb/d120/f155 0 2024-03-14T13:02:27.965 INFO:tasks.workunit.client.1.smithi137.stdout:0/892: rename d1/d44/c68 to d1/d5/d1b/d1e/d25/d26/c12e 0 2024-03-14T13:02:27.966 INFO:tasks.workunit.client.0.smithi111.stdout:7/819: truncate d0/d1/d21/d2a/f8a 2636274 0 2024-03-14T13:02:27.966 INFO:tasks.workunit.client.1.smithi137.stdout:6/855: link d2/d54/d55/d6d/d76/d57/lde d2/d54/d55/d9c/dbf/l123 0 2024-03-14T13:02:27.967 INFO:tasks.workunit.client.0.smithi111.stdout:3/868: readlink d0/d2d/d35/d50/da2/ldd 0 2024-03-14T13:02:27.967 INFO:tasks.workunit.client.1.smithi137.stdout:0/893: truncate d1/d5/d1b/d42/d3a/dc6/fe4 560892 0 2024-03-14T13:02:27.968 INFO:tasks.workunit.client.1.smithi137.stdout:0/894: fdatasync d1/d5/d1b/d1e/fe1 0 2024-03-14T13:02:27.976 INFO:tasks.workunit.client.0.smithi111.stdout:3/869: read d0/d2d/d35/d69/de6/df4/ffd [1724937,122094] 0 2024-03-14T13:02:27.980 INFO:tasks.workunit.client.1.smithi137.stdout:9/930: sync 2024-03-14T13:02:27.980 INFO:tasks.workunit.client.0.smithi111.stdout:5/993: dwrite d2/d45/d76/d9a/d9f/d12e/d9e/fbe [0,4194304] 0 2024-03-14T13:02:27.980 INFO:tasks.workunit.client.0.smithi111.stdout:2/792: dwrite dd/d17/d34/d73/d49/f90 [0,4194304] 0 2024-03-14T13:02:27.981 INFO:tasks.workunit.client.0.smithi111.stdout:1/998: dwrite d20/f24 [0,4194304] 0 2024-03-14T13:02:27.981 INFO:tasks.workunit.client.1.smithi137.stdout:6/856: dread d2/d54/d55/fa3 [0,4194304] 0 2024-03-14T13:02:27.984 INFO:tasks.workunit.client.1.smithi137.stdout:6/857: write d2/d6/de/d45/f116 [866233,67887] 0 2024-03-14T13:02:27.985 INFO:tasks.workunit.client.1.smithi137.stdout:9/931: mknod d6/df/d88/de5/df6/d40/c136 0 2024-03-14T13:02:27.988 INFO:tasks.workunit.client.0.smithi111.stdout:9/835: dwrite d5/d6/d8/f12 [0,4194304] 0 2024-03-14T13:02:27.988 INFO:tasks.workunit.client.1.smithi137.stdout:0/895: getdents d1/d5/d1b/d1e/d25/d10d/d112 0 2024-03-14T13:02:27.990 INFO:tasks.workunit.client.0.smithi111.stdout:6/739: truncate d0/d17/d20/f6b 1380781 0 2024-03-14T13:02:27.991 INFO:tasks.workunit.client.1.smithi137.stdout:9/932: write d6/df/d88/de5/df6/d40/f5c [707491,126210] 0 2024-03-14T13:02:27.991 INFO:tasks.workunit.client.0.smithi111.stdout:1/999: creat d20/d21/da7/f152 x:0 0 0 2024-03-14T13:02:27.995 INFO:tasks.workunit.client.0.smithi111.stdout:6/740: symlink d0/dfb/db8/de9/l111 0 2024-03-14T13:02:27.999 INFO:tasks.workunit.client.0.smithi111.stdout:2/793: mknod dd/d17/d34/d74/dc1/c108 0 2024-03-14T13:02:28.000 INFO:tasks.workunit.client.0.smithi111.stdout:9/836: truncate f4 8310100 0 2024-03-14T13:02:28.003 INFO:tasks.workunit.client.0.smithi111.stdout:5/994: truncate d2/d45/d76/d9a/d9f/d12e/fe6 890321 0 2024-03-14T13:02:28.005 INFO:tasks.workunit.client.0.smithi111.stdout:2/794: creat dd/d17/d34/dbf/f109 x:0 0 0 2024-03-14T13:02:28.006 INFO:tasks.workunit.client.0.smithi111.stdout:2/795: truncate dd/da9/fb2 4469261 0 2024-03-14T13:02:28.006 INFO:tasks.workunit.client.0.smithi111.stdout:9/837: rename d5/d42/da4/fcc to d5/d6/d2a/f126 0 2024-03-14T13:02:28.008 INFO:tasks.workunit.client.0.smithi111.stdout:9/838: chown d5/d6/d2a/d5a/f6f 35 1 2024-03-14T13:02:28.009 INFO:tasks.workunit.client.0.smithi111.stdout:5/995: rename d2/d16/d49/d33/de5/f118 to d2/d16/d49/d7c/d117/f13d 0 2024-03-14T13:02:28.010 INFO:tasks.workunit.client.0.smithi111.stdout:2/796: truncate dd/d8d/dc2/fb9 536760 0 2024-03-14T13:02:28.015 INFO:tasks.workunit.client.0.smithi111.stdout:9/839: rename d5/d6/d8/l44 to d5/d6/d8/d74/l127 0 2024-03-14T13:02:28.018 INFO:tasks.workunit.client.0.smithi111.stdout:2/797: write dd/d11/d1a/d56/dab/fee [9582,38243] 0 2024-03-14T13:02:28.018 INFO:tasks.workunit.client.0.smithi111.stdout:7/820: dread d0/d1/d5/f4c [0,4194304] 0 2024-03-14T13:02:28.019 INFO:tasks.workunit.client.0.smithi111.stdout:2/798: chown dd/d11 1154787894 1 2024-03-14T13:02:28.024 INFO:tasks.workunit.client.0.smithi111.stdout:6/741: write d0/d50/df2/fea [3330497,83567] 0 2024-03-14T13:02:28.033 INFO:tasks.workunit.client.0.smithi111.stdout:5/996: truncate d2/d16/d49/d44/d72/de0/f120 2139917 0 2024-03-14T13:02:28.035 INFO:tasks.workunit.client.0.smithi111.stdout:9/840: dwrite d5/d42/f119 [0,4194304] 0 2024-03-14T13:02:28.035 INFO:tasks.workunit.client.0.smithi111.stdout:6/742: truncate d0/d17/d20/d4a/d68/f88 441983 0 2024-03-14T13:02:28.035 INFO:tasks.workunit.client.0.smithi111.stdout:7/821: dwrite d0/d7e/f3d [4194304,4194304] 0 2024-03-14T13:02:28.036 INFO:tasks.workunit.client.0.smithi111.stdout:2/799: dwrite dd/d11/d1a/f1b [0,4194304] 0 2024-03-14T13:02:28.037 INFO:tasks.workunit.client.0.smithi111.stdout:6/743: chown d0/f4 45 1 2024-03-14T13:02:28.038 INFO:tasks.workunit.client.0.smithi111.stdout:2/800: chown lc 2 1 2024-03-14T13:02:28.041 INFO:tasks.workunit.client.0.smithi111.stdout:6/744: rename d0/d17/d21/d69/d3d/d53/d81/da1 to d0/d17/d21/d69/d3d/d53/d81/da1/dc8/d112 22 2024-03-14T13:02:28.043 INFO:tasks.workunit.client.0.smithi111.stdout:7/822: mkdir d0/d1/d22/d3e/df1 0 2024-03-14T13:02:28.043 INFO:tasks.workunit.client.0.smithi111.stdout:5/997: creat d2/d16/d49/d7c/d117/f13e x:0 0 0 2024-03-14T13:02:28.046 INFO:tasks.workunit.client.0.smithi111.stdout:9/841: creat d5/d6/d2a/d5a/f128 x:0 0 0 2024-03-14T13:02:28.048 INFO:tasks.workunit.client.0.smithi111.stdout:2/801: rename dd/d17/d53/da4/da6/c101 to dd/d17/d53/da4/c10a 0 2024-03-14T13:02:28.050 INFO:tasks.workunit.client.0.smithi111.stdout:6/745: mknod d0/d17/d21/d42/c113 0 2024-03-14T13:02:28.051 INFO:tasks.workunit.client.0.smithi111.stdout:9/842: symlink d5/d42/d71/dbd/de0/l129 0 2024-03-14T13:02:28.051 INFO:tasks.workunit.client.0.smithi111.stdout:2/802: mknod dd/d17/d34/dbf/c10b 0 2024-03-14T13:02:28.053 INFO:tasks.workunit.client.0.smithi111.stdout:2/803: mkdir dd/d8d/d10c 0 2024-03-14T13:02:28.054 INFO:tasks.workunit.client.1.smithi137.stdout:6/858: creat d2/d54/d55/d33/f124 x:0 0 0 2024-03-14T13:02:28.055 INFO:tasks.workunit.client.0.smithi111.stdout:9/843: creat d5/d42/f12a x:0 0 0 2024-03-14T13:02:28.056 INFO:tasks.workunit.client.0.smithi111.stdout:2/804: dread dd/d11/d1a/d56/dab/fee [0,4194304] 0 2024-03-14T13:02:28.056 INFO:tasks.workunit.client.0.smithi111.stdout:2/805: read - dd/fdf zero size 2024-03-14T13:02:28.057 INFO:tasks.workunit.client.1.smithi137.stdout:9/933: creat d6/df/d19/d1f/de8/f137 x:0 0 0 2024-03-14T13:02:28.057 INFO:tasks.workunit.client.1.smithi137.stdout:0/896: getdents d1/d5/d1b/d1e/d25/d10d/d112 0 2024-03-14T13:02:28.058 INFO:tasks.workunit.client.1.smithi137.stdout:6/859: dread - d2/d6/de/d45/d59/f79 zero size 2024-03-14T13:02:28.059 INFO:tasks.workunit.client.0.smithi111.stdout:2/806: mkdir dd/d17/d53/da4/d10d 0 2024-03-14T13:02:28.059 INFO:tasks.workunit.client.1.smithi137.stdout:6/860: fdatasync d2/d54/d55/d6d/f87 0 2024-03-14T13:02:28.063 INFO:tasks.workunit.client.0.smithi111.stdout:9/844: dwrite d5/d6/d2a/f4e [0,4194304] 0 2024-03-14T13:02:28.063 INFO:tasks.workunit.client.0.smithi111.stdout:2/807: creat dd/d17/d34/f10e x:0 0 0 2024-03-14T13:02:28.065 INFO:tasks.workunit.client.1.smithi137.stdout:6/861: link d2/d54/d55/d6d/d76/d57/f9a d2/d54/d55/d33/f125 0 2024-03-14T13:02:28.071 INFO:tasks.workunit.client.1.smithi137.stdout:3/703: write d4/de/d16/d1c/d8d/dcd/f8e [909360,72860] 0 2024-03-14T13:02:28.074 INFO:tasks.workunit.client.0.smithi111.stdout:9/845: rmdir d5/d6/d2a/d86 39 2024-03-14T13:02:28.075 INFO:tasks.workunit.client.1.smithi137.stdout:0/897: rmdir d1/d5/d1b/d1e/d25/d26 39 2024-03-14T13:02:28.080 INFO:tasks.workunit.client.0.smithi111.stdout:9/846: creat d5/d42/d71/d96/d105/f12b x:0 0 0 2024-03-14T13:02:28.081 INFO:tasks.workunit.client.0.smithi111.stdout:5/998: dwrite d2/d16/d49/d44/fca [0,4194304] 0 2024-03-14T13:02:28.084 INFO:tasks.workunit.client.0.smithi111.stdout:2/808: dwrite dd/d17/d34/f10e [0,4194304] 0 2024-03-14T13:02:28.084 INFO:tasks.workunit.client.0.smithi111.stdout:7/823: dread d0/d1/d5/f1a [0,4194304] 0 2024-03-14T13:02:28.086 INFO:tasks.workunit.client.1.smithi137.stdout:3/704: write d4/de/d16/d27/f73 [1034501,100388] 0 2024-03-14T13:02:28.088 INFO:tasks.workunit.client.0.smithi111.stdout:7/824: fsync d0/d1/d4a/d52/f62 0 2024-03-14T13:02:28.088 INFO:tasks.workunit.client.1.smithi137.stdout:9/934: dwrite d6/d26/f93 [0,4194304] 0 2024-03-14T13:02:28.091 INFO:tasks.workunit.client.0.smithi111.stdout:2/809: getdents dd/d11/d1a 0 2024-03-14T13:02:28.091 INFO:tasks.workunit.client.0.smithi111.stdout:6/746: dread d0/d17/d21/d69/d3d/f41 [0,4194304] 0 2024-03-14T13:02:28.091 INFO:tasks.workunit.client.1.smithi137.stdout:3/705: mkdir d4/de/d16/d1c/d5d/d85/d6b/de3 0 2024-03-14T13:02:28.098 INFO:tasks.workunit.client.0.smithi111.stdout:7/825: read d0/d1/d21/f3f [10945056,72084] 0 2024-03-14T13:02:28.100 INFO:tasks.workunit.client.0.smithi111.stdout:7/826: mkdir d0/d1/d22/d3e/df1/df2 0 2024-03-14T13:02:28.100 INFO:tasks.workunit.client.1.smithi137.stdout:0/898: dwrite d1/d5/d1b/d1e/f3d [0,4194304] 0 2024-03-14T13:02:28.101 INFO:tasks.workunit.client.1.smithi137.stdout:5/786: sync 2024-03-14T13:02:28.107 INFO:tasks.workunit.client.1.smithi137.stdout:9/935: dwrite d6/df/d88/df4/ffc [0,4194304] 0 2024-03-14T13:02:28.109 INFO:tasks.workunit.client.1.smithi137.stdout:5/787: mknod d7/d16/d18/d25/d28/d3c/d7a/c101 0 2024-03-14T13:02:28.110 INFO:tasks.workunit.client.1.smithi137.stdout:3/706: dwrite d4/de/d16/d1c/d5d/d85/d6b/f6d [0,4194304] 0 2024-03-14T13:02:28.113 INFO:tasks.workunit.client.1.smithi137.stdout:5/788: symlink d7/d36/l102 0 2024-03-14T13:02:28.115 INFO:tasks.workunit.client.0.smithi111.stdout:7/827: dread d0/d1/d5/d38/f94 [0,4194304] 0 2024-03-14T13:02:28.116 INFO:tasks.workunit.client.1.smithi137.stdout:3/707: stat d4/de/d16/d1c/d8d/dcd/fa2 0 2024-03-14T13:02:28.116 INFO:tasks.workunit.client.1.smithi137.stdout:5/789: mkdir d7/d16/d18/d48/d103 0 2024-03-14T13:02:28.118 INFO:tasks.workunit.client.0.smithi111.stdout:7/828: mknod d0/d1/d5/cf3 0 2024-03-14T13:02:28.120 INFO:tasks.workunit.client.1.smithi137.stdout:3/708: creat d4/de/d16/d1c/db8/dd1/fe4 x:0 0 0 2024-03-14T13:02:28.120 INFO:tasks.workunit.client.1.smithi137.stdout:5/790: rmdir d7/d1d/d74 39 2024-03-14T13:02:28.121 INFO:tasks.workunit.client.0.smithi111.stdout:7/829: mknod d0/d1/d22/d3e/df1/df2/cf4 0 2024-03-14T13:02:28.121 INFO:tasks.workunit.client.1.smithi137.stdout:9/936: truncate d6/df/d71/fc4 378405 0 2024-03-14T13:02:28.124 INFO:tasks.workunit.client.0.smithi111.stdout:9/847: truncate d5/d6/d14/f54 477181 0 2024-03-14T13:02:28.125 INFO:tasks.workunit.client.0.smithi111.stdout:7/830: truncate d0/d1/d5/d38/d5c/db4/fcd 5150516 0 2024-03-14T13:02:28.126 INFO:tasks.workunit.client.1.smithi137.stdout:9/937: unlink d6/c9 0 2024-03-14T13:02:28.128 INFO:tasks.workunit.client.0.smithi111.stdout:7/831: dread - d0/d1/d21/fd7 zero size 2024-03-14T13:02:28.129 INFO:tasks.workunit.client.1.smithi137.stdout:5/791: mkdir d7/d16/d18/d25/d104 0 2024-03-14T13:02:28.131 INFO:tasks.workunit.client.1.smithi137.stdout:3/709: dwrite d4/de/d16/d1c/d5d/d85/d6b/fc2 [0,4194304] 0 2024-03-14T13:02:28.132 INFO:tasks.workunit.client.0.smithi111.stdout:9/848: creat d5/d42/d71/d96/dd3/f12c x:0 0 0 2024-03-14T13:02:28.132 INFO:tasks.workunit.client.0.smithi111.stdout:2/810: dwrite dd/d17/d53/f84 [0,4194304] 0 2024-03-14T13:02:28.132 INFO:tasks.workunit.client.0.smithi111.stdout:7/832: read d0/d11/f29 [1684697,64430] 0 2024-03-14T13:02:28.133 INFO:tasks.workunit.client.1.smithi137.stdout:9/938: rename d6/df/d19/d1f/dff to d6/df/d88/df4/d138 0 2024-03-14T13:02:28.133 INFO:tasks.workunit.client.1.smithi137.stdout:0/899: dread d1/d5/d1b/d1e/d25/d55/fb4 [0,4194304] 0 2024-03-14T13:02:28.135 INFO:tasks.workunit.client.0.smithi111.stdout:5/999: dread d2/d16/d49/d44/d72/de0/f120 [0,4194304] 0 2024-03-14T13:02:28.135 INFO:tasks.workunit.client.0.smithi111.stdout:2/811: write dd/d11/d1a/d56/fc9 [184713,23739] 0 2024-03-14T13:02:28.136 INFO:tasks.workunit.client.0.smithi111.stdout:7/833: mknod d0/d1/d4a/d87/cf5 0 2024-03-14T13:02:28.138 INFO:tasks.workunit.client.0.smithi111.stdout:2/812: mknod dd/d11/d1a/d56/dab/c10f 0 2024-03-14T13:02:28.139 INFO:tasks.workunit.client.0.smithi111.stdout:2/813: chown dd/d17/d53/f98 43949612 1 2024-03-14T13:02:28.139 INFO:tasks.workunit.client.0.smithi111.stdout:7/834: mkdir d0/d11/da8/dd4/df6 0 2024-03-14T13:02:28.142 INFO:tasks.workunit.client.1.smithi137.stdout:3/710: readlink d4/de/d13/ld4 0 2024-03-14T13:02:28.142 INFO:tasks.workunit.client.1.smithi137.stdout:0/900: unlink d1/d1f/d80/f86 0 2024-03-14T13:02:28.145 INFO:tasks.workunit.client.0.smithi111.stdout:2/814: creat dd/d17/d53/da4/d105/f110 x:0 0 0 2024-03-14T13:02:28.147 INFO:tasks.workunit.client.0.smithi111.stdout:2/815: chown dd/d17/d34/d73 25611 1 2024-03-14T13:02:28.147 INFO:tasks.workunit.client.0.smithi111.stdout:7/835: rename d0/d1/d21/d2a/c2d to d0/d1/d4a/d52/cf7 0 2024-03-14T13:02:28.147 INFO:tasks.workunit.client.1.smithi137.stdout:9/939: dwrite d6/df/d19/d1f/f95 [4194304,4194304] 0 2024-03-14T13:02:28.147 INFO:tasks.workunit.client.1.smithi137.stdout:0/901: mkdir d1/d5/d1b/d42/d61/dae/d12f 0 2024-03-14T13:02:28.149 INFO:tasks.workunit.client.1.smithi137.stdout:3/711: getdents d4/de/d16/d1c/d5d/d4b 0 2024-03-14T13:02:28.149 INFO:tasks.workunit.client.0.smithi111.stdout:2/816: read dd/d17/d34/d73/d49/fe9 [1501647,116495] 0 2024-03-14T13:02:28.149 INFO:tasks.workunit.client.0.smithi111.stdout:7/836: write d0/d11/d98/fea [1426097,38671] 0 2024-03-14T13:02:28.150 INFO:tasks.workunit.client.0.smithi111.stdout:7/837: write d0/f72 [1275627,110017] 0 2024-03-14T13:02:28.153 INFO:tasks.workunit.client.0.smithi111.stdout:2/817: creat dd/d11/d1a/d56/f111 x:0 0 0 2024-03-14T13:02:28.156 INFO:tasks.workunit.client.1.smithi137.stdout:9/940: getdents d6/df/d71 0 2024-03-14T13:02:28.156 INFO:tasks.workunit.client.0.smithi111.stdout:2/818: creat dd/d17/d53/f112 x:0 0 0 2024-03-14T13:02:28.159 INFO:tasks.workunit.client.0.smithi111.stdout:2/819: creat dd/d8d/d100/f113 x:0 0 0 2024-03-14T13:02:28.159 INFO:tasks.workunit.client.1.smithi137.stdout:0/902: dwrite d1/d44/f124 [0,4194304] 0 2024-03-14T13:02:28.160 INFO:tasks.workunit.client.1.smithi137.stdout:0/903: read - d1/d5/d1b/d1e/d25/d8a/d4a/f113 zero size 2024-03-14T13:02:28.161 INFO:tasks.workunit.client.0.smithi111.stdout:2/820: truncate dd/d11/d1a/f38 997104 0 2024-03-14T13:02:28.162 INFO:tasks.workunit.client.0.smithi111.stdout:2/821: mkdir dd/d17/d34/d74/dca/dcb/d114 0 2024-03-14T13:02:28.164 INFO:tasks.workunit.client.0.smithi111.stdout:2/822: creat dd/d8d/f115 x:0 0 0 2024-03-14T13:02:28.166 INFO:tasks.workunit.client.0.smithi111.stdout:2/823: unlink dd/d17/d34/d73/d49/fe9 0 2024-03-14T13:02:28.167 INFO:tasks.workunit.client.1.smithi137.stdout:0/904: dread d1/d44/d72/df8/f127 [0,4194304] 0 2024-03-14T13:02:28.169 INFO:tasks.workunit.client.0.smithi111.stdout:2/824: getdents dd/d11/d1a 0 2024-03-14T13:02:28.170 INFO:tasks.workunit.client.0.smithi111.stdout:2/825: write dd/d17/d34/d74/dca/ff7 [3547778,40797] 0 2024-03-14T13:02:28.170 INFO:tasks.workunit.client.1.smithi137.stdout:0/905: rmdir d1/d5/d1b/d1e/d25 39 2024-03-14T13:02:28.171 INFO:tasks.workunit.client.0.smithi111.stdout:2/826: stat dd/d17/d53/da4/d10d 0 2024-03-14T13:02:28.174 INFO:tasks.workunit.client.1.smithi137.stdout:0/906: mknod d1/d5/d1b/d1e/d53/d6d/d102/d128/c130 0 2024-03-14T13:02:28.174 INFO:tasks.workunit.client.1.smithi137.stdout:5/792: dread d7/d16/f38 [0,4194304] 0 2024-03-14T13:02:28.176 INFO:tasks.workunit.client.1.smithi137.stdout:5/793: write d7/d16/d18/d48/d9d/fda [1025651,70415] 0 2024-03-14T13:02:28.176 INFO:tasks.workunit.client.1.smithi137.stdout:0/907: creat d1/d44/d72/f131 x:0 0 0 2024-03-14T13:02:28.177 INFO:tasks.workunit.client.1.smithi137.stdout:5/794: truncate d7/d33/fee 246992 0 2024-03-14T13:02:28.180 INFO:tasks.workunit.client.1.smithi137.stdout:5/795: creat d7/d16/d18/d25/d28/dc5/f105 x:0 0 0 2024-03-14T13:02:28.184 INFO:tasks.workunit.client.1.smithi137.stdout:5/796: dread d7/d1d/d74/d83/f86 [0,4194304] 0 2024-03-14T13:02:28.190 INFO:tasks.workunit.client.1.smithi137.stdout:3/712: dread d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:28.191 INFO:tasks.workunit.client.1.smithi137.stdout:3/713: write d4/fd [1758439,8212] 0 2024-03-14T13:02:28.194 INFO:tasks.workunit.client.0.smithi111.stdout:2/827: dread dd/d17/d53/f92 [0,4194304] 0 2024-03-14T13:02:28.196 INFO:tasks.workunit.client.1.smithi137.stdout:3/714: fsync d4/de/d93/fb0 0 2024-03-14T13:02:28.197 INFO:tasks.workunit.client.0.smithi111.stdout:2/828: rmdir dd/d17/d34/de0 39 2024-03-14T13:02:28.197 INFO:tasks.workunit.client.0.smithi111.stdout:7/838: dwrite d0/d1/d4a/d52/f62 [0,4194304] 0 2024-03-14T13:02:28.199 INFO:tasks.workunit.client.1.smithi137.stdout:5/797: getdents d7/d16/d18/d2d 0 2024-03-14T13:02:28.201 INFO:tasks.workunit.client.0.smithi111.stdout:7/839: symlink d0/dc1/lf8 0 2024-03-14T13:02:28.205 INFO:tasks.workunit.client.0.smithi111.stdout:7/840: creat d0/d11/d98/ff9 x:0 0 0 2024-03-14T13:02:28.206 INFO:tasks.workunit.client.1.smithi137.stdout:3/715: write d4/de/d16/d1c/d5d/f43 [4827315,12805] 0 2024-03-14T13:02:28.208 INFO:tasks.workunit.client.0.smithi111.stdout:2/829: dwrite dd/d11/fed [0,4194304] 0 2024-03-14T13:02:28.211 INFO:tasks.workunit.client.0.smithi111.stdout:2/830: fsync dd/d17/d34/d73/d49/f90 0 2024-03-14T13:02:28.211 INFO:tasks.workunit.client.0.smithi111.stdout:7/841: creat d0/ffa x:0 0 0 2024-03-14T13:02:28.213 INFO:tasks.workunit.client.1.smithi137.stdout:9/941: dwrite d6/df/d19/f37 [0,4194304] 0 2024-03-14T13:02:28.214 INFO:tasks.workunit.client.1.smithi137.stdout:5/798: rename d7/d16/d18/d48/d9d to d7/d1d/d106 0 2024-03-14T13:02:28.216 INFO:tasks.workunit.client.1.smithi137.stdout:9/942: chown d6/df/c50 27603 1 2024-03-14T13:02:28.218 INFO:tasks.workunit.client.1.smithi137.stdout:9/943: write d6/df/d88/de5/df6/f43 [4459019,31716] 0 2024-03-14T13:02:28.218 INFO:tasks.workunit.client.0.smithi111.stdout:2/831: dwrite dd/d11/d1a/d88/f8a [0,4194304] 0 2024-03-14T13:02:28.219 INFO:tasks.workunit.client.1.smithi137.stdout:5/799: rmdir d7/d16/d18/d25/d28/d9e 39 2024-03-14T13:02:28.219 INFO:tasks.workunit.client.1.smithi137.stdout:9/944: dread - d6/df/d88/de5/df6/da9/fd6 zero size 2024-03-14T13:02:28.221 INFO:tasks.workunit.client.1.smithi137.stdout:5/800: write d7/fb [1476238,126562] 0 2024-03-14T13:02:28.225 INFO:tasks.workunit.client.0.smithi111.stdout:2/832: creat dd/d8d/d10c/f116 x:0 0 0 2024-03-14T13:02:28.225 INFO:tasks.workunit.client.0.smithi111.stdout:7/842: dread d0/d1/d22/d3e/fd1 [0,4194304] 0 2024-03-14T13:02:28.225 INFO:tasks.workunit.client.1.smithi137.stdout:3/716: rename d4/de/d16/f17 to d4/de/d16/d1c/d5d/d85/d6b/fe5 0 2024-03-14T13:02:28.226 INFO:tasks.workunit.client.1.smithi137.stdout:3/717: stat d4/de/d16/d1c/ld5 0 2024-03-14T13:02:28.229 INFO:tasks.workunit.client.0.smithi111.stdout:7/843: symlink d0/dc1/lfb 0 2024-03-14T13:02:28.229 INFO:tasks.workunit.client.0.smithi111.stdout:2/833: read dd/d17/d34/d73/f60 [423197,18026] 0 2024-03-14T13:02:28.230 INFO:tasks.workunit.client.0.smithi111.stdout:2/834: write dd/d17/d34/d73/d49/fe2 [743164,35218] 0 2024-03-14T13:02:28.231 INFO:tasks.workunit.client.0.smithi111.stdout:7/844: creat d0/d7e/ffc x:0 0 0 2024-03-14T13:02:28.233 INFO:tasks.workunit.client.1.smithi137.stdout:5/801: read d7/d49/d75/fb2 [1904166,77453] 0 2024-03-14T13:02:28.234 INFO:tasks.workunit.client.0.smithi111.stdout:7/845: rename d0/d1/d5/d56/d70/cca to d0/d11/cfd 0 2024-03-14T13:02:28.235 INFO:tasks.workunit.client.1.smithi137.stdout:9/945: dwrite d6/df/d19/d1f/f106 [0,4194304] 0 2024-03-14T13:02:28.236 INFO:tasks.workunit.client.0.smithi111.stdout:7/846: rmdir d0/d11/d98 39 2024-03-14T13:02:28.236 INFO:tasks.workunit.client.1.smithi137.stdout:5/802: stat d7/d33/d65/la7 0 2024-03-14T13:02:28.237 INFO:tasks.workunit.client.0.smithi111.stdout:7/847: fsync d0/d1/d5/d38/d1b/fa1 0 2024-03-14T13:02:28.238 INFO:tasks.workunit.client.0.smithi111.stdout:7/848: write d0/d1/d5/d38/d1b/d9c/dee/f5b [3440592,7494] 0 2024-03-14T13:02:28.241 INFO:tasks.workunit.client.1.smithi137.stdout:9/946: mkdir d6/df/d19/d33/d8f/d124/d139 0 2024-03-14T13:02:28.241 INFO:tasks.workunit.client.1.smithi137.stdout:5/803: fdatasync d7/d49/d75/fb2 0 2024-03-14T13:02:28.241 INFO:tasks.workunit.client.0.smithi111.stdout:7/849: symlink d0/d11/lfe 0 2024-03-14T13:02:28.243 INFO:tasks.workunit.client.0.smithi111.stdout:7/850: rmdir d0/d1/d4a 39 2024-03-14T13:02:28.244 INFO:tasks.workunit.client.1.smithi137.stdout:3/718: dwrite d4/de/d16/d1c/d8d/dcd/f8e [0,4194304] 0 2024-03-14T13:02:28.246 INFO:tasks.workunit.client.0.smithi111.stdout:7/851: mkdir d0/d1/d5/d56/d70/dd5/dff 0 2024-03-14T13:02:28.247 INFO:tasks.workunit.client.1.smithi137.stdout:5/804: creat d7/d1d/d74/d83/f107 x:0 0 0 2024-03-14T13:02:28.247 INFO:tasks.workunit.client.1.smithi137.stdout:9/947: mknod d6/df/d88/de5/df6/da9/dde/c13a 0 2024-03-14T13:02:28.248 INFO:tasks.workunit.client.1.smithi137.stdout:3/719: mkdir d4/de/d16/d1c/d5d/d4b/de6 0 2024-03-14T13:02:28.250 INFO:tasks.workunit.client.1.smithi137.stdout:3/720: chown d4/df/f83 142702 1 2024-03-14T13:02:28.251 INFO:tasks.workunit.client.0.smithi111.stdout:7/852: link d0/d1/d4a/d52/cf7 d0/d1/d22/d3e/df1/df2/c100 0 2024-03-14T13:02:28.252 INFO:tasks.workunit.client.0.smithi111.stdout:7/853: stat d0/d1/d4a/f6a 0 2024-03-14T13:02:28.252 INFO:tasks.workunit.client.1.smithi137.stdout:5/805: creat d7/d16/d18/d25/d28/d3c/dd8/f108 x:0 0 0 2024-03-14T13:02:28.254 INFO:tasks.workunit.client.0.smithi111.stdout:7/854: write d0/d1/d21/f3f [2760122,61886] 0 2024-03-14T13:02:28.254 INFO:tasks.workunit.client.1.smithi137.stdout:5/806: symlink d7/d33/l109 0 2024-03-14T13:02:28.257 INFO:tasks.workunit.client.1.smithi137.stdout:5/807: rmdir d7/d16/d18/d2d/dd1 39 2024-03-14T13:02:28.258 INFO:tasks.workunit.client.1.smithi137.stdout:9/948: dread d6/df/d19/d33/f61 [0,4194304] 0 2024-03-14T13:02:28.258 INFO:tasks.workunit.client.0.smithi111.stdout:7/855: mknod d0/c101 0 2024-03-14T13:02:28.260 INFO:tasks.workunit.client.1.smithi137.stdout:5/808: dread d7/d49/fbd [0,4194304] 0 2024-03-14T13:02:28.260 INFO:tasks.workunit.client.1.smithi137.stdout:9/949: mkdir d6/df/d71/d94/d13b 0 2024-03-14T13:02:28.261 INFO:tasks.workunit.client.1.smithi137.stdout:9/950: chown d6/d53/db7/l102 281 1 2024-03-14T13:02:28.262 INFO:tasks.workunit.client.1.smithi137.stdout:9/951: chown d6/c4e 3535228 1 2024-03-14T13:02:28.262 INFO:tasks.workunit.client.1.smithi137.stdout:5/809: symlink d7/d16/dff/l10a 0 2024-03-14T13:02:28.264 INFO:tasks.workunit.client.1.smithi137.stdout:5/810: fsync d7/d16/d18/d2d/fd4 0 2024-03-14T13:02:28.265 INFO:tasks.workunit.client.0.smithi111.stdout:7/856: rename d0/d1/d5/d38/d1b/d9c/dee/l57 to d0/d1/d5/d38/l102 0 2024-03-14T13:02:28.266 INFO:tasks.workunit.client.1.smithi137.stdout:5/811: creat d7/d33/d65/f10b x:0 0 0 2024-03-14T13:02:28.267 INFO:tasks.workunit.client.0.smithi111.stdout:7/857: creat d0/d11/da8/dd4/f103 x:0 0 0 2024-03-14T13:02:28.271 INFO:tasks.workunit.client.1.smithi137.stdout:5/812: dwrite d7/d16/d18/d25/d28/f43 [0,4194304] 0 2024-03-14T13:02:28.273 INFO:tasks.workunit.client.0.smithi111.stdout:7/858: dwrite d0/d1/d5/d38/d1b/d9c/fbb [4194304,4194304] 0 2024-03-14T13:02:28.273 INFO:tasks.workunit.client.0.smithi111.stdout:2/835: dread dd/d11/d1a/d56/f85 [0,4194304] 0 2024-03-14T13:02:28.275 INFO:tasks.workunit.client.0.smithi111.stdout:2/836: chown dd/d17/d53/d9f/cb3 80 1 2024-03-14T13:02:28.275 INFO:tasks.workunit.client.0.smithi111.stdout:7/859: truncate d0/d1/d22/d3e/faf 1031341 0 2024-03-14T13:02:28.275 INFO:tasks.workunit.client.0.smithi111.stdout:7/860: stat d0/dc1/lf8 0 2024-03-14T13:02:28.280 INFO:tasks.workunit.client.1.smithi137.stdout:5/813: dread d7/d16/d18/f34 [4194304,4194304] 0 2024-03-14T13:02:28.281 INFO:tasks.workunit.client.0.smithi111.stdout:7/861: creat d0/d11/da8/dd4/df6/f104 x:0 0 0 2024-03-14T13:02:28.283 INFO:tasks.workunit.client.0.smithi111.stdout:2/837: dwrite dd/d8d/f115 [0,4194304] 0 2024-03-14T13:02:28.284 INFO:tasks.workunit.client.1.smithi137.stdout:9/952: write d6/df/d88/de5/df6/da9/fe7 [818468,124099] 0 2024-03-14T13:02:28.284 INFO:tasks.workunit.client.1.smithi137.stdout:5/814: mkdir d7/d16/d18/d25/d28/d3c/dd8/d10c 0 2024-03-14T13:02:28.287 INFO:tasks.workunit.client.1.smithi137.stdout:9/953: readlink d6/l135 0 2024-03-14T13:02:28.287 INFO:tasks.workunit.client.0.smithi111.stdout:7/862: symlink d0/d11/d98/l105 0 2024-03-14T13:02:28.289 INFO:tasks.workunit.client.1.smithi137.stdout:5/815: mkdir d7/d16/d18/d48/d10d 0 2024-03-14T13:02:28.290 INFO:tasks.workunit.client.1.smithi137.stdout:9/954: write d6/d91/dc8/f104 [370805,105411] 0 2024-03-14T13:02:28.292 INFO:tasks.workunit.client.1.smithi137.stdout:3/721: dwrite d4/d19/f23 [4194304,4194304] 0 2024-03-14T13:02:28.293 INFO:tasks.workunit.client.1.smithi137.stdout:9/955: chown d6/df/f6a 138060064 1 2024-03-14T13:02:28.293 INFO:tasks.workunit.client.0.smithi111.stdout:2/838: rename dd/d17/d34/d73/d75/f95 to dd/d17/d34/de0/f117 0 2024-03-14T13:02:28.294 INFO:tasks.workunit.client.1.smithi137.stdout:3/722: chown d4/de/d13 35833318 1 2024-03-14T13:02:28.295 INFO:tasks.workunit.client.1.smithi137.stdout:9/956: write d6/df/d19/d33/ddf/d114/f122 [357833,123571] 0 2024-03-14T13:02:28.299 INFO:tasks.workunit.client.1.smithi137.stdout:5/816: rename d7/c44 to d7/d33/c10e 0 2024-03-14T13:02:28.300 INFO:tasks.workunit.client.1.smithi137.stdout:3/723: symlink d4/de/d16/d1c/d8d/le7 0 2024-03-14T13:02:28.300 INFO:tasks.workunit.client.1.smithi137.stdout:9/957: write d6/df/d88/de5/df6/d40/f10d [521873,106940] 0 2024-03-14T13:02:28.305 INFO:tasks.workunit.client.1.smithi137.stdout:9/958: read d6/df/d19/d1f/f6c [244852,52685] 0 2024-03-14T13:02:28.305 INFO:tasks.workunit.client.1.smithi137.stdout:3/724: symlink d4/de/d16/le8 0 2024-03-14T13:02:28.306 INFO:tasks.workunit.client.1.smithi137.stdout:3/725: fdatasync d4/d19/f70 0 2024-03-14T13:02:28.307 INFO:tasks.workunit.client.1.smithi137.stdout:9/959: mkdir d6/df/d19/d33/dec/d13c 0 2024-03-14T13:02:28.309 INFO:tasks.workunit.client.1.smithi137.stdout:9/960: write d6/df/d71/f8b [2020448,625] 0 2024-03-14T13:02:28.313 INFO:tasks.workunit.client.1.smithi137.stdout:3/726: symlink d4/de/dc7/ddf/le9 0 2024-03-14T13:02:28.314 INFO:tasks.workunit.client.1.smithi137.stdout:9/961: getdents d6/df/d19/d33/d8f 0 2024-03-14T13:02:28.316 INFO:tasks.workunit.client.1.smithi137.stdout:9/962: symlink d6/df/d88/df4/l13d 0 2024-03-14T13:02:28.316 INFO:tasks.workunit.client.1.smithi137.stdout:3/727: creat d4/de/d16/d1c/d5d/d85/d6b/de3/fea x:0 0 0 2024-03-14T13:02:28.317 INFO:tasks.workunit.client.1.smithi137.stdout:3/728: chown d4/de/d16/d1c/d8d/le7 137072973 1 2024-03-14T13:02:28.319 INFO:tasks.workunit.client.1.smithi137.stdout:8/937: sync 2024-03-14T13:02:28.319 INFO:tasks.workunit.client.1.smithi137.stdout:3/729: creat d4/df/d82/feb x:0 0 0 2024-03-14T13:02:28.321 INFO:tasks.workunit.client.1.smithi137.stdout:8/938: read d1/dc/d71/db0/dca/d105/f120 [3983295,3495] 0 2024-03-14T13:02:28.321 INFO:tasks.workunit.client.1.smithi137.stdout:3/730: mknod d4/de/d16/d1c/cec 0 2024-03-14T13:02:28.322 INFO:tasks.workunit.client.1.smithi137.stdout:3/731: chown d4/de/d16/fb9 1853003304 1 2024-03-14T13:02:28.323 INFO:tasks.workunit.client.1.smithi137.stdout:8/939: creat d1/dc/d71/f136 x:0 0 0 2024-03-14T13:02:28.325 INFO:tasks.workunit.client.1.smithi137.stdout:8/940: creat d1/dc/d10/dcf/dd2/f137 x:0 0 0 2024-03-14T13:02:28.326 INFO:tasks.workunit.client.1.smithi137.stdout:3/732: dread d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:28.330 INFO:tasks.workunit.client.0.smithi111.stdout:7/863: rmdir d0 39 2024-03-14T13:02:28.333 INFO:tasks.workunit.client.0.smithi111.stdout:2/839: write dd/d17/d34/d73/d49/f82 [2056722,92681] 0 2024-03-14T13:02:28.337 INFO:tasks.workunit.client.0.smithi111.stdout:2/840: write dd/d11/f6d [3412776,30524] 0 2024-03-14T13:02:28.338 INFO:tasks.workunit.client.0.smithi111.stdout:2/841: fsync dd/d11/d1a/f1c 0 2024-03-14T13:02:28.344 INFO:tasks.workunit.client.0.smithi111.stdout:2/842: dwrite dd/d17/d34/d73/d49/f51 [0,4194304] 0 2024-03-14T13:02:28.345 INFO:tasks.workunit.client.0.smithi111.stdout:2/843: fsync dd/d11/d8b/ff8 0 2024-03-14T13:02:28.347 INFO:tasks.workunit.client.0.smithi111.stdout:2/844: mknod dd/d8d/d100/c118 0 2024-03-14T13:02:28.347 INFO:tasks.workunit.client.0.smithi111.stdout:2/845: fdatasync dd/d17/d34/d74/f9e 0 2024-03-14T13:02:28.356 INFO:tasks.workunit.client.0.smithi111.stdout:2/846: dread dd/d11/d1a/f62 [0,4194304] 0 2024-03-14T13:02:28.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/847: creat dd/d17/d34/d74/dca/f119 x:0 0 0 2024-03-14T13:02:28.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/848: stat dd/d11/l6c 0 2024-03-14T13:02:28.360 INFO:tasks.workunit.client.0.smithi111.stdout:7/864: dread d0/d1/d5/d38/d5c/fc0 [0,4194304] 0 2024-03-14T13:02:28.362 INFO:tasks.workunit.client.0.smithi111.stdout:2/849: mknod dd/d11/d1a/c11a 0 2024-03-14T13:02:28.364 INFO:tasks.workunit.client.0.smithi111.stdout:7/865: rename d0/d1/d21/dba to d0/d1/d5/d38/d1b/d9c/dee/d106 0 2024-03-14T13:02:28.365 INFO:tasks.workunit.client.0.smithi111.stdout:2/850: stat dd/d17/d34/d73/d49/ca3 0 2024-03-14T13:02:28.365 INFO:tasks.workunit.client.0.smithi111.stdout:2/851: chown dd/d17/d34/l1e 820844 1 2024-03-14T13:02:28.367 INFO:tasks.workunit.client.0.smithi111.stdout:2/852: write dd/d17/d34/d74/dca/f119 [225886,109336] 0 2024-03-14T13:02:28.368 INFO:tasks.workunit.client.0.smithi111.stdout:2/853: read - dd/d17/d34/d73/fce zero size 2024-03-14T13:02:28.370 INFO:tasks.workunit.client.0.smithi111.stdout:7/866: dwrite d0/d1/d21/d5d/f84 [0,4194304] 0 2024-03-14T13:02:28.374 INFO:tasks.workunit.client.0.smithi111.stdout:7/867: fdatasync d0/d1/d22/d3e/f64 0 2024-03-14T13:02:28.376 INFO:tasks.workunit.client.0.smithi111.stdout:7/868: readlink d0/d1/d5/d38/d5c/l91 0 2024-03-14T13:02:28.377 INFO:tasks.workunit.client.0.smithi111.stdout:2/854: rename dd/d11/d1a/d56/f78 to dd/d17/d53/da4/df5/f11b 0 2024-03-14T13:02:28.378 INFO:tasks.workunit.client.0.smithi111.stdout:2/855: write dd/d17/d34/d74/dc1/ff0 [50471,11774] 0 2024-03-14T13:02:28.379 INFO:tasks.workunit.client.0.smithi111.stdout:2/856: fdatasync dd/d17/ffd 0 2024-03-14T13:02:28.379 INFO:tasks.workunit.client.0.smithi111.stdout:7/869: creat d0/d1/d21/f107 x:0 0 0 2024-03-14T13:02:28.381 INFO:tasks.workunit.client.0.smithi111.stdout:2/857: mkdir dd/d17/d34/d73/d75/d11c 0 2024-03-14T13:02:28.383 INFO:tasks.workunit.client.0.smithi111.stdout:2/858: creat dd/d11/d1a/d88/f11d x:0 0 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:3/733: creat d4/de/d16/d1c/d5d/fed x:0 0 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:5/817: dwrite d7/d16/d18/d25/d28/f3d [0,4194304] 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:8/941: dread d1/d5e/d11a/f65 [0,4194304] 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:5/818: mknod d7/d36/c10f 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:5/819: write d7/f7b [2596656,3548] 0 2024-03-14T13:02:28.437 INFO:tasks.workunit.client.1.smithi137.stdout:5/820: creat d7/d33/dab/f110 x:0 0 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:3/870: sync 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/859: symlink dd/d17/d34/d74/dca/dcb/dd6/l11e 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/860: truncate dd/d11/fed 4584637 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/861: stat dd/d17/d34/f6b 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:2/862: creat dd/d17/d34/d74/f11f x:0 0 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:7/870: rename d0/d1 to d0/d11/da8/dd4/df6/d108 0 2024-03-14T13:02:28.438 INFO:tasks.workunit.client.0.smithi111.stdout:3/871: rename d0/d2d/d35/d50/da2/l107 to d0/d2d/d35/d4c/dad/d117/l13a 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/871: creat d0/d11/da8/dd4/df6/d108/d21/d5d/f109 x:0 0 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/872: chown d0/f72 1 1 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/873: fsync d0/dc1/fc4 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/874: dwrite d0/d7e/ffc [0,4194304] 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:3/872: write d0/d2d/d35/d69/f9d [5292715,91220] 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:3/873: stat d0/d2d/d35/d4c/d82/f7e 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:3/874: write d0/f102 [177448,123143] 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/875: dwrite d0/d11/da8/dd4/f103 [0,4194304] 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/876: write d0/d7e/f49 [1265047,60925] 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/877: readlink d0/d7e/l1d 0 2024-03-14T13:02:28.439 INFO:tasks.workunit.client.0.smithi111.stdout:7/878: dwrite d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:28.441 INFO:tasks.workunit.client.1.smithi137.stdout:5/821: dwrite d7/d1d/d74/d81/f89 [0,4194304] 0 2024-03-14T13:02:28.446 INFO:tasks.workunit.client.0.smithi111.stdout:3/875: dread d0/d2d/d35/d50/da2/fec [0,4194304] 0 2024-03-14T13:02:28.447 INFO:tasks.workunit.client.1.smithi137.stdout:8/942: dread d1/d5e/d11a/d82/d94/fdb [0,4194304] 0 2024-03-14T13:02:28.447 INFO:tasks.workunit.client.0.smithi111.stdout:3/876: fsync d0/d2d/d35/d4c/dad/d117/f130 0 2024-03-14T13:02:28.454 INFO:tasks.workunit.client.1.smithi137.stdout:8/943: dwrite d1/d5e/d11a/d82/de6/f128 [0,4194304] 0 2024-03-14T13:02:28.455 INFO:tasks.workunit.client.0.smithi111.stdout:3/877: dwrite d0/d2d/d35/d50/da2/db3/d71/f120 [0,4194304] 0 2024-03-14T13:02:28.457 INFO:tasks.workunit.client.1.smithi137.stdout:3/734: write d4/de/d13/f21 [3520576,49767] 0 2024-03-14T13:02:28.457 INFO:tasks.workunit.client.0.smithi111.stdout:3/878: getdents d0/d30/df1/d100 0 2024-03-14T13:02:28.459 INFO:tasks.workunit.client.1.smithi137.stdout:8/944: creat d1/dc/da9/f138 x:0 0 0 2024-03-14T13:02:28.462 INFO:tasks.workunit.client.1.smithi137.stdout:8/945: getdents d1/dc/d77 0 2024-03-14T13:02:28.462 INFO:tasks.workunit.client.0.smithi111.stdout:3/879: rmdir d0/d2d/d35/d4c 39 2024-03-14T13:02:28.463 INFO:tasks.workunit.client.1.smithi137.stdout:2/918: sync 2024-03-14T13:02:28.463 INFO:tasks.workunit.client.1.smithi137.stdout:6/862: sync 2024-03-14T13:02:28.463 INFO:tasks.workunit.client.1.smithi137.stdout:7/875: sync 2024-03-14T13:02:28.463 INFO:tasks.workunit.client.1.smithi137.stdout:4/869: sync 2024-03-14T13:02:28.463 INFO:tasks.workunit.client.0.smithi111.stdout:7/879: dread d0/d11/da8/dd4/df6/d108/d21/fa4 [0,4194304] 0 2024-03-14T13:02:28.464 INFO:tasks.workunit.client.1.smithi137.stdout:6/863: readlink d2/d54/l4b 0 2024-03-14T13:02:28.464 INFO:tasks.workunit.client.1.smithi137.stdout:7/876: stat d3/d6/d59/l107 0 2024-03-14T13:02:28.465 INFO:tasks.workunit.client.1.smithi137.stdout:4/870: chown d4/d10/d42/d73/d9d/dbc 7551679 1 2024-03-14T13:02:28.466 INFO:tasks.workunit.client.1.smithi137.stdout:6/864: write d2/d6/f34 [5931697,117789] 0 2024-03-14T13:02:28.467 INFO:tasks.workunit.client.0.smithi111.stdout:7/880: fdatasync d0/d11/da8/dd4/df6/d108/d5/d38/d5c/fc0 0 2024-03-14T13:02:28.468 INFO:tasks.workunit.client.0.smithi111.stdout:3/880: mkdir d0/d2d/d35/d50/ddf/d118/d13b 0 2024-03-14T13:02:28.468 INFO:tasks.workunit.client.1.smithi137.stdout:8/946: creat d1/dc/d3c/f139 x:0 0 0 2024-03-14T13:02:28.468 INFO:tasks.workunit.client.0.smithi111.stdout:7/881: symlink d0/d11/da8/dd4/df6/d108/l10a 0 2024-03-14T13:02:28.469 INFO:tasks.workunit.client.1.smithi137.stdout:5/822: truncate d7/d33/d65/f84 934719 0 2024-03-14T13:02:28.469 INFO:tasks.workunit.client.0.smithi111.stdout:3/881: creat d0/d2d/d35/d50/d6c/d98/f13c x:0 0 0 2024-03-14T13:02:28.470 INFO:tasks.workunit.client.1.smithi137.stdout:4/871: write d4/d27/d5c/d6d/d58/fc1 [2674828,65964] 0 2024-03-14T13:02:28.471 INFO:tasks.workunit.client.0.smithi111.stdout:3/882: write d0/d2d/d35/d69/de6/df4/ffd [204408,73359] 0 2024-03-14T13:02:28.474 INFO:tasks.workunit.client.0.smithi111.stdout:3/883: truncate d0/d2d/d35/d69/d84/fb2 581509 0 2024-03-14T13:02:28.475 INFO:tasks.workunit.client.1.smithi137.stdout:2/919: truncate d2/d7/d2e/d3e/daa/fb0 515784 0 2024-03-14T13:02:28.475 INFO:tasks.workunit.client.1.smithi137.stdout:5/823: dread - d7/d33/dab/f110 zero size 2024-03-14T13:02:28.475 INFO:tasks.workunit.client.0.smithi111.stdout:3/884: stat d0/d30/df1/l12a 0 2024-03-14T13:02:28.476 INFO:tasks.workunit.client.0.smithi111.stdout:3/885: creat d0/d2d/f13d x:0 0 0 2024-03-14T13:02:28.477 INFO:tasks.workunit.client.1.smithi137.stdout:6/865: write d2/d6/de/d45/d59/fe1 [2046149,54160] 0 2024-03-14T13:02:28.479 INFO:tasks.workunit.client.1.smithi137.stdout:7/877: unlink d3/d6/d59/daa/dcf/d6a/d75/d78/fdc 0 2024-03-14T13:02:28.480 INFO:tasks.workunit.client.1.smithi137.stdout:8/947: dread d1/d4a/f66 [0,4194304] 0 2024-03-14T13:02:28.482 INFO:tasks.workunit.client.0.smithi111.stdout:3/886: dread d0/d2d/d35/d69/de6/f9e [0,4194304] 0 2024-03-14T13:02:28.483 INFO:tasks.workunit.client.1.smithi137.stdout:2/920: creat d2/d7/d2e/d3e/daa/f12c x:0 0 0 2024-03-14T13:02:28.483 INFO:tasks.workunit.client.1.smithi137.stdout:6/866: chown d2/d54/d55/d33/f125 1 1 2024-03-14T13:02:28.484 INFO:tasks.workunit.client.1.smithi137.stdout:8/948: read - d1/dc/d71/f136 zero size 2024-03-14T13:02:28.485 INFO:tasks.workunit.client.0.smithi111.stdout:3/887: symlink d0/d2d/d35/d50/da2/db3/l13e 0 2024-03-14T13:02:28.487 INFO:tasks.workunit.client.1.smithi137.stdout:7/878: creat d3/d52/d77/f124 x:0 0 0 2024-03-14T13:02:28.488 INFO:tasks.workunit.client.0.smithi111.stdout:3/888: mknod d0/d30/df1/c13f 0 2024-03-14T13:02:28.489 INFO:tasks.workunit.client.1.smithi137.stdout:4/872: dwrite d4/d27/d100/d8d/dc0/f101 [0,4194304] 0 2024-03-14T13:02:28.490 INFO:tasks.workunit.client.0.smithi111.stdout:3/889: creat d0/d2d/d35/d69/de6/dd8/f140 x:0 0 0 2024-03-14T13:02:28.491 INFO:tasks.workunit.client.1.smithi137.stdout:5/824: dwrite d7/d33/d65/dc2/feb [0,4194304] 0 2024-03-14T13:02:28.492 INFO:tasks.workunit.client.0.smithi111.stdout:9/849: sync 2024-03-14T13:02:28.492 INFO:tasks.workunit.client.0.smithi111.stdout:6/747: sync 2024-03-14T13:02:28.494 INFO:tasks.workunit.client.0.smithi111.stdout:9/850: dread - d5/d6/d14/d7a/f124 zero size 2024-03-14T13:02:28.495 INFO:tasks.workunit.client.1.smithi137.stdout:6/867: creat d2/d54/d55/d6d/dc3/deb/f126 x:0 0 0 2024-03-14T13:02:28.495 INFO:tasks.workunit.client.1.smithi137.stdout:7/879: readlink d3/d6/d64/d97/lf2 0 2024-03-14T13:02:28.495 INFO:tasks.workunit.client.1.smithi137.stdout:6/868: chown d2/d6/de/d45/d97/d9f 2 1 2024-03-14T13:02:28.496 INFO:tasks.workunit.client.1.smithi137.stdout:2/921: getdents d2/d7/df8 0 2024-03-14T13:02:28.497 INFO:tasks.workunit.client.0.smithi111.stdout:6/748: mkdir d0/dfb/d106/d114 0 2024-03-14T13:02:28.500 INFO:tasks.workunit.client.1.smithi137.stdout:7/880: creat d3/d6/d122/f125 x:0 0 0 2024-03-14T13:02:28.501 INFO:tasks.workunit.client.1.smithi137.stdout:5/825: mkdir d7/d1d/d111 0 2024-03-14T13:02:28.502 INFO:tasks.workunit.client.1.smithi137.stdout:6/869: dread - d2/d54/d55/d6d/dc3/deb/ffa zero size 2024-03-14T13:02:28.503 INFO:tasks.workunit.client.0.smithi111.stdout:6/749: unlink d0/d17/lab 0 2024-03-14T13:02:28.503 INFO:tasks.workunit.client.0.smithi111.stdout:9/851: symlink d5/d42/l12d 0 2024-03-14T13:02:28.503 INFO:tasks.workunit.client.1.smithi137.stdout:7/881: chown d3/d6/d64/d97/dd2 3 1 2024-03-14T13:02:28.504 INFO:tasks.workunit.client.0.smithi111.stdout:7/882: dread d0/f2e [0,4194304] 0 2024-03-14T13:02:28.506 INFO:tasks.workunit.client.1.smithi137.stdout:7/882: chown d3/d6/d59/daa/f117 4001716 1 2024-03-14T13:02:28.506 INFO:tasks.workunit.client.1.smithi137.stdout:2/922: dread - d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/fb6 zero size 2024-03-14T13:02:28.506 INFO:tasks.workunit.client.1.smithi137.stdout:8/949: dwrite d1/d5e/d11a/d97/fc0 [0,4194304] 0 2024-03-14T13:02:28.507 INFO:tasks.workunit.client.1.smithi137.stdout:6/870: chown d2/d54/d55/d6d/dc3/c120 1561529013 1 2024-03-14T13:02:28.508 INFO:tasks.workunit.client.0.smithi111.stdout:6/750: creat d0/d17/d20/d4a/d59/dc3/f115 x:0 0 0 2024-03-14T13:02:28.510 INFO:tasks.workunit.client.0.smithi111.stdout:7/883: getdents d0/d11/da8/dd4/df6/d108/d4a/dc7 0 2024-03-14T13:02:28.512 INFO:tasks.workunit.client.0.smithi111.stdout:7/884: fdatasync d0/d11/da8/dd4/df6/d108/d21/d5d/f84 0 2024-03-14T13:02:28.513 INFO:tasks.workunit.client.0.smithi111.stdout:6/751: unlink d0/d17/d20/d4a/d68/ca8 0 2024-03-14T13:02:28.513 INFO:tasks.workunit.client.1.smithi137.stdout:7/883: write d3/d5/f37 [5186887,80127] 0 2024-03-14T13:02:28.516 INFO:tasks.workunit.client.1.smithi137.stdout:3/735: dread d4/de/d16/d1c/d5d/f7c [0,4194304] 0 2024-03-14T13:02:28.517 INFO:tasks.workunit.client.0.smithi111.stdout:7/885: symlink d0/d11/da8/dd4/df6/d108/d5/d38/l10b 0 2024-03-14T13:02:28.518 INFO:tasks.workunit.client.1.smithi137.stdout:2/923: symlink d2/d7/d2e/d3e/d44/d6b/d22/l12d 0 2024-03-14T13:02:28.519 INFO:tasks.workunit.client.1.smithi137.stdout:7/884: mkdir d3/d5/d23/d2c/d126 0 2024-03-14T13:02:28.519 INFO:tasks.workunit.client.0.smithi111.stdout:6/752: getdents d0/d17/d21/d69/d3d/d53/d81/da1 0 2024-03-14T13:02:28.520 INFO:tasks.workunit.client.1.smithi137.stdout:3/736: symlink d4/de/d16/d1c/d5d/de0/lee 0 2024-03-14T13:02:28.521 INFO:tasks.workunit.client.0.smithi111.stdout:7/886: mknod d0/d11/da8/dd4/df6/d108/d22/c10c 0 2024-03-14T13:02:28.521 INFO:tasks.workunit.client.1.smithi137.stdout:6/871: truncate d2/d6/d86/d8a/ff2 495333 0 2024-03-14T13:02:28.523 INFO:tasks.workunit.client.1.smithi137.stdout:8/950: dwrite d1/dc/d71/db0/f10f [0,4194304] 0 2024-03-14T13:02:28.523 INFO:tasks.workunit.client.1.smithi137.stdout:3/737: symlink d4/de/dc7/lef 0 2024-03-14T13:02:28.528 INFO:tasks.workunit.client.0.smithi111.stdout:6/753: dwrite d0/d17/d20/d4a/d59/d82/fe8 [4194304,4194304] 0 2024-03-14T13:02:28.529 INFO:tasks.workunit.client.1.smithi137.stdout:8/951: chown d1/dc/d71/da3/fd0 7987773 1 2024-03-14T13:02:28.530 INFO:tasks.workunit.client.0.smithi111.stdout:6/754: stat d0/d17/d21/d42/ca0 0 2024-03-14T13:02:28.530 INFO:tasks.workunit.client.0.smithi111.stdout:3/890: dread d0/d2d/d35/d50/d6c/d98/f10f [0,4194304] 0 2024-03-14T13:02:28.532 INFO:tasks.workunit.client.1.smithi137.stdout:8/952: read d1/d5e/d11a/d97/d7f/fa0 [1194875,37304] 0 2024-03-14T13:02:28.532 INFO:tasks.workunit.client.0.smithi111.stdout:3/891: chown d0/d2d/f13d 2688 1 2024-03-14T13:02:28.536 INFO:tasks.workunit.client.1.smithi137.stdout:4/873: truncate d4/d9/d3a/d41/f49 423104 0 2024-03-14T13:02:28.537 INFO:tasks.workunit.client.0.smithi111.stdout:3/892: dread - d0/d2d/d35/d4c/db0/f12f zero size 2024-03-14T13:02:28.538 INFO:tasks.workunit.client.1.smithi137.stdout:5/826: truncate d7/d16/f98 223876 0 2024-03-14T13:02:28.539 INFO:tasks.workunit.client.1.smithi137.stdout:4/874: mknod d4/d27/d100/c117 0 2024-03-14T13:02:28.540 INFO:tasks.workunit.client.0.smithi111.stdout:9/852: dwrite d5/d6/d8/d1a/f80 [0,4194304] 0 2024-03-14T13:02:28.541 INFO:tasks.workunit.client.0.smithi111.stdout:3/893: truncate d0/d2d/d35/d50/da2/fec 4740673 0 2024-03-14T13:02:28.542 INFO:tasks.workunit.client.1.smithi137.stdout:4/875: chown d4/d9/d3a/d41/d38/l74 1494 1 2024-03-14T13:02:28.543 INFO:tasks.workunit.client.0.smithi111.stdout:3/894: mknod d0/d2d/d35/d4c/d82/c141 0 2024-03-14T13:02:28.545 INFO:tasks.workunit.client.0.smithi111.stdout:9/853: mkdir d5/d6/d8/d5b/d84/db6/de7/d12e 0 2024-03-14T13:02:28.546 INFO:tasks.workunit.client.0.smithi111.stdout:9/854: chown d5/d6/c18 158315 1 2024-03-14T13:02:28.547 INFO:tasks.workunit.client.1.smithi137.stdout:7/885: dread d3/d6/d2e/f3e [0,4194304] 0 2024-03-14T13:02:28.548 INFO:tasks.workunit.client.1.smithi137.stdout:4/876: symlink d4/d9/d3a/d41/d38/l118 0 2024-03-14T13:02:28.549 INFO:tasks.workunit.client.1.smithi137.stdout:5/827: link d7/d1d/d74/d83/dc4/ld9 d7/d16/d18/d2d/dd1/ddc/l112 0 2024-03-14T13:02:28.549 INFO:tasks.workunit.client.0.smithi111.stdout:9/855: creat d5/d6/d8/f12f x:0 0 0 2024-03-14T13:02:28.550 INFO:tasks.workunit.client.1.smithi137.stdout:7/886: fsync d3/d6/d59/daa/fb3 0 2024-03-14T13:02:28.552 INFO:tasks.workunit.client.1.smithi137.stdout:5/828: symlink d7/d16/l113 0 2024-03-14T13:02:28.552 INFO:tasks.workunit.client.0.smithi111.stdout:9/856: creat d5/d6/d14/d7a/d102/f130 x:0 0 0 2024-03-14T13:02:28.553 INFO:tasks.workunit.client.1.smithi137.stdout:7/887: mkdir d3/d6/d64/d9e/d127 0 2024-03-14T13:02:28.554 INFO:tasks.workunit.client.1.smithi137.stdout:7/888: chown d3/d6/d59/daa/l102 462728057 1 2024-03-14T13:02:28.555 INFO:tasks.workunit.client.1.smithi137.stdout:7/889: fdatasync d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fef 0 2024-03-14T13:02:28.556 INFO:tasks.workunit.client.1.smithi137.stdout:7/890: dread - d3/d54/db4/feb zero size 2024-03-14T13:02:28.557 INFO:tasks.workunit.client.0.smithi111.stdout:7/887: dread d0/d11/da8/dd4/df6/d108/d22/f46 [0,4194304] 0 2024-03-14T13:02:28.558 INFO:tasks.workunit.client.1.smithi137.stdout:3/738: write f1 [1884509,100440] 0 2024-03-14T13:02:28.559 INFO:tasks.workunit.client.1.smithi137.stdout:2/924: truncate d2/d7/d2e/d3e/d44/d6b/d22/dbb/f10a 489891 0 2024-03-14T13:02:28.560 INFO:tasks.workunit.client.0.smithi111.stdout:9/857: dread d5/d6/d14/d7a/fba [4194304,4194304] 0 2024-03-14T13:02:28.560 INFO:tasks.workunit.client.1.smithi137.stdout:6/872: truncate d2/d54/d55/d38/f51 2840435 0 2024-03-14T13:02:28.562 INFO:tasks.workunit.client.0.smithi111.stdout:9/858: chown d5/d6/d2a/d5a/cd7 422 1 2024-03-14T13:02:28.563 INFO:tasks.workunit.client.1.smithi137.stdout:6/873: chown d2/d54/d55/d9c/f11a 10 1 2024-03-14T13:02:28.564 INFO:tasks.workunit.client.1.smithi137.stdout:3/739: mknod d4/de/d16/cf0 0 2024-03-14T13:02:28.565 INFO:tasks.workunit.client.1.smithi137.stdout:7/891: dwrite d3/d6/d59/daa/fb3 [0,4194304] 0 2024-03-14T13:02:28.565 INFO:tasks.workunit.client.0.smithi111.stdout:7/888: dread d0/d11/da8/dd4/df6/d108/d5/d38/f7f [0,4194304] 0 2024-03-14T13:02:28.568 INFO:tasks.workunit.client.1.smithi137.stdout:3/740: rmdir d4/de/d16/d1c/d8d 39 2024-03-14T13:02:28.568 INFO:tasks.workunit.client.1.smithi137.stdout:6/874: write d2/d6/de/d45/d97/d9f/fb8 [1610344,28032] 0 2024-03-14T13:02:28.569 INFO:tasks.workunit.client.1.smithi137.stdout:7/892: write d3/f32 [2646410,105646] 0 2024-03-14T13:02:28.570 INFO:tasks.workunit.client.0.smithi111.stdout:7/889: mkdir d0/d11/da8/dd4/df6/d108/d21/d5d/d10d 0 2024-03-14T13:02:28.570 INFO:tasks.workunit.client.1.smithi137.stdout:0/908: sync 2024-03-14T13:02:28.570 INFO:tasks.workunit.client.0.smithi111.stdout:9/859: dwrite d5/d6/d8/d5b/f11d [0,4194304] 0 2024-03-14T13:02:28.572 INFO:tasks.workunit.client.1.smithi137.stdout:2/925: dwrite d2/d7/d2e/ff5 [0,4194304] 0 2024-03-14T13:02:28.572 INFO:tasks.workunit.client.1.smithi137.stdout:7/893: chown d3/d5/d5a/d86/dc5/dc7 121917505 1 2024-03-14T13:02:28.576 INFO:tasks.workunit.client.0.smithi111.stdout:7/890: creat d0/d11/da8/dd4/df6/d108/d4a/dc7/f10e x:0 0 0 2024-03-14T13:02:28.578 INFO:tasks.workunit.client.0.smithi111.stdout:9/860: fsync d5/f11f 0 2024-03-14T13:02:28.584 INFO:tasks.workunit.client.0.smithi111.stdout:9/861: creat d5/d6/d8/f131 x:0 0 0 2024-03-14T13:02:28.585 INFO:tasks.workunit.client.0.smithi111.stdout:9/862: fsync d5/d42/d71/dbd/dcd/fb4 0 2024-03-14T13:02:28.585 INFO:tasks.workunit.client.1.smithi137.stdout:0/909: mknod d1/d1f/d82/c132 0 2024-03-14T13:02:28.587 INFO:tasks.workunit.client.0.smithi111.stdout:9/863: dread - d5/d6/d8/fd9 zero size 2024-03-14T13:02:28.589 INFO:tasks.workunit.client.0.smithi111.stdout:3/895: dread d0/d2d/d35/d69/f9d [0,4194304] 0 2024-03-14T13:02:28.590 INFO:tasks.workunit.client.0.smithi111.stdout:3/896: write d0/d2d/dc7/f127 [392178,93505] 0 2024-03-14T13:02:28.591 INFO:tasks.workunit.client.1.smithi137.stdout:6/875: dread d2/d4c/f84 [0,4194304] 0 2024-03-14T13:02:28.591 INFO:tasks.workunit.client.0.smithi111.stdout:9/864: rename c3 to d5/d42/d52/c132 0 2024-03-14T13:02:28.591 INFO:tasks.workunit.client.0.smithi111.stdout:9/865: fsync d5/d6/d8/f16 0 2024-03-14T13:02:28.594 INFO:tasks.workunit.client.1.smithi137.stdout:2/926: mkdir d2/d11e/d12e 0 2024-03-14T13:02:28.594 INFO:tasks.workunit.client.1.smithi137.stdout:0/910: mknod d1/d5/d1b/d42/d61/c133 0 2024-03-14T13:02:28.594 INFO:tasks.workunit.client.1.smithi137.stdout:7/894: dwrite d3/d6/d14/f1a [0,4194304] 0 2024-03-14T13:02:28.594 INFO:tasks.workunit.client.0.smithi111.stdout:3/897: unlink d0/d2d/d35/d50/da2/db3/d67/l8c 0 2024-03-14T13:02:28.598 INFO:tasks.workunit.client.0.smithi111.stdout:9/866: dwrite d5/d6/d8/f16 [0,4194304] 0 2024-03-14T13:02:28.602 INFO:tasks.workunit.client.0.smithi111.stdout:9/867: mknod d5/d6/d8/d5b/c133 0 2024-03-14T13:02:28.603 INFO:tasks.workunit.client.1.smithi137.stdout:0/911: creat d1/d5/d1b/d42/d3a/db0/d84/f134 x:0 0 0 2024-03-14T13:02:28.603 INFO:tasks.workunit.client.1.smithi137.stdout:7/895: write d3/d6/d59/daa/dcf/f6b [5788981,10656] 0 2024-03-14T13:02:28.604 INFO:tasks.workunit.client.1.smithi137.stdout:6/876: mknod d2/d6/de/d45/d59/d39/d3c/dfe/c127 0 2024-03-14T13:02:28.605 INFO:tasks.workunit.client.1.smithi137.stdout:7/896: creat d3/d6/d59/daa/dcf/d6a/f128 x:0 0 0 2024-03-14T13:02:28.607 INFO:tasks.workunit.client.0.smithi111.stdout:9/868: creat d5/d6/d8/d5b/d84/db6/f134 x:0 0 0 2024-03-14T13:02:28.608 INFO:tasks.workunit.client.0.smithi111.stdout:9/869: dread - d5/d42/d71/d96/d105/f117 zero size 2024-03-14T13:02:28.608 INFO:tasks.workunit.client.0.smithi111.stdout:9/870: fdatasync d5/d6/faa 0 2024-03-14T13:02:28.609 INFO:tasks.workunit.client.1.smithi137.stdout:6/877: dread - d2/d54/d55/d38/f119 zero size 2024-03-14T13:02:28.610 INFO:tasks.workunit.client.0.smithi111.stdout:9/871: symlink d5/d42/dd4/l135 0 2024-03-14T13:02:28.613 INFO:tasks.workunit.client.0.smithi111.stdout:9/872: rename d5/d42/d71/d96/d105 to d5/d42/d71/d96/dd3/d136 0 2024-03-14T13:02:28.614 INFO:tasks.workunit.client.1.smithi137.stdout:2/927: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/lfc to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/l12f 0 2024-03-14T13:02:28.614 INFO:tasks.workunit.client.1.smithi137.stdout:0/912: truncate d1/d5/d1b/d1e/d25/d8a/d4a/f11e 368631 0 2024-03-14T13:02:28.614 INFO:tasks.workunit.client.1.smithi137.stdout:7/897: symlink d3/d6/d14/d27/dfe/l129 0 2024-03-14T13:02:28.615 INFO:tasks.workunit.client.0.smithi111.stdout:9/873: symlink d5/d6/d2a/d5a/l137 0 2024-03-14T13:02:28.617 INFO:tasks.workunit.client.1.smithi137.stdout:6/878: creat d2/d54/d55/d6d/dc3/deb/f128 x:0 0 0 2024-03-14T13:02:28.619 INFO:tasks.workunit.client.1.smithi137.stdout:7/898: symlink d3/d6/d59/l12a 0 2024-03-14T13:02:28.621 INFO:tasks.workunit.client.1.smithi137.stdout:4/877: dread d4/d27/d5c/d6d/f10e [4194304,4194304] 0 2024-03-14T13:02:28.621 INFO:tasks.workunit.client.1.smithi137.stdout:3/741: dread d4/de/f1f [0,4194304] 0 2024-03-14T13:02:28.622 INFO:tasks.workunit.client.1.smithi137.stdout:6/879: creat d2/d6/d86/f129 x:0 0 0 2024-03-14T13:02:28.623 INFO:tasks.workunit.client.1.smithi137.stdout:0/913: creat d1/f135 x:0 0 0 2024-03-14T13:02:28.624 INFO:tasks.workunit.client.1.smithi137.stdout:7/899: fsync d3/d6/d14/f1a 0 2024-03-14T13:02:28.624 INFO:tasks.workunit.client.1.smithi137.stdout:4/878: fdatasync d4/f89 0 2024-03-14T13:02:28.624 INFO:tasks.workunit.client.0.smithi111.stdout:7/891: dread d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/dc9/dce/fe7 [0,4194304] 0 2024-03-14T13:02:28.625 INFO:tasks.workunit.client.1.smithi137.stdout:6/880: chown d2/d6/lef 702990 1 2024-03-14T13:02:28.626 INFO:tasks.workunit.client.1.smithi137.stdout:3/742: symlink d4/df/lf1 0 2024-03-14T13:02:28.626 INFO:tasks.workunit.client.1.smithi137.stdout:7/900: dread - d3/d5/d68/f115 zero size 2024-03-14T13:02:28.628 INFO:tasks.workunit.client.1.smithi137.stdout:3/743: write d4/de/d16/d27/dd3/fc4 [1040469,126271] 0 2024-03-14T13:02:28.628 INFO:tasks.workunit.client.0.smithi111.stdout:7/892: dread d0/d11/da8/dd4/df6/d108/d22/f46 [0,4194304] 0 2024-03-14T13:02:28.628 INFO:tasks.workunit.client.1.smithi137.stdout:0/914: creat d1/d44/d72/d120/f136 x:0 0 0 2024-03-14T13:02:28.629 INFO:tasks.workunit.client.0.smithi111.stdout:9/874: write d5/d6/d8/f49 [1043111,82773] 0 2024-03-14T13:02:28.630 INFO:tasks.workunit.client.1.smithi137.stdout:7/901: fsync d3/d5/f116 0 2024-03-14T13:02:28.630 INFO:tasks.workunit.client.1.smithi137.stdout:4/879: symlink d4/d9/d8e/l119 0 2024-03-14T13:02:28.630 INFO:tasks.workunit.client.1.smithi137.stdout:0/915: fsync d1/d5/d1b/d1e/d25/d8a/d4a/fd9 0 2024-03-14T13:02:28.632 INFO:tasks.workunit.client.0.smithi111.stdout:7/893: creat d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/ddb/f10f x:0 0 0 2024-03-14T13:02:28.633 INFO:tasks.workunit.client.1.smithi137.stdout:6/881: unlink d2/d54/d55/l10a 0 2024-03-14T13:02:28.633 INFO:tasks.workunit.client.1.smithi137.stdout:4/880: truncate d4/d10/d5a/f114 248703 0 2024-03-14T13:02:28.634 INFO:tasks.workunit.client.1.smithi137.stdout:3/744: chown d4/de/d16/d1c/d8d/cac 2953328 1 2024-03-14T13:02:28.635 INFO:tasks.workunit.client.0.smithi111.stdout:7/894: creat d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/dc9/f110 x:0 0 0 2024-03-14T13:02:28.635 INFO:tasks.workunit.client.1.smithi137.stdout:2/928: link d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/cfb d2/d7/d2e/d3e/d4e/c130 0 2024-03-14T13:02:28.641 INFO:tasks.workunit.client.0.smithi111.stdout:7/895: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:28.641 INFO:tasks.workunit.client.1.smithi137.stdout:7/902: mkdir d3/d5/d23/d3f/d5b/d111/d12b 0 2024-03-14T13:02:28.643 INFO:tasks.workunit.client.1.smithi137.stdout:4/881: dwrite d4/f8 [0,4194304] 0 2024-03-14T13:02:28.645 INFO:tasks.workunit.client.1.smithi137.stdout:3/745: unlink d4/de/d16/d1c/db8/fdb 0 2024-03-14T13:02:28.645 INFO:tasks.workunit.client.0.smithi111.stdout:7/896: truncate d0/d11/da8/dd4/df6/d108/d4a/fe9 1043672 0 2024-03-14T13:02:28.647 INFO:tasks.workunit.client.0.smithi111.stdout:7/897: read d0/d11/da8/dd4/df6/d108/d4a/f60 [334777,65631] 0 2024-03-14T13:02:28.648 INFO:tasks.workunit.client.0.smithi111.stdout:7/898: chown d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5 624020 1 2024-03-14T13:02:28.649 INFO:tasks.workunit.client.0.smithi111.stdout:7/899: chown d0/d11/da8/dd4/df6/d108/d5/l41 3372 1 2024-03-14T13:02:28.652 INFO:tasks.workunit.client.0.smithi111.stdout:7/900: truncate d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/ddb/f10f 737489 0 2024-03-14T13:02:28.653 INFO:tasks.workunit.client.0.smithi111.stdout:9/875: dread d5/f20 [0,4194304] 0 2024-03-14T13:02:28.656 INFO:tasks.workunit.client.0.smithi111.stdout:7/901: creat d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5/f111 x:0 0 0 2024-03-14T13:02:28.657 INFO:tasks.workunit.client.1.smithi137.stdout:4/882: creat d4/d9/d3a/d41/d6a/dff/f11a x:0 0 0 2024-03-14T13:02:28.659 INFO:tasks.workunit.client.1.smithi137.stdout:4/883: write d4/d10/d5a/f114 [1124541,58305] 0 2024-03-14T13:02:28.661 INFO:tasks.workunit.client.1.smithi137.stdout:6/882: dread d2/d6/de/d45/fa8 [0,4194304] 0 2024-03-14T13:02:28.663 INFO:tasks.workunit.client.0.smithi111.stdout:7/902: read d0/d11/da8/dd4/df6/d108/d5/d38/d5c/f8d [3669610,130248] 0 2024-03-14T13:02:28.664 INFO:tasks.workunit.client.1.smithi137.stdout:0/916: dread d1/d5/d1b/d1e/d25/d8a/d4a/fb6 [0,4194304] 0 2024-03-14T13:02:28.665 INFO:tasks.workunit.client.1.smithi137.stdout:4/884: creat d4/d10/d42/daf/f11b x:0 0 0 2024-03-14T13:02:28.666 INFO:tasks.workunit.client.1.smithi137.stdout:6/883: mknod d2/d6/d1d/d88/c12a 0 2024-03-14T13:02:28.666 INFO:tasks.workunit.client.1.smithi137.stdout:0/917: write d1/d44/ffa [535454,93558] 0 2024-03-14T13:02:28.668 INFO:tasks.workunit.client.1.smithi137.stdout:7/903: write d3/d5/f116 [778661,123596] 0 2024-03-14T13:02:28.668 INFO:tasks.workunit.client.1.smithi137.stdout:3/746: dwrite d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:28.668 INFO:tasks.workunit.client.1.smithi137.stdout:4/885: dread - d4/d27/d5c/d77/fbd zero size 2024-03-14T13:02:28.670 INFO:tasks.workunit.client.0.smithi111.stdout:7/903: dwrite d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/fbb [8388608,4194304] 0 2024-03-14T13:02:28.673 INFO:tasks.workunit.client.1.smithi137.stdout:4/886: creat d4/d10/d26/f11c x:0 0 0 2024-03-14T13:02:28.675 INFO:tasks.workunit.client.0.smithi111.stdout:7/904: creat d0/d11/da8/dd4/df6/d108/d5/f112 x:0 0 0 2024-03-14T13:02:28.675 INFO:tasks.workunit.client.0.smithi111.stdout:7/905: fdatasync d0/d11/f73 0 2024-03-14T13:02:28.677 INFO:tasks.workunit.client.1.smithi137.stdout:6/884: unlink d2/d54/d55/d6d/fc7 0 2024-03-14T13:02:28.678 INFO:tasks.workunit.client.1.smithi137.stdout:2/929: write d2/d7/d2e/d3e/d44/d6b/f4a [53837,19247] 0 2024-03-14T13:02:28.679 INFO:tasks.workunit.client.0.smithi111.stdout:7/906: read d0/d11/da8/dd4/df6/d108/d21/d5d/f84 [2363365,73655] 0 2024-03-14T13:02:28.683 INFO:tasks.workunit.client.1.smithi137.stdout:7/904: mknod d3/d6/d59/daa/dcf/c12c 0 2024-03-14T13:02:28.683 INFO:tasks.workunit.client.0.smithi111.stdout:7/907: creat d0/d11/da8/dd4/f113 x:0 0 0 2024-03-14T13:02:28.686 INFO:tasks.workunit.client.0.smithi111.stdout:7/908: rename d0/d11/da8/dd4/cd6 to d0/d11/da8/dd4/df6/d108/d5/d38/d1b/c114 0 2024-03-14T13:02:28.689 INFO:tasks.workunit.client.0.smithi111.stdout:7/909: creat d0/d11/da8/dd4/df6/d108/d21/d5d/d10d/f115 x:0 0 0 2024-03-14T13:02:28.689 INFO:tasks.workunit.client.0.smithi111.stdout:7/910: dread - d0/ffa zero size 2024-03-14T13:02:28.690 INFO:tasks.workunit.client.1.smithi137.stdout:0/918: rename d1/d1f/c106 to d1/d5/d1b/d1e/d53/df0/d121/c137 0 2024-03-14T13:02:28.690 INFO:tasks.workunit.client.1.smithi137.stdout:4/887: mknod d4/d9/d3a/c11d 0 2024-03-14T13:02:28.690 INFO:tasks.workunit.client.1.smithi137.stdout:2/930: creat d2/d11e/f131 x:0 0 0 2024-03-14T13:02:28.690 INFO:tasks.workunit.client.1.smithi137.stdout:7/905: truncate d3/d5/d23/d2c/f53 895632 0 2024-03-14T13:02:28.691 INFO:tasks.workunit.client.1.smithi137.stdout:4/888: creat d4/d27/d100/d8d/f11e x:0 0 0 2024-03-14T13:02:28.693 INFO:tasks.workunit.client.0.smithi111.stdout:7/911: getdents d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/ddb 0 2024-03-14T13:02:28.693 INFO:tasks.workunit.client.1.smithi137.stdout:2/931: readlink d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/dd7/l101 0 2024-03-14T13:02:28.694 INFO:tasks.workunit.client.1.smithi137.stdout:7/906: creat d3/d6/d59/daa/dcf/d6a/da1/f12d x:0 0 0 2024-03-14T13:02:28.696 INFO:tasks.workunit.client.1.smithi137.stdout:0/919: getdents d1/d44/d72/de8/d125 0 2024-03-14T13:02:28.696 INFO:tasks.workunit.client.0.smithi111.stdout:7/912: rename d0/d11/da8/dd4/df6/d108/d5/f1a to d0/d11/da8/dd4/df6/d108/f116 0 2024-03-14T13:02:28.697 INFO:tasks.workunit.client.1.smithi137.stdout:7/907: mkdir d3/d6/d64/d9e/d10d/d12e 0 2024-03-14T13:02:28.698 INFO:tasks.workunit.client.1.smithi137.stdout:4/889: read d4/d27/d5c/d77/f108 [1234608,12554] 0 2024-03-14T13:02:28.699 INFO:tasks.workunit.client.0.smithi111.stdout:7/913: getdents d0/d11/da8/dd4/df6/d108/d5/d56 0 2024-03-14T13:02:28.699 INFO:tasks.workunit.client.1.smithi137.stdout:2/932: rename d2/f4 to d2/d7/d2e/d3e/f132 0 2024-03-14T13:02:28.700 INFO:tasks.workunit.client.1.smithi137.stdout:7/908: dread - d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1/ffc zero size 2024-03-14T13:02:28.702 INFO:tasks.workunit.client.0.smithi111.stdout:7/914: write d0/d11/da8/dd4/df6/d108/d22/d3e/faf [715191,66044] 0 2024-03-14T13:02:28.703 INFO:tasks.workunit.client.1.smithi137.stdout:7/909: write d3/d5/d5a/d86/fe5 [687851,48260] 0 2024-03-14T13:02:28.706 INFO:tasks.workunit.client.0.smithi111.stdout:7/915: dread d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:28.706 INFO:tasks.workunit.client.1.smithi137.stdout:4/890: rename d4/d10/d42 to d4/d9/d3a/d41/d38/d79/dc2/d11f 0 2024-03-14T13:02:28.707 INFO:tasks.workunit.client.0.smithi111.stdout:7/916: chown d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/dee/c79 2870 1 2024-03-14T13:02:28.707 INFO:tasks.workunit.client.1.smithi137.stdout:0/920: dread d1/d5/d1b/d42/d3a/db0/d84/fa4 [0,4194304] 0 2024-03-14T13:02:28.709 INFO:tasks.workunit.client.1.smithi137.stdout:2/933: unlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/f98 0 2024-03-14T13:02:28.709 INFO:tasks.workunit.client.0.smithi111.stdout:7/917: creat d0/d11/da8/dd4/df6/d108/d5/d38/d5c/db4/f117 x:0 0 0 2024-03-14T13:02:28.710 INFO:tasks.workunit.client.1.smithi137.stdout:7/910: creat d3/d6/d59/daa/dcf/d6a/d75/d78/da4/dd1/f12f x:0 0 0 2024-03-14T13:02:28.711 INFO:tasks.workunit.client.0.smithi111.stdout:7/918: chown d0/d11/da8/dd4/df6/d108/d5/d56/d70/f95 13955 1 2024-03-14T13:02:28.713 INFO:tasks.workunit.client.1.smithi137.stdout:4/891: mknod d4/d9/d3a/d41/d6a/dac/c120 0 2024-03-14T13:02:28.713 INFO:tasks.workunit.client.0.smithi111.stdout:7/919: rename d0/dc1/lf8 to d0/d11/da8/dd4/df6/d108/d21/d5d/l118 0 2024-03-14T13:02:28.714 INFO:tasks.workunit.client.1.smithi137.stdout:6/885: write d2/d6/de/d45/d59/d39/d3c/fc8 [165,110218] 0 2024-03-14T13:02:28.715 INFO:tasks.workunit.client.1.smithi137.stdout:2/934: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/f133 x:0 0 0 2024-03-14T13:02:28.718 INFO:tasks.workunit.client.1.smithi137.stdout:0/921: mknod d1/d5/d1b/d1e/c138 0 2024-03-14T13:02:28.720 INFO:tasks.workunit.client.0.smithi111.stdout:7/920: dwrite d0/d11/da8/dd4/df6/d108/d21/d2a/d8c/fb1 [0,4194304] 0 2024-03-14T13:02:28.720 INFO:tasks.workunit.client.1.smithi137.stdout:6/886: symlink d2/d6/d1d/d88/l12b 0 2024-03-14T13:02:28.724 INFO:tasks.workunit.client.1.smithi137.stdout:0/922: mkdir d1/d5/d1b/d42/d3a/dc6/d139 0 2024-03-14T13:02:28.725 INFO:tasks.workunit.client.0.smithi111.stdout:7/921: dread d0/d11/f29 [0,4194304] 0 2024-03-14T13:02:28.729 INFO:tasks.workunit.client.1.smithi137.stdout:6/887: dwrite d2/d54/d55/d6d/f87 [0,4194304] 0 2024-03-14T13:02:28.729 INFO:tasks.workunit.client.1.smithi137.stdout:0/923: write d1/f51 [1097183,83383] 0 2024-03-14T13:02:28.734 INFO:tasks.workunit.client.1.smithi137.stdout:2/935: creat d2/d7/d2e/d3e/d44/d6b/f134 x:0 0 0 2024-03-14T13:02:28.735 INFO:tasks.workunit.client.1.smithi137.stdout:4/892: getdents d4/d9/d3a/d41/dd7 0 2024-03-14T13:02:28.739 INFO:tasks.workunit.client.1.smithi137.stdout:0/924: rename d1/d5/d1b/d42/d3a/f9e to d1/d44/d72/f13a 0 2024-03-14T13:02:28.740 INFO:tasks.workunit.client.1.smithi137.stdout:4/893: fdatasync d4/d27/d5c/d6d/f10e 0 2024-03-14T13:02:28.741 INFO:tasks.workunit.client.1.smithi137.stdout:0/925: readlink d1/d5/d1b/d42/l8d 0 2024-03-14T13:02:28.744 INFO:tasks.workunit.client.1.smithi137.stdout:4/894: mkdir d4/d27/d100/d121 0 2024-03-14T13:02:28.746 INFO:tasks.workunit.client.1.smithi137.stdout:0/926: creat d1/d5/d1b/d42/d61/f13b x:0 0 0 2024-03-14T13:02:28.747 INFO:tasks.workunit.client.1.smithi137.stdout:2/936: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 [0,4194304] 0 2024-03-14T13:02:28.747 INFO:tasks.workunit.client.1.smithi137.stdout:6/888: rmdir d2/d6/d1d 39 2024-03-14T13:02:28.749 INFO:tasks.workunit.client.1.smithi137.stdout:7/911: dwrite d3/d5/d5a/fc6 [0,4194304] 0 2024-03-14T13:02:28.750 INFO:tasks.workunit.client.1.smithi137.stdout:4/895: symlink d4/d27/d5c/d77/l122 0 2024-03-14T13:02:28.751 INFO:tasks.workunit.client.1.smithi137.stdout:6/889: fdatasync d2/d6/d86/f109 0 2024-03-14T13:02:28.752 INFO:tasks.workunit.client.1.smithi137.stdout:4/896: fdatasync d4/f8 0 2024-03-14T13:02:28.756 INFO:tasks.workunit.client.1.smithi137.stdout:7/912: rmdir d3/d6/d14/d27 39 2024-03-14T13:02:28.758 INFO:tasks.workunit.client.1.smithi137.stdout:6/890: mknod d2/d6/de/d45/d59/d39/c12c 0 2024-03-14T13:02:28.761 INFO:tasks.workunit.client.1.smithi137.stdout:6/891: mkdir d2/d54/d55/d33/d12d 0 2024-03-14T13:02:28.761 INFO:tasks.workunit.client.0.smithi111.stdout:2/863: sync 2024-03-14T13:02:28.767 INFO:tasks.workunit.client.1.smithi137.stdout:7/913: dwrite d3/d6/d59/daa/dcf/d6a/fc8 [0,4194304] 0 2024-03-14T13:02:28.771 INFO:tasks.workunit.client.1.smithi137.stdout:7/914: write d3/d5/d23/fb5 [3591681,1331] 0 2024-03-14T13:02:28.777 INFO:tasks.workunit.client.1.smithi137.stdout:4/897: dread d4/d9/d3a/d41/d38/d79/dc2/d11f/d73/fb5 [0,4194304] 0 2024-03-14T13:02:28.779 INFO:tasks.workunit.client.1.smithi137.stdout:0/927: dread d1/d5/d1b/d1e/d53/d6d/f107 [0,4194304] 0 2024-03-14T13:02:28.779 INFO:tasks.workunit.client.1.smithi137.stdout:4/898: fsync d4/f8 0 2024-03-14T13:02:28.780 INFO:tasks.workunit.client.1.smithi137.stdout:2/937: truncate d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/f123 3916923 0 2024-03-14T13:02:28.782 INFO:tasks.workunit.client.1.smithi137.stdout:7/915: dwrite d3/d6/d59/daa/fce [0,4194304] 0 2024-03-14T13:02:28.790 INFO:tasks.workunit.client.0.smithi111.stdout:2/864: rename dd/d11/d1a/fb5 to dd/d8d/f120 0 2024-03-14T13:02:28.791 INFO:tasks.workunit.client.1.smithi137.stdout:0/928: dwrite d1/d5/d1b/d1e/d53/d6d/faa [0,4194304] 0 2024-03-14T13:02:28.791 INFO:tasks.workunit.client.1.smithi137.stdout:4/899: link d4/ce9 d4/d9/d3a/d41/d38/d79/dc2/d11f/d73/d9d/dbc/dd1/c123 0 2024-03-14T13:02:28.795 INFO:tasks.workunit.client.0.smithi111.stdout:2/865: getdents dd/d17/d34/d74/dca 0 2024-03-14T13:02:28.797 INFO:tasks.workunit.client.0.smithi111.stdout:2/866: read dd/f87 [45041,78811] 0 2024-03-14T13:02:28.797 INFO:tasks.workunit.client.1.smithi137.stdout:4/900: symlink d4/d9/d3a/d41/d38/d79/dc2/d11f/d73/l124 0 2024-03-14T13:02:28.800 INFO:tasks.workunit.client.0.smithi111.stdout:2/867: fsync dd/d17/d34/d73/d75/f8e 0 2024-03-14T13:02:28.801 INFO:tasks.workunit.client.0.smithi111.stdout:2/868: read - dd/d8d/d10c/f116 zero size 2024-03-14T13:02:28.802 INFO:tasks.workunit.client.1.smithi137.stdout:6/892: dread d2/d54/d55/d6d/d76/d57/f74 [0,4194304] 0 2024-03-14T13:02:28.807 INFO:tasks.workunit.client.0.smithi111.stdout:2/869: dwrite dd/d8d/f115 [0,4194304] 0 2024-03-14T13:02:28.809 INFO:tasks.workunit.client.1.smithi137.stdout:2/938: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/f63 [0,4194304] 0 2024-03-14T13:02:28.810 INFO:tasks.workunit.client.0.smithi111.stdout:2/870: creat dd/d8d/f121 x:0 0 0 2024-03-14T13:02:28.810 INFO:tasks.workunit.client.1.smithi137.stdout:7/916: dwrite d3/d5/d5a/f98 [0,4194304] 0 2024-03-14T13:02:28.813 INFO:tasks.workunit.client.1.smithi137.stdout:7/917: chown d3/d54 63 1 2024-03-14T13:02:28.814 INFO:tasks.workunit.client.1.smithi137.stdout:0/929: write d1/d44/d56/f5a [327555,115859] 0 2024-03-14T13:02:28.816 INFO:tasks.workunit.client.1.smithi137.stdout:6/893: write d2/d54/d55/d6d/dc3/dce/fdc [727480,14489] 0 2024-03-14T13:02:28.818 INFO:tasks.workunit.client.0.smithi111.stdout:2/871: dwrite dd/d17/d34/ddb/f65 [0,4194304] 0 2024-03-14T13:02:28.819 INFO:tasks.workunit.client.1.smithi137.stdout:2/939: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/fbf [1108567,69280] 0 2024-03-14T13:02:28.823 INFO:tasks.workunit.client.1.smithi137.stdout:7/918: rename d3/d5/f46 to d3/d5/d5a/d86/dc5/f130 0 2024-03-14T13:02:28.825 INFO:tasks.workunit.client.1.smithi137.stdout:0/930: getdents d1/d44/d72/df8 0 2024-03-14T13:02:28.825 INFO:tasks.workunit.client.1.smithi137.stdout:6/894: write d2/d6/de/f68 [882399,104800] 0 2024-03-14T13:02:28.828 INFO:tasks.workunit.client.1.smithi137.stdout:0/931: chown d1/d44/d72/fc0 13 1 2024-03-14T13:02:28.831 INFO:tasks.workunit.client.1.smithi137.stdout:6/895: write d2/d6/de/d45/d59/f40 [5008594,41926] 0 2024-03-14T13:02:28.833 INFO:tasks.workunit.client.1.smithi137.stdout:7/919: creat d3/d6/d14/d22/ddf/f131 x:0 0 0 2024-03-14T13:02:28.834 INFO:tasks.workunit.client.1.smithi137.stdout:0/932: mknod d1/d5/d1b/d42/d3a/db0/c13c 0 2024-03-14T13:02:28.835 INFO:tasks.workunit.client.0.smithi111.stdout:2/872: dwrite dd/d11/d8b/ff8 [0,4194304] 0 2024-03-14T13:02:28.835 INFO:tasks.workunit.client.1.smithi137.stdout:2/940: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/dee/f103 [0,4194304] 0 2024-03-14T13:02:28.839 INFO:tasks.workunit.client.1.smithi137.stdout:7/920: symlink d3/d4a/l132 0 2024-03-14T13:02:28.841 INFO:tasks.workunit.client.1.smithi137.stdout:4/901: dread d4/d27/d5c/d6d/d58/d65/fce [0,4194304] 0 2024-03-14T13:02:28.841 INFO:tasks.workunit.client.1.smithi137.stdout:0/933: symlink d1/d5/d1b/d1e/d53/d6d/l13d 0 2024-03-14T13:02:28.843 INFO:tasks.workunit.client.0.smithi111.stdout:2/873: dwrite dd/d17/d34/d74/f9e [0,4194304] 0 2024-03-14T13:02:28.843 INFO:tasks.workunit.client.1.smithi137.stdout:7/921: rmdir d3/d5/d23/d3f 39 2024-03-14T13:02:28.844 INFO:tasks.workunit.client.1.smithi137.stdout:0/934: write d1/d1f/f12d [446811,8590] 0 2024-03-14T13:02:28.846 INFO:tasks.workunit.client.1.smithi137.stdout:0/935: fsync d1/f46 0 2024-03-14T13:02:28.847 INFO:tasks.workunit.client.0.smithi111.stdout:2/874: mkdir dd/d8d/dc2/dec/d122 0 2024-03-14T13:02:28.850 INFO:tasks.workunit.client.1.smithi137.stdout:7/922: dread - d3/d5/d23/d3f/f105 zero size 2024-03-14T13:02:28.850 INFO:tasks.workunit.client.1.smithi137.stdout:4/902: unlink d4/d9/d3a/d41/d38/d79/dc2/d11f/fef 0 2024-03-14T13:02:28.855 INFO:tasks.workunit.client.1.smithi137.stdout:7/923: fdatasync d3/d54/db4/f121 0 2024-03-14T13:02:28.855 INFO:tasks.workunit.client.1.smithi137.stdout:0/936: unlink d1/d5/d1b/d1e/d25/l6b 0 2024-03-14T13:02:28.857 INFO:tasks.workunit.client.1.smithi137.stdout:7/924: write d3/d5/d23/d3f/d5b/f79 [1845125,47716] 0 2024-03-14T13:02:28.861 INFO:tasks.workunit.client.1.smithi137.stdout:0/937: mknod d1/d5/d1b/c13e 0 2024-03-14T13:02:28.864 INFO:tasks.workunit.client.1.smithi137.stdout:7/925: rename d3/d5/d5a/d86/dc5/f130 to d3/d5/d5a/d86/dc5/dc7/f133 0 2024-03-14T13:02:28.866 INFO:tasks.workunit.client.1.smithi137.stdout:7/926: write d3/d5/f67 [1890243,99454] 0 2024-03-14T13:02:28.867 INFO:tasks.workunit.client.1.smithi137.stdout:7/927: write d3/d6/d59/daa/dcf/d6a/da1/f12d [163970,99474] 0 2024-03-14T13:02:28.870 INFO:tasks.workunit.client.1.smithi137.stdout:7/928: mkdir d3/d6/d64/d97/d134 0 2024-03-14T13:02:28.871 INFO:tasks.workunit.client.1.smithi137.stdout:6/896: truncate d2/d6/de/d45/f77 3492416 0 2024-03-14T13:02:28.871 INFO:tasks.workunit.client.1.smithi137.stdout:2/941: dread d2/d7/d2e/d3e/d44/fab [0,4194304] 0 2024-03-14T13:02:28.874 INFO:tasks.workunit.client.1.smithi137.stdout:4/903: truncate d4/d9/d3a/d41/d6a/dac/fb0 2540496 0 2024-03-14T13:02:28.875 INFO:tasks.workunit.client.1.smithi137.stdout:4/904: chown d4/d27/d100/lee 3 1 2024-03-14T13:02:28.876 INFO:tasks.workunit.client.1.smithi137.stdout:6/897: creat d2/d4c/f12e x:0 0 0 2024-03-14T13:02:28.880 INFO:tasks.workunit.client.1.smithi137.stdout:4/905: mkdir d4/d9/d3a/d70/d125 0 2024-03-14T13:02:28.880 INFO:tasks.workunit.client.1.smithi137.stdout:4/906: stat d4/d10 0 2024-03-14T13:02:28.881 INFO:tasks.workunit.client.1.smithi137.stdout:0/938: dread d1/d44/d56/f9c [0,4194304] 0 2024-03-14T13:02:28.883 INFO:tasks.workunit.client.1.smithi137.stdout:4/907: fdatasync d4/d9/d3a/d41/d38/d79/dc2/d11f/f6c 0 2024-03-14T13:02:28.885 INFO:tasks.workunit.client.1.smithi137.stdout:4/908: write d4/d10/f3f [4088835,30710] 0 2024-03-14T13:02:28.886 INFO:tasks.workunit.client.1.smithi137.stdout:6/898: rename d2/d6/de/d45/d59/f79 to d2/d6/d1d/d88/f12f 0 2024-03-14T13:02:28.889 INFO:tasks.workunit.client.1.smithi137.stdout:0/939: mkdir d1/d5/d1b/d1e/d25/d55/d13f 0 2024-03-14T13:02:28.890 INFO:tasks.workunit.client.1.smithi137.stdout:0/940: fdatasync d1/f7b 0 2024-03-14T13:02:28.890 INFO:tasks.workunit.client.1.smithi137.stdout:7/929: write d3/d5/d23/d2c/f53 [859736,60052] 0 2024-03-14T13:02:28.891 INFO:tasks.workunit.client.1.smithi137.stdout:4/909: readlink d4/d9/d3a/d41/d38/d55/db6/le5 0 2024-03-14T13:02:28.897 INFO:tasks.workunit.client.1.smithi137.stdout:2/942: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f45 [0,4194304] 0 2024-03-14T13:02:28.898 INFO:tasks.workunit.client.1.smithi137.stdout:6/899: unlink d2/d54/d55/d6d/d76/d57/lde 0 2024-03-14T13:02:28.899 INFO:tasks.workunit.client.1.smithi137.stdout:0/941: creat d1/d5/d1b/d1e/d25/d55/f140 x:0 0 0 2024-03-14T13:02:28.900 INFO:tasks.workunit.client.1.smithi137.stdout:4/910: dwrite d4/d10/f3f [0,4194304] 0 2024-03-14T13:02:28.901 INFO:tasks.workunit.client.1.smithi137.stdout:7/930: mkdir d3/d6/d14/d22/d135 0 2024-03-14T13:02:28.903 INFO:tasks.workunit.client.1.smithi137.stdout:0/942: stat d1/d5/d1b/d42/d61/dae/lc1 0 2024-03-14T13:02:28.906 INFO:tasks.workunit.client.1.smithi137.stdout:2/943: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/f135 x:0 0 0 2024-03-14T13:02:28.906 INFO:tasks.workunit.client.1.smithi137.stdout:4/911: fdatasync d4/d27/d5c/d6d/d58/f7e 0 2024-03-14T13:02:28.907 INFO:tasks.workunit.client.1.smithi137.stdout:7/931: fdatasync d3/d6/d59/daa/dcf/d6a/fc8 0 2024-03-14T13:02:28.911 INFO:tasks.workunit.client.1.smithi137.stdout:0/943: unlink d1/d5/d1b/d42/d3a/db0/cfb 0 2024-03-14T13:02:28.912 INFO:tasks.workunit.client.1.smithi137.stdout:7/932: stat d3/d6/d59/daa/dcf/d6a/d75/d78/ffa 0 2024-03-14T13:02:28.912 INFO:tasks.workunit.client.1.smithi137.stdout:6/900: dwrite d2/d54/d55/d6d/dc3/deb/ffa [0,4194304] 0 2024-03-14T13:02:28.921 INFO:tasks.workunit.client.1.smithi137.stdout:7/933: mknod d3/d5/d5a/d86/dc5/dc7/c136 0 2024-03-14T13:02:28.921 INFO:tasks.workunit.client.1.smithi137.stdout:6/901: creat d2/d6/de/d45/d97/d9f/f130 x:0 0 0 2024-03-14T13:02:28.923 INFO:tasks.workunit.client.1.smithi137.stdout:4/912: dwrite d4/d9/d3a/d41/d6a/dff/f11a [0,4194304] 0 2024-03-14T13:02:28.925 INFO:tasks.workunit.client.1.smithi137.stdout:0/944: link d1/d5/d1b/d42/d61/dae/lc1 d1/d5/d1b/d1e/d53/d6d/d102/d128/l141 0 2024-03-14T13:02:28.928 INFO:tasks.workunit.client.1.smithi137.stdout:2/944: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/fc5 [0,4194304] 0 2024-03-14T13:02:28.933 INFO:tasks.workunit.client.1.smithi137.stdout:6/902: mknod d2/d6/d1d/c131 0 2024-03-14T13:02:28.935 INFO:tasks.workunit.client.1.smithi137.stdout:7/934: mknod d3/d6/d14/d22/ddf/c137 0 2024-03-14T13:02:28.936 INFO:tasks.workunit.client.1.smithi137.stdout:9/963: sync 2024-03-14T13:02:28.952 INFO:tasks.workunit.client.1.smithi137.stdout:9/964: dread d6/f96 [0,4194304] 0 2024-03-14T13:02:28.954 INFO:tasks.workunit.client.1.smithi137.stdout:9/965: truncate d6/df/d88/de5/df6/d40/f60 1305499 0 2024-03-14T13:02:28.966 INFO:tasks.workunit.client.1.smithi137.stdout:8/953: sync 2024-03-14T13:02:28.966 INFO:tasks.workunit.client.1.smithi137.stdout:5/829: sync 2024-03-14T13:02:28.982 INFO:tasks.workunit.client.1.smithi137.stdout:5/830: creat d7/d16/d41/f114 x:0 0 0 2024-03-14T13:02:28.984 INFO:tasks.workunit.client.1.smithi137.stdout:8/954: unlink d1/dc/d3c/d87/f48 0 2024-03-14T13:02:28.987 INFO:tasks.workunit.client.1.smithi137.stdout:0/945: getdents d1/d5/d1b/d1e/d25/d8a 0 2024-03-14T13:02:28.988 INFO:tasks.workunit.client.1.smithi137.stdout:2/945: getdents d2/d7/d2e/d3e/d44/d6b 0 2024-03-14T13:02:28.988 INFO:tasks.workunit.client.1.smithi137.stdout:0/946: chown d1/d44/d72/d7e/fcb 114834578 1 2024-03-14T13:02:28.990 INFO:tasks.workunit.client.1.smithi137.stdout:0/947: write d1/d44/f124 [200213,92156] 0 2024-03-14T13:02:28.990 INFO:tasks.workunit.client.1.smithi137.stdout:7/935: link d3/d5/d23/d3f/d5b/fc1 d3/d6/d64/d97/dd2/d101/f138 0 2024-03-14T13:02:28.991 INFO:tasks.workunit.client.1.smithi137.stdout:0/948: readlink d1/d5/d1b/d42/lcc 0 2024-03-14T13:02:28.993 INFO:tasks.workunit.client.1.smithi137.stdout:9/966: rename d6/df/d19/c63 to d6/df/d88/de5/df6/d40/c13e 0 2024-03-14T13:02:28.994 INFO:tasks.workunit.client.1.smithi137.stdout:9/967: write d6/d56/f12a [684195,39680] 0 2024-03-14T13:02:29.000 INFO:tasks.workunit.client.1.smithi137.stdout:4/913: link d4/d9/d3a/d41/d38/d79/dc2/d11f/d73/d9d/dbc/l102 d4/d9/d8e/dae/l126 0 2024-03-14T13:02:29.002 INFO:tasks.workunit.client.1.smithi137.stdout:7/936: creat d3/d54/db4/f139 x:0 0 0 2024-03-14T13:02:29.005 INFO:tasks.workunit.client.1.smithi137.stdout:6/903: getdents d2/d54/d55/d6d/d76/d57 0 2024-03-14T13:02:29.006 INFO:tasks.workunit.client.1.smithi137.stdout:9/968: mknod d6/df/d88/de5/df6/da9/c13f 0 2024-03-14T13:02:29.007 INFO:tasks.workunit.client.1.smithi137.stdout:3/747: sync 2024-03-14T13:02:29.010 INFO:tasks.workunit.client.1.smithi137.stdout:7/937: dwrite d3/d4a/f90 [0,4194304] 0 2024-03-14T13:02:29.012 INFO:tasks.workunit.client.1.smithi137.stdout:0/949: rename d1/d5/d1b/d1e/d25/d55/c7c to d1/d5/d1b/d1e/d25/d8a/c142 0 2024-03-14T13:02:29.014 INFO:tasks.workunit.client.1.smithi137.stdout:7/938: dread d3/d6/d59/daa/dcf/d6a/da1/f12d [0,4194304] 0 2024-03-14T13:02:29.015 INFO:tasks.workunit.client.1.smithi137.stdout:7/939: chown d3/d6/d14/d22/ddf/f131 31669002 1 2024-03-14T13:02:29.016 INFO:tasks.workunit.client.1.smithi137.stdout:2/946: getdents d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b 0 2024-03-14T13:02:29.018 INFO:tasks.workunit.client.1.smithi137.stdout:2/947: chown d2/d7/d2e/d3e/d44/d6b/d22/d26/f27 2853 1 2024-03-14T13:02:29.018 INFO:tasks.workunit.client.1.smithi137.stdout:8/955: rename d1/dc/d71/fb6 to d1/d5e/d11a/d97/d11d/f13a 0 2024-03-14T13:02:29.019 INFO:tasks.workunit.client.1.smithi137.stdout:3/748: mknod d4/de/d16/d1c/d5d/d4b/de6/cf2 0 2024-03-14T13:02:29.019 INFO:tasks.workunit.client.1.smithi137.stdout:8/956: chown d1/l93 4516639 1 2024-03-14T13:02:29.020 INFO:tasks.workunit.client.1.smithi137.stdout:0/950: symlink d1/d5/d1b/d1e/d53/d6d/d102/d128/l143 0 2024-03-14T13:02:29.023 INFO:tasks.workunit.client.1.smithi137.stdout:4/914: rename d4/d9/d3a/d70/d125 to d4/d27/d5c/d77/d127 0 2024-03-14T13:02:29.025 INFO:tasks.workunit.client.1.smithi137.stdout:8/957: fdatasync d1/dc/d77/d9f/de4/f7c 0 2024-03-14T13:02:29.026 INFO:tasks.workunit.client.1.smithi137.stdout:0/951: symlink d1/d44/d72/de8/l144 0 2024-03-14T13:02:29.029 INFO:tasks.workunit.client.1.smithi137.stdout:2/948: mknod d2/d7/d2e/d3e/d44/d6b/d1d/d41/dad/c136 0 2024-03-14T13:02:29.032 INFO:tasks.workunit.client.1.smithi137.stdout:4/915: creat d4/d9/d3a/d41/d38/d55/f128 x:0 0 0 2024-03-14T13:02:29.032 INFO:tasks.workunit.client.1.smithi137.stdout:8/958: creat d1/d5e/d11a/f13b x:0 0 0 2024-03-14T13:02:29.036 INFO:tasks.workunit.client.1.smithi137.stdout:0/952: unlink d1/d5/d1b/d1e/d53/df0/d121/c123 0 2024-03-14T13:02:29.040 INFO:tasks.workunit.client.1.smithi137.stdout:6/904: rename d2/d6 to d2/d54/d55/d6d/d76/d132 0 2024-03-14T13:02:29.041 INFO:tasks.workunit.client.1.smithi137.stdout:3/749: dread d4/de/d16/d1c/f41 [0,4194304] 0 2024-03-14T13:02:29.041 INFO:tasks.workunit.client.1.smithi137.stdout:8/959: fsync d1/d5e/d11a/d82/f110 0 2024-03-14T13:02:29.043 INFO:tasks.workunit.client.1.smithi137.stdout:8/960: write d1/dc/d71/db0/f10f [1520308,68830] 0 2024-03-14T13:02:29.043 INFO:tasks.workunit.client.1.smithi137.stdout:2/949: fsync d2/d7/d2e/d3e/d44/d6b/d22/f5a 0 2024-03-14T13:02:29.046 INFO:tasks.workunit.client.1.smithi137.stdout:2/950: stat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/l90 0 2024-03-14T13:02:29.047 INFO:tasks.workunit.client.1.smithi137.stdout:3/750: chown d4/de/d16/c6c 10962 1 2024-03-14T13:02:29.048 INFO:tasks.workunit.client.1.smithi137.stdout:2/951: fdatasync d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/df0/f122 0 2024-03-14T13:02:29.048 INFO:tasks.workunit.client.0.smithi111.stdout:6/755: sync 2024-03-14T13:02:29.048 INFO:tasks.workunit.client.0.smithi111.stdout:3/898: sync 2024-03-14T13:02:29.049 INFO:tasks.workunit.client.1.smithi137.stdout:2/952: chown d2/d7/d2e/d3e/d44/d115 1 1 2024-03-14T13:02:29.052 INFO:tasks.workunit.client.1.smithi137.stdout:5/831: dwrite d7/d36/fce [0,4194304] 0 2024-03-14T13:02:29.052 INFO:tasks.workunit.client.1.smithi137.stdout:0/953: mknod d1/d1f/c145 0 2024-03-14T13:02:29.052 INFO:tasks.workunit.client.1.smithi137.stdout:3/751: write d4/de/d16/d1c/d5d/d85/d6b/f6d [5006849,50475] 0 2024-03-14T13:02:29.053 INFO:tasks.workunit.client.0.smithi111.stdout:6/756: rename d0/d17/d21/d42/da5/fb4 to d0/d17/d20/d4a/d68/dc2/f116 0 2024-03-14T13:02:29.053 INFO:tasks.workunit.client.1.smithi137.stdout:8/961: chown d1/dc/fcd 26 1 2024-03-14T13:02:29.056 INFO:tasks.workunit.client.0.smithi111.stdout:6/757: dread d0/d50/f56 [0,4194304] 0 2024-03-14T13:02:29.059 INFO:tasks.workunit.client.0.smithi111.stdout:6/758: getdents d0/d17/d20/d4a/d68 0 2024-03-14T13:02:29.060 INFO:tasks.workunit.client.1.smithi137.stdout:6/905: mknod d2/d54/d55/d6d/d76/d132/d1d/d88/dcd/c133 0 2024-03-14T13:02:29.062 INFO:tasks.workunit.client.1.smithi137.stdout:0/954: dread - d1/d44/d72/d7e/f81 zero size 2024-03-14T13:02:29.062 INFO:tasks.workunit.client.0.smithi111.stdout:6/759: symlink d0/d17/d21/d69/d3d/dca/l117 0 2024-03-14T13:02:29.062 INFO:tasks.workunit.client.0.smithi111.stdout:6/760: dread - d0/d17/f10c zero size 2024-03-14T13:02:29.063 INFO:tasks.workunit.client.1.smithi137.stdout:0/955: dread - d1/d1f/d80/f117 zero size 2024-03-14T13:02:29.065 INFO:tasks.workunit.client.0.smithi111.stdout:6/761: mkdir d0/dfb/d106/d114/d118 0 2024-03-14T13:02:29.065 INFO:tasks.workunit.client.1.smithi137.stdout:2/953: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/ffe [0,4194304] 0 2024-03-14T13:02:29.066 INFO:tasks.workunit.client.0.smithi111.stdout:6/762: dread - d0/d17/d21/d69/d3d/f5c zero size 2024-03-14T13:02:29.067 INFO:tasks.workunit.client.1.smithi137.stdout:5/832: fsync d7/d16/d18/d25/d28/fad 0 2024-03-14T13:02:29.068 INFO:tasks.workunit.client.0.smithi111.stdout:6/763: symlink d0/d17/d20/d4a/d68/l119 0 2024-03-14T13:02:29.068 INFO:tasks.workunit.client.1.smithi137.stdout:9/969: rename d6/df/d19/d33/ddf/d116 to d6/df/d140 0 2024-03-14T13:02:29.070 INFO:tasks.workunit.client.0.smithi111.stdout:6/764: truncate d0/d17/f2b 319864 0 2024-03-14T13:02:29.073 INFO:tasks.workunit.client.1.smithi137.stdout:0/956: dwrite d1/d1f/d82/f118 [0,4194304] 0 2024-03-14T13:02:29.074 INFO:tasks.workunit.client.1.smithi137.stdout:2/954: symlink d2/d11e/l137 0 2024-03-14T13:02:29.075 INFO:tasks.workunit.client.0.smithi111.stdout:6/765: truncate d0/d17/d21/d69/d3d/d53/d81/da1/dc8/ff3 813772 0 2024-03-14T13:02:29.076 INFO:tasks.workunit.client.0.smithi111.stdout:3/899: dread d0/d2d/d35/d69/f110 [0,4194304] 0 2024-03-14T13:02:29.078 INFO:tasks.workunit.client.0.smithi111.stdout:3/900: write d0/d30/f116 [697408,113040] 0 2024-03-14T13:02:29.079 INFO:tasks.workunit.client.0.smithi111.stdout:6/766: link d0/l27 d0/dfb/d106/l11a 0 2024-03-14T13:02:29.083 INFO:tasks.workunit.client.1.smithi137.stdout:7/940: rename d3/d6/d2e/f49 to d3/d52/d77/f13a 0 2024-03-14T13:02:29.083 INFO:tasks.workunit.client.1.smithi137.stdout:6/906: mkdir d2/d54/d55/d6d/d76/d132/de/d45/d59/d102/d134 0 2024-03-14T13:02:29.083 INFO:tasks.workunit.client.1.smithi137.stdout:0/957: rename d1/d44/d72 to d1/d44/d72/de8/d146 22 2024-03-14T13:02:29.084 INFO:tasks.workunit.client.0.smithi111.stdout:3/901: dwrite d0/d2d/d35/d50/d5e/f5f [0,4194304] 0 2024-03-14T13:02:29.087 INFO:tasks.workunit.client.0.smithi111.stdout:3/902: creat d0/d2d/d35/d50/ddf/f142 x:0 0 0 2024-03-14T13:02:29.089 INFO:tasks.workunit.client.0.smithi111.stdout:3/903: chown d0/d2d/d35/d50/d6c/d98/l11f 235838993 1 2024-03-14T13:02:29.093 INFO:tasks.workunit.client.1.smithi137.stdout:0/958: symlink d1/d5/d1b/d1e/d25/d10d/l147 0 2024-03-14T13:02:29.095 INFO:tasks.workunit.client.1.smithi137.stdout:8/962: write d1/dc/d4e/f7a [1000763,122594] 0 2024-03-14T13:02:29.097 INFO:tasks.workunit.client.1.smithi137.stdout:7/941: write d3/fe [94767,93205] 0 2024-03-14T13:02:29.097 INFO:tasks.workunit.client.1.smithi137.stdout:2/955: truncate d2/d7/d2e/d3e/d44/d6b/d22/dbb/f10a 511089 0 2024-03-14T13:02:29.099 INFO:tasks.workunit.client.1.smithi137.stdout:9/970: write d6/df/d88/de5/df6/f76 [741732,64041] 0 2024-03-14T13:02:29.099 INFO:tasks.workunit.client.1.smithi137.stdout:5/833: fsync d7/d33/d65/f7c 0 2024-03-14T13:02:29.099 INFO:tasks.workunit.client.1.smithi137.stdout:4/916: dwrite d4/d4d/f6b [0,4194304] 0 2024-03-14T13:02:29.101 INFO:tasks.workunit.client.0.smithi111.stdout:6/767: rmdir d0/d17/d21/d69/d3d/d53/d81/da1/dc8 39 2024-03-14T13:02:29.102 INFO:tasks.workunit.client.1.smithi137.stdout:3/752: dwrite d4/de/d16/d1c/f41 [4194304,4194304] 0 2024-03-14T13:02:29.104 INFO:tasks.workunit.client.1.smithi137.stdout:0/959: dread - d1/d44/d72/fc0 zero size 2024-03-14T13:02:29.106 INFO:tasks.workunit.client.0.smithi111.stdout:6/768: truncate d0/d17/d21/d69/d3d/d53/d81/f8b 25595 0 2024-03-14T13:02:29.107 INFO:tasks.workunit.client.1.smithi137.stdout:8/963: write d1/dc/d77/d9f/de4/f73 [4318459,111682] 0 2024-03-14T13:02:29.114 INFO:tasks.workunit.client.1.smithi137.stdout:5/834: mknod d7/d16/d18/d2d/dd1/c115 0 2024-03-14T13:02:29.114 INFO:tasks.workunit.client.0.smithi111.stdout:3/904: dwrite d0/d2d/d35/d50/da2/db3/d71/fda [0,4194304] 0 2024-03-14T13:02:29.120 INFO:tasks.workunit.client.1.smithi137.stdout:7/942: creat d3/d6/d59/daa/dcf/d6a/d75/d78/da4/f13b x:0 0 0 2024-03-14T13:02:29.125 INFO:tasks.workunit.client.1.smithi137.stdout:4/917: dwrite d4/d10/d5a/f114 [0,4194304] 0 2024-03-14T13:02:29.131 INFO:tasks.workunit.client.1.smithi137.stdout:0/960: getdents d1/d5/d1b/d42/d3a/dc6 0 2024-03-14T13:02:29.136 INFO:tasks.workunit.client.1.smithi137.stdout:6/907: dread d2/d54/d55/d6d/d76/d57/f81 [0,4194304] 0 2024-03-14T13:02:29.136 INFO:tasks.workunit.client.1.smithi137.stdout:8/964: unlink d1/d5e/d11a/d82/d94/ff4 0 2024-03-14T13:02:29.137 INFO:tasks.workunit.client.1.smithi137.stdout:5/835: getdents d7/d1d/d106 0 2024-03-14T13:02:29.137 INFO:tasks.workunit.client.1.smithi137.stdout:7/943: mkdir d3/d6/d59/da9/d13c 0 2024-03-14T13:02:29.142 INFO:tasks.workunit.client.1.smithi137.stdout:7/944: mkdir d3/d6/d59/daa/dcf/d6a/da1/d13d 0 2024-03-14T13:02:29.147 INFO:tasks.workunit.client.1.smithi137.stdout:6/908: rmdir d2/d54/d55/d6d/d76/d132/d1d/d88 39 2024-03-14T13:02:29.148 INFO:tasks.workunit.client.1.smithi137.stdout:2/956: write d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/ffd [2780243,7877] 0 2024-03-14T13:02:29.151 INFO:tasks.workunit.client.1.smithi137.stdout:9/971: dwrite d6/d56/fba [0,4194304] 0 2024-03-14T13:02:29.151 INFO:tasks.workunit.client.0.smithi111.stdout:9/876: sync 2024-03-14T13:02:29.152 INFO:tasks.workunit.client.1.smithi137.stdout:5/836: link d7/d33/d65/dc2/fe4 d7/d1d/d74/d81/f116 0 2024-03-14T13:02:29.157 INFO:tasks.workunit.client.1.smithi137.stdout:8/965: getdents d1/dc/d10/d102 0 2024-03-14T13:02:29.158 INFO:tasks.workunit.client.1.smithi137.stdout:2/957: chown d2/d7/d2e/d3e/d44/d6b/d22/dac/faf 5 1 2024-03-14T13:02:29.165 INFO:tasks.workunit.client.0.smithi111.stdout:3/905: mkdir d0/d2d/d35/d50/ddf/d118/d13b/d143 0 2024-03-14T13:02:29.165 INFO:tasks.workunit.client.1.smithi137.stdout:8/966: mknod d1/dc/d77/c13c 0 2024-03-14T13:02:29.167 INFO:tasks.workunit.client.0.smithi111.stdout:9/877: getdents d5/d6/d8/d5b/d84/db6/de7/d12e 0 2024-03-14T13:02:29.167 INFO:tasks.workunit.client.1.smithi137.stdout:6/909: read d2/d54/d55/d6d/d76/d57/ff9 [319356,49064] 0 2024-03-14T13:02:29.168 INFO:tasks.workunit.client.0.smithi111.stdout:9/878: fdatasync d5/d6/d2a/f3e 0 2024-03-14T13:02:29.171 INFO:tasks.workunit.client.0.smithi111.stdout:3/906: dread d0/d2d/d35/d50/da2/fec [0,4194304] 0 2024-03-14T13:02:29.177 INFO:tasks.workunit.client.1.smithi137.stdout:3/753: dread d4/d19/f70 [0,4194304] 0 2024-03-14T13:02:29.177 INFO:tasks.workunit.client.1.smithi137.stdout:5/837: dwrite d7/d36/fce [0,4194304] 0 2024-03-14T13:02:29.177 INFO:tasks.workunit.client.1.smithi137.stdout:9/972: dwrite d6/fa3 [0,4194304] 0 2024-03-14T13:02:29.180 INFO:tasks.workunit.client.1.smithi137.stdout:8/967: rename d1/d5e/d11a/d97/d7f to d1/d5e/d11a/d82/d13d 0 2024-03-14T13:02:29.181 INFO:tasks.workunit.client.0.smithi111.stdout:6/769: creat d0/dfb/f11b x:0 0 0 2024-03-14T13:02:29.184 INFO:tasks.workunit.client.1.smithi137.stdout:3/754: creat d4/de/dc7/ff3 x:0 0 0 2024-03-14T13:02:29.184 INFO:tasks.workunit.client.1.smithi137.stdout:7/945: dwrite d3/d5/d5a/f7f [4194304,4194304] 0 2024-03-14T13:02:29.193 INFO:tasks.workunit.client.1.smithi137.stdout:0/961: dread d1/d1f/fc4 [0,4194304] 0 2024-03-14T13:02:29.193 INFO:tasks.workunit.client.1.smithi137.stdout:9/973: write d6/df/d19/d33/f46 [7633402,100321] 0 2024-03-14T13:02:29.196 INFO:tasks.workunit.client.1.smithi137.stdout:9/974: truncate d6/d26/f10f 496101 0 2024-03-14T13:02:29.199 INFO:tasks.workunit.client.1.smithi137.stdout:3/755: dwrite d4/de/d16/d1c/d5d/d85/d6b/f6d [0,4194304] 0 2024-03-14T13:02:29.202 INFO:tasks.workunit.client.1.smithi137.stdout:3/756: readlink d4/de/d16/d1c/d5d/d85/l3b 0 2024-03-14T13:02:29.202 INFO:tasks.workunit.client.1.smithi137.stdout:2/958: write d2/d7/d2e/d3e/d44/d6b/d1d/fdd [587614,85916] 0 2024-03-14T13:02:29.203 INFO:tasks.workunit.client.1.smithi137.stdout:5/838: rename d7/d16/l51 to d7/d49/d75/l117 0 2024-03-14T13:02:29.204 INFO:tasks.workunit.client.1.smithi137.stdout:0/962: creat d1/d44/d72/d120/f148 x:0 0 0 2024-03-14T13:02:29.204 INFO:tasks.workunit.client.0.smithi111.stdout:9/879: write d5/d6/d14/f91 [746127,69249] 0 2024-03-14T13:02:29.209 INFO:tasks.workunit.client.0.smithi111.stdout:2/875: sync 2024-03-14T13:02:29.211 INFO:tasks.workunit.client.0.smithi111.stdout:3/907: truncate d0/d2d/d35/d4c/dad/dee/ff6 316445 0 2024-03-14T13:02:29.212 INFO:tasks.workunit.client.1.smithi137.stdout:0/963: read d1/d5/d1b/d1e/d53/d6d/f9a [683521,113507] 0 2024-03-14T13:02:29.212 INFO:tasks.workunit.client.0.smithi111.stdout:2/876: write dd/d17/d34/fd2 [298009,34857] 0 2024-03-14T13:02:29.212 INFO:tasks.workunit.client.1.smithi137.stdout:2/959: symlink d2/d7/d2e/d3e/d44/d6b/d22/d26/l138 0 2024-03-14T13:02:29.213 INFO:tasks.workunit.client.0.smithi111.stdout:2/877: dread - dd/d17/d34/d74/dca/dcb/fef zero size 2024-03-14T13:02:29.214 INFO:tasks.workunit.client.0.smithi111.stdout:7/922: sync 2024-03-14T13:02:29.215 INFO:tasks.workunit.client.1.smithi137.stdout:3/757: dwrite d4/de/d16/d1c/d5d/d4b/f5e [0,4194304] 0 2024-03-14T13:02:29.216 INFO:tasks.workunit.client.0.smithi111.stdout:7/923: rmdir d0/d11/da8/dd4/df6/d108/d21 39 2024-03-14T13:02:29.218 INFO:tasks.workunit.client.0.smithi111.stdout:7/924: read d0/d11/da8/dd4/df6/d108/d4a/fa7 [788128,64547] 0 2024-03-14T13:02:29.221 INFO:tasks.workunit.client.0.smithi111.stdout:7/925: mknod d0/d11/d98/c119 0 2024-03-14T13:02:29.221 INFO:tasks.workunit.client.1.smithi137.stdout:2/960: truncate d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/f12b 206943 0 2024-03-14T13:02:29.224 INFO:tasks.workunit.client.1.smithi137.stdout:0/964: link d1/d1f/fc4 d1/d44/f149 0 2024-03-14T13:02:29.224 INFO:tasks.workunit.client.0.smithi111.stdout:2/878: dread dd/d17/d34/fd2 [0,4194304] 0 2024-03-14T13:02:29.225 INFO:tasks.workunit.client.0.smithi111.stdout:7/926: dread d0/d11/da8/dd4/df6/d108/d5/d38/f7f [0,4194304] 0 2024-03-14T13:02:29.226 INFO:tasks.workunit.client.1.smithi137.stdout:8/968: dread d1/d5e/d11a/fa6 [0,4194304] 0 2024-03-14T13:02:29.226 INFO:tasks.workunit.client.1.smithi137.stdout:3/758: getdents d4/de/d16/d74 0 2024-03-14T13:02:29.226 INFO:tasks.workunit.client.0.smithi111.stdout:2/879: write dd/d11/d1a/fcd [884090,113110] 0 2024-03-14T13:02:29.226 INFO:tasks.workunit.client.1.smithi137.stdout:6/910: dread d2/d54/d55/d33/fcc [0,4194304] 0 2024-03-14T13:02:29.228 INFO:tasks.workunit.client.1.smithi137.stdout:0/965: truncate d1/d44/d56/f5a 1300473 0 2024-03-14T13:02:29.228 INFO:tasks.workunit.client.1.smithi137.stdout:2/961: creat d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f139 x:0 0 0 2024-03-14T13:02:29.228 INFO:tasks.workunit.client.0.smithi111.stdout:7/927: write d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/fa3 [1993451,62497] 0 2024-03-14T13:02:29.229 INFO:tasks.workunit.client.0.smithi111.stdout:2/880: mknod dd/d8d/db7/c123 0 2024-03-14T13:02:29.233 INFO:tasks.workunit.client.0.smithi111.stdout:7/928: read d0/d11/da8/dd4/df6/d108/d22/f46 [3382553,125601] 0 2024-03-14T13:02:29.233 INFO:tasks.workunit.client.0.smithi111.stdout:2/881: read dd/d8d/dc2/fb9 [295115,69093] 0 2024-03-14T13:02:29.234 INFO:tasks.workunit.client.0.smithi111.stdout:9/880: dwrite d5/d6/d8/d1a/f64 [0,4194304] 0 2024-03-14T13:02:29.236 INFO:tasks.workunit.client.0.smithi111.stdout:6/770: dread d0/d17/d20/fb1 [0,4194304] 0 2024-03-14T13:02:29.236 INFO:tasks.workunit.client.0.smithi111.stdout:9/881: read - d5/d6/d8/d5b/f11c zero size 2024-03-14T13:02:29.237 INFO:tasks.workunit.client.1.smithi137.stdout:8/969: fsync d1/dc/d4e/d6d/f8b 0 2024-03-14T13:02:29.237 INFO:tasks.workunit.client.1.smithi137.stdout:2/962: symlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d9d/l13a 0 2024-03-14T13:02:29.238 INFO:tasks.workunit.client.0.smithi111.stdout:9/882: write d5/d6/d8/f16 [1509888,47590] 0 2024-03-14T13:02:29.238 INFO:tasks.workunit.client.0.smithi111.stdout:2/882: unlink dd/d11/d1a/d56/f85 0 2024-03-14T13:02:29.238 INFO:tasks.workunit.client.0.smithi111.stdout:7/929: dread d0/d11/da8/dd4/df6/d108/d5/d38/f7f [0,4194304] 0 2024-03-14T13:02:29.239 INFO:tasks.workunit.client.0.smithi111.stdout:6/771: symlink d0/d17/d21/d69/d3d/d53/l11c 0 2024-03-14T13:02:29.245 INFO:tasks.workunit.client.1.smithi137.stdout:0/966: dwrite d1/d44/feb [0,4194304] 0 2024-03-14T13:02:29.245 INFO:tasks.workunit.client.0.smithi111.stdout:2/883: unlink dd/d11/d1a/d56/c106 0 2024-03-14T13:02:29.245 INFO:tasks.workunit.client.1.smithi137.stdout:8/970: rmdir d1/dc/d4e/d6d 39 2024-03-14T13:02:29.247 INFO:tasks.workunit.client.0.smithi111.stdout:7/930: dwrite d0/f7 [0,4194304] 0 2024-03-14T13:02:29.248 INFO:tasks.workunit.client.0.smithi111.stdout:2/884: write dd/d8d/f115 [2992609,59940] 0 2024-03-14T13:02:29.249 INFO:tasks.workunit.client.0.smithi111.stdout:6/772: truncate d0/d17/d21/d42/da5/fad 2546018 0 2024-03-14T13:02:29.249 INFO:tasks.workunit.client.0.smithi111.stdout:7/931: rmdir d0/d11 39 2024-03-14T13:02:29.250 INFO:tasks.workunit.client.1.smithi137.stdout:0/967: rename d1/d5/d1b/fc5 to d1/d5/d1b/d1e/d25/d9f/f14a 0 2024-03-14T13:02:29.251 INFO:tasks.workunit.client.0.smithi111.stdout:2/885: fsync dd/d17/d34/fbe 0 2024-03-14T13:02:29.251 INFO:tasks.workunit.client.1.smithi137.stdout:8/971: rmdir d1/d5e/d11a/d82/d94 39 2024-03-14T13:02:29.253 INFO:tasks.workunit.client.0.smithi111.stdout:6/773: mknod d0/d17/d21/d69/d3d/d10b/c11d 0 2024-03-14T13:02:29.254 INFO:tasks.workunit.client.1.smithi137.stdout:8/972: write d1/d5e/d11a/d97/d11d/f122 [66422,57786] 0 2024-03-14T13:02:29.254 INFO:tasks.workunit.client.0.smithi111.stdout:6/774: chown d0/f4 3109110 1 2024-03-14T13:02:29.256 INFO:tasks.workunit.client.0.smithi111.stdout:6/775: symlink d0/d50/df2/dc6/l11e 0 2024-03-14T13:02:29.259 INFO:tasks.workunit.client.0.smithi111.stdout:6/776: fsync d0/d17/d21/f2d 0 2024-03-14T13:02:29.261 INFO:tasks.workunit.client.0.smithi111.stdout:2/886: dwrite dd/d17/d34/d73/d49/f5c [4194304,4194304] 0 2024-03-14T13:02:29.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/777: write d0/d17/d21/d69/f39 [2233841,104546] 0 2024-03-14T13:02:29.265 INFO:tasks.workunit.client.0.smithi111.stdout:6/778: dread - d0/d17/d21/d69/d3d/d53/ffd zero size 2024-03-14T13:02:29.266 INFO:tasks.workunit.client.0.smithi111.stdout:6/779: fdatasync d0/d17/d21/ffe 0 2024-03-14T13:02:29.267 INFO:tasks.workunit.client.0.smithi111.stdout:6/780: stat d0/dfb/d106 0 2024-03-14T13:02:29.270 INFO:tasks.workunit.client.0.smithi111.stdout:2/887: dwrite dd/d17/d34/d74/dca/ff7 [4194304,4194304] 0 2024-03-14T13:02:29.272 INFO:tasks.workunit.client.0.smithi111.stdout:2/888: write dd/d8d/f121 [719311,57393] 0 2024-03-14T13:02:29.274 INFO:tasks.workunit.client.0.smithi111.stdout:9/883: write d5/d6/d8/d5b/d84/fa2 [227773,108099] 0 2024-03-14T13:02:29.276 INFO:tasks.workunit.client.0.smithi111.stdout:9/884: write d5/d6/d14/f115 [4204683,124216] 0 2024-03-14T13:02:29.277 INFO:tasks.workunit.client.0.smithi111.stdout:2/889: symlink dd/d17/d34/d73/d49/l124 0 2024-03-14T13:02:29.277 INFO:tasks.workunit.client.0.smithi111.stdout:2/890: fdatasync dd/d17/f37 0 2024-03-14T13:02:29.281 INFO:tasks.workunit.client.0.smithi111.stdout:9/885: mknod d5/d6/d8/c138 0 2024-03-14T13:02:29.284 INFO:tasks.workunit.client.0.smithi111.stdout:9/886: creat d5/d42/d71/dbd/d123/f139 x:0 0 0 2024-03-14T13:02:29.285 INFO:tasks.workunit.client.0.smithi111.stdout:9/887: chown d5/lc4 156960592 1 2024-03-14T13:02:29.287 INFO:tasks.workunit.client.0.smithi111.stdout:7/932: truncate d0/d11/da8/dd4/df6/d108/d22/d3e/faf 949903 0 2024-03-14T13:02:29.290 INFO:tasks.workunit.client.0.smithi111.stdout:7/933: mknod d0/d11/d98/c11a 0 2024-03-14T13:02:29.295 INFO:tasks.workunit.client.0.smithi111.stdout:6/781: dwrite d0/d17/d21/f44 [0,4194304] 0 2024-03-14T13:02:29.300 INFO:tasks.workunit.client.0.smithi111.stdout:2/891: dwrite dd/d8d/dc2/f9a [0,4194304] 0 2024-03-14T13:02:29.304 INFO:tasks.workunit.client.0.smithi111.stdout:7/934: truncate d0/d11/da8/dd4/df6/d108/d5/d56/f5e 2186622 0 2024-03-14T13:02:29.305 INFO:tasks.workunit.client.0.smithi111.stdout:9/888: dwrite d5/d6/d2a/d2d/f4a [0,4194304] 0 2024-03-14T13:02:29.310 INFO:tasks.workunit.client.0.smithi111.stdout:7/935: symlink d0/d11/da8/dd4/df6/d108/d21/l11b 0 2024-03-14T13:02:29.311 INFO:tasks.workunit.client.0.smithi111.stdout:7/936: readlink d0/d11/d98/l105 0 2024-03-14T13:02:29.311 INFO:tasks.workunit.client.0.smithi111.stdout:9/889: dread d5/f20 [0,4194304] 0 2024-03-14T13:02:29.312 INFO:tasks.workunit.client.0.smithi111.stdout:7/937: stat d0/d11/d98/dab 0 2024-03-14T13:02:29.315 INFO:tasks.workunit.client.0.smithi111.stdout:7/938: dread - d0/d11/da8/dd4/df6/d108/d5/f112 zero size 2024-03-14T13:02:29.318 INFO:tasks.workunit.client.0.smithi111.stdout:2/892: dwrite dd/d17/d34/ddb/f97 [0,4194304] 0 2024-03-14T13:02:29.319 INFO:tasks.workunit.client.0.smithi111.stdout:9/890: rename d5/d6/d8/d5b/d84/l85 to d5/d6/d8/d5b/d7c/l13a 0 2024-03-14T13:02:29.324 INFO:tasks.workunit.client.0.smithi111.stdout:6/782: dread d0/f3e [0,4194304] 0 2024-03-14T13:02:29.325 INFO:tasks.workunit.client.0.smithi111.stdout:6/783: chown d0/d17/d20/l83 0 1 2024-03-14T13:02:29.325 INFO:tasks.workunit.client.0.smithi111.stdout:9/891: readlink d5/d6/d8/d5b/d87/l9c 0 2024-03-14T13:02:29.325 INFO:tasks.workunit.client.0.smithi111.stdout:2/893: mknod dd/d17/d53/da4/d105/c125 0 2024-03-14T13:02:29.327 INFO:tasks.workunit.client.0.smithi111.stdout:7/939: rename d0/cc5 to d0/d11/da8/dd4/df6/d108/d21/c11c 0 2024-03-14T13:02:29.330 INFO:tasks.workunit.client.0.smithi111.stdout:9/892: rename d5/d6/d14/d7a/c92 to d5/d6/d2a/d2d/d8b/c13b 0 2024-03-14T13:02:29.331 INFO:tasks.workunit.client.0.smithi111.stdout:2/894: dread dd/d11/d8b/ff8 [0,4194304] 0 2024-03-14T13:02:29.331 INFO:tasks.workunit.client.0.smithi111.stdout:9/893: readlink d5/d42/l12d 0 2024-03-14T13:02:29.331 INFO:tasks.workunit.client.0.smithi111.stdout:7/940: mkdir d0/d11d 0 2024-03-14T13:02:29.333 INFO:tasks.workunit.client.0.smithi111.stdout:6/784: link d0/d17/d20/f26 d0/d17/d21/f11f 0 2024-03-14T13:02:29.334 INFO:tasks.workunit.client.0.smithi111.stdout:2/895: mknod dd/d8d/db7/c126 0 2024-03-14T13:02:29.334 INFO:tasks.workunit.client.0.smithi111.stdout:6/785: stat d0/d17/d21/d69/d3d/d53/l11c 0 2024-03-14T13:02:29.334 INFO:tasks.workunit.client.0.smithi111.stdout:9/894: read d5/d6/d14/d7a/fba [4023498,121903] 0 2024-03-14T13:02:29.335 INFO:tasks.workunit.client.0.smithi111.stdout:6/786: readlink d0/dfb/db8/de9/l101 0 2024-03-14T13:02:29.337 INFO:tasks.workunit.client.0.smithi111.stdout:7/941: creat d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5/dff/f11e x:0 0 0 2024-03-14T13:02:29.339 INFO:tasks.workunit.client.0.smithi111.stdout:2/896: creat dd/d17/d34/d74/dc1/dd1/f127 x:0 0 0 2024-03-14T13:02:29.340 INFO:tasks.workunit.client.0.smithi111.stdout:6/787: symlink d0/d17/d20/d4a/d59/d82/l120 0 2024-03-14T13:02:29.340 INFO:tasks.workunit.client.0.smithi111.stdout:9/895: dread d5/f20 [0,4194304] 0 2024-03-14T13:02:29.341 INFO:tasks.workunit.client.0.smithi111.stdout:9/896: stat d5/d6/d14/l1f 0 2024-03-14T13:02:29.342 INFO:tasks.workunit.client.0.smithi111.stdout:2/897: dread dd/d8d/dc2/fb9 [0,4194304] 0 2024-03-14T13:02:29.343 INFO:tasks.workunit.client.0.smithi111.stdout:7/942: dread d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/dee/f5b [0,4194304] 0 2024-03-14T13:02:29.345 INFO:tasks.workunit.client.0.smithi111.stdout:6/788: truncate d0/d17/d21/d69/d3d/d53/d81/f8b 107536 0 2024-03-14T13:02:29.348 INFO:tasks.workunit.client.0.smithi111.stdout:6/789: fsync d0/d17/f1e 0 2024-03-14T13:02:29.350 INFO:tasks.workunit.client.0.smithi111.stdout:2/898: dwrite dd/d17/d34/f10e [0,4194304] 0 2024-03-14T13:02:29.352 INFO:tasks.workunit.client.0.smithi111.stdout:7/943: rename d0/d11/da8/dd4/df6/d108/d22/d3e/l4e to d0/d11/da8/dd4/df6/d108/d5/d38/d5c/l11f 0 2024-03-14T13:02:29.356 INFO:tasks.workunit.client.0.smithi111.stdout:7/944: symlink d0/d11/da8/dd4/df6/l120 0 2024-03-14T13:02:29.357 INFO:tasks.workunit.client.0.smithi111.stdout:6/790: truncate d0/f35 1118167 0 2024-03-14T13:02:29.359 INFO:tasks.workunit.client.0.smithi111.stdout:2/899: dwrite dd/d11/d1a/d88/f11d [0,4194304] 0 2024-03-14T13:02:29.362 INFO:tasks.workunit.client.1.smithi137.stdout:0/968: unlink d1/fb3 0 2024-03-14T13:02:29.363 INFO:tasks.workunit.client.1.smithi137.stdout:5/839: dread d7/d16/d18/d2d/f53 [4194304,4194304] 0 2024-03-14T13:02:29.363 INFO:tasks.workunit.client.1.smithi137.stdout:8/973: unlink d1/cac 0 2024-03-14T13:02:29.366 INFO:tasks.workunit.client.1.smithi137.stdout:8/974: fdatasync d1/dc/d3c/f139 0 2024-03-14T13:02:29.366 INFO:tasks.workunit.client.1.smithi137.stdout:5/840: dread - d7/d33/dab/f110 zero size 2024-03-14T13:02:29.367 INFO:tasks.workunit.client.1.smithi137.stdout:9/975: dread d6/d56/fb3 [0,4194304] 0 2024-03-14T13:02:29.367 INFO:tasks.workunit.client.1.smithi137.stdout:7/946: dread d3/d5/d23/d3f/d5b/f79 [0,4194304] 0 2024-03-14T13:02:29.367 INFO:tasks.workunit.client.0.smithi111.stdout:2/900: rename dd/d8d/dc2/dec/d122 to dd/d17/d34/d74/dca/d128 0 2024-03-14T13:02:29.368 INFO:tasks.workunit.client.0.smithi111.stdout:7/945: dwrite d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/dc9/f110 [0,4194304] 0 2024-03-14T13:02:29.371 INFO:tasks.workunit.client.1.smithi137.stdout:7/947: dread - d3/d54/db4/f139 zero size 2024-03-14T13:02:29.372 INFO:tasks.workunit.client.1.smithi137.stdout:9/976: mkdir d6/df/d71/d141 0 2024-03-14T13:02:29.373 INFO:tasks.workunit.client.0.smithi111.stdout:6/791: rename d0/d17/d21/d69/d3d/d53/d81/da1/fef to d0/d17/d20/d94/f121 0 2024-03-14T13:02:29.374 INFO:tasks.workunit.client.0.smithi111.stdout:6/792: chown d0/d17/le6 436893058 1 2024-03-14T13:02:29.376 INFO:tasks.workunit.client.1.smithi137.stdout:0/969: dwrite d1/d44/ffa [0,4194304] 0 2024-03-14T13:02:29.376 INFO:tasks.workunit.client.1.smithi137.stdout:7/948: chown d3/d5/d23/d3f/d5b/fab 12115562 1 2024-03-14T13:02:29.377 INFO:tasks.workunit.client.0.smithi111.stdout:2/901: dwrite dd/d8d/fc0 [0,4194304] 0 2024-03-14T13:02:29.378 INFO:tasks.workunit.client.1.smithi137.stdout:5/841: mknod d7/d16/d18/d25/d28/d9e/c118 0 2024-03-14T13:02:29.380 INFO:tasks.workunit.client.1.smithi137.stdout:2/963: dread d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/f5d [0,4194304] 0 2024-03-14T13:02:29.381 INFO:tasks.workunit.client.0.smithi111.stdout:7/946: link d0/d11/leb d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5/dff/l121 0 2024-03-14T13:02:29.382 INFO:tasks.workunit.client.1.smithi137.stdout:0/970: dread d1/d5/d1b/d42/d3a/fc2 [0,4194304] 0 2024-03-14T13:02:29.385 INFO:tasks.workunit.client.0.smithi111.stdout:7/947: fsync d0/d11/da8/dd4/df6/d108/d5/d56/f7c 0 2024-03-14T13:02:29.386 INFO:tasks.workunit.client.0.smithi111.stdout:7/948: read - d0/d11/da8/dd4/f113 zero size 2024-03-14T13:02:29.388 INFO:tasks.workunit.client.1.smithi137.stdout:9/977: write d6/df/f70 [1934748,62030] 0 2024-03-14T13:02:29.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:29.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:29.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:02:29.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:29 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:02:29.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:29.393 INFO:tasks.workunit.client.1.smithi137.stdout:9/978: mknod d6/d26/c142 0 2024-03-14T13:02:29.393 INFO:tasks.workunit.client.1.smithi137.stdout:0/971: dwrite d1/d5/d1b/d1e/d53/fed [0,4194304] 0 2024-03-14T13:02:29.396 INFO:tasks.workunit.client.0.smithi111.stdout:7/949: dwrite d0/d11/da8/dd4/df6/d108/d4a/d52/f6c [0,4194304] 0 2024-03-14T13:02:29.396 INFO:tasks.workunit.client.1.smithi137.stdout:0/972: mkdir d1/d44/d56/d14b 0 2024-03-14T13:02:29.397 INFO:tasks.workunit.client.1.smithi137.stdout:8/975: dread d1/d5e/d11a/d97/f2a [4194304,4194304] 0 2024-03-14T13:02:29.399 INFO:tasks.workunit.client.0.smithi111.stdout:7/950: symlink d0/d11/da8/dd4/df6/d108/d22/d3e/l122 0 2024-03-14T13:02:29.402 INFO:tasks.workunit.client.0.smithi111.stdout:7/951: write d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/ddb/f10f [164749,95603] 0 2024-03-14T13:02:29.403 INFO:tasks.workunit.client.1.smithi137.stdout:5/842: dread d7/d33/d65/f84 [0,4194304] 0 2024-03-14T13:02:29.404 INFO:tasks.workunit.client.1.smithi137.stdout:5/843: fsync d7/d33/d65/dc2/feb 0 2024-03-14T13:02:29.409 INFO:tasks.workunit.client.1.smithi137.stdout:5/844: rename d7/d16/d18/d25/fca to d7/d16/d18/d25/d28/d3c/dd8/f119 0 2024-03-14T13:02:29.410 INFO:tasks.workunit.client.1.smithi137.stdout:5/845: getdents d7/d16/d18/d25/d104 0 2024-03-14T13:02:29.412 INFO:tasks.workunit.client.1.smithi137.stdout:5/846: getdents d7/d1d/d106 0 2024-03-14T13:02:29.421 INFO:tasks.workunit.client.0.smithi111.stdout:2/902: truncate dd/fe4 2736041 0 2024-03-14T13:02:29.423 INFO:tasks.workunit.client.0.smithi111.stdout:2/903: chown dd/d11/d1a/f1b 0 1 2024-03-14T13:02:29.425 INFO:tasks.workunit.client.1.smithi137.stdout:9/979: write d6/df/d19/d33/d8f/f9a [585786,55646] 0 2024-03-14T13:02:29.426 INFO:tasks.workunit.client.0.smithi111.stdout:6/793: dwrite d0/d17/d21/d42/f77 [0,4194304] 0 2024-03-14T13:02:29.427 INFO:tasks.workunit.client.0.smithi111.stdout:2/904: creat dd/f129 x:0 0 0 2024-03-14T13:02:29.427 INFO:tasks.workunit.client.0.smithi111.stdout:6/794: chown d0 347 1 2024-03-14T13:02:29.431 INFO:tasks.workunit.client.0.smithi111.stdout:2/905: creat dd/d17/d53/d9f/f12a x:0 0 0 2024-03-14T13:02:29.434 INFO:tasks.workunit.client.0.smithi111.stdout:2/906: creat dd/d8d/dc2/dec/f12b x:0 0 0 2024-03-14T13:02:29.435 INFO:tasks.workunit.client.0.smithi111.stdout:2/907: write dd/d17/d53/f92 [896866,74692] 0 2024-03-14T13:02:29.435 INFO:tasks.workunit.client.1.smithi137.stdout:9/980: link d6/df/d71/f12f d6/df/d19/d33/d8f/f143 0 2024-03-14T13:02:29.436 INFO:tasks.workunit.client.1.smithi137.stdout:0/973: dwrite d1/d44/d72/d7e/f81 [0,4194304] 0 2024-03-14T13:02:29.438 INFO:tasks.workunit.client.1.smithi137.stdout:5/847: dwrite d7/d1d/d74/d83/fd2 [0,4194304] 0 2024-03-14T13:02:29.442 INFO:tasks.workunit.client.0.smithi111.stdout:2/908: dread dd/d11/d1a/d88/f8a [0,4194304] 0 2024-03-14T13:02:29.442 INFO:tasks.workunit.client.1.smithi137.stdout:0/974: getdents d1/d44/d72/de8/d125 0 2024-03-14T13:02:29.442 INFO:tasks.workunit.client.0.smithi111.stdout:3/908: sync 2024-03-14T13:02:29.444 INFO:tasks.workunit.client.1.smithi137.stdout:5/848: creat d7/d16/d18/d25/d28/d3c/dd8/d10c/f11a x:0 0 0 2024-03-14T13:02:29.445 INFO:tasks.workunit.client.1.smithi137.stdout:5/849: fsync d7/d16/fb9 0 2024-03-14T13:02:29.445 INFO:tasks.workunit.client.0.smithi111.stdout:2/909: chown dd/d11/l6c 19636272 1 2024-03-14T13:02:29.446 INFO:tasks.workunit.client.1.smithi137.stdout:0/975: unlink d1/d5/d1b/d42/d3a/f48 0 2024-03-14T13:02:29.446 INFO:tasks.workunit.client.0.smithi111.stdout:6/795: dread d0/f4 [0,4194304] 0 2024-03-14T13:02:29.448 INFO:tasks.workunit.client.1.smithi137.stdout:0/976: creat d1/da6/f14c x:0 0 0 2024-03-14T13:02:29.450 INFO:tasks.workunit.client.0.smithi111.stdout:2/910: mknod dd/d17/d34/dbf/c12c 0 2024-03-14T13:02:29.451 INFO:tasks.workunit.client.1.smithi137.stdout:0/977: mkdir d1/d5/d1b/d42/d61/dae/d12f/d14d 0 2024-03-14T13:02:29.452 INFO:tasks.workunit.client.0.smithi111.stdout:3/909: write d0/f1c [3654169,59658] 0 2024-03-14T13:02:29.453 INFO:tasks.workunit.client.1.smithi137.stdout:0/978: write d1/d5/d1b/d42/f10e [202557,95825] 0 2024-03-14T13:02:29.454 INFO:tasks.workunit.client.0.smithi111.stdout:6/796: rename d0/d17/d20/d4a/d68/dc2/f116 to d0/d17/d20/d4a/d68/d10e/f122 0 2024-03-14T13:02:29.456 INFO:tasks.workunit.client.0.smithi111.stdout:6/797: mknod d0/dfb/db8/c123 0 2024-03-14T13:02:29.456 INFO:tasks.workunit.client.1.smithi137.stdout:0/979: symlink d1/d5/d1b/d1e/d53/d6d/d102/l14e 0 2024-03-14T13:02:29.458 INFO:tasks.workunit.client.0.smithi111.stdout:6/798: rmdir d0/d17/d21/d42/da5 39 2024-03-14T13:02:29.459 INFO:tasks.workunit.client.0.smithi111.stdout:3/910: link d0/d2d/d35/d50/d5e/l85 d0/d2d/d35/d50/d5e/d19/l144 0 2024-03-14T13:02:29.460 INFO:tasks.workunit.client.0.smithi111.stdout:6/799: getdents d0/d50/df2/dc6/de5 0 2024-03-14T13:02:29.461 INFO:tasks.workunit.client.1.smithi137.stdout:0/980: creat d1/d5/d1b/d1e/d25/f14f x:0 0 0 2024-03-14T13:02:29.461 INFO:tasks.workunit.client.1.smithi137.stdout:9/981: dread d6/df/d19/d33/f80 [0,4194304] 0 2024-03-14T13:02:29.463 INFO:tasks.workunit.client.1.smithi137.stdout:9/982: fsync d6/d53/f99 0 2024-03-14T13:02:29.463 INFO:tasks.workunit.client.0.smithi111.stdout:3/911: dread d0/f1 [0,4194304] 0 2024-03-14T13:02:29.464 INFO:tasks.workunit.client.0.smithi111.stdout:6/800: rename d0/d17/d21/le7 to d0/d17/d20/d4a/d68/dc2/l124 0 2024-03-14T13:02:29.465 INFO:tasks.workunit.client.1.smithi137.stdout:0/981: symlink d1/d5/d1b/d1e/d25/d55/d13f/l150 0 2024-03-14T13:02:29.466 INFO:tasks.workunit.client.1.smithi137.stdout:9/983: rename d6/l135 to d6/df/d19/dcf/l144 0 2024-03-14T13:02:29.467 INFO:tasks.workunit.client.1.smithi137.stdout:9/984: chown d6/d91/dc8/ld3 199554 1 2024-03-14T13:02:29.472 INFO:tasks.workunit.client.0.smithi111.stdout:6/801: creat d0/d17/d20/d94/dbd/f125 x:0 0 0 2024-03-14T13:02:29.472 INFO:tasks.workunit.client.1.smithi137.stdout:0/982: dwrite d1/d5/d1b/d1e/d53/d6d/f74 [0,4194304] 0 2024-03-14T13:02:29.475 INFO:tasks.workunit.client.1.smithi137.stdout:0/983: rmdir d1/d5/d1b/d42/d61 39 2024-03-14T13:02:29.476 INFO:tasks.workunit.client.0.smithi111.stdout:6/802: rename d0/d17/d21/d69/d3d/d53/d81/da1/df6 to d0/d50/df2/dc6/d126 0 2024-03-14T13:02:29.477 INFO:tasks.workunit.client.0.smithi111.stdout:3/912: dwrite d0/d2d/d35/d69/d84/fb2 [0,4194304] 0 2024-03-14T13:02:29.479 INFO:tasks.workunit.client.1.smithi137.stdout:0/984: symlink d1/d44/d72/de8/l151 0 2024-03-14T13:02:29.481 INFO:tasks.workunit.client.1.smithi137.stdout:9/985: dwrite d6/d53/f74 [0,4194304] 0 2024-03-14T13:02:29.482 INFO:tasks.workunit.client.0.smithi111.stdout:2/911: dread dd/d17/d34/d73/d75/f5d [0,4194304] 0 2024-03-14T13:02:29.484 INFO:tasks.workunit.client.0.smithi111.stdout:6/803: rename d0/d17/d21/d69/d3d/d53/d81 to d0/dfb/db8/d127 0 2024-03-14T13:02:29.485 INFO:tasks.workunit.client.1.smithi137.stdout:9/986: dread d6/df/d71/f128 [0,4194304] 0 2024-03-14T13:02:29.485 INFO:tasks.workunit.client.0.smithi111.stdout:3/913: mkdir d0/d2d/d35/d50/d5e/d145 0 2024-03-14T13:02:29.487 INFO:tasks.workunit.client.1.smithi137.stdout:9/987: readlink d6/df/d88/de5/df6/lb9 0 2024-03-14T13:02:29.488 INFO:tasks.workunit.client.0.smithi111.stdout:6/804: creat d0/d17/d20/d5a/f128 x:0 0 0 2024-03-14T13:02:29.488 INFO:tasks.workunit.client.1.smithi137.stdout:9/988: mknod d6/d53/c145 0 2024-03-14T13:02:29.490 INFO:tasks.workunit.client.0.smithi111.stdout:3/914: mknod d0/d2d/d35/d4c/dad/c146 0 2024-03-14T13:02:29.491 INFO:tasks.workunit.client.0.smithi111.stdout:2/912: creat dd/f12d x:0 0 0 2024-03-14T13:02:29.494 INFO:tasks.workunit.client.0.smithi111.stdout:6/805: rename d0/d17/d20/d65/l78 to d0/d17/d20/d4a/d68/d10e/l129 0 2024-03-14T13:02:29.496 INFO:tasks.workunit.client.0.smithi111.stdout:2/913: link dd/f12d dd/d17/d34/d74/dca/dcb/d114/f12e 0 2024-03-14T13:02:29.496 INFO:tasks.workunit.client.1.smithi137.stdout:0/985: dread d1/d5/d1b/d1e/d53/d6d/f9a [0,4194304] 0 2024-03-14T13:02:29.497 INFO:tasks.workunit.client.0.smithi111.stdout:3/915: rename d0/d2d/d35/d50/da2/fec to d0/d2d/d35/d4c/dad/f147 0 2024-03-14T13:02:29.499 INFO:tasks.workunit.client.0.smithi111.stdout:2/914: creat dd/d17/d53/da4/df5/f12f x:0 0 0 2024-03-14T13:02:29.501 INFO:tasks.workunit.client.1.smithi137.stdout:0/986: read d1/d1f/f24 [1424983,72478] 0 2024-03-14T13:02:29.501 INFO:tasks.workunit.client.0.smithi111.stdout:3/916: mkdir d0/d2d/d35/d4c/dad/d117/d148 0 2024-03-14T13:02:29.501 INFO:tasks.workunit.client.0.smithi111.stdout:2/915: truncate dd/d17/d34/d73/d49/f90 9237047 0 2024-03-14T13:02:29.502 INFO:tasks.workunit.client.1.smithi137.stdout:0/987: read d1/d5/d1b/d1e/d25/d55/f60 [1548223,51950] 0 2024-03-14T13:02:29.502 INFO:tasks.workunit.client.0.smithi111.stdout:2/916: chown dd/d17/d34/d74/dca/dcb/c104 1745305 1 2024-03-14T13:02:29.504 INFO:tasks.workunit.client.0.smithi111.stdout:3/917: truncate d0/d2d/d35/d4c/d82/f7e 74964 0 2024-03-14T13:02:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:02:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:29 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:02:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:29.506 INFO:tasks.workunit.client.1.smithi137.stdout:0/988: link d1/f46 d1/d5/d1b/d1e/f152 0 2024-03-14T13:02:29.508 INFO:tasks.workunit.client.0.smithi111.stdout:6/806: rename d0/d17/d21/d69/d3d to d0/d17/d21/d42/da5/d12a 0 2024-03-14T13:02:29.508 INFO:tasks.workunit.client.0.smithi111.stdout:3/918: truncate d0/d2d/fdc 992940 0 2024-03-14T13:02:29.509 INFO:tasks.workunit.client.1.smithi137.stdout:0/989: write d1/d5/d1b/d1e/d53/d6d/f107 [2323521,40067] 0 2024-03-14T13:02:29.509 INFO:tasks.workunit.client.0.smithi111.stdout:6/807: dread - d0/d17/d21/d42/da5/fed zero size 2024-03-14T13:02:29.510 INFO:tasks.workunit.client.0.smithi111.stdout:6/808: chown d0/d17/d20/d5a 65027934 1 2024-03-14T13:02:29.510 INFO:tasks.workunit.client.1.smithi137.stdout:0/990: dread - d1/d5/d1b/d1e/d25/d9f/fce zero size 2024-03-14T13:02:29.512 INFO:tasks.workunit.client.0.smithi111.stdout:6/809: creat d0/dfb/db8/de9/f12b x:0 0 0 2024-03-14T13:02:29.514 INFO:tasks.workunit.client.1.smithi137.stdout:0/991: rename d1/d5/d1b/d1e/d53/d6d/d102/d128 to d1/d5/d1b/d1e/d25/d10d/d153 0 2024-03-14T13:02:29.516 INFO:tasks.workunit.client.1.smithi137.stdout:0/992: fdatasync d1/da6/ffc 0 2024-03-14T13:02:29.517 INFO:tasks.workunit.client.0.smithi111.stdout:2/917: getdents dd/d17/d53/da4/df5 0 2024-03-14T13:02:29.518 INFO:tasks.workunit.client.1.smithi137.stdout:0/993: truncate d1/d44/f124 5176660 0 2024-03-14T13:02:29.520 INFO:tasks.workunit.client.0.smithi111.stdout:6/810: dwrite d0/d17/d20/d4a/d68/ff9 [0,4194304] 0 2024-03-14T13:02:29.520 INFO:tasks.workunit.client.0.smithi111.stdout:3/919: read d0/d2d/d35/d4c/d82/f7e [37530,89979] 0 2024-03-14T13:02:29.521 INFO:tasks.workunit.client.1.smithi137.stdout:0/994: creat d1/d5/d1b/d42/d3a/db0/d84/f154 x:0 0 0 2024-03-14T13:02:29.523 INFO:tasks.workunit.client.1.smithi137.stdout:0/995: getdents d1/d44/d72/df8 0 2024-03-14T13:02:29.525 INFO:tasks.workunit.client.0.smithi111.stdout:2/918: getdents dd/d11/d1a 0 2024-03-14T13:02:29.526 INFO:tasks.workunit.client.0.smithi111.stdout:2/919: chown dd/d17/d34/d73/d49/l124 0 1 2024-03-14T13:02:29.526 INFO:tasks.workunit.client.0.smithi111.stdout:6/811: rename d0/d17/d21/d69/c7a to d0/d17/d20/d94/c12c 0 2024-03-14T13:02:29.528 INFO:tasks.workunit.client.0.smithi111.stdout:6/812: dread - d0/d17/d20/d5a/f128 zero size 2024-03-14T13:02:29.530 INFO:tasks.workunit.client.0.smithi111.stdout:3/920: rename d0/d2d/d35/d50/d5e/l51 to d0/d2d/d35/d69/d10a/l149 0 2024-03-14T13:02:29.530 INFO:tasks.workunit.client.1.smithi137.stdout:0/996: dwrite d1/d44/d72/d7e/f78 [4194304,4194304] 0 2024-03-14T13:02:29.531 INFO:tasks.workunit.client.0.smithi111.stdout:3/921: chown d0/d2d/d35/d4c/d82/f7e 0 1 2024-03-14T13:02:29.531 INFO:tasks.workunit.client.0.smithi111.stdout:6/813: creat d0/d50/df2/f12d x:0 0 0 2024-03-14T13:02:29.532 INFO:tasks.workunit.client.0.smithi111.stdout:2/920: write dd/d17/f3a [873290,125293] 0 2024-03-14T13:02:29.536 INFO:tasks.workunit.client.0.smithi111.stdout:6/814: rename d0/d17/d21/ffe to d0/d17/d21/d69/f12e 0 2024-03-14T13:02:29.539 INFO:tasks.workunit.client.0.smithi111.stdout:2/921: creat dd/d17/d34/ddb/f130 x:0 0 0 2024-03-14T13:02:29.540 INFO:tasks.workunit.client.0.smithi111.stdout:2/922: chown dd/d17/d34/d73/d75/ff4 124982 1 2024-03-14T13:02:29.540 INFO:tasks.workunit.client.1.smithi137.stdout:0/997: dwrite d1/d5/d1b/d1e/fe1 [0,4194304] 0 2024-03-14T13:02:29.541 INFO:tasks.workunit.client.0.smithi111.stdout:2/923: readlink dd/d17/d34/ddb/l81 0 2024-03-14T13:02:29.543 INFO:tasks.workunit.client.1.smithi137.stdout:0/998: mknod d1/d5/d1b/d1e/d25/d10d/d112/c155 0 2024-03-14T13:02:29.548 INFO:tasks.workunit.client.0.smithi111.stdout:2/924: dwrite dd/d8d/f9c [0,4194304] 0 2024-03-14T13:02:29.550 INFO:tasks.workunit.client.1.smithi137.stdout:0/999: dwrite d1/f51 [0,4194304] 0 2024-03-14T13:02:29.553 INFO:tasks.workunit.client.0.smithi111.stdout:3/922: rename d0/d2d/d35/d69/d10a to d0/d2d/d35/d50/d14a 0 2024-03-14T13:02:29.556 INFO:tasks.workunit.client.0.smithi111.stdout:3/923: chown d0/d2d/d35/d50/ddf/f131 51301757 1 2024-03-14T13:02:29.557 INFO:tasks.workunit.client.0.smithi111.stdout:2/925: dwrite dd/d17/d53/da4/d105/f110 [0,4194304] 0 2024-03-14T13:02:29.561 INFO:tasks.workunit.client.0.smithi111.stdout:3/924: rmdir d0/d2d/d35/d4c/dad 39 2024-03-14T13:02:29.565 INFO:tasks.workunit.client.0.smithi111.stdout:2/926: dwrite dd/d17/d34/d74/dc1/ff0 [0,4194304] 0 2024-03-14T13:02:29.566 INFO:tasks.workunit.client.0.smithi111.stdout:2/927: chown dd/d11/d1a/d88 106857 1 2024-03-14T13:02:29.567 INFO:tasks.workunit.client.0.smithi111.stdout:3/925: rename d0/d30/c3b to d0/d2d/d35/d4c/d82/c14b 0 2024-03-14T13:02:29.568 INFO:tasks.workunit.client.0.smithi111.stdout:2/928: write dd/d17/d34/d73/d49/fe2 [459691,90361] 0 2024-03-14T13:02:29.571 INFO:tasks.workunit.client.0.smithi111.stdout:3/926: mknod d0/d2d/d35/d50/da2/db3/d71/dbc/c14c 0 2024-03-14T13:02:29.575 INFO:tasks.workunit.client.0.smithi111.stdout:2/929: link dd/d11/c2f dd/d17/d34/d73/c131 0 2024-03-14T13:02:29.579 INFO:tasks.workunit.client.0.smithi111.stdout:3/927: link d0/d2d/d35/d50/da2/db3/d67/le0 d0/d2d/d35/d4c/d82/l14d 0 2024-03-14T13:02:29.580 INFO:tasks.workunit.client.0.smithi111.stdout:2/930: link dd/d17/cd9 dd/d8d/c132 0 2024-03-14T13:02:29.582 INFO:tasks.workunit.client.0.smithi111.stdout:3/928: creat d0/d2d/d35/d50/da2/db3/d71/f14e x:0 0 0 2024-03-14T13:02:29.582 INFO:tasks.workunit.client.0.smithi111.stdout:2/931: symlink dd/d8d/dc2/dec/l133 0 2024-03-14T13:02:29.583 INFO:tasks.workunit.client.0.smithi111.stdout:3/929: fsync d0/d2d/d35/d69/d84/f87 0 2024-03-14T13:02:29.585 INFO:tasks.workunit.client.0.smithi111.stdout:2/932: unlink l9 0 2024-03-14T13:02:29.590 INFO:tasks.workunit.client.0.smithi111.stdout:3/930: truncate d0/d2d/d35/d50/da2/db3/da7/fbb 2158694 0 2024-03-14T13:02:29.594 INFO:tasks.workunit.client.0.smithi111.stdout:3/931: rename d0/d2d/d35/d69/de6/l96 to d0/d2d/d35/d50/d6c/da3/daa/l14f 0 2024-03-14T13:02:29.595 INFO:tasks.workunit.client.0.smithi111.stdout:2/933: rmdir dd/d17 39 2024-03-14T13:02:29.600 INFO:tasks.workunit.client.0.smithi111.stdout:2/934: rename dd/d17/d34/d74/dca/dcb to dd/d17/d34/d74/dca/dcb/d114/d134 22 2024-03-14T13:02:29.603 INFO:tasks.workunit.client.0.smithi111.stdout:3/932: dwrite d0/f1 [4194304,4194304] 0 2024-03-14T13:02:29.608 INFO:tasks.workunit.client.0.smithi111.stdout:2/935: symlink dd/d17/d34/d73/l135 0 2024-03-14T13:02:29.615 INFO:tasks.workunit.client.0.smithi111.stdout:2/936: dwrite dd/d17/d34/dbf/f109 [0,4194304] 0 2024-03-14T13:02:29.618 INFO:tasks.workunit.client.0.smithi111.stdout:2/937: truncate dd/d17/d34/fbe 985367 0 2024-03-14T13:02:29.619 INFO:tasks.workunit.client.0.smithi111.stdout:2/938: readlink l2 0 2024-03-14T13:02:29.621 INFO:tasks.workunit.client.0.smithi111.stdout:2/939: mkdir dd/d17/d34/d73/d75/d11c/d136 0 2024-03-14T13:02:29.623 INFO:tasks.workunit.client.0.smithi111.stdout:2/940: write dd/d17/d34/d73/d75/f5d [4206308,30241] 0 2024-03-14T13:02:29.624 INFO:tasks.workunit.client.0.smithi111.stdout:2/941: chown dd/d17/d34/dbf/de6 17 1 2024-03-14T13:02:29.625 INFO:tasks.workunit.client.1.smithi137.stdout:4/918: sync 2024-03-14T13:02:29.627 INFO:tasks.workunit.client.1.smithi137.stdout:4/919: chown d4/d10/c8a 17 1 2024-03-14T13:02:29.631 INFO:tasks.workunit.client.1.smithi137.stdout:4/920: rename d4/d27/d100/d8d to d4/d9/d3a/d41/d38/d79/dc2/d129 0 2024-03-14T13:02:29.635 INFO:tasks.workunit.client.1.smithi137.stdout:4/921: rename d4/d10/d5a/f85 to d4/d9/d3a/d41/d38/d79/dc2/d11f/f12a 0 2024-03-14T13:02:29.637 INFO:tasks.workunit.client.1.smithi137.stdout:4/922: unlink d4/d9/d3a/d41/d6a/dac/ff1 0 2024-03-14T13:02:29.646 INFO:tasks.workunit.client.1.smithi137.stdout:4/923: dwrite d4/d9/f103 [4194304,4194304] 0 2024-03-14T13:02:29.647 INFO:tasks.workunit.client.1.smithi137.stdout:4/924: dread - d4/d9/d3a/d41/fe8 zero size 2024-03-14T13:02:29.651 INFO:tasks.workunit.client.1.smithi137.stdout:4/925: link d4/d27/d5c/d6d/d58/f7e d4/d9/d3a/d41/d38/d55/db6/f12b 0 2024-03-14T13:02:29.656 INFO:tasks.workunit.client.1.smithi137.stdout:4/926: dread d4/d9/d3a/d41/d38/d55/db6/ffd [0,4194304] 0 2024-03-14T13:02:29.657 INFO:tasks.workunit.client.1.smithi137.stdout:4/927: stat d4/d9/d3a/d41/d38/d79/dc2/d129/fad 0 2024-03-14T13:02:29.658 INFO:tasks.workunit.client.1.smithi137.stdout:4/928: dread - d4/d9/d3a/d70/fd0 zero size 2024-03-14T13:02:29.661 INFO:tasks.workunit.client.1.smithi137.stdout:4/929: creat d4/d9/d3a/d41/d38/d79/dc2/d11f/d73/d9d/f12c x:0 0 0 2024-03-14T13:02:29.666 INFO:tasks.workunit.client.0.smithi111.stdout:9/897: sync 2024-03-14T13:02:29.668 INFO:tasks.workunit.client.0.smithi111.stdout:9/898: mknod d5/d42/d71/d96/dd3/d136/c13c 0 2024-03-14T13:02:29.674 INFO:tasks.workunit.client.0.smithi111.stdout:9/899: dwrite d5/d6/d8/d5b/f11c [0,4194304] 0 2024-03-14T13:02:29.676 INFO:tasks.workunit.client.1.smithi137.stdout:4/930: write d4/d9/d3a/f4f [1049629,6522] 0 2024-03-14T13:02:29.676 INFO:tasks.workunit.client.0.smithi111.stdout:9/900: stat d5/d42/d71/d96/dd3/f12c 0 2024-03-14T13:02:29.677 INFO:tasks.workunit.client.0.smithi111.stdout:9/901: write d5/d6/d14/d7a/d102/f130 [940849,4584] 0 2024-03-14T13:02:29.679 INFO:tasks.workunit.client.1.smithi137.stdout:4/931: unlink d4/d9/d3a/d41/l35 0 2024-03-14T13:02:29.679 INFO:tasks.workunit.client.0.smithi111.stdout:9/902: readlink d5/d6/d8/d5b/d7c/le8 0 2024-03-14T13:02:29.684 INFO:tasks.workunit.client.1.smithi137.stdout:4/932: dread d4/d9/d3a/d41/f49 [0,4194304] 0 2024-03-14T13:02:29.686 INFO:tasks.workunit.client.1.smithi137.stdout:4/933: write d4/d9/d3a/d41/d6a/ff8 [843305,45121] 0 2024-03-14T13:02:29.688 INFO:tasks.workunit.client.1.smithi137.stdout:4/934: write d4/d9/d3a/f4f [4087170,75193] 0 2024-03-14T13:02:29.689 INFO:tasks.workunit.client.1.smithi137.stdout:4/935: mknod d4/d27/d100/d121/c12d 0 2024-03-14T13:02:29.691 INFO:tasks.workunit.client.0.smithi111.stdout:9/903: dwrite d5/d6/d8/d5b/fbe [0,4194304] 0 2024-03-14T13:02:29.692 INFO:tasks.workunit.client.1.smithi137.stdout:4/936: mknod d4/d27/d5c/d77/c12e 0 2024-03-14T13:02:29.694 INFO:tasks.workunit.client.0.smithi111.stdout:9/904: readlink d5/d6/d8/d5b/d7c/lbb 0 2024-03-14T13:02:29.700 INFO:tasks.workunit.client.0.smithi111.stdout:9/905: dwrite d5/d42/f12a [0,4194304] 0 2024-03-14T13:02:29.702 INFO:tasks.workunit.client.0.smithi111.stdout:9/906: symlink d5/d42/d71/l13d 0 2024-03-14T13:02:29.709 INFO:tasks.workunit.client.1.smithi137.stdout:4/937: dread d4/d27/d5c/d6d/d58/f7e [4194304,4194304] 0 2024-03-14T13:02:29.711 INFO:tasks.workunit.client.0.smithi111.stdout:9/907: dwrite d5/d6/d2a/d5a/dee/f10d [0,4194304] 0 2024-03-14T13:02:29.712 INFO:tasks.workunit.client.0.smithi111.stdout:9/908: chown d5/d6/d2a/d2d/la0 5750 1 2024-03-14T13:02:29.712 INFO:tasks.workunit.client.1.smithi137.stdout:4/938: creat d4/d9/d3a/d41/d38/f12f x:0 0 0 2024-03-14T13:02:29.713 INFO:tasks.workunit.client.1.smithi137.stdout:4/939: mkdir d4/d10/d130 0 2024-03-14T13:02:29.725 INFO:tasks.workunit.client.1.smithi137.stdout:4/940: dread d4/d9/d3a/d41/d6a/dac/f109 [0,4194304] 0 2024-03-14T13:02:29.726 INFO:tasks.workunit.client.1.smithi137.stdout:4/941: write d4/d9/d3a/d41/fe8 [100463,101277] 0 2024-03-14T13:02:29.727 INFO:tasks.workunit.client.0.smithi111.stdout:9/909: dwrite d5/d6/f2f [4194304,4194304] 0 2024-03-14T13:02:29.728 INFO:tasks.workunit.client.1.smithi137.stdout:4/942: write d4/d9/d3a/d41/f110 [869791,107767] 0 2024-03-14T13:02:29.728 INFO:tasks.workunit.client.0.smithi111.stdout:9/910: dread - d5/d42/d71/d96/dd3/d136/f117 zero size 2024-03-14T13:02:29.729 INFO:tasks.workunit.client.1.smithi137.stdout:4/943: fsync d4/f3c 0 2024-03-14T13:02:29.733 INFO:tasks.workunit.client.0.smithi111.stdout:9/911: creat d5/d42/d71/dbd/d120/f13e x:0 0 0 2024-03-14T13:02:29.733 INFO:tasks.workunit.client.0.smithi111.stdout:9/912: dread - d5/d42/d71/d96/dd3/d136/f117 zero size 2024-03-14T13:02:29.734 INFO:tasks.workunit.client.0.smithi111.stdout:9/913: stat d5/d6/d14/f115 0 2024-03-14T13:02:29.734 INFO:tasks.workunit.client.1.smithi137.stdout:4/944: mknod d4/d9/d3a/d41/d38/d79/dc2/d11f/daf/c131 0 2024-03-14T13:02:29.735 INFO:tasks.workunit.client.1.smithi137.stdout:4/945: fdatasync d4/d9/d3a/d41/f110 0 2024-03-14T13:02:29.737 INFO:tasks.workunit.client.0.smithi111.stdout:9/914: symlink d5/d6/d14/l13f 0 2024-03-14T13:02:29.740 INFO:tasks.workunit.client.1.smithi137.stdout:4/946: rename d4/ld to d4/d9/d3a/d41/d38/d79/dc2/d11f/daf/l132 0 2024-03-14T13:02:29.742 INFO:tasks.workunit.client.1.smithi137.stdout:4/947: symlink d4/d9/d3a/d41/l133 0 2024-03-14T13:02:29.743 INFO:tasks.workunit.client.1.smithi137.stdout:4/948: mknod d4/d4d/c134 0 2024-03-14T13:02:29.756 INFO:tasks.workunit.client.0.smithi111.stdout:9/915: rmdir d5/d42/d71 39 2024-03-14T13:02:29.759 INFO:tasks.workunit.client.0.smithi111.stdout:9/916: creat d5/d6/d2a/d86/f140 x:0 0 0 2024-03-14T13:02:29.766 INFO:tasks.workunit.client.0.smithi111.stdout:9/917: dwrite d5/d42/d71/dbd/dcd/f88 [0,4194304] 0 2024-03-14T13:02:29.771 INFO:tasks.workunit.client.0.smithi111.stdout:9/918: link d5/d42/f5c d5/d6/f141 0 2024-03-14T13:02:29.800 INFO:tasks.workunit.client.1.smithi137.stdout:3/759: sync 2024-03-14T13:02:29.803 INFO:tasks.workunit.client.1.smithi137.stdout:3/760: dread d4/de/d16/d1c/d5d/d85/d6b/f6d [0,4194304] 0 2024-03-14T13:02:29.852 INFO:tasks.workunit.client.0.smithi111.stdout:7/952: sync 2024-03-14T13:02:29.866 INFO:tasks.workunit.client.0.smithi111.stdout:7/953: dread d0/d11/da8/dd4/df6/d108/d4a/d52/f62 [0,4194304] 0 2024-03-14T13:02:29.872 INFO:tasks.workunit.client.0.smithi111.stdout:7/954: dwrite d0/d11/da8/dd4/df6/d108/d4a/d52/f6c [4194304,4194304] 0 2024-03-14T13:02:29.872 INFO:tasks.workunit.client.0.smithi111.stdout:7/955: fsync d0/dc1/fc4 0 2024-03-14T13:02:29.875 INFO:tasks.workunit.client.0.smithi111.stdout:7/956: creat d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d99/f123 x:0 0 0 2024-03-14T13:02:29.879 INFO:tasks.workunit.client.0.smithi111.stdout:7/957: getdents d0/d11/da8/dd4/df6/d108/d5/d38/d5c 0 2024-03-14T13:02:29.882 INFO:tasks.workunit.client.0.smithi111.stdout:7/958: read d0/d7e/ffc [1426039,12748] 0 2024-03-14T13:02:29.886 INFO:tasks.workunit.client.0.smithi111.stdout:7/959: link d0/d7e/f49 d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/dee/d106/f124 0 2024-03-14T13:02:29.889 INFO:tasks.workunit.client.0.smithi111.stdout:7/960: creat d0/d11/da8/dd4/df6/d108/d5/f125 x:0 0 0 2024-03-14T13:02:29.891 INFO:tasks.workunit.client.0.smithi111.stdout:7/961: mknod d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5/dff/c126 0 2024-03-14T13:02:29.893 INFO:tasks.workunit.client.0.smithi111.stdout:7/962: mknod d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d99/c127 0 2024-03-14T13:02:29.894 INFO:tasks.workunit.client.0.smithi111.stdout:7/963: getdents d0/d11d 0 2024-03-14T13:02:29.897 INFO:tasks.workunit.client.0.smithi111.stdout:7/964: rename d0/d11/da8/dd4/df6/d108/d4a/d52/cd0 to d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/dee/d106/c128 0 2024-03-14T13:02:29.899 INFO:tasks.workunit.client.0.smithi111.stdout:7/965: mknod d0/d11/da8/dd4/df6/d108/d22/d3e/df1/df2/c129 0 2024-03-14T13:02:29.901 INFO:tasks.workunit.client.0.smithi111.stdout:7/966: write d0/d11/da8/dd4/df6/d108/d22/f46 [5140856,94139] 0 2024-03-14T13:02:29.901 INFO:tasks.workunit.client.0.smithi111.stdout:2/942: sync 2024-03-14T13:02:29.901 INFO:tasks.workunit.client.0.smithi111.stdout:6/815: sync 2024-03-14T13:02:29.901 INFO:tasks.workunit.client.0.smithi111.stdout:3/933: sync 2024-03-14T13:02:29.901 INFO:tasks.workunit.client.0.smithi111.stdout:9/919: sync 2024-03-14T13:02:29.904 INFO:tasks.workunit.client.0.smithi111.stdout:6/816: creat d0/d17/d20/d94/dbd/f12f x:0 0 0 2024-03-14T13:02:29.905 INFO:tasks.workunit.client.0.smithi111.stdout:7/967: mknod d0/d11/da8/dd4/df6/d108/d5/d38/d1b/d9c/c12a 0 2024-03-14T13:02:29.905 INFO:tasks.workunit.client.0.smithi111.stdout:3/934: read - d0/d2d/d35/d50/da2/db3/d71/f14e zero size 2024-03-14T13:02:29.911 INFO:tasks.workunit.client.0.smithi111.stdout:3/935: creat d0/d2d/d35/d4c/db0/f150 x:0 0 0 2024-03-14T13:02:29.912 INFO:tasks.workunit.client.0.smithi111.stdout:7/968: dwrite d0/d11/da8/dd4/df6/d108/d5/d38/d1b/fa1 [0,4194304] 0 2024-03-14T13:02:29.914 INFO:tasks.workunit.client.0.smithi111.stdout:6/817: symlink d0/d17/d20/d4a/d59/dc3/l130 0 2024-03-14T13:02:29.915 INFO:tasks.workunit.client.0.smithi111.stdout:6/818: readlink d0/d17/d21/d42/da5/d12a/dca/l117 0 2024-03-14T13:02:29.916 INFO:tasks.workunit.client.0.smithi111.stdout:3/936: truncate d0/fc 4471058 0 2024-03-14T13:02:29.917 INFO:tasks.workunit.client.0.smithi111.stdout:2/943: truncate dd/d17/d34/ddb/f97 724287 0 2024-03-14T13:02:29.919 INFO:tasks.workunit.client.0.smithi111.stdout:7/969: creat d0/f12b x:0 0 0 2024-03-14T13:02:29.921 INFO:tasks.workunit.client.0.smithi111.stdout:2/944: write dd/da9/fb2 [955818,99844] 0 2024-03-14T13:02:29.923 INFO:tasks.workunit.client.0.smithi111.stdout:9/920: dwrite d5/f65 [0,4194304] 0 2024-03-14T13:02:29.923 INFO:tasks.workunit.client.0.smithi111.stdout:3/937: fdatasync d0/d2d/d35/d50/d5e/fd4 0 2024-03-14T13:02:29.925 INFO:tasks.workunit.client.0.smithi111.stdout:3/938: fsync d0/d2d/d35/ff2 0 2024-03-14T13:02:29.930 INFO:tasks.workunit.client.0.smithi111.stdout:2/945: unlink dd/d17/d34/dbf/lc3 0 2024-03-14T13:02:29.931 INFO:tasks.workunit.client.0.smithi111.stdout:9/921: dread d5/d42/d71/d96/fde [0,4194304] 0 2024-03-14T13:02:29.933 INFO:tasks.workunit.client.0.smithi111.stdout:6/819: write d0/d17/d21/d42/da5/d12a/f41 [3030057,110173] 0 2024-03-14T13:02:29.936 INFO:tasks.workunit.client.0.smithi111.stdout:6/820: creat d0/d17/d20/d94/dbd/f131 x:0 0 0 2024-03-14T13:02:29.936 INFO:tasks.workunit.client.0.smithi111.stdout:2/946: link dd/d17/d34/d73/faf dd/da9/dd5/f137 0 2024-03-14T13:02:29.936 INFO:tasks.workunit.client.0.smithi111.stdout:9/922: read d5/d42/d71/d96/fac [433334,45836] 0 2024-03-14T13:02:29.938 INFO:tasks.workunit.client.0.smithi111.stdout:6/821: fsync d0/dfb/f38 0 2024-03-14T13:02:29.939 INFO:tasks.workunit.client.0.smithi111.stdout:9/923: fdatasync d5/d6/d8/f10b 0 2024-03-14T13:02:29.942 INFO:tasks.workunit.client.0.smithi111.stdout:6/822: mknod d0/c132 0 2024-03-14T13:02:29.942 INFO:tasks.workunit.client.0.smithi111.stdout:9/924: read d5/d6/d8/f93 [1264886,1437] 0 2024-03-14T13:02:29.943 INFO:tasks.workunit.client.1.smithi137.stdout:6/911: sync 2024-03-14T13:02:29.945 INFO:tasks.workunit.client.0.smithi111.stdout:7/970: write d0/d11/da8/dd4/df6/d108/f116 [4707127,10085] 0 2024-03-14T13:02:29.945 INFO:tasks.workunit.client.1.smithi137.stdout:6/912: rmdir d2/d54/d55/d6d/d76/d132/de 39 2024-03-14T13:02:29.946 INFO:tasks.workunit.client.1.smithi137.stdout:6/913: chown d2/d54/d55/d6d/d76/d132/d1d/c96 14890 1 2024-03-14T13:02:29.950 INFO:tasks.workunit.client.0.smithi111.stdout:6/823: creat d0/d17/d20/d94/dbd/f133 x:0 0 0 2024-03-14T13:02:29.951 INFO:tasks.workunit.client.0.smithi111.stdout:9/925: mknod d5/d42/d52/c142 0 2024-03-14T13:02:29.951 INFO:tasks.workunit.client.0.smithi111.stdout:2/947: dwrite dd/d17/d34/d74/dca/dcb/fef [0,4194304] 0 2024-03-14T13:02:29.953 INFO:tasks.workunit.client.0.smithi111.stdout:9/926: chown d5/d6/d8/d1a/cb5 10 1 2024-03-14T13:02:29.955 INFO:tasks.workunit.client.0.smithi111.stdout:2/948: fdatasync dd/d11/d1a/d88/f8a 0 2024-03-14T13:02:29.956 INFO:tasks.workunit.client.0.smithi111.stdout:3/939: dread d0/f20 [0,4194304] 0 2024-03-14T13:02:29.957 INFO:tasks.workunit.client.0.smithi111.stdout:2/949: fsync dd/d8d/fc0 0 2024-03-14T13:02:29.961 INFO:tasks.workunit.client.0.smithi111.stdout:7/971: getdents d0/d11/da8/dd4/df6/d108/d22/d3e 0 2024-03-14T13:02:29.967 INFO:tasks.workunit.client.0.smithi111.stdout:2/950: rename dd/d11/d1a/lfb to dd/d17/d34/d73/d75/d11c/d136/l138 0 2024-03-14T13:02:29.968 INFO:tasks.workunit.client.0.smithi111.stdout:3/940: dwrite d0/d2d/d35/d50/ddf/f131 [0,4194304] 0 2024-03-14T13:02:29.971 INFO:tasks.workunit.client.1.smithi137.stdout:7/949: sync 2024-03-14T13:02:29.973 INFO:tasks.workunit.client.0.smithi111.stdout:7/972: truncate d0/d11/da8/dd4/df6/d108/d5/d38/f7f 636734 0 2024-03-14T13:02:29.976 INFO:tasks.workunit.client.0.smithi111.stdout:3/941: creat d0/d2d/d35/d50/da2/f151 x:0 0 0 2024-03-14T13:02:29.976 INFO:tasks.workunit.client.0.smithi111.stdout:2/951: creat dd/d11/f139 x:0 0 0 2024-03-14T13:02:29.977 INFO:tasks.workunit.client.0.smithi111.stdout:9/927: dwrite d5/d6/d14/d7a/d102/f108 [0,4194304] 0 2024-03-14T13:02:29.978 INFO:tasks.workunit.client.1.smithi137.stdout:6/914: mkdir d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/dfe/d135 0 2024-03-14T13:02:29.979 INFO:tasks.workunit.client.1.smithi137.stdout:6/915: chown d2/cfd 97008 1 2024-03-14T13:02:29.980 INFO:tasks.workunit.client.0.smithi111.stdout:3/942: chown d0/d2d/d35/d50/f88 414429 1 2024-03-14T13:02:29.983 INFO:tasks.workunit.client.1.smithi137.stdout:6/916: symlink d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/d83/l136 0 2024-03-14T13:02:29.985 INFO:tasks.workunit.client.1.smithi137.stdout:6/917: dread - d2/d54/d55/d6d/d76/d132/da7/f117 zero size 2024-03-14T13:02:29.986 INFO:tasks.workunit.client.0.smithi111.stdout:9/928: fdatasync d5/d42/d71/dbd/dcd/fb4 0 2024-03-14T13:02:29.989 INFO:tasks.workunit.client.0.smithi111.stdout:9/929: chown d5/d6/d8/c40 380 1 2024-03-14T13:02:29.989 INFO:tasks.workunit.client.1.smithi137.stdout:6/918: mknod d2/d54/d55/d6d/d76/d132/de/d45/d59/c137 0 2024-03-14T13:02:29.991 INFO:tasks.workunit.client.0.smithi111.stdout:2/952: dwrite dd/f129 [0,4194304] 0 2024-03-14T13:02:29.994 INFO:tasks.workunit.client.0.smithi111.stdout:9/930: creat d5/d6/d8/d5b/d84/db6/de7/f143 x:0 0 0 2024-03-14T13:02:29.996 INFO:tasks.workunit.client.1.smithi137.stdout:6/919: getdents d2/d54/d55/d6d/d76/d132/d86 0 2024-03-14T13:02:29.999 INFO:tasks.workunit.client.0.smithi111.stdout:2/953: creat dd/d17/d34/d74/dca/d128/f13a x:0 0 0 2024-03-14T13:02:30.001 INFO:tasks.workunit.client.0.smithi111.stdout:2/954: stat dd/d8d/dc2/f9a 0 2024-03-14T13:02:30.002 INFO:tasks.workunit.client.0.smithi111.stdout:3/943: dwrite d0/d2d/d35/d50/d5e/f37 [0,4194304] 0 2024-03-14T13:02:30.007 INFO:tasks.workunit.client.0.smithi111.stdout:2/955: write dd/d17/d34/d74/f11f [79649,111500] 0 2024-03-14T13:02:30.013 INFO:tasks.workunit.client.0.smithi111.stdout:9/931: dwrite d5/d6/fe1 [0,4194304] 0 2024-03-14T13:02:30.013 INFO:tasks.workunit.client.0.smithi111.stdout:7/973: dread d0/d11/da8/dd4/df6/d108/d21/fa4 [0,4194304] 0 2024-03-14T13:02:30.014 INFO:tasks.workunit.client.0.smithi111.stdout:2/956: mkdir dd/d17/d34/d74/dca/dcb/d13b 0 2024-03-14T13:02:30.015 INFO:tasks.workunit.client.0.smithi111.stdout:2/957: write dd/d8d/f115 [2756409,88293] 0 2024-03-14T13:02:30.018 INFO:tasks.workunit.client.0.smithi111.stdout:7/974: mkdir d0/d11/da8/d12c 0 2024-03-14T13:02:30.019 INFO:tasks.workunit.client.1.smithi137.stdout:7/950: truncate d3/d6/f45 3677004 0 2024-03-14T13:02:30.019 INFO:tasks.workunit.client.1.smithi137.stdout:7/951: fdatasync d3/f32 0 2024-03-14T13:02:30.020 INFO:tasks.workunit.client.0.smithi111.stdout:2/958: creat dd/d17/d34/d73/d75/f13c x:0 0 0 2024-03-14T13:02:30.021 INFO:tasks.workunit.client.1.smithi137.stdout:7/952: write d3/d5/d23/d3f/f105 [52296,90019] 0 2024-03-14T13:02:30.023 INFO:tasks.workunit.client.0.smithi111.stdout:2/959: symlink dd/d11/d8b/l13d 0 2024-03-14T13:02:30.024 INFO:tasks.workunit.client.0.smithi111.stdout:7/975: getdents d0/d11/da8/d12c 0 2024-03-14T13:02:30.025 INFO:tasks.workunit.client.1.smithi137.stdout:7/953: symlink d3/d6/d59/daa/dcf/d6a/da1/d13d/l13e 0 2024-03-14T13:02:30.026 INFO:tasks.workunit.client.0.smithi111.stdout:2/960: creat dd/d11/d8b/f13e x:0 0 0 2024-03-14T13:02:30.026 INFO:tasks.workunit.client.0.smithi111.stdout:9/932: link d5/l19 d5/d6/d8/d5b/d7c/l144 0 2024-03-14T13:02:30.027 INFO:tasks.workunit.client.1.smithi137.stdout:7/954: write f1 [1855349,64181] 0 2024-03-14T13:02:30.028 INFO:tasks.workunit.client.0.smithi111.stdout:9/933: read d5/f20 [1883158,8757] 0 2024-03-14T13:02:30.029 INFO:tasks.workunit.client.1.smithi137.stdout:6/920: dwrite d2/d54/d55/d6d/d76/d132/d1d/d88/f12f [0,4194304] 0 2024-03-14T13:02:30.029 INFO:tasks.workunit.client.0.smithi111.stdout:2/961: rmdir dd/d8d/db7 39 2024-03-14T13:02:30.031 INFO:tasks.workunit.client.1.smithi137.stdout:7/955: creat d3/d6/d64/d9e/d10d/d12e/f13f x:0 0 0 2024-03-14T13:02:30.031 INFO:tasks.workunit.client.1.smithi137.stdout:2/964: sync 2024-03-14T13:02:30.031 INFO:tasks.workunit.client.1.smithi137.stdout:4/949: sync 2024-03-14T13:02:30.032 INFO:tasks.workunit.client.0.smithi111.stdout:7/976: truncate d0/d11/da8/dd4/df6/d108/d5/d56/f5e 1404562 0 2024-03-14T13:02:30.032 INFO:tasks.workunit.client.1.smithi137.stdout:9/989: sync 2024-03-14T13:02:30.032 INFO:tasks.workunit.client.1.smithi137.stdout:5/850: sync 2024-03-14T13:02:30.032 INFO:tasks.workunit.client.1.smithi137.stdout:8/976: sync 2024-03-14T13:02:30.033 INFO:tasks.workunit.client.1.smithi137.stdout:6/921: mknod d2/d54/d55/d6d/d76/d132/c138 0 2024-03-14T13:02:30.035 INFO:tasks.workunit.client.1.smithi137.stdout:2/965: fsync d2/d7/d2e/d3e/d44/d6b/f134 0 2024-03-14T13:02:30.035 INFO:tasks.workunit.client.1.smithi137.stdout:9/990: fdatasync d6/d91/dc8/f120 0 2024-03-14T13:02:30.039 INFO:tasks.workunit.client.0.smithi111.stdout:9/934: dwrite d5/d6/fd5 [4194304,4194304] 0 2024-03-14T13:02:30.039 INFO:tasks.workunit.client.1.smithi137.stdout:9/991: stat d6/df/d88/de5/df6/da9/ldd 0 2024-03-14T13:02:30.045 INFO:tasks.workunit.client.0.smithi111.stdout:7/977: rename d0/d11/da8/dd4/df6/d108/d21/d2a to d0/d11/da8/dd4/df6/d108/d22/d3e/df1/df2/d12d 0 2024-03-14T13:02:30.046 INFO:tasks.workunit.client.1.smithi137.stdout:5/851: rmdir d7/d1d/d74 39 2024-03-14T13:02:30.046 INFO:tasks.workunit.client.1.smithi137.stdout:3/761: sync 2024-03-14T13:02:30.047 INFO:tasks.workunit.client.0.smithi111.stdout:3/944: dread d0/d2d/d35/d50/d6c/da3/daa/fb1 [0,4194304] 0 2024-03-14T13:02:30.047 INFO:tasks.workunit.client.0.smithi111.stdout:7/978: readlink d0/d11/da8/dd4/df6/d108/d5/d38/l10b 0 2024-03-14T13:02:30.048 INFO:tasks.workunit.client.1.smithi137.stdout:8/977: mkdir d1/dc/d71/db0/dca/d13e 0 2024-03-14T13:02:30.048 INFO:tasks.workunit.client.1.smithi137.stdout:9/992: write d6/df/d88/de5/df6/da9/fd6 [413755,74271] 0 2024-03-14T13:02:30.048 INFO:tasks.workunit.client.1.smithi137.stdout:7/956: creat d3/d6/d64/d9e/d127/f140 x:0 0 0 2024-03-14T13:02:30.049 INFO:tasks.workunit.client.0.smithi111.stdout:3/945: chown d0/d2d/d35/d4c/dad/c146 58 1 2024-03-14T13:02:30.050 INFO:tasks.workunit.client.0.smithi111.stdout:7/979: write d0/d11/da8/dd4/df6/d108/d4a/d52/fd2 [482365,94202] 0 2024-03-14T13:02:30.050 INFO:tasks.workunit.client.1.smithi137.stdout:6/922: symlink d2/d54/d55/d6d/d76/d132/d86/l139 0 2024-03-14T13:02:30.052 INFO:tasks.workunit.client.0.smithi111.stdout:3/946: fdatasync d0/d2d/d35/d4c/dad/f147 0 2024-03-14T13:02:30.054 INFO:tasks.workunit.client.1.smithi137.stdout:2/966: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d26/f100 [0,4194304] 0 2024-03-14T13:02:30.055 INFO:tasks.workunit.client.0.smithi111.stdout:7/980: creat d0/d11/da8/dd4/df6/d108/d22/d3e/df1/df2/d12d/d8c/f12e x:0 0 0 2024-03-14T13:02:30.059 INFO:tasks.workunit.client.0.smithi111.stdout:9/935: dwrite d5/d6/d8/d74/f77 [4194304,4194304] 0 2024-03-14T13:02:30.060 INFO:tasks.workunit.client.1.smithi137.stdout:5/852: mknod d7/d16/d18/d25/d28/d3c/d7a/c11b 0 2024-03-14T13:02:30.063 INFO:tasks.workunit.client.1.smithi137.stdout:3/762: dread d4/de/d16/d27/f73 [0,4194304] 0 2024-03-14T13:02:30.064 INFO:tasks.workunit.client.0.smithi111.stdout:7/981: creat d0/d11/da8/dd4/df6/d108/d22/d3e/f12f x:0 0 0 2024-03-14T13:02:30.066 INFO:tasks.workunit.client.1.smithi137.stdout:9/993: link d6/f96 d6/df/d71/d94/d12d/f146 0 2024-03-14T13:02:30.066 INFO:tasks.workunit.client.0.smithi111.stdout:3/947: dwrite d0/d2d/d35/d50/da2/db3/da7/fe7 [0,4194304] 0 2024-03-14T13:02:30.067 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:30 smithi137 ceph-mon[50336]: pgmap v29: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 158 MiB/s rd, 411 MiB/s wr, 838 op/s 2024-03-14T13:02:30.068 INFO:tasks.workunit.client.0.smithi111.stdout:3/948: creat d0/d2d/f152 x:0 0 0 2024-03-14T13:02:30.070 INFO:tasks.workunit.client.1.smithi137.stdout:7/957: dwrite d3/d5/f116 [0,4194304] 0 2024-03-14T13:02:30.070 INFO:tasks.workunit.client.0.smithi111.stdout:9/936: dread d5/d6/d2a/d2d/fa8 [0,4194304] 0 2024-03-14T13:02:30.072 INFO:tasks.workunit.client.1.smithi137.stdout:3/763: write d4/df/d82/feb [322644,46602] 0 2024-03-14T13:02:30.073 INFO:tasks.workunit.client.0.smithi111.stdout:9/937: mkdir d5/d6/d2a/d86/d145 0 2024-03-14T13:02:30.074 INFO:tasks.workunit.client.0.smithi111.stdout:9/938: readlink d5/l98 0 2024-03-14T13:02:30.074 INFO:tasks.workunit.client.0.smithi111.stdout:3/949: fsync d0/d2d/d35/d50/d5e/d19/f5a 0 2024-03-14T13:02:30.075 INFO:tasks.workunit.client.0.smithi111.stdout:9/939: chown d5/d42/d52/ld0 701767 1 2024-03-14T13:02:30.076 INFO:tasks.workunit.client.0.smithi111.stdout:7/982: link d0/d11/da8/dd4/df6/d108/d22/d3e/df1/df2/d12d/f8a d0/d7e/f130 0 2024-03-14T13:02:30.076 INFO:tasks.workunit.client.1.smithi137.stdout:5/853: mkdir d7/d16/d41/d11c 0 2024-03-14T13:02:30.077 INFO:tasks.workunit.client.0.smithi111.stdout:9/940: write d5/d6/d2a/d5a/f114 [353863,65004] 0 2024-03-14T13:02:30.078 INFO:tasks.workunit.client.1.smithi137.stdout:2/967: unlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dda/ff7 0 2024-03-14T13:02:30.078 INFO:tasks.workunit.client.0.smithi111.stdout:3/950: mkdir d0/d2d/d35/d50/da2/db3/d67/d153 0 2024-03-14T13:02:30.080 INFO:tasks.workunit.client.1.smithi137.stdout:4/950: dread d4/d9/d8e/fd8 [0,4194304] 0 2024-03-14T13:02:30.080 INFO:tasks.workunit.client.0.smithi111.stdout:2/962: dread dd/d17/d53/f84 [0,4194304] 0 2024-03-14T13:02:30.083 INFO:tasks.workunit.client.1.smithi137.stdout:8/978: getdents d1/dc/d10/dcf/dd2 0 2024-03-14T13:02:30.084 INFO:tasks.workunit.client.1.smithi137.stdout:8/979: stat d1/dc/d71/l108 0 2024-03-14T13:02:30.086 INFO:tasks.workunit.client.0.smithi111.stdout:9/941: rmdir d5/d42/d71/dbd/de0 39 2024-03-14T13:02:30.086 INFO:tasks.workunit.client.1.smithi137.stdout:9/994: mkdir d6/df/d19/d33/dec/d147 0 2024-03-14T13:02:30.087 INFO:tasks.workunit.client.1.smithi137.stdout:2/968: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/f9b 22 1 2024-03-14T13:02:30.087 INFO:tasks.workunit.client.1.smithi137.stdout:7/958: mkdir d3/d5/d5a/d86/d141 0 2024-03-14T13:02:30.087 INFO:tasks.workunit.client.0.smithi111.stdout:7/983: mkdir d0/d11/da8/dd4/df6/d108/d22/d131 0 2024-03-14T13:02:30.088 INFO:tasks.workunit.client.1.smithi137.stdout:9/995: chown d6/df/d88/de5/df6/da9/dde/c13a 29063690 1 2024-03-14T13:02:30.090 INFO:tasks.workunit.client.1.smithi137.stdout:3/764: creat d4/de/d16/d1c/db8/ff4 x:0 0 0 2024-03-14T13:02:30.092 INFO:tasks.workunit.client.1.smithi137.stdout:4/951: mkdir d4/d10/d130/d135 0 2024-03-14T13:02:30.092 INFO:tasks.workunit.client.0.smithi111.stdout:7/984: mknod d0/d11/da8/dd4/df6/d108/d5/d38/d5c/d6d/c132 0 2024-03-14T13:02:30.093 INFO:tasks.workunit.client.1.smithi137.stdout:5/854: mkdir d7/d16/d18/d25/d28/d9e/de5/d11d 0 2024-03-14T13:02:30.094 INFO:tasks.workunit.client.1.smithi137.stdout:8/980: dread d1/d5e/d11a/d97/d11d/f122 [0,4194304] 0 2024-03-14T13:02:30.094 INFO:tasks.workunit.client.1.smithi137.stdout:6/923: link d2/d54/d55/d6d/d76/d132/d1d/d88/ce4 d2/d54/d55/d6d/d76/d132/c13a 0 2024-03-14T13:02:30.096 INFO:tasks.workunit.client.1.smithi137.stdout:2/969: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dda/f13b x:0 0 0 2024-03-14T13:02:30.097 INFO:tasks.workunit.client.1.smithi137.stdout:3/765: truncate d4/de/fcb 185353 0 2024-03-14T13:02:30.097 INFO:tasks.workunit.client.1.smithi137.stdout:5/855: write d7/d16/d18/d25/d28/d3c/d7a/fc0 [138582,7222] 0 2024-03-14T13:02:30.099 INFO:tasks.workunit.client.1.smithi137.stdout:7/959: creat d3/d6/d14/d22/f142 x:0 0 0 2024-03-14T13:02:30.100 INFO:tasks.workunit.client.1.smithi137.stdout:4/952: unlink d4/d9/d3a/d41/d38/d55/c91 0 2024-03-14T13:02:30.104 INFO:tasks.workunit.client.0.smithi111.stdout:3/951: rmdir d0/d2d/d35 39 2024-03-14T13:02:30.106 INFO:tasks.workunit.client.1.smithi137.stdout:5/856: creat d7/d49/d75/f11e x:0 0 0 2024-03-14T13:02:30.107 INFO:tasks.workunit.client.0.smithi111.stdout:7/985: write d0/d11/da8/dd4/df6/d108/d22/d3e/f64 [1346889,76727] 0 2024-03-14T13:02:30.108 INFO:tasks.workunit.client.1.smithi137.stdout:2/970: dwrite d2/d7/d2e/d3e/d44/d6b/d1d/fdd [4194304,4194304] 0 2024-03-14T13:02:30.108 INFO:tasks.workunit.client.1.smithi137.stdout:7/960: readlink d3/d4a/l58 0 2024-03-14T13:02:30.108 INFO:tasks.workunit.client.1.smithi137.stdout:4/953: dread d4/d9/d3a/d41/d38/f4c [4194304,4194304] 0 2024-03-14T13:02:30.115 INFO:tasks.workunit.client.1.smithi137.stdout:3/766: mknod d4/de/d16/d1c/d8d/dcd/cf5 0 2024-03-14T13:02:30.115 INFO:tasks.workunit.client.1.smithi137.stdout:2/971: mkdir d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d13c 0 2024-03-14T13:02:30.116 INFO:tasks.workunit.client.0.smithi111.stdout:3/952: dwrite d0/f1c [0,4194304] 0 2024-03-14T13:02:30.118 INFO:tasks.workunit.client.0.smithi111.stdout:9/942: dwrite d5/d42/da4/fea [0,4194304] 0 2024-03-14T13:02:30.120 INFO:tasks.workunit.client.0.smithi111.stdout:2/963: dwrite dd/d17/d34/d73/faf [0,4194304] 0 2024-03-14T13:02:30.122 INFO:tasks.workunit.client.0.smithi111.stdout:2/964: chown dd/d17/d34/d74/dc1/dd1 210 1 2024-03-14T13:02:30.123 INFO:tasks.workunit.client.1.smithi137.stdout:8/981: write d1/dc/da9/f10c [2283143,42040] 0 2024-03-14T13:02:30.124 INFO:tasks.workunit.client.1.smithi137.stdout:2/972: rename d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/c85 to d2/d7/d2e/d3e/d44/c13d 0 2024-03-14T13:02:30.126 INFO:tasks.workunit.client.1.smithi137.stdout:8/982: stat d1/d5e/dc9/dda/f134 0 2024-03-14T13:02:30.128 INFO:tasks.workunit.client.1.smithi137.stdout:9/996: dwrite d6/df/d88/de5/df6/d40/fb0 [4194304,4194304] 0 2024-03-14T13:02:30.129 INFO:tasks.workunit.client.0.smithi111.stdout:3/953: creat d0/d2d/d35/d50/da2/db3/d67/f154 x:0 0 0 2024-03-14T13:02:30.131 INFO:tasks.workunit.client.0.smithi111.stdout:2/965: rename dd/d17/d53/da4/d105/c125 to dd/d17/d34/d73/d75/d11c/d136/c13f 0 2024-03-14T13:02:30.131 INFO:tasks.workunit.client.1.smithi137.stdout:3/767: dread d4/de/f2a [0,4194304] 0 2024-03-14T13:02:30.132 INFO:tasks.workunit.client.1.smithi137.stdout:8/983: readlink d1/l3a 0 2024-03-14T13:02:30.133 INFO:tasks.workunit.client.0.smithi111.stdout:3/954: symlink d0/d2d/d35/d69/de6/dd8/l155 0 2024-03-14T13:02:30.133 INFO:tasks.workunit.client.0.smithi111.stdout:2/966: mknod dd/d8d/d10c/c140 0 2024-03-14T13:02:30.135 INFO:tasks.workunit.client.1.smithi137.stdout:2/973: truncate d2/f12 3162629 0 2024-03-14T13:02:30.135 INFO:tasks.workunit.client.1.smithi137.stdout:6/924: dwrite d2/d54/d55/d33/fcc [0,4194304] 0 2024-03-14T13:02:30.137 INFO:tasks.workunit.client.1.smithi137.stdout:8/984: creat d1/dc/d71/da3/f13f x:0 0 0 2024-03-14T13:02:30.138 INFO:tasks.workunit.client.1.smithi137.stdout:3/768: mkdir d4/de/d16/d1c/d5d/d4b/df6 0 2024-03-14T13:02:30.138 INFO:tasks.workunit.client.0.smithi111.stdout:3/955: mknod d0/d2d/d35/d50/da2/db3/d71/dbc/c156 0 2024-03-14T13:02:30.139 INFO:tasks.workunit.client.1.smithi137.stdout:9/997: rename d6/cf7 to d6/df/d19/d33/d8f/d124/c148 0 2024-03-14T13:02:30.144 INFO:tasks.workunit.client.1.smithi137.stdout:2/974: mkdir d2/d7/d2e/d3e/daa/dcd/d13e 0 2024-03-14T13:02:30.144 INFO:tasks.workunit.client.1.smithi137.stdout:6/925: creat d2/d54/d55/d6d/d76/d132/dee/f13b x:0 0 0 2024-03-14T13:02:30.146 INFO:tasks.workunit.client.0.smithi111.stdout:7/986: dwrite d0/d11/da8/dd4/df6/d108/d22/d3e/df1/df2/d12d/f8a [0,4194304] 0 2024-03-14T13:02:30.147 INFO:tasks.workunit.client.1.smithi137.stdout:8/985: mkdir d1/dc/d4e/d6d/d140 0 2024-03-14T13:02:30.150 INFO:tasks.workunit.client.0.smithi111.stdout:7/987: creat d0/d11/da8/dd4/df6/d108/d4a/dc7/f133 x:0 0 0 2024-03-14T13:02:30.151 INFO:tasks.workunit.client.1.smithi137.stdout:3/769: dwrite d4/de/d16/d1c/db8/dd1/fdd [0,4194304] 0 2024-03-14T13:02:30.152 INFO:tasks.workunit.client.1.smithi137.stdout:9/998: dread d6/d56/fb3 [0,4194304] 0 2024-03-14T13:02:30.154 INFO:tasks.workunit.client.1.smithi137.stdout:8/986: dread - d1/d5e/d11a/d82/d96/f125 zero size 2024-03-14T13:02:30.157 INFO:tasks.workunit.client.0.smithi111.stdout:6/824: sync 2024-03-14T13:02:30.161 INFO:tasks.workunit.client.0.smithi111.stdout:6/825: dread - d0/dfb/db8/d127/f95 zero size 2024-03-14T13:02:30.162 INFO:tasks.workunit.client.1.smithi137.stdout:5/857: write d7/d16/d18/d25/d28/d3c/dd8/f119 [766175,70319] 0 2024-03-14T13:02:30.164 INFO:tasks.workunit.client.0.smithi111.stdout:6/826: symlink d0/d50/df2/dc6/d126/l134 0 2024-03-14T13:02:30.166 INFO:tasks.workunit.client.0.smithi111.stdout:6/827: chown d0/d50/df2/dc6/d126/ff8 12 1 2024-03-14T13:02:30.166 INFO:tasks.workunit.client.1.smithi137.stdout:5/858: fsync d7/d33/d65/f7c 0 2024-03-14T13:02:30.166 INFO:tasks.workunit.client.1.smithi137.stdout:9/999: chown d6/df/d88/de5/df6/d40/lb1 179 1 2024-03-14T13:02:30.167 INFO:tasks.workunit.client.0.smithi111.stdout:9/943: dwrite d5/d6/f141 [0,4194304] 0 2024-03-14T13:02:30.167 INFO:tasks.workunit.client.1.smithi137.stdout:4/954: truncate d4/d9/d3a/d70/fdf 2046957 0 2024-03-14T13:02:30.168 INFO:tasks.workunit.client.1.smithi137.stdout:3/770: mkdir d4/de/d16/d1c/d5d/d85/d6b/df7 0 2024-03-14T13:02:30.169 INFO:tasks.workunit.client.1.smithi137.stdout:6/926: dwrite d2/d54/d55/d6d/d76/d132/de/d45/d59/f75 [0,4194304] 0 2024-03-14T13:02:30.170 INFO:tasks.workunit.client.0.smithi111.stdout:7/988: truncate d0/d11/da8/dd4/df6/d108/d5/d38/d5c/fb8 317205 0 2024-03-14T13:02:30.172 INFO:tasks.workunit.client.1.smithi137.stdout:4/955: chown d4/d27/d100/db7/fde 21849186 1 2024-03-14T13:02:30.173 INFO:tasks.workunit.client.1.smithi137.stdout:8/987: dwrite d1/d5e/dc9/dda/f134 [0,4194304] 0 2024-03-14T13:02:30.174 INFO:tasks.workunit.client.1.smithi137.stdout:2/975: link d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/lb3 d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d13c/l13f 0 2024-03-14T13:02:30.174 INFO:tasks.workunit.client.0.smithi111.stdout:6/828: dread - d0/d17/d20/d4a/d59/d82/fcf zero size 2024-03-14T13:02:30.175 INFO:tasks.workunit.client.1.smithi137.stdout:3/771: creat d4/de/d16/d1c/d5d/d85/d6b/ff8 x:0 0 0 2024-03-14T13:02:30.176 INFO:tasks.workunit.client.0.smithi111.stdout:9/944: unlink d5/d42/d71/ddc/c100 0 2024-03-14T13:02:30.179 INFO:tasks.workunit.client.1.smithi137.stdout:4/956: mkdir d4/d27/d5c/d6d/d136 0 2024-03-14T13:02:30.183 INFO:tasks.workunit.client.0.smithi111.stdout:2/967: dwrite dd/d11/d1a/f62 [0,4194304] 0 2024-03-14T13:02:30.186 INFO:tasks.workunit.client.0.smithi111.stdout:3/956: dread d0/d2d/d35/d50/d6c/d98/fd6 [0,4194304] 0 2024-03-14T13:02:30.188 INFO:tasks.workunit.client.1.smithi137.stdout:2/976: unlink d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/f73 0 2024-03-14T13:02:30.189 INFO:tasks.workunit.client.0.smithi111.stdout:2/968: symlink dd/d17/d34/l141 0 2024-03-14T13:02:30.190 INFO:tasks.workunit.client.1.smithi137.stdout:4/957: creat d4/d27/d5c/d6d/d58/d65/f137 x:0 0 0 2024-03-14T13:02:30.190 INFO:tasks.workunit.client.1.smithi137.stdout:8/988: truncate d1/dc/f4b 1118730 0 2024-03-14T13:02:30.191 INFO:tasks.workunit.client.1.smithi137.stdout:5/859: getdents d7/d16/d18/d25/d28 0 2024-03-14T13:02:30.193 INFO:tasks.workunit.client.1.smithi137.stdout:6/927: symlink d2/d54/d55/d6d/d76/d132/de/d45/d59/d102/d134/l13c 0 2024-03-14T13:02:30.193 INFO:tasks.workunit.client.1.smithi137.stdout:5/860: truncate d7/fa2 80587 0 2024-03-14T13:02:30.195 INFO:tasks.workunit.client.0.smithi111.stdout:2/969: unlink dd/l4a 0 2024-03-14T13:02:30.196 INFO:tasks.workunit.client.0.smithi111.stdout:6/829: dwrite d0/d17/d21/d42/f77 [4194304,4194304] 0 2024-03-14T13:02:30.197 INFO:tasks.workunit.client.1.smithi137.stdout:4/958: dread - d4/d9/d3a/d41/d38/d55/f105 zero size 2024-03-14T13:02:30.198 INFO:tasks.workunit.client.0.smithi111.stdout:3/957: rmdir d0/d2d/d35/d50/da2/db3/da7 39 2024-03-14T13:02:30.200 INFO:tasks.workunit.client.0.smithi111.stdout:9/945: dwrite d5/d6/d2a/d5a/f114 [0,4194304] 0 2024-03-14T13:02:30.200 INFO:tasks.workunit.client.0.smithi111.stdout:9/946: stat d5/d42/d52 0 2024-03-14T13:02:30.202 INFO:tasks.workunit.client.1.smithi137.stdout:8/989: symlink d1/d5e/d11a/d82/l141 0 2024-03-14T13:02:30.202 INFO:tasks.workunit.client.1.smithi137.stdout:5/861: dread - d7/d16/d18/d25/d28/d3c/d76/ff8 zero size 2024-03-14T13:02:30.203 INFO:tasks.workunit.client.0.smithi111.stdout:7/989: truncate d0/f72 772464 0 2024-03-14T13:02:30.203 INFO:tasks.workunit.client.0.smithi111.stdout:2/970: write dd/d17/d53/da4/df5/df6/f107 [329412,45899] 0 2024-03-14T13:02:30.205 INFO:tasks.workunit.client.0.smithi111.stdout:6/830: symlink d0/d17/d99/l135 0 2024-03-14T13:02:30.209 INFO:tasks.workunit.client.0.smithi111.stdout:7/990: symlink d0/d11/da8/dd4/df6/d108/d21/d5d/d10d/l134 0 2024-03-14T13:02:30.211 INFO:tasks.workunit.client.0.smithi111.stdout:2/971: truncate dd/f10 2773876 0 2024-03-14T13:02:30.212 INFO:tasks.workunit.client.0.smithi111.stdout:3/958: link d0/d2d/d35/d50/d6c/fbf d0/d30/df1/d100/f157 0 2024-03-14T13:02:30.212 INFO:tasks.workunit.client.1.smithi137.stdout:3/772: dread d4/d19/f90 [0,4194304] 0 2024-03-14T13:02:30.213 INFO:tasks.workunit.client.0.smithi111.stdout:7/991: chown d0/d11/da8/dd4/df6/d108/lf 88743 1 2024-03-14T13:02:30.217 INFO:tasks.workunit.client.1.smithi137.stdout:3/773: stat d4/de/d16/d1c/db8/ff4 0 2024-03-14T13:02:30.217 INFO:tasks.workunit.client.1.smithi137.stdout:8/990: dwrite d1/dc/d71/db0/dca/f112 [0,4194304] 0 2024-03-14T13:02:30.217 INFO:tasks.workunit.client.0.smithi111.stdout:2/972: rmdir dd/d11 39 2024-03-14T13:02:30.217 INFO:tasks.workunit.client.1.smithi137.stdout:3/774: chown d4/de/d16/d1c/la7 7 1 2024-03-14T13:02:30.218 INFO:tasks.workunit.client.0.smithi111.stdout:2/973: write dd/d17/d53/f92 [3662936,23423] 0 2024-03-14T13:02:30.219 INFO:tasks.workunit.client.1.smithi137.stdout:5/862: mknod d7/d16/d18/d25/c11f 0 2024-03-14T13:02:30.229 INFO:tasks.workunit.client.1.smithi137.stdout:8/991: dread d1/dc/fce [0,4194304] 0 2024-03-14T13:02:30.229 INFO:tasks.workunit.client.0.smithi111.stdout:2/974: dwrite dd/d17/d53/fe1 [0,4194304] 0 2024-03-14T13:02:30.232 INFO:tasks.workunit.client.1.smithi137.stdout:3/775: unlink d4/de/d16/d1c/d5d/d4b/f5e 0 2024-03-14T13:02:30.234 INFO:tasks.workunit.client.0.smithi111.stdout:9/947: write d5/d6/d8/fd9 [1045399,5524] 0 2024-03-14T13:02:30.234 INFO:tasks.workunit.client.1.smithi137.stdout:2/977: dread d2/d7/d2e/d3e/d44/d6b/fc2 [0,4194304] 0 2024-03-14T13:02:30.235 INFO:tasks.workunit.client.0.smithi111.stdout:7/992: rename d0/d11/da8/dd4/df6/d108/d21 to d0/dc1/d135 0 2024-03-14T13:02:30.236 INFO:tasks.workunit.client.1.smithi137.stdout:3/776: truncate d4/de/d16/d27/dd3/fc4 1988899 0 2024-03-14T13:02:30.236 INFO:tasks.workunit.client.1.smithi137.stdout:6/928: creat d2/d54/d55/f13d x:0 0 0 2024-03-14T13:02:30.240 INFO:tasks.workunit.client.0.smithi111.stdout:7/993: read d0/d7e/fb [952600,16795] 0 2024-03-14T13:02:30.243 INFO:tasks.workunit.client.1.smithi137.stdout:4/959: dread d4/d9/f18 [0,4194304] 0 2024-03-14T13:02:30.243 INFO:tasks.workunit.client.0.smithi111.stdout:2/975: dwrite dd/d17/d34/de0/f117 [0,4194304] 0 2024-03-14T13:02:30.246 INFO:tasks.workunit.client.0.smithi111.stdout:6/831: dwrite d0/dfb/db8/d127/da1/fbc [0,4194304] 0 2024-03-14T13:02:30.246 INFO:tasks.workunit.client.0.smithi111.stdout:2/976: rmdir dd/d17 39 2024-03-14T13:02:30.247 INFO:tasks.workunit.client.1.smithi137.stdout:6/929: mkdir d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/d83/d13e 0 2024-03-14T13:02:30.249 INFO:tasks.workunit.client.1.smithi137.stdout:3/777: mkdir d4/de/d16/d27/df9 0 2024-03-14T13:02:30.250 INFO:tasks.workunit.client.0.smithi111.stdout:6/832: truncate d0/d17/d20/fb1 5089702 0 2024-03-14T13:02:30.251 INFO:tasks.workunit.client.1.smithi137.stdout:5/863: creat d7/d16/d18/d25/f120 x:0 0 0 2024-03-14T13:02:30.253 INFO:tasks.workunit.client.0.smithi111.stdout:3/959: dwrite d0/d2d/d35/d50/d6c/fbf [0,4194304] 0 2024-03-14T13:02:30.255 INFO:tasks.workunit.client.0.smithi111.stdout:2/977: write dd/d11/d8b/ff8 [3534986,104362] 0 2024-03-14T13:02:30.259 INFO:tasks.workunit.client.1.smithi137.stdout:6/930: getdents d2/d54/d55/d6d 0 2024-03-14T13:02:30.259 INFO:tasks.workunit.client.0.smithi111.stdout:6/833: creat d0/d17/d20/f136 x:0 0 0 2024-03-14T13:02:30.260 INFO:tasks.workunit.client.1.smithi137.stdout:4/960: dwrite d4/d10/d5a/f63 [4194304,4194304] 0 2024-03-14T13:02:30.262 INFO:tasks.workunit.client.1.smithi137.stdout:2/978: creat d2/d7/d2e/d3e/d44/d6b/f140 x:0 0 0 2024-03-14T13:02:30.263 INFO:tasks.workunit.client.0.smithi111.stdout:9/948: dread d5/d42/d71/dbd/dcd/fb4 [0,4194304] 0 2024-03-14T13:02:30.264 INFO:tasks.workunit.client.0.smithi111.stdout:2/978: fsync dd/d17/d34/d74/f11f 0 2024-03-14T13:02:30.265 INFO:tasks.workunit.client.0.smithi111.stdout:9/949: dread - d5/d42/d71/d96/dd3/d136/f117 zero size 2024-03-14T13:02:30.266 INFO:tasks.workunit.client.1.smithi137.stdout:3/778: rename d4/de/d16/d1c/d8d/dcd/cb1 to d4/de/dc7/cfa 0 2024-03-14T13:02:30.267 INFO:tasks.workunit.client.1.smithi137.stdout:4/961: dread - d4/d27/d5c/d6d/d58/d65/f137 zero size 2024-03-14T13:02:30.267 INFO:tasks.workunit.client.1.smithi137.stdout:5/864: dread d7/f96 [0,4194304] 0 2024-03-14T13:02:30.268 INFO:tasks.workunit.client.0.smithi111.stdout:9/950: write d5/d6/d2a/d2d/d8b/f8e [910227,82739] 0 2024-03-14T13:02:30.270 INFO:tasks.workunit.client.0.smithi111.stdout:6/834: dwrite d0/d17/d20/d65/d93/fe0 [0,4194304] 0 2024-03-14T13:02:30.271 INFO:tasks.workunit.client.0.smithi111.stdout:9/951: dread - d5/d6/d2a/d5a/f128 zero size 2024-03-14T13:02:30.271 INFO:tasks.workunit.client.1.smithi137.stdout:3/779: rename d4/de/d16 to d4/de/d16/d1c/d8d/dfb 22 2024-03-14T13:02:30.273 INFO:tasks.workunit.client.1.smithi137.stdout:6/931: dwrite d2/d54/d55/d6d/d76/d132/d86/d8a/ff2 [0,4194304] 0 2024-03-14T13:02:30.274 INFO:tasks.workunit.client.1.smithi137.stdout:5/865: symlink d7/d16/d18/d2d/dd1/l121 0 2024-03-14T13:02:30.276 INFO:tasks.workunit.client.0.smithi111.stdout:9/952: write d5/d6/d8/d5b/d84/db6/f134 [161130,37093] 0 2024-03-14T13:02:30.278 INFO:tasks.workunit.client.0.smithi111.stdout:9/953: stat d5/d6/d2a/f3e 0 2024-03-14T13:02:30.282 INFO:tasks.workunit.client.1.smithi137.stdout:8/992: truncate d1/d5e/dc9/dda/f134 2568860 0 2024-03-14T13:02:30.283 INFO:tasks.workunit.client.0.smithi111.stdout:3/960: truncate d0/d2d/d35/d69/d84/fb2 706557 0 2024-03-14T13:02:30.284 INFO:tasks.workunit.client.1.smithi137.stdout:5/866: truncate d7/d33/fb1 1469495 0 2024-03-14T13:02:30.286 INFO:tasks.workunit.client.1.smithi137.stdout:3/780: creat d4/ffc x:0 0 0 2024-03-14T13:02:30.286 INFO:tasks.workunit.client.0.smithi111.stdout:7/994: dwrite d0/d11/da8/dd4/df6/d108/d5/d56/f5e [0,4194304] 0 2024-03-14T13:02:30.287 INFO:tasks.workunit.client.0.smithi111.stdout:3/961: chown d0/d30/df1/d100/f157 3495911 1 2024-03-14T13:02:30.291 INFO:tasks.workunit.client.0.smithi111.stdout:3/962: write d0/f102 [616169,42295] 0 2024-03-14T13:02:30.291 INFO:tasks.workunit.client.0.smithi111.stdout:6/835: dread d0/dfb/db8/d127/da1/fbc [0,4194304] 0 2024-03-14T13:02:30.294 INFO:tasks.workunit.client.0.smithi111.stdout:9/954: dwrite d5/d6/f2f [4194304,4194304] 0 2024-03-14T13:02:30.294 INFO:tasks.workunit.client.1.smithi137.stdout:3/781: dread d4/de/d16/fbc [0,4194304] 0 2024-03-14T13:02:30.295 INFO:tasks.workunit.client.1.smithi137.stdout:5/867: mknod d7/d16/d18/d2d/dbe/c122 0 2024-03-14T13:02:30.295 INFO:tasks.workunit.client.1.smithi137.stdout:8/993: creat d1/d5e/d11a/d82/f142 x:0 0 0 2024-03-14T13:02:30.296 INFO:tasks.workunit.client.1.smithi137.stdout:6/932: getdents d2/d54/d55/d6d/d76/d132/de/d45/d97/d9f 0 2024-03-14T13:02:30.297 INFO:tasks.workunit.client.1.smithi137.stdout:2/979: mknod d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/c141 0 2024-03-14T13:02:30.297 INFO:tasks.workunit.client.0.smithi111.stdout:9/955: fdatasync d5/d6/d8/f12f 0 2024-03-14T13:02:30.301 INFO:tasks.workunit.client.0.smithi111.stdout:2/979: dwrite dd/f61 [0,4194304] 0 2024-03-14T13:02:30.304 INFO:tasks.workunit.client.0.smithi111.stdout:6/836: link d0/d17/d20/f136 d0/d17/d21/d69/f137 0 2024-03-14T13:02:30.304 INFO:tasks.workunit.client.0.smithi111.stdout:3/963: rename d0/d2d/d35/f113 to d0/d2d/d35/d4c/dad/d117/f158 0 2024-03-14T13:02:30.305 INFO:tasks.workunit.client.1.smithi137.stdout:3/782: symlink d4/lfd 0 2024-03-14T13:02:30.305 INFO:tasks.workunit.client.1.smithi137.stdout:5/868: truncate d7/d33/d65/dc2/fe4 4278764 0 2024-03-14T13:02:30.307 INFO:tasks.workunit.client.0.smithi111.stdout:7/995: dwrite d0/d11/da8/dd4/df6/d108/d5/d56/f7c [0,4194304] 0 2024-03-14T13:02:30.308 INFO:tasks.workunit.client.0.smithi111.stdout:6/837: write d0/d17/d21/d42/da5/d12a/f41 [2439215,36912] 0 2024-03-14T13:02:30.309 INFO:tasks.workunit.client.0.smithi111.stdout:3/964: readlink d0/d2d/d35/d50/d14a/l149 0 2024-03-14T13:02:30.309 INFO:tasks.workunit.client.1.smithi137.stdout:8/994: truncate d1/d5e/d11a/d82/f8d 2396995 0 2024-03-14T13:02:30.310 INFO:tasks.workunit.client.1.smithi137.stdout:6/933: rename d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/f122 to d2/d54/d55/d6d/dc3/dce/f13f 0 2024-03-14T13:02:30.312 INFO:tasks.workunit.client.1.smithi137.stdout:6/934: stat d2/d4c 0 2024-03-14T13:02:30.313 INFO:tasks.workunit.client.0.smithi111.stdout:2/980: rename dd/d17/d34/d74/dca/d128 to dd/d17/d53/da4/d10d/d142 0 2024-03-14T13:02:30.314 INFO:tasks.workunit.client.1.smithi137.stdout:6/935: chown d2/d54/d55/d6d/d76/d132 1 1 2024-03-14T13:02:30.316 INFO:tasks.workunit.client.0.smithi111.stdout:6/838: write d0/d17/d20/f136 [70615,70073] 0 2024-03-14T13:02:30.316 INFO:tasks.workunit.client.0.smithi111.stdout:3/965: dread - d0/d2d/d35/d50/d5e/f103 zero size 2024-03-14T13:02:30.316 INFO:tasks.workunit.client.1.smithi137.stdout:3/783: rename d4/de/d16/d1c/d8d/cac to d4/de/dc7/ddf/cfe 0 2024-03-14T13:02:30.316 INFO:tasks.workunit.client.0.smithi111.stdout:7/996: dread d0/d7e/f130 [0,4194304] 0 2024-03-14T13:02:30.318 INFO:tasks.workunit.client.0.smithi111.stdout:3/966: chown d0/d2d/d35/f61 174342 1 2024-03-14T13:02:30.319 INFO:tasks.workunit.client.1.smithi137.stdout:5/869: link d7/d16/d18/d25/d28/d3c/d76/f90 d7/d1d/d74/d81/f123 0 2024-03-14T13:02:30.320 INFO:tasks.workunit.client.0.smithi111.stdout:2/981: creat dd/d17/d34/d74/dca/f143 x:0 0 0 2024-03-14T13:02:30.320 INFO:tasks.workunit.client.0.smithi111.stdout:7/997: creat d0/d11/da8/dd4/df6/d108/d5/d56/d70/dd5/f136 x:0 0 0 2024-03-14T13:02:30.321 INFO:tasks.workunit.client.0.smithi111.stdout:3/967: truncate d0/d2d/d35/d50/d5e/d19/f26 4258448 0 2024-03-14T13:02:30.321 INFO:tasks.workunit.client.0.smithi111.stdout:6/839: mkdir d0/d138 0 2024-03-14T13:02:30.323 INFO:tasks.workunit.client.1.smithi137.stdout:5/870: chown d7/d16/d18/d25/d28/dc5 205 1 2024-03-14T13:02:30.323 INFO:tasks.workunit.client.1.smithi137.stdout:3/784: getdents d4/de/dc7 0 2024-03-14T13:02:30.324 INFO:tasks.workunit.client.0.smithi111.stdout:6/840: readlink d0/d17/d20/d94/lb2 0 2024-03-14T13:02:30.325 INFO:tasks.workunit.client.1.smithi137.stdout:4/962: dread d4/d9/d3a/d41/d38/d55/db6/f12b [0,4194304] 0 2024-03-14T13:02:30.325 INFO:tasks.workunit.client.1.smithi137.stdout:6/936: dwrite d2/d54/d55/d33/f64 [0,4194304] 0 2024-03-14T13:02:30.326 INFO:tasks.workunit.client.0.smithi111.stdout:3/968: symlink d0/d2d/d35/d50/da2/db3/d71/dbc/l159 0 2024-03-14T13:02:30.327 INFO:tasks.workunit.client.1.smithi137.stdout:5/871: stat d7/d16/d18/d2d/dd1/l121 0 2024-03-14T13:02:30.330 INFO:tasks.workunit.client.1.smithi137.stdout:4/963: chown d4/d9/d3a/d41/d38/d79/dc2/d129/f106 0 1 2024-03-14T13:02:30.332 INFO:tasks.workunit.client.1.smithi137.stdout:7/961: sync 2024-03-14T13:02:30.333 INFO:tasks.workunit.client.0.smithi111.stdout:3/969: dwrite d0/d2d/d35/d69/de6/df4/ffd [4194304,4194304] 0 2024-03-14T13:02:30.334 INFO:tasks.workunit.client.1.smithi137.stdout:3/785: dwrite f1 [0,4194304] 0 2024-03-14T13:02:30.334 INFO:tasks.workunit.client.0.smithi111.stdout:3/970: chown d0/d2d/d35/d50/d6c/cfb 5 1 2024-03-14T13:02:30.337 INFO:tasks.workunit.client.0.smithi111.stdout:3/971: creat d0/d2d/d35/d50/da2/db3/d67/f15a x:0 0 0 2024-03-14T13:02:30.338 INFO:tasks.workunit.client.0.smithi111.stdout:7/998: dread d0/d11/da8/dd4/df6/d108/d22/f42 [0,4194304] 0 2024-03-14T13:02:30.340 INFO:tasks.workunit.client.1.smithi137.stdout:7/962: chown d3/d6/d122/dfd/l10e 830037492 1 2024-03-14T13:02:30.340 INFO:tasks.workunit.client.1.smithi137.stdout:2/980: write d2/d7/d2e/d3e/d44/d6b/d22/f5a [3715459,119503] 0 2024-03-14T13:02:30.341 INFO:tasks.workunit.client.0.smithi111.stdout:3/972: link d0/d30/f57 d0/d2d/d35/d50/da2/db3/d6f/f15b 0 2024-03-14T13:02:30.342 INFO:tasks.workunit.client.1.smithi137.stdout:8/995: rmdir d1/d5e/d11a 39 2024-03-14T13:02:30.354 INFO:tasks.workunit.client.1.smithi137.stdout:3/786: dwrite d4/df/fbe [0,4194304] 0 2024-03-14T13:02:30.354 INFO:tasks.workunit.client.1.smithi137.stdout:2/981: symlink d2/d7/d2e/d3e/daa/de7/l142 0 2024-03-14T13:02:30.355 INFO:tasks.workunit.client.1.smithi137.stdout:7/963: chown d3/d6/d59/daa/dcf/d6a/d75/c10a 24613 1 2024-03-14T13:02:30.359 INFO:tasks.workunit.client.1.smithi137.stdout:7/964: write d3/d6/d59/daa/dcf/d6a/d75/fcb [847129,99894] 0 2024-03-14T13:02:30.359 INFO:tasks.workunit.client.1.smithi137.stdout:4/964: dread d4/d9/d3a/d41/d38/d55/db6/f12b [0,4194304] 0 2024-03-14T13:02:30.359 INFO:tasks.workunit.client.1.smithi137.stdout:8/996: symlink d1/d4a/dc4/l143 0 2024-03-14T13:02:30.361 INFO:tasks.workunit.client.1.smithi137.stdout:2/982: creat d2/d11e/f143 x:0 0 0 2024-03-14T13:02:30.362 INFO:tasks.workunit.client.1.smithi137.stdout:7/965: mkdir d3/d5/d5a/d143 0 2024-03-14T13:02:30.362 INFO:tasks.workunit.client.1.smithi137.stdout:6/937: write d2/d54/d55/d6d/d76/d132/d1d/d88/fa2 [420727,105027] 0 2024-03-14T13:02:30.366 INFO:tasks.workunit.client.1.smithi137.stdout:7/966: chown d3/d6/d2e/f8b 36270125 1 2024-03-14T13:02:30.367 INFO:tasks.workunit.client.1.smithi137.stdout:2/983: symlink d2/d7/d2e/d3e/d4e/l144 0 2024-03-14T13:02:30.371 INFO:tasks.workunit.client.0.smithi111.stdout:2/982: dread dd/fe4 [0,4194304] 0 2024-03-14T13:02:30.371 INFO:tasks.workunit.client.1.smithi137.stdout:2/984: mkdir d2/d7/d2e/d3e/daa/de7/d145 0 2024-03-14T13:02:30.371 INFO:tasks.workunit.client.0.smithi111.stdout:2/983: chown dd/l39 3273 1 2024-03-14T13:02:30.372 INFO:tasks.workunit.client.1.smithi137.stdout:7/967: mkdir d3/d52/d144 0 2024-03-14T13:02:30.373 INFO:tasks.workunit.client.1.smithi137.stdout:5/872: dread d7/d16/d18/d2d/f8e [0,4194304] 0 2024-03-14T13:02:30.373 INFO:tasks.workunit.client.1.smithi137.stdout:7/968: fdatasync d3/d5/f116 0 2024-03-14T13:02:30.375 INFO:tasks.workunit.client.1.smithi137.stdout:5/873: dread - d7/d16/d18/d25/d28/d3c/d76/f79 zero size 2024-03-14T13:02:30.376 INFO:tasks.workunit.client.1.smithi137.stdout:2/985: symlink d2/d7/df8/l146 0 2024-03-14T13:02:30.378 INFO:tasks.workunit.client.1.smithi137.stdout:5/874: write d7/d49/d75/fb2 [4758663,103388] 0 2024-03-14T13:02:30.379 INFO:tasks.workunit.client.1.smithi137.stdout:7/969: mkdir d3/d6/d59/daa/dcf/d6a/da1/d13d/d145 0 2024-03-14T13:02:30.380 INFO:tasks.workunit.client.1.smithi137.stdout:6/938: link d2/d54/d55/d6d/d76/d132/d1d/d88/ce4 d2/d54/d55/d6d/dc3/deb/c140 0 2024-03-14T13:02:30.383 INFO:tasks.workunit.client.0.smithi111.stdout:9/956: truncate d5/d42/d71/dbd/dcd/f10c 24740 0 2024-03-14T13:02:30.383 INFO:tasks.workunit.client.1.smithi137.stdout:6/939: readlink d2/d54/d55/d6d/d76/d132/de/d45/d97/d9f/l95 0 2024-03-14T13:02:30.384 INFO:tasks.workunit.client.0.smithi111.stdout:9/957: chown d5/d6/d2a/d2d/d8b 0 1 2024-03-14T13:02:30.384 INFO:tasks.workunit.client.1.smithi137.stdout:5/875: creat d7/d1d/d106/f124 x:0 0 0 2024-03-14T13:02:30.385 INFO:tasks.workunit.client.0.smithi111.stdout:3/973: dread d0/d2d/d35/d69/fc4 [0,4194304] 0 2024-03-14T13:02:30.385 INFO:tasks.workunit.client.0.smithi111.stdout:6/841: dread d0/d17/d20/d4a/d59/f7e [0,4194304] 0 2024-03-14T13:02:30.385 INFO:tasks.workunit.client.1.smithi137.stdout:5/876: chown d7/d16/d18/d2d/f8f 1 1 2024-03-14T13:02:30.387 INFO:tasks.workunit.client.0.smithi111.stdout:6/842: chown d0/d17/d21/d42/da5/d12a/f5c 163534 1 2024-03-14T13:02:30.387 INFO:tasks.workunit.client.1.smithi137.stdout:6/940: unlink d2/d54/d55/f13d 0 2024-03-14T13:02:30.388 INFO:tasks.workunit.client.1.smithi137.stdout:3/787: dread d4/de/d16/d1c/d8d/dcd/fba [0,4194304] 0 2024-03-14T13:02:30.389 INFO:tasks.workunit.client.0.smithi111.stdout:3/974: mkdir d0/d30/df1/d100/d15c 0 2024-03-14T13:02:30.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:30 smithi111 ceph-mon[39688]: pgmap v29: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 158 MiB/s rd, 411 MiB/s wr, 838 op/s 2024-03-14T13:02:30.390 INFO:tasks.workunit.client.1.smithi137.stdout:2/986: rename d2/d7/d2e/d3e/d44/d6b/d1d/d3a/f93 to d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b/d3c/d9d/d109/f147 0 2024-03-14T13:02:30.392 INFO:tasks.workunit.client.1.smithi137.stdout:7/970: creat d3/d5/d5a/d80/f146 x:0 0 0 2024-03-14T13:02:30.397 INFO:tasks.workunit.client.1.smithi137.stdout:2/987: mknod d2/d7/d2e/d3e/d44/d6b/d22/c148 0 2024-03-14T13:02:30.398 INFO:tasks.workunit.client.1.smithi137.stdout:6/941: dread d2/d54/d55/d6d/dc3/dce/f13f [0,4194304] 0 2024-03-14T13:02:30.398 INFO:tasks.workunit.client.1.smithi137.stdout:4/965: dwrite d4/f48 [0,4194304] 0 2024-03-14T13:02:30.400 INFO:tasks.workunit.client.1.smithi137.stdout:7/971: mknod d3/d6/d64/c147 0 2024-03-14T13:02:30.400 INFO:tasks.workunit.client.1.smithi137.stdout:5/877: symlink d7/d16/d18/d25/d28/d3c/dfb/l125 0 2024-03-14T13:02:30.404 INFO:tasks.workunit.client.1.smithi137.stdout:2/988: write d2/d7/d2e/d3e/d44/d6b/f134 [942446,101689] 0 2024-03-14T13:02:30.406 INFO:tasks.workunit.client.1.smithi137.stdout:2/989: chown d2/d7/d2e/d3e/d44/fab 14 1 2024-03-14T13:02:30.408 INFO:tasks.workunit.client.1.smithi137.stdout:7/972: symlink d3/d6/d64/d9e/d127/l148 0 2024-03-14T13:02:30.409 INFO:tasks.workunit.client.1.smithi137.stdout:3/788: link d4/lad d4/de/d16/d1c/d5d/d85/d6b/lff 0 2024-03-14T13:02:30.410 INFO:tasks.workunit.client.1.smithi137.stdout:3/789: fsync d4/de/d13/f36 0 2024-03-14T13:02:30.410 INFO:tasks.workunit.client.1.smithi137.stdout:5/878: mkdir d7/d1d/d74/d83/d126 0 2024-03-14T13:02:30.414 INFO:tasks.workunit.client.1.smithi137.stdout:2/990: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f149 x:0 0 0 2024-03-14T13:02:30.416 INFO:tasks.workunit.client.1.smithi137.stdout:7/973: dwrite d3/d5/d5a/fc6 [0,4194304] 0 2024-03-14T13:02:30.417 INFO:tasks.workunit.client.1.smithi137.stdout:6/942: write d2/d54/d55/d6d/d76/d132/d86/d8a/fbb [1420726,111367] 0 2024-03-14T13:02:30.417 INFO:tasks.workunit.client.1.smithi137.stdout:3/790: symlink d4/de/d93/l100 0 2024-03-14T13:02:30.425 INFO:tasks.workunit.client.1.smithi137.stdout:3/791: dread d4/de/d16/d27/dd3/fab [0,4194304] 0 2024-03-14T13:02:30.427 INFO:tasks.workunit.client.1.smithi137.stdout:7/974: dwrite f1 [0,4194304] 0 2024-03-14T13:02:30.427 INFO:tasks.workunit.client.1.smithi137.stdout:4/966: write d4/d9/d3a/d70/fdf [3028936,122497] 0 2024-03-14T13:02:30.432 INFO:tasks.workunit.client.1.smithi137.stdout:2/991: getdents d2/d7/d2e/d3e/d44/d6b/d22/d4d/d10b 0 2024-03-14T13:02:30.433 INFO:tasks.workunit.client.1.smithi137.stdout:6/943: getdents d2/d54/d55/d6d/d76/d132/de/d45/d59 0 2024-03-14T13:02:30.433 INFO:tasks.workunit.client.1.smithi137.stdout:4/967: write d4/d27/d5c/d6d/f10e [364687,96373] 0 2024-03-14T13:02:30.436 INFO:tasks.workunit.client.1.smithi137.stdout:3/792: rename d4/d19/l1e to d4/de/d16/d27/dd3/l101 0 2024-03-14T13:02:30.436 INFO:tasks.workunit.client.1.smithi137.stdout:7/975: rmdir d3/d5/d23/d3f/d5b/d111 39 2024-03-14T13:02:30.438 INFO:tasks.workunit.client.1.smithi137.stdout:3/793: write d4/df/f96 [2968098,40576] 0 2024-03-14T13:02:30.440 INFO:tasks.workunit.client.1.smithi137.stdout:4/968: unlink d4/d9/d3a/d41/d38/d79/dc2/d11f/l111 0 2024-03-14T13:02:30.447 INFO:tasks.workunit.client.1.smithi137.stdout:7/976: creat d3/d5/d5a/d11b/f149 x:0 0 0 2024-03-14T13:02:30.449 INFO:tasks.workunit.client.1.smithi137.stdout:6/944: dwrite d2/d54/d55/d6d/d76/d132/de/d45/d59/f5e [0,4194304] 0 2024-03-14T13:02:30.449 INFO:tasks.workunit.client.1.smithi137.stdout:4/969: mknod d4/d9/d3a/d41/d38/d79/dc2/d11f/c138 0 2024-03-14T13:02:30.451 INFO:tasks.workunit.client.1.smithi137.stdout:5/879: dread d7/d16/d18/d25/d28/d3c/d7a/fc0 [0,4194304] 0 2024-03-14T13:02:30.452 INFO:tasks.workunit.client.1.smithi137.stdout:3/794: symlink d4/de/d16/d1c/l102 0 2024-03-14T13:02:30.454 INFO:tasks.workunit.client.1.smithi137.stdout:7/977: mkdir d3/d6/d122/dfd/d14a 0 2024-03-14T13:02:30.457 INFO:tasks.workunit.client.1.smithi137.stdout:6/945: creat d2/d54/d55/d6d/d76/d132/d1d/d88/dcd/f141 x:0 0 0 2024-03-14T13:02:30.461 INFO:tasks.workunit.client.1.smithi137.stdout:5/880: link d7/d16/d18/d25/d28/d3c/d7a/fc0 d7/d49/f127 0 2024-03-14T13:02:30.463 INFO:tasks.workunit.client.1.smithi137.stdout:6/946: rmdir d2/d54/d55/d6d/d76/d132/de/d45/d59/d102 39 2024-03-14T13:02:30.466 INFO:tasks.workunit.client.1.smithi137.stdout:2/992: dwrite d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d9f/fd6 [0,4194304] 0 2024-03-14T13:02:30.468 INFO:tasks.workunit.client.1.smithi137.stdout:7/978: dwrite d3/d6/d14/f2f [0,4194304] 0 2024-03-14T13:02:30.469 INFO:tasks.workunit.client.1.smithi137.stdout:5/881: rmdir d7/d16/d18/d48/d103 0 2024-03-14T13:02:30.478 INFO:tasks.workunit.client.1.smithi137.stdout:2/993: chown d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d74/d84/dcc/d10f 209616251 1 2024-03-14T13:02:30.479 INFO:tasks.workunit.client.1.smithi137.stdout:7/979: dwrite d3/d54/db4/f121 [0,4194304] 0 2024-03-14T13:02:30.479 INFO:tasks.workunit.client.0.smithi111.stdout:6/843: write d0/dfb/db8/d127/da1/fbc [79611,31188] 0 2024-03-14T13:02:30.482 INFO:tasks.workunit.client.0.smithi111.stdout:9/958: dwrite d5/d6/d8/d5b/f11d [4194304,4194304] 0 2024-03-14T13:02:30.483 INFO:tasks.workunit.client.0.smithi111.stdout:9/959: chown d5/d6/d8/d5b/f11d 101 1 2024-03-14T13:02:30.484 INFO:tasks.workunit.client.1.smithi137.stdout:5/882: rename d7/d16/d18/d25/d28/d3c/d76/f90 to d7/d16/d18/d25/d28/d3c/dd8/d10c/f128 0 2024-03-14T13:02:30.485 INFO:tasks.workunit.client.1.smithi137.stdout:4/970: dwrite d4/d9/d3a/d41/d38/d55/db6/fc6 [4194304,4194304] 0 2024-03-14T13:02:30.486 INFO:tasks.workunit.client.1.smithi137.stdout:7/980: fsync d3/d5/f19 0 2024-03-14T13:02:30.487 INFO:tasks.workunit.client.1.smithi137.stdout:2/994: symlink d2/d11e/l14a 0 2024-03-14T13:02:30.489 INFO:tasks.workunit.client.1.smithi137.stdout:6/947: rmdir d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c 39 2024-03-14T13:02:30.494 INFO:tasks.workunit.client.1.smithi137.stdout:2/995: chown d2/d7/d2e/d3e/d44/d6b/d22/da5/db4/ff2 2 1 2024-03-14T13:02:30.494 INFO:tasks.workunit.client.1.smithi137.stdout:3/795: dwrite d4/de/d16/d1c/d8d/dcd/fba [0,4194304] 0 2024-03-14T13:02:30.494 INFO:tasks.workunit.client.0.smithi111.stdout:6/844: rename d0/d17/d21/d69/c43 to d0/d17/d20/c139 0 2024-03-14T13:02:30.496 INFO:tasks.workunit.client.1.smithi137.stdout:5/883: rename d7/d16/d18/d25/c11f to d7/d1d/d74/df7/c129 0 2024-03-14T13:02:30.500 INFO:tasks.workunit.client.1.smithi137.stdout:4/971: mkdir d4/d9/d8e/d95/d139 0 2024-03-14T13:02:30.502 INFO:tasks.workunit.client.1.smithi137.stdout:3/796: truncate d4/de/dc7/ff3 85054 0 2024-03-14T13:02:30.502 INFO:tasks.workunit.client.1.smithi137.stdout:5/884: fdatasync d7/d36/f95 0 2024-03-14T13:02:30.504 INFO:tasks.workunit.client.1.smithi137.stdout:2/996: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d64/f14b x:0 0 0 2024-03-14T13:02:30.506 INFO:tasks.workunit.client.1.smithi137.stdout:4/972: write d4/d27/d5c/d6d/d58/f76 [5280354,105635] 0 2024-03-14T13:02:30.506 INFO:tasks.workunit.client.1.smithi137.stdout:5/885: chown d7/d16/d18/d2d/f53 6389948 1 2024-03-14T13:02:30.507 INFO:tasks.workunit.client.1.smithi137.stdout:6/948: creat d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/f142 x:0 0 0 2024-03-14T13:02:30.509 INFO:tasks.workunit.client.1.smithi137.stdout:7/981: dwrite d3/d6/d59/daa/dcf/f6b [0,4194304] 0 2024-03-14T13:02:30.509 INFO:tasks.workunit.client.1.smithi137.stdout:3/797: creat d4/de/d16/d1c/d5d/d4b/de6/f103 x:0 0 0 2024-03-14T13:02:30.514 INFO:tasks.workunit.client.1.smithi137.stdout:7/982: readlink d3/d6/d14/d22/l41 0 2024-03-14T13:02:30.516 INFO:tasks.workunit.client.1.smithi137.stdout:2/997: truncate d2/d7/d2e/d3e/d44/d6b/d22/d26/f11c 734549 0 2024-03-14T13:02:30.518 INFO:tasks.workunit.client.1.smithi137.stdout:3/798: chown d4/de/d16/d1c/d5d/d85/l3b 14814 1 2024-03-14T13:02:30.520 INFO:tasks.workunit.client.1.smithi137.stdout:7/983: mknod d3/d5/d23/d2c/d126/c14b 0 2024-03-14T13:02:30.523 INFO:tasks.workunit.client.1.smithi137.stdout:6/949: dwrite d2/d54/fff [0,4194304] 0 2024-03-14T13:02:30.523 INFO:tasks.workunit.client.1.smithi137.stdout:7/984: symlink d3/d54/db4/l14c 0 2024-03-14T13:02:30.527 INFO:tasks.workunit.client.1.smithi137.stdout:2/998: creat d2/d7/d2e/d3e/d44/d6b/d22/d4d/d57/d8d/dbc/d2f/d66/d10c/f14c x:0 0 0 2024-03-14T13:02:30.528 INFO:tasks.workunit.client.0.smithi111.stdout:3/975: dread d0/d2d/d35/d50/d5e/f12 [0,4194304] 0 2024-03-14T13:02:30.531 INFO:tasks.workunit.client.0.smithi111.stdout:3/976: mknod d0/d2d/d35/d50/d6c/da3/c15d 0 2024-03-14T13:02:30.531 INFO:tasks.workunit.client.1.smithi137.stdout:7/985: rename d3/d6/d59/daa/dcf/d6a/da1/f123 to d3/d6/d64/d97/dd2/f14d 0 2024-03-14T13:02:30.531 INFO:tasks.workunit.client.1.smithi137.stdout:2/999: dread d2/d7/d2e/d3e/d44/d6b/d1d/d2d/d8b/d99/f12b [0,4194304] 0 2024-03-14T13:02:30.532 INFO:tasks.workunit.client.1.smithi137.stdout:6/950: mknod d2/d54/d55/d6d/dc3/dce/c143 0 2024-03-14T13:02:30.532 INFO:tasks.workunit.client.0.smithi111.stdout:9/960: dread d5/d6/d14/d7a/fba [0,4194304] 0 2024-03-14T13:02:30.534 INFO:tasks.workunit.client.1.smithi137.stdout:6/951: rmdir d2/d54/d55/d6d/d76/d132/d86 39 2024-03-14T13:02:30.539 INFO:tasks.workunit.client.1.smithi137.stdout:6/952: dwrite d2/f5a [4194304,4194304] 0 2024-03-14T13:02:30.541 INFO:tasks.workunit.client.1.smithi137.stdout:6/953: write d2/d54/d55/d9c/dc0/dc4/f11c [1001638,88394] 0 2024-03-14T13:02:30.541 INFO:tasks.workunit.client.1.smithi137.stdout:6/954: readlink d2/d54/d55/d33/l104 0 2024-03-14T13:02:30.545 INFO:tasks.workunit.client.1.smithi137.stdout:6/955: creat d2/d54/d55/d6d/d76/d132/de/d45/d59/d102/d134/f144 x:0 0 0 2024-03-14T13:02:30.550 INFO:tasks.workunit.client.1.smithi137.stdout:6/956: dwrite d2/d54/d55/d6d/dc3/dce/f11e [0,4194304] 0 2024-03-14T13:02:30.551 INFO:tasks.workunit.client.1.smithi137.stdout:6/957: chown d2/d4c/f69 22236 1 2024-03-14T13:02:30.556 INFO:tasks.workunit.client.1.smithi137.stdout:6/958: dread d2/d54/d55/d6d/d76/d57/d58/fa5 [0,4194304] 0 2024-03-14T13:02:30.558 INFO:tasks.workunit.client.1.smithi137.stdout:6/959: rmdir d2/d54/d55/d6d/d76/d132/d1d/d88/df4 39 2024-03-14T13:02:30.564 INFO:tasks.workunit.client.1.smithi137.stdout:6/960: dwrite d2/d54/d55/d6d/d76/d132/d1d/d88/dcd/f141 [0,4194304] 0 2024-03-14T13:02:30.566 INFO:tasks.workunit.client.1.smithi137.stdout:6/961: creat d2/d54/d55/d38/f145 x:0 0 0 2024-03-14T13:02:30.575 INFO:tasks.workunit.client.1.smithi137.stdout:6/962: rename d2/f11f to d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/dfe/d135/f146 0 2024-03-14T13:02:30.576 INFO:tasks.workunit.client.1.smithi137.stdout:6/963: readlink d2/d54/d55/d6d/d76/d57/ld5 0 2024-03-14T13:02:30.581 INFO:tasks.workunit.client.0.smithi111.stdout:6/845: dwrite d0/d17/d21/f22 [0,4194304] 0 2024-03-14T13:02:30.583 INFO:tasks.workunit.client.1.smithi137.stdout:4/973: write d4/d9/d3a/d41/d38/d79/dbe/fe7 [458239,88247] 0 2024-03-14T13:02:30.583 INFO:tasks.workunit.client.1.smithi137.stdout:6/964: dwrite d2/d54/d55/d33/f64 [0,4194304] 0 2024-03-14T13:02:30.590 INFO:tasks.workunit.client.1.smithi137.stdout:6/965: mkdir d2/d54/d55/d6d/d76/d57/d58/d147 0 2024-03-14T13:02:30.593 INFO:tasks.workunit.client.1.smithi137.stdout:6/966: write d2/d54/d55/d6d/dc3/dec/ffc [842443,86415] 0 2024-03-14T13:02:30.593 INFO:tasks.workunit.client.1.smithi137.stdout:4/974: getdents d4/d27/d5c/d6d/d58/d65 0 2024-03-14T13:02:30.595 INFO:tasks.workunit.client.1.smithi137.stdout:6/967: stat d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/dfe/d135/f146 0 2024-03-14T13:02:30.596 INFO:tasks.workunit.client.1.smithi137.stdout:4/975: symlink d4/d9/d3a/d41/d38/d79/dc2/l13a 0 2024-03-14T13:02:30.597 INFO:tasks.workunit.client.1.smithi137.stdout:4/976: fsync d4/d27/d100/db7/fde 0 2024-03-14T13:02:30.602 INFO:tasks.workunit.client.0.smithi111.stdout:7/999: sync 2024-03-14T13:02:30.603 INFO:tasks.workunit.client.0.smithi111.stdout:9/961: truncate d5/d6/d8/f3a 4751228 0 2024-03-14T13:02:30.604 INFO:tasks.workunit.client.0.smithi111.stdout:6/846: write d0/d17/d21/d69/f45 [1838264,59400] 0 2024-03-14T13:02:30.606 INFO:tasks.workunit.client.0.smithi111.stdout:9/962: symlink d5/d42/d71/l146 0 2024-03-14T13:02:30.606 INFO:tasks.workunit.client.0.smithi111.stdout:3/977: dwrite d0/d2d/d35/fea [0,4194304] 0 2024-03-14T13:02:30.608 INFO:tasks.workunit.client.1.smithi137.stdout:6/968: truncate d2/d54/d55/d6d/d76/d57/f81 1340436 0 2024-03-14T13:02:30.609 INFO:tasks.workunit.client.1.smithi137.stdout:4/977: truncate d4/d9/d3a/d41/d6a/dff/f11a 396009 0 2024-03-14T13:02:30.610 INFO:tasks.workunit.client.0.smithi111.stdout:6/847: creat d0/d17/f13a x:0 0 0 2024-03-14T13:02:30.610 INFO:tasks.workunit.client.0.smithi111.stdout:3/978: dread d0/d2d/d35/d50/da2/feb [0,4194304] 0 2024-03-14T13:02:30.613 INFO:tasks.workunit.client.1.smithi137.stdout:4/978: getdents d4/d4d 0 2024-03-14T13:02:30.614 INFO:tasks.workunit.client.0.smithi111.stdout:9/963: creat d5/d6/d8/d5b/f147 x:0 0 0 2024-03-14T13:02:30.614 INFO:tasks.workunit.client.1.smithi137.stdout:4/979: chown d4/d9/d3a/d41/dd7 146628 1 2024-03-14T13:02:30.615 INFO:tasks.workunit.client.0.smithi111.stdout:3/979: symlink d0/l15e 0 2024-03-14T13:02:30.615 INFO:tasks.workunit.client.0.smithi111.stdout:9/964: write d5/d42/d71/dbd/dcd/f88 [1004483,82399] 0 2024-03-14T13:02:30.617 INFO:tasks.workunit.client.1.smithi137.stdout:6/969: dwrite d2/d54/d55/d6d/d76/d132/f34 [0,4194304] 0 2024-03-14T13:02:30.617 INFO:tasks.workunit.client.0.smithi111.stdout:6/848: rmdir d0/d17/d21/d42/da5/d12a/d10b/d110 0 2024-03-14T13:02:30.620 INFO:tasks.workunit.client.0.smithi111.stdout:9/965: creat d5/d42/d71/ddc/f148 x:0 0 0 2024-03-14T13:02:30.620 INFO:tasks.workunit.client.1.smithi137.stdout:6/970: stat d2/d54/d55/d6d/d76/d132/d1d/d88/df4 0 2024-03-14T13:02:30.620 INFO:tasks.workunit.client.1.smithi137.stdout:4/980: link d4/d9/d3a/d41/d6a/fb3 d4/d10/d5a/f13b 0 2024-03-14T13:02:30.621 INFO:tasks.workunit.client.1.smithi137.stdout:4/981: chown d4/d27/d5c/d77/fbd 367215 1 2024-03-14T13:02:30.621 INFO:tasks.workunit.client.0.smithi111.stdout:6/849: creat d0/dfb/db8/d127/f13b x:0 0 0 2024-03-14T13:02:30.622 INFO:tasks.workunit.client.1.smithi137.stdout:6/971: rmdir d2/d54/d55/d9c/dc0/dc4 39 2024-03-14T13:02:30.623 INFO:tasks.workunit.client.0.smithi111.stdout:9/966: chown d5/d6/d2a/l53 9150 1 2024-03-14T13:02:30.624 INFO:tasks.workunit.client.1.smithi137.stdout:4/982: mkdir d4/d9/d3a/d41/d38/d79/dc2/d129/d13c 0 2024-03-14T13:02:30.626 INFO:tasks.workunit.client.0.smithi111.stdout:3/980: dwrite d0/f1c [0,4194304] 0 2024-03-14T13:02:30.626 INFO:tasks.workunit.client.1.smithi137.stdout:6/972: creat d2/d54/d55/d6d/d76/d132/de/f148 x:0 0 0 2024-03-14T13:02:30.627 INFO:tasks.workunit.client.0.smithi111.stdout:6/850: dread d0/d17/d21/d69/f76 [0,4194304] 0 2024-03-14T13:02:30.628 INFO:tasks.workunit.client.0.smithi111.stdout:6/851: stat d0/d17/l4e 0 2024-03-14T13:02:30.628 INFO:tasks.workunit.client.0.smithi111.stdout:9/967: dread - d5/d42/d71/dbd/fc8 zero size 2024-03-14T13:02:30.630 INFO:tasks.workunit.client.1.smithi137.stdout:6/973: stat d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/dfe/d135 0 2024-03-14T13:02:30.630 INFO:tasks.workunit.client.0.smithi111.stdout:3/981: write d0/d2d/d35/d50/da2/feb [67637,120631] 0 2024-03-14T13:02:30.633 INFO:tasks.workunit.client.0.smithi111.stdout:3/982: write d0/d2d/d35/d69/f94 [595742,44197] 0 2024-03-14T13:02:30.637 INFO:tasks.workunit.client.0.smithi111.stdout:6/852: dread d0/d17/d20/d65/d93/fe0 [0,4194304] 0 2024-03-14T13:02:30.637 INFO:tasks.workunit.client.1.smithi137.stdout:4/983: dwrite d4/d9/d3a/d41/fe8 [0,4194304] 0 2024-03-14T13:02:30.638 INFO:tasks.workunit.client.1.smithi137.stdout:6/974: symlink d2/d54/d55/d6d/d76/d132/l149 0 2024-03-14T13:02:30.641 INFO:tasks.workunit.client.0.smithi111.stdout:9/968: dwrite d5/d42/d71/dbd/d123/f139 [0,4194304] 0 2024-03-14T13:02:30.643 INFO:tasks.workunit.client.1.smithi137.stdout:4/984: mknod d4/d27/d100/c13d 0 2024-03-14T13:02:30.647 INFO:tasks.workunit.client.0.smithi111.stdout:3/983: dwrite d0/d2d/d35/d50/da2/feb [0,4194304] 0 2024-03-14T13:02:30.647 INFO:tasks.workunit.client.0.smithi111.stdout:9/969: getdents d5/d6/d8/d1a 0 2024-03-14T13:02:30.649 INFO:tasks.workunit.client.1.smithi137.stdout:6/975: unlink d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/dfe/d135/f146 0 2024-03-14T13:02:30.651 INFO:tasks.workunit.client.1.smithi137.stdout:4/985: fsync d4/d9/d3a/d41/d6a/dac/fb0 0 2024-03-14T13:02:30.651 INFO:tasks.workunit.client.0.smithi111.stdout:6/853: dwrite d0/d17/d21/d69/f45 [0,4194304] 0 2024-03-14T13:02:30.653 INFO:tasks.workunit.client.0.smithi111.stdout:2/984: sync 2024-03-14T13:02:30.653 INFO:tasks.workunit.client.0.smithi111.stdout:6/854: write d0/d17/d20/d4a/d59/d82/f102 [2262063,5709] 0 2024-03-14T13:02:30.653 INFO:tasks.workunit.client.0.smithi111.stdout:3/984: rmdir d0/d2d/d35/d50/d6c/da3/daa 39 2024-03-14T13:02:30.654 INFO:tasks.workunit.client.1.smithi137.stdout:4/986: creat d4/d9/d3a/d41/dd7/f13e x:0 0 0 2024-03-14T13:02:30.657 INFO:tasks.workunit.client.0.smithi111.stdout:6/855: chown d0/d17/d20/d4a/d68/d10e/l129 250936915 1 2024-03-14T13:02:30.659 INFO:tasks.workunit.client.0.smithi111.stdout:2/985: write dd/d11/d1a/d56/dab/fee [658912,65178] 0 2024-03-14T13:02:30.660 INFO:tasks.workunit.client.0.smithi111.stdout:6/856: mknod d0/dfb/db8/c13c 0 2024-03-14T13:02:30.663 INFO:tasks.workunit.client.0.smithi111.stdout:6/857: stat d0/d17/d20/d4a/d59/f7e 0 2024-03-14T13:02:30.666 INFO:tasks.workunit.client.0.smithi111.stdout:2/986: dwrite dd/d8d/dc2/fc7 [4194304,4194304] 0 2024-03-14T13:02:30.677 INFO:tasks.workunit.client.0.smithi111.stdout:6/858: dread d0/d17/d20/d4a/d59/fbb [0,4194304] 0 2024-03-14T13:02:30.680 INFO:tasks.workunit.client.0.smithi111.stdout:6/859: creat d0/d17/d99/f13d x:0 0 0 2024-03-14T13:02:30.683 INFO:tasks.workunit.client.0.smithi111.stdout:6/860: creat d0/dfb/d106/d114/f13e x:0 0 0 2024-03-14T13:02:30.687 INFO:tasks.workunit.client.1.smithi137.stdout:4/987: dread d4/d9/d3a/d41/d38/d79/dc2/d129/fd4 [0,4194304] 0 2024-03-14T13:02:30.687 INFO:tasks.workunit.client.0.smithi111.stdout:6/861: read d0/d17/d20/d94/f121 [1541623,82615] 0 2024-03-14T13:02:30.691 INFO:tasks.workunit.client.0.smithi111.stdout:6/862: creat d0/d17/d20/d94/dbd/f13f x:0 0 0 2024-03-14T13:02:30.691 INFO:tasks.workunit.client.0.smithi111.stdout:3/985: write d0/d2d/d35/d69/d84/fb2 [25745,63397] 0 2024-03-14T13:02:30.692 INFO:tasks.workunit.client.0.smithi111.stdout:6/863: readlink d0/l9 0 2024-03-14T13:02:30.692 INFO:tasks.workunit.client.0.smithi111.stdout:2/987: write dd/d17/d34/f6b [4662768,22156] 0 2024-03-14T13:02:30.693 INFO:tasks.workunit.client.0.smithi111.stdout:9/970: dread d5/d6/d8/d74/fdd [0,4194304] 0 2024-03-14T13:02:30.694 INFO:tasks.workunit.client.0.smithi111.stdout:3/986: dread - d0/d2d/d35/d50/da2/fd9 zero size 2024-03-14T13:02:30.694 INFO:tasks.workunit.client.0.smithi111.stdout:2/988: rmdir dd/d17/d34/ddb 39 2024-03-14T13:02:30.697 INFO:tasks.workunit.client.0.smithi111.stdout:2/989: write dd/d17/d34/f6b [5297846,73219] 0 2024-03-14T13:02:30.698 INFO:tasks.workunit.client.0.smithi111.stdout:3/987: write d0/d2d/d35/d50/da2/db3/da7/fe7 [3019895,85483] 0 2024-03-14T13:02:30.699 INFO:tasks.workunit.client.0.smithi111.stdout:9/971: truncate d5/d6/d8/f3a 1535071 0 2024-03-14T13:02:30.700 INFO:tasks.workunit.client.0.smithi111.stdout:6/864: getdents d0/d17/d20/d4a/d59/dc3 0 2024-03-14T13:02:30.702 INFO:tasks.workunit.client.0.smithi111.stdout:2/990: creat dd/d17/d34/d74/dc1/f144 x:0 0 0 2024-03-14T13:02:30.704 INFO:tasks.workunit.client.1.smithi137.stdout:4/988: dread d4/f3c [0,4194304] 0 2024-03-14T13:02:30.704 INFO:tasks.workunit.client.0.smithi111.stdout:2/991: write dd/d17/d53/f92 [8744965,123434] 0 2024-03-14T13:02:30.705 INFO:tasks.workunit.client.0.smithi111.stdout:9/972: creat d5/d6/d8/d5b/d84/f149 x:0 0 0 2024-03-14T13:02:30.706 INFO:tasks.workunit.client.1.smithi137.stdout:4/989: getdents d4/d9/d3a/d41/d6a/dff 0 2024-03-14T13:02:30.708 INFO:tasks.workunit.client.1.smithi137.stdout:4/990: chown d4/d9/d3a/d41/f49 428 1 2024-03-14T13:02:30.709 INFO:tasks.workunit.client.1.smithi137.stdout:8/997: sync 2024-03-14T13:02:30.709 INFO:tasks.workunit.client.0.smithi111.stdout:2/992: creat dd/d8d/d10c/f145 x:0 0 0 2024-03-14T13:02:30.710 INFO:tasks.workunit.client.1.smithi137.stdout:4/991: stat d4/d9/l1d 0 2024-03-14T13:02:30.710 INFO:tasks.workunit.client.0.smithi111.stdout:6/865: write d0/d17/d21/f2d [1736950,62489] 0 2024-03-14T13:02:30.710 INFO:tasks.workunit.client.0.smithi111.stdout:3/988: dwrite d0/d2d/d35/d50/da2/db3/d71/f14e [0,4194304] 0 2024-03-14T13:02:30.712 INFO:tasks.workunit.client.1.smithi137.stdout:8/998: readlink d1/d5e/d11a/d82/d13d/l85 0 2024-03-14T13:02:30.713 INFO:tasks.workunit.client.0.smithi111.stdout:9/973: creat d5/d42/d71/dbd/de0/f14a x:0 0 0 2024-03-14T13:02:30.714 INFO:tasks.workunit.client.1.smithi137.stdout:4/992: rename d4/d4d/l62 to d4/d9/d3a/d41/d38/d79/dbe/l13f 0 2024-03-14T13:02:30.715 INFO:tasks.workunit.client.0.smithi111.stdout:6/866: write d0/d17/d21/f44 [1200413,65547] 0 2024-03-14T13:02:30.717 INFO:tasks.workunit.client.0.smithi111.stdout:2/993: mkdir dd/d17/d53/da4/df5/df6/d146 0 2024-03-14T13:02:30.718 INFO:tasks.workunit.client.1.smithi137.stdout:4/993: mkdir d4/d27/d5c/d6d/d58/d140 0 2024-03-14T13:02:30.718 INFO:tasks.workunit.client.0.smithi111.stdout:3/989: mkdir d0/d2d/d35/d4c/dad/d117/d15f 0 2024-03-14T13:02:30.720 INFO:tasks.workunit.client.0.smithi111.stdout:9/974: mknod d5/d6/d8/d5b/d84/db6/c14b 0 2024-03-14T13:02:30.720 INFO:tasks.workunit.client.1.smithi137.stdout:8/999: write d1/d5e/f63 [366013,118395] 0 2024-03-14T13:02:30.721 INFO:tasks.workunit.client.0.smithi111.stdout:3/990: mknod d0/d2d/d35/d50/d5e/c160 0 2024-03-14T13:02:30.724 INFO:tasks.workunit.client.0.smithi111.stdout:6/867: rename d0/d17/d20/d4a/d59/d82/l8c to d0/dfb/db8/d127/da1/l140 0 2024-03-14T13:02:30.728 INFO:tasks.workunit.client.0.smithi111.stdout:9/975: dwrite d5/d6/feb [0,4194304] 0 2024-03-14T13:02:30.728 INFO:tasks.workunit.client.0.smithi111.stdout:3/991: rename d0/d2d/d35/d50/da2/db3/d67/f15a to d0/d2d/d35/d50/da2/f161 0 2024-03-14T13:02:30.729 INFO:tasks.workunit.client.0.smithi111.stdout:9/976: chown d5/d6/d14/d7a 96 1 2024-03-14T13:02:30.733 INFO:tasks.workunit.client.0.smithi111.stdout:9/977: mkdir d5/d6/d2a/d2d/db8/d14c 0 2024-03-14T13:02:30.734 INFO:tasks.workunit.client.0.smithi111.stdout:6/868: getdents d0/dfb/db8/de9 0 2024-03-14T13:02:30.736 INFO:tasks.workunit.client.0.smithi111.stdout:3/992: rename d0/d30/l9a to d0/d2d/d35/d50/d5e/l162 0 2024-03-14T13:02:30.737 INFO:tasks.workunit.client.1.smithi137.stdout:4/994: dread d4/d27/d5c/d77/f7c [0,4194304] 0 2024-03-14T13:02:30.740 INFO:tasks.workunit.client.1.smithi137.stdout:4/995: link d4/d10/f3f d4/d27/d5c/d6d/d58/d104/f141 0 2024-03-14T13:02:30.742 INFO:tasks.workunit.client.1.smithi137.stdout:4/996: fdatasync d4/d9/d8e/fd8 0 2024-03-14T13:02:30.745 INFO:tasks.workunit.client.0.smithi111.stdout:6/869: creat d0/dfb/d106/d108/f141 x:0 0 0 2024-03-14T13:02:30.746 INFO:tasks.workunit.client.0.smithi111.stdout:3/993: write d0/d2d/d35/d50/da2/db3/da7/f12c [435360,85496] 0 2024-03-14T13:02:30.759 INFO:tasks.workunit.client.1.smithi137.stdout:4/997: dread d4/d9/d3a/d41/d38/d79/f94 [4194304,4194304] 0 2024-03-14T13:02:30.765 INFO:tasks.workunit.client.1.smithi137.stdout:4/998: dwrite d4/d9/d3a/d41/d38/d79/fe0 [0,4194304] 0 2024-03-14T13:02:30.768 INFO:tasks.workunit.client.1.smithi137.stdout:4/999: rmdir d4/d9/d8e/dae 39 2024-03-14T13:02:30.776 INFO:tasks.workunit.client.0.smithi111.stdout:9/978: rename d5/d6/d2a/d5a/cbf to d5/d6/d14/d7a/c14d 0 2024-03-14T13:02:30.777 INFO:tasks.workunit.client.0.smithi111.stdout:6/870: mkdir d0/dfb/d106/d114/d142 0 2024-03-14T13:02:30.778 INFO:tasks.workunit.client.0.smithi111.stdout:6/871: fdatasync d0/d50/f66 0 2024-03-14T13:02:30.782 INFO:tasks.workunit.client.0.smithi111.stdout:6/872: creat d0/d50/df2/dc6/d126/f143 x:0 0 0 2024-03-14T13:02:30.783 INFO:tasks.workunit.client.0.smithi111.stdout:6/873: symlink d0/dfb/l144 0 2024-03-14T13:02:30.786 INFO:tasks.workunit.client.0.smithi111.stdout:6/874: mknod d0/d17/c145 0 2024-03-14T13:02:30.789 INFO:tasks.workunit.client.0.smithi111.stdout:6/875: creat d0/d17/d21/d42/f146 x:0 0 0 2024-03-14T13:02:30.795 INFO:tasks.workunit.client.0.smithi111.stdout:6/876: dwrite d0/dfb/f11b [0,4194304] 0 2024-03-14T13:02:30.807 INFO:tasks.workunit.client.0.smithi111.stdout:6/877: rmdir d0/dfb/d106/d108 39 2024-03-14T13:02:30.813 INFO:tasks.workunit.client.1.smithi137.stdout:3/799: sync 2024-03-14T13:02:30.813 INFO:tasks.workunit.client.1.smithi137.stdout:5/886: sync 2024-03-14T13:02:30.813 INFO:tasks.workunit.client.1.smithi137.stdout:6/976: sync 2024-03-14T13:02:30.813 INFO:tasks.workunit.client.1.smithi137.stdout:7/986: sync 2024-03-14T13:02:30.814 INFO:tasks.workunit.client.1.smithi137.stdout:3/800: readlink d4/df/lf1 0 2024-03-14T13:02:30.815 INFO:tasks.workunit.client.1.smithi137.stdout:7/987: chown d3/d5/d23/d3f/d5b/d111/d12b 48461 1 2024-03-14T13:02:30.819 INFO:tasks.workunit.client.0.smithi111.stdout:9/979: truncate d5/d6/fd5 3546761 0 2024-03-14T13:02:30.821 INFO:tasks.workunit.client.0.smithi111.stdout:9/980: mkdir d5/d6/d8/d5b/d84/db6/de7/d14e 0 2024-03-14T13:02:30.821 INFO:tasks.workunit.client.0.smithi111.stdout:6/878: write d0/f1c [4731124,110257] 0 2024-03-14T13:02:30.823 INFO:tasks.workunit.client.1.smithi137.stdout:6/977: rename d2/d54/d55/d9c/dbf/l123 to d2/d54/d55/d33/l14a 0 2024-03-14T13:02:30.823 INFO:tasks.workunit.client.1.smithi137.stdout:3/801: dwrite d4/de/f1f [0,4194304] 0 2024-03-14T13:02:30.823 INFO:tasks.workunit.client.0.smithi111.stdout:9/981: chown d5/d6/d8/d5b/d87/cb0 3878 1 2024-03-14T13:02:30.824 INFO:tasks.workunit.client.0.smithi111.stdout:6/879: read d0/d17/d20/d4a/d68/f88 [397324,57062] 0 2024-03-14T13:02:30.825 INFO:tasks.workunit.client.0.smithi111.stdout:6/880: chown d0/d17/d20/d4a/d59/d82 8319 1 2024-03-14T13:02:30.826 INFO:tasks.workunit.client.0.smithi111.stdout:9/982: dread d5/d6/d8/f3a [0,4194304] 0 2024-03-14T13:02:30.827 INFO:tasks.workunit.client.0.smithi111.stdout:9/983: fdatasync d5/d6/d8/f12 0 2024-03-14T13:02:30.829 INFO:tasks.workunit.client.0.smithi111.stdout:9/984: write d5/d6/d14/d7a/d102/f130 [145714,125745] 0 2024-03-14T13:02:30.834 INFO:tasks.workunit.client.1.smithi137.stdout:7/988: dwrite d3/d6/d14/d27/f6c [0,4194304] 0 2024-03-14T13:02:30.834 INFO:tasks.workunit.client.1.smithi137.stdout:5/887: dwrite d7/d33/f61 [0,4194304] 0 2024-03-14T13:02:30.837 INFO:tasks.workunit.client.1.smithi137.stdout:6/978: dread d2/d54/d55/d6d/dc3/dce/f13f [0,4194304] 0 2024-03-14T13:02:30.840 INFO:tasks.workunit.client.0.smithi111.stdout:9/985: truncate d5/d42/da4/fb1 3407485 0 2024-03-14T13:02:30.842 INFO:tasks.workunit.client.0.smithi111.stdout:9/986: read d5/d6/faa [446598,40641] 0 2024-03-14T13:02:30.843 INFO:tasks.workunit.client.0.smithi111.stdout:6/881: dread d0/d17/d20/d65/fc7 [0,4194304] 0 2024-03-14T13:02:30.845 INFO:tasks.workunit.client.1.smithi137.stdout:6/979: read d2/d54/d55/d6d/d76/d132/de/d45/f116 [288206,89140] 0 2024-03-14T13:02:30.846 INFO:tasks.workunit.client.1.smithi137.stdout:7/989: link d3/d6/d59/daa/dcf/d6a/d75/c8f d3/d6/d64/d97/dd2/c14e 0 2024-03-14T13:02:30.847 INFO:tasks.workunit.client.1.smithi137.stdout:3/802: dwrite d4/d19/fb7 [0,4194304] 0 2024-03-14T13:02:30.847 INFO:tasks.workunit.client.1.smithi137.stdout:5/888: dwrite d7/d16/fb9 [0,4194304] 0 2024-03-14T13:02:30.849 INFO:tasks.workunit.client.0.smithi111.stdout:9/987: creat d5/d6/d2a/d5a/dee/f14f x:0 0 0 2024-03-14T13:02:30.850 INFO:tasks.workunit.client.0.smithi111.stdout:6/882: creat d0/dfb/df5/f147 x:0 0 0 2024-03-14T13:02:30.851 INFO:tasks.workunit.client.0.smithi111.stdout:6/883: read - d0/dfb/df5/f105 zero size 2024-03-14T13:02:30.852 INFO:tasks.workunit.client.0.smithi111.stdout:6/884: write d0/dfb/df5/f147 [522800,8797] 0 2024-03-14T13:02:30.852 INFO:tasks.workunit.client.0.smithi111.stdout:9/988: unlink d5/d6/d2a/d86/ca6 0 2024-03-14T13:02:30.855 INFO:tasks.workunit.client.1.smithi137.stdout:7/990: read d3/d6/d59/daa/dcf/d6a/da1/f112 [2855602,59132] 0 2024-03-14T13:02:30.857 INFO:tasks.workunit.client.0.smithi111.stdout:9/989: unlink d5/d6/d2a/d2d/db8/fd1 0 2024-03-14T13:02:30.857 INFO:tasks.workunit.client.0.smithi111.stdout:6/885: symlink d0/dfb/db8/d127/da1/dc8/l148 0 2024-03-14T13:02:30.858 INFO:tasks.workunit.client.1.smithi137.stdout:7/991: dread - d3/d6/d59/daa/dcf/d6a/d75/d78/da4/fef zero size 2024-03-14T13:02:30.858 INFO:tasks.workunit.client.0.smithi111.stdout:6/886: stat d0/dfb/df5/f105 0 2024-03-14T13:02:30.859 INFO:tasks.workunit.client.0.smithi111.stdout:6/887: stat d0/d17/d20/l96 0 2024-03-14T13:02:30.859 INFO:tasks.workunit.client.1.smithi137.stdout:3/803: rmdir d4/de/d16/d27/dd3 39 2024-03-14T13:02:30.861 INFO:tasks.workunit.client.0.smithi111.stdout:9/990: rename d5/d6/d2a/d5a/fad to d5/d6/d8/d1a/f150 0 2024-03-14T13:02:30.862 INFO:tasks.workunit.client.1.smithi137.stdout:5/889: getdents d7/d16/dff 0 2024-03-14T13:02:30.863 INFO:tasks.workunit.client.1.smithi137.stdout:7/992: read - d3/d6/d59/daa/dcf/d6a/d75/d78/da4/f13b zero size 2024-03-14T13:02:30.865 INFO:tasks.workunit.client.1.smithi137.stdout:6/980: dwrite d2/d54/d55/d6d/d76/d132/d1d/d88/fa2 [0,4194304] 0 2024-03-14T13:02:30.865 INFO:tasks.workunit.client.0.smithi111.stdout:9/991: truncate d5/d6/d8/d5b/d87/ffe 24040 0 2024-03-14T13:02:30.865 INFO:tasks.workunit.client.0.smithi111.stdout:9/992: chown d5/d6/d8/fd9 3152 1 2024-03-14T13:02:30.868 INFO:tasks.workunit.client.0.smithi111.stdout:9/993: rename d5/d6/fe1 to d5/d6/d8/d1a/f151 0 2024-03-14T13:02:30.868 INFO:tasks.workunit.client.1.smithi137.stdout:5/890: unlink d7/d16/d18/d25/f55 0 2024-03-14T13:02:30.870 INFO:tasks.workunit.client.1.smithi137.stdout:7/993: dread d3/d5/f116 [0,4194304] 0 2024-03-14T13:02:30.871 INFO:tasks.workunit.client.1.smithi137.stdout:6/981: rename d2/d54/d55/f78 to d2/d54/d55/d6d/d76/d132/de/d45/d59/d102/d134/f14b 0 2024-03-14T13:02:30.874 INFO:tasks.workunit.client.1.smithi137.stdout:5/891: creat d7/d33/dab/f12a x:0 0 0 2024-03-14T13:02:30.875 INFO:tasks.workunit.client.0.smithi111.stdout:9/994: dwrite f4 [4194304,4194304] 0 2024-03-14T13:02:30.876 INFO:tasks.workunit.client.1.smithi137.stdout:6/982: creat d2/d54/d55/d33/f14c x:0 0 0 2024-03-14T13:02:30.876 INFO:tasks.workunit.client.1.smithi137.stdout:5/892: truncate d7/f7b 4396341 0 2024-03-14T13:02:30.879 INFO:tasks.workunit.client.1.smithi137.stdout:7/994: rename d3/d5/d5a/d11b to d3/d5/d23/d14f 0 2024-03-14T13:02:30.880 INFO:tasks.workunit.client.0.smithi111.stdout:9/995: dread d5/d6/d8/d74/fdd [0,4194304] 0 2024-03-14T13:02:30.880 INFO:tasks.workunit.client.0.smithi111.stdout:6/888: write d0/d17/d20/d4a/d59/fda [345610,101185] 0 2024-03-14T13:02:30.881 INFO:tasks.workunit.client.0.smithi111.stdout:9/996: write d5/f11f [4211360,103907] 0 2024-03-14T13:02:30.883 INFO:tasks.workunit.client.0.smithi111.stdout:9/997: dread - d5/d42/d71/d96/dd3/d136/f12b zero size 2024-03-14T13:02:30.883 INFO:tasks.workunit.client.1.smithi137.stdout:6/983: dread - d2/d54/d55/d6d/d76/d132/de/d45/d97/d9f/f111 zero size 2024-03-14T13:02:30.884 INFO:tasks.workunit.client.0.smithi111.stdout:6/889: symlink d0/dfb/d106/d108/l149 0 2024-03-14T13:02:30.886 INFO:tasks.workunit.client.0.smithi111.stdout:6/890: mkdir d0/d17/d21/d69/d14a 0 2024-03-14T13:02:30.891 INFO:tasks.workunit.client.1.smithi137.stdout:5/893: creat d7/d16/d18/d2d/f12b x:0 0 0 2024-03-14T13:02:30.891 INFO:tasks.workunit.client.1.smithi137.stdout:6/984: stat d2/d54/d55/d6d/d76/d57/cb5 0 2024-03-14T13:02:30.894 INFO:tasks.workunit.client.1.smithi137.stdout:7/995: dread f1 [0,4194304] 0 2024-03-14T13:02:30.896 INFO:tasks.workunit.client.1.smithi137.stdout:5/894: chown d7/d16/d18/d25/d28/d3c/c92 54 1 2024-03-14T13:02:30.898 INFO:tasks.workunit.client.1.smithi137.stdout:3/804: dwrite d4/de/d13/f61 [4194304,4194304] 0 2024-03-14T13:02:30.898 INFO:tasks.workunit.client.1.smithi137.stdout:6/985: mkdir d2/d54/d55/d6d/dc3/dec/d14d 0 2024-03-14T13:02:30.900 INFO:tasks.workunit.client.1.smithi137.stdout:7/996: mkdir d3/d5/d68/d150 0 2024-03-14T13:02:30.905 INFO:tasks.workunit.client.1.smithi137.stdout:6/986: symlink d2/d54/d55/d6d/d76/d132/d1d/d88/df4/l14e 0 2024-03-14T13:02:30.905 INFO:tasks.workunit.client.1.smithi137.stdout:3/805: rename d4/de/d16/d1c/cec to d4/de/d16/d1c/d8d/c104 0 2024-03-14T13:02:30.906 INFO:tasks.workunit.client.1.smithi137.stdout:7/997: creat d3/d6/d64/d97/dd2/f151 x:0 0 0 2024-03-14T13:02:30.910 INFO:tasks.workunit.client.1.smithi137.stdout:5/895: dwrite d7/d16/d18/d25/d28/dc5/f105 [0,4194304] 0 2024-03-14T13:02:30.914 INFO:tasks.workunit.client.0.smithi111.stdout:9/998: dread d5/d6/d2a/d5a/f5d [0,4194304] 0 2024-03-14T13:02:30.914 INFO:tasks.workunit.client.1.smithi137.stdout:6/987: mkdir d2/d54/d55/d6d/d76/d132/d1d/d88/dcd/d14f 0 2024-03-14T13:02:30.916 INFO:tasks.workunit.client.0.smithi111.stdout:9/999: unlink d5/f11f 0 2024-03-14T13:02:30.925 INFO:tasks.workunit.client.1.smithi137.stdout:7/998: rename d3/d6/ce9 to d3/d5/d5a/c152 0 2024-03-14T13:02:30.926 INFO:tasks.workunit.client.1.smithi137.stdout:5/896: dwrite d7/d33/d65/f7c [0,4194304] 0 2024-03-14T13:02:30.936 INFO:tasks.workunit.client.1.smithi137.stdout:6/988: dwrite d2/d54/d55/f1e [4194304,4194304] 0 2024-03-14T13:02:30.939 INFO:tasks.workunit.client.1.smithi137.stdout:7/999: dwrite d3/d4a/f5e [0,4194304] 0 2024-03-14T13:02:30.939 INFO:tasks.workunit.client.1.smithi137.stdout:3/806: dread d4/de/d16/d1c/d8d/dcd/f8e [0,4194304] 0 2024-03-14T13:02:30.943 INFO:tasks.workunit.client.1.smithi137.stdout:3/807: mkdir d4/de/d16/d1c/d5d/d85/d6b/de3/d105 0 2024-03-14T13:02:30.949 INFO:tasks.workunit.client.1.smithi137.stdout:3/808: creat d4/de/d16/d74/f106 x:0 0 0 2024-03-14T13:02:30.949 INFO:tasks.workunit.client.1.smithi137.stdout:6/989: write d2/d54/d55/d6d/d76/d132/de/d45/d97/d9f/f10b [5007026,4377] 0 2024-03-14T13:02:30.949 INFO:tasks.workunit.client.1.smithi137.stdout:6/990: chown d2/f25 2 1 2024-03-14T13:02:30.949 INFO:tasks.workunit.client.1.smithi137.stdout:3/809: unlink f1 0 2024-03-14T13:02:30.963 INFO:tasks.workunit.client.1.smithi137.stdout:3/810: fdatasync d4/de/d16/d1c/d8d/dcd/f64 0 2024-03-14T13:02:30.966 INFO:tasks.workunit.client.1.smithi137.stdout:5/897: dwrite d7/f1b [0,4194304] 0 2024-03-14T13:02:30.969 INFO:tasks.workunit.client.1.smithi137.stdout:6/991: dwrite d2/d54/d55/d6d/d76/d132/de/d45/d97/d9f/f10b [0,4194304] 0 2024-03-14T13:02:30.969 INFO:tasks.workunit.client.1.smithi137.stdout:6/992: readlink d2/d54/d55/d6d/d76/d132/de/laf 0 2024-03-14T13:02:30.991 INFO:tasks.workunit.client.0.smithi111.stdout:2/994: sync 2024-03-14T13:02:30.993 INFO:tasks.workunit.client.0.smithi111.stdout:2/995: creat dd/d8d/d10c/f147 x:0 0 0 2024-03-14T13:02:30.999 INFO:tasks.workunit.client.1.smithi137.stdout:5/898: creat d7/d1d/d111/f12c x:0 0 0 2024-03-14T13:02:31.001 INFO:tasks.workunit.client.0.smithi111.stdout:2/996: dwrite dd/d11/d1a/f1c [4194304,4194304] 0 2024-03-14T13:02:31.006 INFO:tasks.workunit.client.0.smithi111.stdout:2/997: mknod dd/c148 0 2024-03-14T13:02:31.007 INFO:tasks.workunit.client.0.smithi111.stdout:2/998: readlink dd/d11/d1a/l2b 0 2024-03-14T13:02:31.009 INFO:tasks.workunit.client.0.smithi111.stdout:2/999: unlink dd/d17/d34/lcf 0 2024-03-14T13:02:31.036 INFO:tasks.workunit.client.1.smithi137.stdout:6/993: dwrite d2/d54/d55/d6d/d76/d132/de/d45/d97/fc6 [0,4194304] 0 2024-03-14T13:02:31.036 INFO:tasks.workunit.client.1.smithi137.stdout:5/899: dwrite d7/d16/d18/d25/f5b [4194304,4194304] 0 2024-03-14T13:02:31.036 INFO:tasks.workunit.client.0.smithi111.stdout:3/994: sync 2024-03-14T13:02:31.037 INFO:tasks.workunit.client.0.smithi111.stdout:6/891: sync 2024-03-14T13:02:31.037 INFO:tasks.workunit.client.0.smithi111.stdout:3/995: fdatasync d0/f23 0 2024-03-14T13:02:31.040 INFO:tasks.workunit.client.0.smithi111.stdout:6/892: rename d0/dfb/db8/c13c to d0/d17/d20/d4a/d68/c14b 0 2024-03-14T13:02:31.041 INFO:tasks.workunit.client.0.smithi111.stdout:6/893: stat d0/dfb/db8/d127/da1/dc8/fc9 0 2024-03-14T13:02:31.042 INFO:tasks.workunit.client.0.smithi111.stdout:3/996: getdents d0/d2d/d35/d50 0 2024-03-14T13:02:31.045 INFO:tasks.workunit.client.1.smithi137.stdout:6/994: write d2/d54/d55/d6d/d76/d132/d86/d8a/fbb [2238372,70457] 0 2024-03-14T13:02:31.045 INFO:tasks.workunit.client.1.smithi137.stdout:6/995: mkdir d2/d54/d55/d6d/d76/d57/d150 0 2024-03-14T13:02:31.045 INFO:tasks.workunit.client.1.smithi137.stdout:5/900: creat d7/d16/d18/f12d x:0 0 0 2024-03-14T13:02:31.046 INFO:tasks.workunit.client.0.smithi111.stdout:3/997: creat d0/d2d/d35/d4c/dad/d117/d148/f163 x:0 0 0 2024-03-14T13:02:31.048 INFO:tasks.workunit.client.1.smithi137.stdout:5/901: write d7/d16/d18/f2a [718782,97119] 0 2024-03-14T13:02:31.048 INFO:tasks.workunit.client.1.smithi137.stdout:6/996: truncate d2/d54/d55/d6d/d76/d57/d58/fa5 575646 0 2024-03-14T13:02:31.049 INFO:tasks.workunit.client.0.smithi111.stdout:6/894: read d0/d17/d21/d42/da5/d12a/fe4 [2428803,64737] 0 2024-03-14T13:02:31.053 INFO:tasks.workunit.client.0.smithi111.stdout:3/998: dwrite d0/d2d/d35/d4c/dad/d117/f130 [0,4194304] 0 2024-03-14T13:02:31.053 INFO:tasks.workunit.client.0.smithi111.stdout:6/895: readlink d0/d17/d20/d4a/d59/d80/lfc 0 2024-03-14T13:02:31.056 INFO:tasks.workunit.client.1.smithi137.stdout:6/997: rename d2/d54/d55/d6d/d76/d132/de/d45/d59/daa to d2/d54/d55/d6d/d76/d132/de/d45/d151 0 2024-03-14T13:02:31.058 INFO:tasks.workunit.client.0.smithi111.stdout:6/896: symlink d0/d17/d20/l14c 0 2024-03-14T13:02:31.061 INFO:tasks.workunit.client.0.smithi111.stdout:3/999: rmdir d0/d2d 39 2024-03-14T13:02:31.062 INFO:tasks.workunit.client.0.smithi111.stdout:6/897: mkdir d0/d17/d20/d65/d103/d14d 0 2024-03-14T13:02:31.065 INFO:tasks.workunit.client.0.smithi111.stdout:6/898: mknod d0/d17/d20/d4a/c14e 0 2024-03-14T13:02:31.065 INFO:tasks.workunit.client.1.smithi137.stdout:5/902: dwrite d7/d36/fed [0,4194304] 0 2024-03-14T13:02:31.066 INFO:tasks.workunit.client.0.smithi111.stdout:6/899: creat d0/d17/d20/d4a/d59/d82/f14f x:0 0 0 2024-03-14T13:02:31.068 INFO:tasks.workunit.client.1.smithi137.stdout:6/998: creat d2/d54/d55/d6d/d76/d132/de/d45/d97/f152 x:0 0 0 2024-03-14T13:02:31.071 INFO:tasks.workunit.client.0.smithi111.stdout:6/900: read d0/dfb/f38 [3666520,16107] 0 2024-03-14T13:02:31.072 INFO:tasks.workunit.client.1.smithi137.stdout:6/999: mknod d2/d54/d55/d6d/d76/d132/de/d45/d59/d39/d3c/d83/c153 0 2024-03-14T13:02:31.079 INFO:tasks.workunit.client.1.smithi137.stdout:5/903: dwrite d7/d16/d18/f8a [0,4194304] 0 2024-03-14T13:02:31.081 INFO:tasks.workunit.client.1.smithi137.stdout:5/904: chown d7/d16/d18/d25/d28/fad 1 1 2024-03-14T13:02:31.082 INFO:tasks.workunit.client.0.smithi111.stdout:6/901: truncate d0/d17/d20/d65/d93/fe0 159245 0 2024-03-14T13:02:31.083 INFO:tasks.workunit.client.0.smithi111.stdout:6/902: readlink d0/d17/d20/l3a 0 2024-03-14T13:02:31.085 INFO:tasks.workunit.client.0.smithi111.stdout:6/903: mkdir d0/d17/d20/d150 0 2024-03-14T13:02:31.087 INFO:tasks.workunit.client.0.smithi111.stdout:6/904: mknod d0/d17/d20/d4a/d59/dc3/c151 0 2024-03-14T13:02:31.099 INFO:tasks.workunit.client.0.smithi111.stdout:6/905: dread d0/d50/f5f [0,4194304] 0 2024-03-14T13:02:31.102 INFO:tasks.workunit.client.0.smithi111.stdout:6/906: dread d0/d50/f5f [0,4194304] 0 2024-03-14T13:02:31.106 INFO:tasks.workunit.client.0.smithi111.stdout:6/907: dread d0/d17/d20/f6b [0,4194304] 0 2024-03-14T13:02:31.108 INFO:tasks.workunit.client.0.smithi111.stdout:6/908: creat d0/dfb/d106/d108/f152 x:0 0 0 2024-03-14T13:02:31.110 INFO:tasks.workunit.client.0.smithi111.stdout:6/909: mknod d0/d17/d20/d65/d103/d14d/c153 0 2024-03-14T13:02:31.113 INFO:tasks.workunit.client.0.smithi111.stdout:6/910: getdents d0/dfb/df5 0 2024-03-14T13:02:31.115 INFO:tasks.workunit.client.0.smithi111.stdout:6/911: mkdir d0/d17/d20/d94/d154 0 2024-03-14T13:02:31.128 INFO:tasks.workunit.client.0.smithi111.stdout:6/912: dread d0/d17/d21/d42/f58 [0,4194304] 0 2024-03-14T13:02:31.132 INFO:tasks.workunit.client.0.smithi111.stdout:6/913: rename d0/d17/d20/d65/d93/dd9/lf7 to d0/dfb/d106/d108/l155 0 2024-03-14T13:02:31.134 INFO:tasks.workunit.client.0.smithi111.stdout:6/914: mknod d0/d17/d20/d4a/d59/c156 0 2024-03-14T13:02:31.138 INFO:tasks.workunit.client.0.smithi111.stdout:6/915: write d0/d17/d21/f11f [3948684,53730] 0 2024-03-14T13:02:31.140 INFO:tasks.workunit.client.0.smithi111.stdout:6/916: mkdir d0/d138/d157 0 2024-03-14T13:02:31.142 INFO:tasks.workunit.client.0.smithi111.stdout:6/917: mkdir d0/dfb/db8/d127/da1/dc8/d158 0 2024-03-14T13:02:31.145 INFO:tasks.workunit.client.0.smithi111.stdout:6/918: rmdir d0/dfb/d106 39 2024-03-14T13:02:31.147 INFO:tasks.workunit.client.0.smithi111.stdout:6/919: dread - d0/d17/d21/d42/da5/d12a/f5c zero size 2024-03-14T13:02:31.148 INFO:tasks.workunit.client.0.smithi111.stdout:6/920: chown d0/dfb/d106 8 1 2024-03-14T13:02:31.149 INFO:tasks.workunit.client.0.smithi111.stdout:6/921: mknod d0/dfb/d106/d108/c159 0 2024-03-14T13:02:31.168 INFO:tasks.workunit.client.0.smithi111.stdout:6/922: dread d0/f57 [0,4194304] 0 2024-03-14T13:02:31.168 INFO:tasks.workunit.client.0.smithi111.stdout:6/923: read d0/f18 [535002,104204] 0 2024-03-14T13:02:31.173 INFO:tasks.workunit.client.0.smithi111.stdout:6/924: write d0/d17/f1f [3668321,104485] 0 2024-03-14T13:02:31.176 INFO:tasks.workunit.client.0.smithi111.stdout:6/925: write d0/dfb/db8/d127/f95 [342577,71495] 0 2024-03-14T13:02:31.182 INFO:tasks.workunit.client.0.smithi111.stdout:6/926: dwrite d0/d17/d21/d69/f39 [0,4194304] 0 2024-03-14T13:02:31.185 INFO:tasks.workunit.client.0.smithi111.stdout:6/927: write d0/d17/d20/d4a/d59/d82/f102 [2962428,94889] 0 2024-03-14T13:02:31.187 INFO:tasks.workunit.client.0.smithi111.stdout:6/928: mkdir d0/d17/d20/d65/d103/d14d/d15a 0 2024-03-14T13:02:31.189 INFO:tasks.workunit.client.0.smithi111.stdout:6/929: mknod d0/dfb/d106/c15b 0 2024-03-14T13:02:31.192 INFO:tasks.workunit.client.0.smithi111.stdout:6/930: truncate d0/d17/d21/d69/f52 511231 0 2024-03-14T13:02:31.198 INFO:tasks.workunit.client.0.smithi111.stdout:6/931: write d0/d17/d21/fe2 [135205,40953] 0 2024-03-14T13:02:31.200 INFO:tasks.workunit.client.0.smithi111.stdout:6/932: chown d0/d17/d20/d4a/d68/dc2 2076 1 2024-03-14T13:02:31.200 INFO:tasks.workunit.client.0.smithi111.stdout:6/933: chown d0/d138/d157 32745629 1 2024-03-14T13:02:31.207 INFO:tasks.workunit.client.0.smithi111.stdout:6/934: dwrite d0/ff [4194304,4194304] 0 2024-03-14T13:02:31.221 INFO:tasks.workunit.client.0.smithi111.stdout:6/935: dwrite d0/f35 [0,4194304] 0 2024-03-14T13:02:31.229 INFO:tasks.workunit.client.0.smithi111.stdout:6/936: dwrite d0/d17/d20/d4a/d68/d10e/f122 [0,4194304] 0 2024-03-14T13:02:31.232 INFO:tasks.workunit.client.0.smithi111.stdout:6/937: creat d0/d50/f15c x:0 0 0 2024-03-14T13:02:31.233 INFO:tasks.workunit.client.0.smithi111.stdout:6/938: stat d0/d17/d21/d42/da5/d12a/dca 0 2024-03-14T13:02:31.245 INFO:tasks.workunit.client.0.smithi111.stdout:6/939: write d0/d17/f1e [2341836,126972] 0 2024-03-14T13:02:31.252 INFO:tasks.workunit.client.0.smithi111.stdout:6/940: dwrite d0/dfb/db8/d127/fa7 [0,4194304] 0 2024-03-14T13:02:31.257 INFO:tasks.workunit.client.0.smithi111.stdout:6/941: mknod d0/d17/d20/d4a/c15d 0 2024-03-14T13:02:31.263 INFO:tasks.workunit.client.0.smithi111.stdout:6/942: dwrite d0/d17/d21/d69/f39 [4194304,4194304] 0 2024-03-14T13:02:31.267 INFO:tasks.workunit.client.0.smithi111.stdout:6/943: mkdir d0/d17/d20/d65/d103/d15e 0 2024-03-14T13:02:31.269 INFO:tasks.workunit.client.0.smithi111.stdout:6/944: fsync d0/d17/d20/d4a/d59/fbb 0 2024-03-14T13:02:31.272 INFO:tasks.workunit.client.0.smithi111.stdout:6/945: mkdir d0/d17/d21/d42/da5/d12a/d15f 0 2024-03-14T13:02:31.279 INFO:tasks.workunit.client.0.smithi111.stdout:6/946: write d0/d17/d20/d65/d93/fe0 [454502,41275] 0 2024-03-14T13:02:31.282 INFO:tasks.workunit.client.0.smithi111.stdout:6/947: creat d0/d17/d20/d4a/d68/f160 x:0 0 0 2024-03-14T13:02:31.283 INFO:tasks.workunit.client.0.smithi111.stdout:6/948: write d0/dfb/d106/d114/f13e [695271,110327] 0 2024-03-14T13:02:31.284 INFO:tasks.workunit.client.0.smithi111.stdout:6/949: creat d0/d138/f161 x:0 0 0 2024-03-14T13:02:31.287 INFO:tasks.workunit.client.0.smithi111.stdout:6/950: mknod d0/d17/d20/d4a/d59/d82/c162 0 2024-03-14T13:02:31.287 INFO:tasks.workunit.client.0.smithi111.stdout:6/951: chown d0/dfb/db8/d127/da1/cd2 15 1 2024-03-14T13:02:31.289 INFO:tasks.workunit.client.1.smithi137.stdout:3/811: sync 2024-03-14T13:02:31.289 INFO:tasks.workunit.client.0.smithi111.stdout:6/952: read d0/d17/d20/d65/f92 [3958783,35657] 0 2024-03-14T13:02:31.292 INFO:tasks.workunit.client.1.smithi137.stdout:3/812: truncate d4/de/d13/faf 3654663 0 2024-03-14T13:02:31.293 INFO:tasks.workunit.client.0.smithi111.stdout:6/953: getdents d0 0 2024-03-14T13:02:31.295 INFO:tasks.workunit.client.0.smithi111.stdout:6/954: mkdir d0/d17/d21/d69/d163 0 2024-03-14T13:02:31.297 INFO:tasks.workunit.client.0.smithi111.stdout:6/955: fdatasync d0/d17/d20/d5a/fcb 0 2024-03-14T13:02:31.298 INFO:tasks.workunit.client.1.smithi137.stdout:3/813: truncate d4/d19/f90 2457587 0 2024-03-14T13:02:31.299 INFO:tasks.workunit.client.0.smithi111.stdout:6/956: fdatasync d0/d17/d21/d42/f58 0 2024-03-14T13:02:31.300 INFO:tasks.workunit.client.1.smithi137.stdout:3/814: fsync d4/de/d16/fbc 0 2024-03-14T13:02:31.301 INFO:tasks.workunit.client.0.smithi111.stdout:6/957: rmdir d0/d17/d99 39 2024-03-14T13:02:31.306 INFO:tasks.workunit.client.0.smithi111.stdout:6/958: dwrite d0/d17/d21/f44 [0,4194304] 0 2024-03-14T13:02:31.307 INFO:tasks.workunit.client.1.smithi137.stdout:3/815: dwrite d4/de/d16/d1c/d8d/dcd/f8e [0,4194304] 0 2024-03-14T13:02:31.308 INFO:tasks.workunit.client.1.smithi137.stdout:3/816: dread - d4/de/d16/d1c/db8/ff4 zero size 2024-03-14T13:02:31.315 INFO:tasks.workunit.client.0.smithi111.stdout:6/959: dwrite d0/d17/d20/d4a/d59/dc3/f115 [0,4194304] 0 2024-03-14T13:02:31.317 INFO:tasks.workunit.client.0.smithi111.stdout:6/960: write d0/d17/d20/d4a/d68/d10e/f122 [3290456,9302] 0 2024-03-14T13:02:31.318 INFO:tasks.workunit.client.1.smithi137.stdout:3/817: dwrite d4/de/d16/d27/f73 [0,4194304] 0 2024-03-14T13:02:31.325 INFO:tasks.workunit.client.1.smithi137.stdout:3/818: unlink d4/de/d16/d1c/d5d/d85/d6b/fe5 0 2024-03-14T13:02:31.327 INFO:tasks.workunit.client.1.smithi137.stdout:3/819: chown d4/de/d16/c7b 0 1 2024-03-14T13:02:31.335 INFO:tasks.workunit.client.1.smithi137.stdout:3/820: dwrite d4/de/d16/d1c/db8/dd1/fdd [0,4194304] 0 2024-03-14T13:02:31.336 INFO:tasks.workunit.client.1.smithi137.stdout:3/821: chown d4/de/d16/cf0 38944802 1 2024-03-14T13:02:31.339 INFO:tasks.workunit.client.0.smithi111.stdout:6/961: dwrite d0/d17/d20/f48 [0,4194304] 0 2024-03-14T13:02:31.342 INFO:tasks.workunit.client.0.smithi111.stdout:6/962: creat d0/d17/d21/d42/da5/d12a/dca/f164 x:0 0 0 2024-03-14T13:02:31.342 INFO:tasks.workunit.client.1.smithi137.stdout:5/905: sync 2024-03-14T13:02:31.343 INFO:tasks.workunit.client.0.smithi111.stdout:6/963: write d0/dfb/db8/de9/f12b [822613,45895] 0 2024-03-14T13:02:31.344 INFO:tasks.workunit.client.1.smithi137.stdout:5/906: chown d7/d16/d18/d25/d28/d3c/dd8/f119 48182 1 2024-03-14T13:02:31.344 INFO:tasks.workunit.client.1.smithi137.stdout:3/822: dwrite d4/de/d16/d1c/d8d/dcd/fba [0,4194304] 0 2024-03-14T13:02:31.347 INFO:tasks.workunit.client.1.smithi137.stdout:3/823: chown d4/de/d16/d1c/d5d/d4b/de6 3 1 2024-03-14T13:02:31.350 INFO:tasks.workunit.client.1.smithi137.stdout:3/824: write d4/de/d16/d1c/d5d/f43 [6578992,61202] 0 2024-03-14T13:02:31.354 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:31 smithi137 ceph-mon[50336]: pgmap v30: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 113 MiB/s rd, 275 MiB/s wr, 569 op/s 2024-03-14T13:02:31.354 INFO:tasks.workunit.client.0.smithi111.stdout:6/964: dread d0/dfb/df5/f147 [0,4194304] 0 2024-03-14T13:02:31.356 INFO:tasks.workunit.client.1.smithi137.stdout:3/825: creat d4/de/d13/f107 x:0 0 0 2024-03-14T13:02:31.357 INFO:tasks.workunit.client.0.smithi111.stdout:6/965: creat d0/d17/d21/f165 x:0 0 0 2024-03-14T13:02:31.520 INFO:tasks.workunit.client.1.smithi137.stdout:5/907: sync 2024-03-14T13:02:31.520 INFO:tasks.workunit.client.1.smithi137.stdout:3/826: sync 2024-03-14T13:02:31.521 INFO:tasks.workunit.client.1.smithi137.stdout:5/908: stat d7/d16/d18/d25/d28/d3c/d76/ff8 0 2024-03-14T13:02:31.522 INFO:tasks.workunit.client.1.smithi137.stdout:3/827: symlink d4/de/l108 0 2024-03-14T13:02:31.523 INFO:tasks.workunit.client.1.smithi137.stdout:5/909: mknod d7/d1d/d106/c12e 0 2024-03-14T13:02:31.524 INFO:tasks.workunit.client.1.smithi137.stdout:5/910: rmdir d7/d1d/d74/d83/dc4 39 2024-03-14T13:02:31.525 INFO:tasks.workunit.client.1.smithi137.stdout:3/828: truncate d4/de/d16/d1c/d8d/fb4 579394 0 2024-03-14T13:02:31.526 INFO:tasks.workunit.client.1.smithi137.stdout:5/911: creat d7/d16/d18/d25/d28/dc5/f12f x:0 0 0 2024-03-14T13:02:31.527 INFO:tasks.workunit.client.1.smithi137.stdout:3/829: mknod d4/de/d16/d1c/d5d/d85/d6b/df7/c109 0 2024-03-14T13:02:31.527 INFO:tasks.workunit.client.1.smithi137.stdout:5/912: write d7/d16/d18/d25/d28/d3c/d76/ff8 [467974,9234] 0 2024-03-14T13:02:31.529 INFO:tasks.workunit.client.1.smithi137.stdout:3/830: fsync d4/de/d16/d1c/f8f 0 2024-03-14T13:02:31.533 INFO:tasks.workunit.client.1.smithi137.stdout:3/831: readlink d4/de/d16/d1c/d5d/d85/d6b/lff 0 2024-03-14T13:02:31.535 INFO:tasks.workunit.client.1.smithi137.stdout:5/913: dwrite d7/d33/dab/f12a [0,4194304] 0 2024-03-14T13:02:31.537 INFO:tasks.workunit.client.1.smithi137.stdout:3/832: read d4/de/d16/d1c/d8d/dcd/fba [1620954,49520] 0 2024-03-14T13:02:31.542 INFO:tasks.workunit.client.1.smithi137.stdout:3/833: mknod d4/de/d16/d1c/d5d/d85/d6b/de3/d105/c10a 0 2024-03-14T13:02:31.544 INFO:tasks.workunit.client.0.smithi111.stdout:6/966: sync 2024-03-14T13:02:31.544 INFO:tasks.workunit.client.1.smithi137.stdout:3/834: read d4/f80 [1860189,17632] 0 2024-03-14T13:02:31.545 INFO:tasks.workunit.client.1.smithi137.stdout:5/914: write d7/d16/d18/f39 [3004130,31165] 0 2024-03-14T13:02:31.546 INFO:tasks.workunit.client.1.smithi137.stdout:3/835: chown d4/de/d16/d1c/d5d/d85/l3b 141744 1 2024-03-14T13:02:31.547 INFO:tasks.workunit.client.1.smithi137.stdout:5/915: creat d7/d33/d65/f130 x:0 0 0 2024-03-14T13:02:31.550 INFO:tasks.workunit.client.1.smithi137.stdout:5/916: fdatasync d7/d1d/d74/d83/dc4/fcc 0 2024-03-14T13:02:31.550 INFO:tasks.workunit.client.0.smithi111.stdout:6/967: dwrite d0/d17/d20/d94/f121 [0,4194304] 0 2024-03-14T13:02:31.552 INFO:tasks.workunit.client.1.smithi137.stdout:5/917: unlink d7/d33/dab/f12a 0 2024-03-14T13:02:31.553 INFO:tasks.workunit.client.0.smithi111.stdout:6/968: symlink d0/d17/d20/d65/d103/d15e/l166 0 2024-03-14T13:02:31.554 INFO:tasks.workunit.client.1.smithi137.stdout:5/918: readlink d7/d16/d18/l1f 0 2024-03-14T13:02:31.555 INFO:tasks.workunit.client.1.smithi137.stdout:5/919: dread - d7/d1d/d74/fe3 zero size 2024-03-14T13:02:31.556 INFO:tasks.workunit.client.1.smithi137.stdout:5/920: readlink d7/d16/d18/d25/l97 0 2024-03-14T13:02:31.559 INFO:tasks.workunit.client.1.smithi137.stdout:5/921: creat d7/d16/d18/d25/d28/d9e/de5/d11d/f131 x:0 0 0 2024-03-14T13:02:31.560 INFO:tasks.workunit.client.0.smithi111.stdout:6/969: write d0/f14 [3816469,63391] 0 2024-03-14T13:02:31.561 INFO:tasks.workunit.client.0.smithi111.stdout:6/970: fdatasync d0/d50/f5f 0 2024-03-14T13:02:31.562 INFO:tasks.workunit.client.0.smithi111.stdout:6/971: fsync d0/d17/d20/d4a/d68/ff9 0 2024-03-14T13:02:31.565 INFO:tasks.workunit.client.0.smithi111.stdout:6/972: getdents d0/d17/d20/d94 0 2024-03-14T13:02:31.566 INFO:tasks.workunit.client.0.smithi111.stdout:6/973: write d0/d50/df2/f12d [316932,115217] 0 2024-03-14T13:02:31.567 INFO:tasks.workunit.client.1.smithi137.stdout:5/922: write d7/d49/fbd [513223,84581] 0 2024-03-14T13:02:31.569 INFO:tasks.workunit.client.1.smithi137.stdout:5/923: write d7/d16/d18/d25/d28/d3c/d76/ff8 [768050,93799] 0 2024-03-14T13:02:31.570 INFO:tasks.workunit.client.0.smithi111.stdout:6/974: unlink d0/d17/d20/l10f 0 2024-03-14T13:02:31.571 INFO:tasks.workunit.client.1.smithi137.stdout:5/924: creat d7/d33/f132 x:0 0 0 2024-03-14T13:02:31.571 INFO:tasks.workunit.client.0.smithi111.stdout:6/975: rmdir d0/d17/d21/d69 39 2024-03-14T13:02:31.573 INFO:tasks.workunit.client.0.smithi111.stdout:6/976: getdents d0/d17/d21/d42/da5/d12a/d10b 0 2024-03-14T13:02:31.575 INFO:tasks.workunit.client.1.smithi137.stdout:5/925: write d7/f30 [1409117,56776] 0 2024-03-14T13:02:31.576 INFO:tasks.workunit.client.0.smithi111.stdout:6/977: truncate d0/d17/d21/d69/f52 1425836 0 2024-03-14T13:02:31.577 INFO:tasks.workunit.client.1.smithi137.stdout:5/926: rename d7/d16/ca6 to d7/d1d/d74/d81/c133 0 2024-03-14T13:02:31.578 INFO:tasks.workunit.client.0.smithi111.stdout:6/978: rmdir d0/d17/d21/d42/da5/d12a/dca 39 2024-03-14T13:02:31.579 INFO:tasks.workunit.client.0.smithi111.stdout:6/979: fsync d0/f35 0 2024-03-14T13:02:31.579 INFO:tasks.workunit.client.1.smithi137.stdout:5/927: dread - d7/d16/d18/d25/d28/d3c/f4f zero size 2024-03-14T13:02:31.580 INFO:tasks.workunit.client.0.smithi111.stdout:6/980: readlink d0/l90 0 2024-03-14T13:02:31.582 INFO:tasks.workunit.client.0.smithi111.stdout:6/981: creat d0/dfb/d106/d114/f167 x:0 0 0 2024-03-14T13:02:31.582 INFO:tasks.workunit.client.1.smithi137.stdout:5/928: link d7/d33/d65/dc2/fef d7/d16/d18/d25/d28/d3c/d76/f134 0 2024-03-14T13:02:31.583 INFO:tasks.workunit.client.1.smithi137.stdout:5/929: fdatasync d7/fb 0 2024-03-14T13:02:31.583 INFO:tasks.workunit.client.0.smithi111.stdout:6/982: write d0/d17/d20/d4a/d68/f160 [885516,60399] 0 2024-03-14T13:02:31.584 INFO:tasks.workunit.client.1.smithi137.stdout:5/930: fsync d7/d49/d75/ff4 0 2024-03-14T13:02:31.585 INFO:tasks.workunit.client.0.smithi111.stdout:6/983: rmdir d0/d17/d21/d42 39 2024-03-14T13:02:31.587 INFO:tasks.workunit.client.0.smithi111.stdout:6/984: unlink d0/f4 0 2024-03-14T13:02:31.588 INFO:tasks.workunit.client.1.smithi137.stdout:5/931: link d7/d16/d18/d2d/f8f d7/d16/d18/d2d/dd1/ddc/f135 0 2024-03-14T13:02:31.589 INFO:tasks.workunit.client.1.smithi137.stdout:5/932: write d7/d1d/d106/f124 [578185,109179] 0 2024-03-14T13:02:31.589 INFO:tasks.workunit.client.1.smithi137.stdout:5/933: readlink d7/d1d/l32 0 2024-03-14T13:02:31.593 INFO:tasks.workunit.client.0.smithi111.stdout:6/985: dwrite d0/f3e [0,4194304] 0 2024-03-14T13:02:31.599 INFO:tasks.workunit.client.1.smithi137.stdout:5/934: truncate d7/d1d/d74/d81/f89 772931 0 2024-03-14T13:02:31.611 INFO:tasks.workunit.client.1.smithi137.stdout:5/935: dwrite d7/d49/f127 [0,4194304] 0 2024-03-14T13:02:31.612 INFO:tasks.workunit.client.0.smithi111.stdout:6/986: dread d0/d17/d20/f4b [0,4194304] 0 2024-03-14T13:02:31.613 INFO:tasks.workunit.client.0.smithi111.stdout:6/987: chown d0/d50/f56 0 1 2024-03-14T13:02:31.615 INFO:tasks.workunit.client.0.smithi111.stdout:6/988: mknod d0/dfb/df5/c168 0 2024-03-14T13:02:31.626 INFO:tasks.workunit.client.1.smithi137.stdout:5/936: dwrite d7/d16/f38 [0,4194304] 0 2024-03-14T13:02:31.628 INFO:tasks.workunit.client.1.smithi137.stdout:5/937: symlink d7/d1d/d74/d83/l136 0 2024-03-14T13:02:31.630 INFO:tasks.workunit.client.1.smithi137.stdout:5/938: dread d7/d1d/d74/d83/f9a [0,4194304] 0 2024-03-14T13:02:31.632 INFO:tasks.workunit.client.1.smithi137.stdout:5/939: getdents d7/d16/d18/d25/d28/d9e/dd0 0 2024-03-14T13:02:31.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:31 smithi111 ceph-mon[39688]: pgmap v30: 65 pgs: 65 active+clean; 3.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 113 MiB/s rd, 275 MiB/s wr, 569 op/s 2024-03-14T13:02:31.642 INFO:tasks.workunit.client.1.smithi137.stdout:5/940: write d7/d33/d65/dc2/fe4 [5131715,81816] 0 2024-03-14T13:02:31.647 INFO:tasks.workunit.client.1.smithi137.stdout:5/941: truncate d7/d1d/d106/f124 273047 0 2024-03-14T13:02:31.649 INFO:tasks.workunit.client.1.smithi137.stdout:5/942: unlink d7/d16/d18/d25/f120 0 2024-03-14T13:02:31.649 INFO:tasks.workunit.client.1.smithi137.stdout:5/943: readlink d7/d33/d65/la7 0 2024-03-14T13:02:31.651 INFO:tasks.workunit.client.1.smithi137.stdout:5/944: read d7/d16/fae [419117,57354] 0 2024-03-14T13:02:31.657 INFO:tasks.workunit.client.1.smithi137.stdout:5/945: dread d7/d1d/d106/f124 [0,4194304] 0 2024-03-14T13:02:31.668 INFO:tasks.workunit.client.1.smithi137.stdout:5/946: dread d7/d49/d75/f8c [0,4194304] 0 2024-03-14T13:02:31.671 INFO:tasks.workunit.client.1.smithi137.stdout:5/947: creat d7/d16/d18/d25/d28/d3c/d7a/f137 x:0 0 0 2024-03-14T13:02:31.673 INFO:tasks.workunit.client.1.smithi137.stdout:5/948: dread - d7/d1d/d74/d81/f123 zero size 2024-03-14T13:02:31.675 INFO:tasks.workunit.client.1.smithi137.stdout:5/949: mknod d7/d1d/d74/dec/c138 0 2024-03-14T13:02:31.677 INFO:tasks.workunit.client.1.smithi137.stdout:5/950: symlink d7/d16/d18/d25/d28/l139 0 2024-03-14T13:02:31.678 INFO:tasks.workunit.client.1.smithi137.stdout:5/951: mkdir d7/d36/d13a 0 2024-03-14T13:02:31.679 INFO:tasks.workunit.client.0.smithi111.stdout:6/989: sync 2024-03-14T13:02:31.683 INFO:tasks.workunit.client.0.smithi111.stdout:6/990: rename d0/dfb/db8/d127/da1/dc8/ff3 to d0/d50/df2/dc6/de5/f169 0 2024-03-14T13:02:31.684 INFO:tasks.workunit.client.1.smithi137.stdout:5/952: read - d7/d16/d18/d25/d28/d3c/dd8/d10c/f128 zero size 2024-03-14T13:02:31.685 INFO:tasks.workunit.client.0.smithi111.stdout:6/991: creat d0/d17/d21/d42/da5/f16a x:0 0 0 2024-03-14T13:02:31.686 INFO:tasks.workunit.client.1.smithi137.stdout:5/953: mkdir d7/d16/d18/d48/d13b 0 2024-03-14T13:02:31.687 INFO:tasks.workunit.client.0.smithi111.stdout:6/992: creat d0/d17/d21/d42/da5/d12a/d15f/f16b x:0 0 0 2024-03-14T13:02:31.687 INFO:tasks.workunit.client.1.smithi137.stdout:5/954: dread d7/d33/f93 [0,4194304] 0 2024-03-14T13:02:31.688 INFO:tasks.workunit.client.1.smithi137.stdout:5/955: chown d7/d33/f61 19103075 1 2024-03-14T13:02:31.688 INFO:tasks.workunit.client.0.smithi111.stdout:6/993: write d0/d17/d20/f26 [5125994,127731] 0 2024-03-14T13:02:31.688 INFO:tasks.workunit.client.1.smithi137.stdout:5/956: chown d7/d16/d18/d2d 0 1 2024-03-14T13:02:31.690 INFO:tasks.workunit.client.0.smithi111.stdout:6/994: mknod d0/dfb/db8/d127/c16c 0 2024-03-14T13:02:31.690 INFO:tasks.workunit.client.0.smithi111.stdout:6/995: fsync d0/d17/d21/f44 0 2024-03-14T13:02:31.691 INFO:tasks.workunit.client.1.smithi137.stdout:5/957: creat d7/d16/d18/d25/d104/f13c x:0 0 0 2024-03-14T13:02:31.692 INFO:tasks.workunit.client.0.smithi111.stdout:6/996: read d0/d50/df2/fea [43680,12287] 0 2024-03-14T13:02:31.692 INFO:tasks.workunit.client.1.smithi137.stdout:5/958: truncate d7/d16/d18/d25/d28/d3c/dd8/d10c/f11a 479200 0 2024-03-14T13:02:31.693 INFO:tasks.workunit.client.0.smithi111.stdout:6/997: write d0/d17/d20/d4a/d59/d82/f102 [619027,106628] 0 2024-03-14T13:02:31.696 INFO:tasks.workunit.client.0.smithi111.stdout:6/998: dread d0/d17/d20/f136 [0,4194304] 0 2024-03-14T13:02:31.696 INFO:tasks.workunit.client.0.smithi111.stdout:6/999: chown d0/l9 2810 1 2024-03-14T13:02:31.699 INFO:tasks.workunit.client.0.smithi111.stderr:+ rm -rf -- ./tmp.IDB5UJWc6S 2024-03-14T13:02:31.700 INFO:tasks.workunit.client.1.smithi137.stdout:5/959: dwrite d7/d16/d18/d25/d28/d3c/f77 [0,4194304] 0 2024-03-14T13:02:31.708 INFO:tasks.workunit.client.1.smithi137.stdout:5/960: dwrite d7/d49/d75/ff4 [0,4194304] 0 2024-03-14T13:02:31.712 INFO:tasks.workunit.client.1.smithi137.stdout:5/961: creat d7/d16/d41/d11c/f13d x:0 0 0 2024-03-14T13:02:31.716 INFO:tasks.workunit.client.1.smithi137.stdout:5/962: rename d7/d36/fed to d7/d16/d18/d2d/f13e 0 2024-03-14T13:02:31.718 INFO:tasks.workunit.client.1.smithi137.stdout:5/963: getdents d7/d1d/d74 0 2024-03-14T13:02:31.719 INFO:tasks.workunit.client.1.smithi137.stdout:5/964: chown d7/d33/fee 5470 1 2024-03-14T13:02:31.720 INFO:tasks.workunit.client.1.smithi137.stdout:5/965: mkdir d7/d16/d41/d13f 0 2024-03-14T13:02:31.722 INFO:tasks.workunit.client.1.smithi137.stdout:5/966: chown d7/d16/d18/d25/d28/d9e/de5 3333 1 2024-03-14T13:02:31.724 INFO:tasks.workunit.client.1.smithi137.stdout:3/836: sync 2024-03-14T13:02:31.724 INFO:tasks.workunit.client.1.smithi137.stdout:5/967: rmdir d7/d16/d18/d2d 39 2024-03-14T13:02:31.727 INFO:tasks.workunit.client.1.smithi137.stdout:5/968: creat d7/d33/d65/f140 x:0 0 0 2024-03-14T13:02:31.727 INFO:tasks.workunit.client.1.smithi137.stdout:3/837: dread d4/de/d16/fbc [0,4194304] 0 2024-03-14T13:02:31.729 INFO:tasks.workunit.client.1.smithi137.stdout:5/969: creat d7/d16/d18/d25/d28/d3c/dd8/d10c/f141 x:0 0 0 2024-03-14T13:02:31.730 INFO:tasks.workunit.client.1.smithi137.stdout:3/838: creat d4/de/d16/d1c/d5d/d85/d6b/de3/f10b x:0 0 0 2024-03-14T13:02:31.732 INFO:tasks.workunit.client.1.smithi137.stdout:3/839: symlink d4/de/dc7/l10c 0 2024-03-14T13:02:31.732 INFO:tasks.workunit.client.1.smithi137.stdout:5/970: mknod d7/d16/d18/d48/dfa/c142 0 2024-03-14T13:02:31.733 INFO:tasks.workunit.client.1.smithi137.stdout:5/971: chown d7/d16/l113 31950524 1 2024-03-14T13:02:31.739 INFO:tasks.workunit.client.1.smithi137.stdout:5/972: dwrite d7/d16/d18/d25/d28/d3c/dd8/fe9 [4194304,4194304] 0 2024-03-14T13:02:31.749 INFO:tasks.workunit.client.1.smithi137.stdout:5/973: dwrite d7/d16/d18/d2d/f12b [0,4194304] 0 2024-03-14T13:02:31.750 INFO:tasks.workunit.client.1.smithi137.stdout:3/840: dread d4/de/d16/d1c/d5d/f43 [0,4194304] 0 2024-03-14T13:02:31.751 INFO:tasks.workunit.client.1.smithi137.stdout:5/974: chown d7/d16/d18/d25/d28/d3c/dd8/f108 10595923 1 2024-03-14T13:02:31.753 INFO:tasks.workunit.client.1.smithi137.stdout:5/975: creat d7/d33/d65/dc2/f143 x:0 0 0 2024-03-14T13:02:31.755 INFO:tasks.workunit.client.1.smithi137.stdout:3/841: link d4/d19/f54 d4/de/d16/d1c/db8/f10d 0 2024-03-14T13:02:31.756 INFO:tasks.workunit.client.1.smithi137.stdout:3/842: chown d4/de/d16/d1c/d5d/l62 173739 1 2024-03-14T13:02:31.757 INFO:tasks.workunit.client.1.smithi137.stdout:5/976: creat d7/d49/d75/f144 x:0 0 0 2024-03-14T13:02:31.759 INFO:tasks.workunit.client.1.smithi137.stdout:5/977: write d7/d16/d18/d25/d28/d3c/d7a/fc0 [4598745,37149] 0 2024-03-14T13:02:31.761 INFO:tasks.workunit.client.1.smithi137.stdout:5/978: rmdir d7/d16/d18/d2d/dd1 39 2024-03-14T13:02:31.765 INFO:tasks.workunit.client.1.smithi137.stdout:3/843: dwrite d4/de/d16/d27/dd3/fc4 [0,4194304] 0 2024-03-14T13:02:31.768 INFO:tasks.workunit.client.1.smithi137.stdout:5/979: read d7/d16/d18/d25/f5b [1939819,121471] 0 2024-03-14T13:02:31.829 INFO:tasks.workunit.client.1.smithi137.stdout:3/844: creat d4/de/dc7/f10e x:0 0 0 2024-03-14T13:02:31.832 INFO:tasks.workunit.client.1.smithi137.stdout:3/845: truncate d4/de/f2a 1359354 0 2024-03-14T13:02:31.833 INFO:tasks.workunit.client.1.smithi137.stdout:3/846: fdatasync d4/de/f1f 0 2024-03-14T13:02:31.838 INFO:tasks.workunit.client.1.smithi137.stdout:3/847: mkdir d4/de/d10f 0 2024-03-14T13:02:31.854 INFO:tasks.workunit.client.1.smithi137.stdout:3/848: creat d4/de/d16/d1c/d8d/f110 x:0 0 0 2024-03-14T13:02:31.879 INFO:tasks.workunit.client.1.smithi137.stdout:3/849: truncate d4/de/f1f 1031918 0 2024-03-14T13:02:31.880 INFO:tasks.workunit.client.1.smithi137.stdout:3/850: write d4/ffc [494483,87461] 0 2024-03-14T13:02:31.883 INFO:tasks.workunit.client.1.smithi137.stdout:3/851: symlink d4/de/d16/d1c/d5d/d85/d6b/df7/l111 0 2024-03-14T13:02:31.886 INFO:tasks.workunit.client.1.smithi137.stdout:3/852: creat d4/f112 x:0 0 0 2024-03-14T13:02:31.891 INFO:tasks.workunit.client.1.smithi137.stdout:3/853: dread d4/de/d16/d27/dd3/fc4 [0,4194304] 0 2024-03-14T13:02:31.891 INFO:tasks.workunit.client.1.smithi137.stdout:5/980: unlink d7/d16/d18/d2d/f63 0 2024-03-14T13:02:31.893 INFO:tasks.workunit.client.1.smithi137.stdout:5/981: write d7/d16/d41/d11c/f13d [708386,69527] 0 2024-03-14T13:02:31.894 INFO:tasks.workunit.client.1.smithi137.stdout:3/854: truncate d4/de/d16/d1c/f8f 300496 0 2024-03-14T13:02:31.895 INFO:tasks.workunit.client.1.smithi137.stdout:5/982: creat d7/d16/d18/d25/d28/d9e/de5/d11d/f145 x:0 0 0 2024-03-14T13:02:31.896 INFO:tasks.workunit.client.1.smithi137.stdout:5/983: write d7/d49/f127 [1805077,582] 0 2024-03-14T13:02:31.898 INFO:tasks.workunit.client.1.smithi137.stdout:5/984: read - d7/d16/d18/d25/d28/d3c/f4f zero size 2024-03-14T13:02:31.899 INFO:tasks.workunit.client.1.smithi137.stdout:3/855: dread d4/de/d16/fb9 [0,4194304] 0 2024-03-14T13:02:31.900 INFO:tasks.workunit.client.1.smithi137.stdout:5/985: write d7/d16/d18/d25/d28/d9e/de5/d11d/f145 [23409,100506] 0 2024-03-14T13:02:31.902 INFO:tasks.workunit.client.1.smithi137.stdout:5/986: mkdir d7/d49/d75/d146 0 2024-03-14T13:02:31.903 INFO:tasks.workunit.client.1.smithi137.stdout:5/987: dread - d7/d16/d18/d25/d104/f13c zero size 2024-03-14T13:02:31.905 INFO:tasks.workunit.client.1.smithi137.stdout:5/988: rmdir d7/d36 39 2024-03-14T13:02:31.906 INFO:tasks.workunit.client.1.smithi137.stdout:3/856: link d4/de/d16/d1c/d5d/d85/ldc d4/de/d16/d1c/d5d/l113 0 2024-03-14T13:02:31.907 INFO:tasks.workunit.client.1.smithi137.stdout:3/857: chown d4/de/d16/d1c/d5d/d85/lbb 1 1 2024-03-14T13:02:31.907 INFO:tasks.workunit.client.1.smithi137.stdout:5/989: creat d7/d33/d65/f147 x:0 0 0 2024-03-14T13:02:31.908 INFO:tasks.workunit.client.1.smithi137.stdout:5/990: fdatasync d7/d16/f38 0 2024-03-14T13:02:31.910 INFO:tasks.workunit.client.1.smithi137.stdout:5/991: truncate d7/d16/d18/d25/d28/d9e/fc7 283610 0 2024-03-14T13:02:31.914 INFO:tasks.workunit.client.1.smithi137.stdout:5/992: creat d7/d1d/d74/d83/d126/f148 x:0 0 0 2024-03-14T13:02:31.914 INFO:tasks.workunit.client.1.smithi137.stdout:3/858: write d4/de/d16/d1c/d8d/dcd/fbd [1051391,47749] 0 2024-03-14T13:02:31.915 INFO:tasks.workunit.client.1.smithi137.stdout:3/859: write d4/f112 [138892,90555] 0 2024-03-14T13:02:31.916 INFO:tasks.workunit.client.1.smithi137.stdout:5/993: unlink d7/d33/d65/f10b 0 2024-03-14T13:02:31.920 INFO:tasks.workunit.client.1.smithi137.stdout:3/860: creat d4/de/d16/d1c/d5d/d85/d6b/df7/f114 x:0 0 0 2024-03-14T13:02:31.920 INFO:tasks.workunit.client.1.smithi137.stdout:5/994: rename d7/d16/d18/d25/d28/d3c/f6f to d7/d16/d18/df6/f149 0 2024-03-14T13:02:31.921 INFO:tasks.workunit.client.1.smithi137.stdout:5/995: stat d7/d16/f1a 0 2024-03-14T13:02:32.212 INFO:tasks.workunit.client.1.smithi137.stdout:5/996: sync 2024-03-14T13:02:32.212 INFO:tasks.workunit.client.1.smithi137.stdout:3/861: sync 2024-03-14T13:02:32.213 INFO:tasks.workunit.client.1.smithi137.stdout:3/862: fsync d4/de/dc7/f10e 0 2024-03-14T13:02:32.214 INFO:tasks.workunit.client.1.smithi137.stdout:5/997: rmdir d7/d49/d75 39 2024-03-14T13:02:32.215 INFO:tasks.workunit.client.1.smithi137.stdout:5/998: stat d7/d49/fbd 0 2024-03-14T13:02:32.218 INFO:tasks.workunit.client.1.smithi137.stdout:5/999: chown d7/d16/c70 2952 1 2024-03-14T13:02:32.220 INFO:tasks.workunit.client.1.smithi137.stdout:3/863: dwrite d4/de/d16/d74/f106 [0,4194304] 0 2024-03-14T13:02:32.226 INFO:tasks.workunit.client.1.smithi137.stdout:3/864: unlink d4/de/d16/d1c/f8f 0 2024-03-14T13:02:32.228 INFO:tasks.workunit.client.1.smithi137.stdout:3/865: dread d4/de/d16/fbc [0,4194304] 0 2024-03-14T13:02:32.231 INFO:tasks.workunit.client.1.smithi137.stdout:3/866: creat d4/de/d16/d1c/d5d/de0/f115 x:0 0 0 2024-03-14T13:02:32.239 INFO:tasks.workunit.client.1.smithi137.stdout:3/867: dwrite d4/de/d16/d1c/d5d/d85/d6b/f8b [0,4194304] 0 2024-03-14T13:02:32.248 INFO:tasks.workunit.client.1.smithi137.stdout:3/868: write d4/de/d16/d1c/d5d/d85/d6b/fca [369802,24716] 0 2024-03-14T13:02:32.249 INFO:tasks.workunit.client.1.smithi137.stdout:3/869: chown d4/de/d16/d1c/d5d 50012107 1 2024-03-14T13:02:32.250 INFO:tasks.workunit.client.1.smithi137.stdout:3/870: write d4/de/d16/d1c/d5d/d85/f42 [2013365,47297] 0 2024-03-14T13:02:32.253 INFO:tasks.workunit.client.1.smithi137.stdout:3/871: mkdir d4/de/d116 0 2024-03-14T13:02:32.254 INFO:tasks.workunit.client.1.smithi137.stdout:3/872: fsync d4/de/d16/d1c/d5d/f43 0 2024-03-14T13:02:32.258 INFO:tasks.workunit.client.1.smithi137.stdout:3/873: write d4/d19/f66 [2278352,20865] 0 2024-03-14T13:02:32.261 INFO:tasks.workunit.client.1.smithi137.stdout:3/874: rmdir d4/de/d116 0 2024-03-14T13:02:32.267 INFO:tasks.workunit.client.1.smithi137.stdout:3/875: dwrite d4/de/d16/d1c/db8/ff4 [0,4194304] 0 2024-03-14T13:02:32.270 INFO:tasks.workunit.client.1.smithi137.stdout:3/876: creat d4/f117 x:0 0 0 2024-03-14T13:02:32.271 INFO:tasks.workunit.client.1.smithi137.stdout:3/877: fdatasync d4/df/d82/feb 0 2024-03-14T13:02:32.274 INFO:tasks.workunit.client.1.smithi137.stdout:3/878: creat d4/de/d16/d1c/d5d/d85/d6b/de3/f118 x:0 0 0 2024-03-14T13:02:32.283 INFO:tasks.workunit.client.1.smithi137.stdout:3/879: truncate d4/de/d16/d27/dd3/fc4 3195244 0 2024-03-14T13:02:32.284 INFO:tasks.workunit.client.1.smithi137.stdout:3/880: mkdir d4/de/d16/d1c/d5d/d4b/d119 0 2024-03-14T13:02:32.285 INFO:tasks.workunit.client.1.smithi137.stdout:3/881: write d4/de/d16/d1c/d5d/d85/d6b/ff8 [732189,18854] 0 2024-03-14T13:02:32.290 INFO:tasks.workunit.client.1.smithi137.stdout:3/882: dwrite d4/de/d13/f21 [0,4194304] 0 2024-03-14T13:02:32.292 INFO:tasks.workunit.client.1.smithi137.stdout:3/883: creat d4/df/d82/f11a x:0 0 0 2024-03-14T13:02:32.295 INFO:tasks.workunit.client.1.smithi137.stdout:3/884: creat d4/de/d16/d27/dd3/f11b x:0 0 0 2024-03-14T13:02:32.296 INFO:tasks.workunit.client.1.smithi137.stdout:3/885: write d4/de/d16/d1c/f41 [7209709,100505] 0 2024-03-14T13:02:32.299 INFO:tasks.workunit.client.1.smithi137.stdout:3/886: unlink d4/de/d16/cf0 0 2024-03-14T13:02:32.301 INFO:tasks.workunit.client.1.smithi137.stdout:3/887: mkdir d4/de/d16/d27/dd3/d11c 0 2024-03-14T13:02:32.303 INFO:tasks.workunit.client.1.smithi137.stdout:3/888: rmdir d4/de/d16/d1c/d5d/d85/d6b/de3/d105 39 2024-03-14T13:02:32.311 INFO:tasks.workunit.client.1.smithi137.stdout:3/889: truncate d4/df/fbe 3109744 0 2024-03-14T13:02:32.312 INFO:tasks.workunit.client.1.smithi137.stdout:3/890: stat d4/de/d16/d1c/d5d/de0 0 2024-03-14T13:02:32.313 INFO:tasks.workunit.client.1.smithi137.stdout:3/891: truncate d4/fd 2003238 0 2024-03-14T13:02:32.315 INFO:tasks.workunit.client.1.smithi137.stdout:3/892: unlink d4/de/d13/c15 0 2024-03-14T13:02:32.317 INFO:tasks.workunit.client.1.smithi137.stdout:3/893: truncate d4/de/d16/d1c/d5d/d4b/de6/f103 48962 0 2024-03-14T13:02:32.326 INFO:tasks.workunit.client.1.smithi137.stdout:3/894: dwrite d4/de/d16/f31 [0,4194304] 0 2024-03-14T13:02:32.327 INFO:tasks.workunit.client.1.smithi137.stdout:3/895: chown d4/de/d16/f1a 168811 1 2024-03-14T13:02:32.338 INFO:tasks.workunit.client.1.smithi137.stdout:3/896: dwrite d4/de/d16/d1c/db8/f10d [0,4194304] 0 2024-03-14T13:02:32.363 INFO:tasks.workunit.client.1.smithi137.stdout:3/897: dread d4/de/d16/d1c/d5d/d85/f42 [0,4194304] 0 2024-03-14T13:02:32.363 INFO:tasks.workunit.client.1.smithi137.stdout:3/898: dread - d4/df/d82/f11a zero size 2024-03-14T13:02:32.364 INFO:tasks.workunit.client.1.smithi137.stdout:3/899: read - d4/de/d16/d1c/d5d/de0/f115 zero size 2024-03-14T13:02:32.369 INFO:tasks.workunit.client.1.smithi137.stdout:3/900: getdents d4/de/d16/d27/dd3 0 2024-03-14T13:02:32.370 INFO:tasks.workunit.client.1.smithi137.stdout:3/901: chown d4/de/d16/d1c/d8d/le7 3 1 2024-03-14T13:02:32.375 INFO:tasks.workunit.client.1.smithi137.stdout:3/902: truncate d4/df/f83 4025109 0 2024-03-14T13:02:32.376 INFO:tasks.workunit.client.1.smithi137.stdout:3/903: chown d4/de/d13/l50 1702804 1 2024-03-14T13:02:32.379 INFO:tasks.workunit.client.1.smithi137.stdout:3/904: rename d4/de/d16/d27/dd3/l101 to d4/de/d16/d27/l11d 0 2024-03-14T13:02:32.380 INFO:tasks.workunit.client.1.smithi137.stdout:3/905: chown d4/de/d16/d1c/d5d/de0/lee 4431 1 2024-03-14T13:02:32.380 INFO:tasks.workunit.client.1.smithi137.stdout:3/906: chown d4/de/d16/d1c/db8 1 1 2024-03-14T13:02:32.381 INFO:tasks.workunit.client.1.smithi137.stdout:3/907: stat d4/de/d16/d27/f73 0 2024-03-14T13:02:32.383 INFO:tasks.workunit.client.1.smithi137.stdout:3/908: creat d4/de/d16/d1c/d8d/dcd/f11e x:0 0 0 2024-03-14T13:02:32.384 INFO:tasks.workunit.client.1.smithi137.stdout:3/909: truncate d4/df/d82/feb 1244697 0 2024-03-14T13:02:32.389 INFO:tasks.workunit.client.1.smithi137.stdout:3/910: dwrite d4/de/d16/d1c/d5d/f7c [0,4194304] 0 2024-03-14T13:02:32.393 INFO:tasks.workunit.client.1.smithi137.stdout:3/911: creat d4/de/d16/d1c/d8d/dcd/f11f x:0 0 0 2024-03-14T13:02:32.396 INFO:tasks.workunit.client.1.smithi137.stdout:3/912: unlink d4/de/d16/d1c/d8d/fb4 0 2024-03-14T13:02:32.396 INFO:tasks.workunit.client.1.smithi137.stdout:3/913: rename d4/de to d4/de/d120 22 2024-03-14T13:02:32.397 INFO:tasks.workunit.client.1.smithi137.stdout:3/914: chown d4/de/d16/d1c/d5d/d85/l3b 26247177 1 2024-03-14T13:02:32.399 INFO:tasks.workunit.client.1.smithi137.stdout:3/915: rmdir d4/de/d16/d1c/d5d/d4b/de6 39 2024-03-14T13:02:32.401 INFO:tasks.workunit.client.1.smithi137.stdout:3/916: write d4/de/d16/d1c/db8/f10d [955354,26849] 0 2024-03-14T13:02:32.403 INFO:tasks.workunit.client.1.smithi137.stdout:3/917: write d4/de/d13/f21 [3886788,117493] 0 2024-03-14T13:02:32.405 INFO:tasks.workunit.client.1.smithi137.stdout:3/918: creat d4/de/d16/d1c/d5d/f121 x:0 0 0 2024-03-14T13:02:32.406 INFO:tasks.workunit.client.1.smithi137.stdout:3/919: creat d4/df/d82/f122 x:0 0 0 2024-03-14T13:02:32.410 INFO:tasks.workunit.client.1.smithi137.stdout:3/920: mknod d4/de/d16/d1c/d5d/d85/d6b/df7/c123 0 2024-03-14T13:02:32.412 INFO:tasks.workunit.client.1.smithi137.stdout:3/921: dread d4/de/d16/d1c/d5d/d4b/de6/f103 [0,4194304] 0 2024-03-14T13:02:32.413 INFO:tasks.workunit.client.1.smithi137.stdout:3/922: dread - d4/de/d16/d1c/d8d/dcd/f11e zero size 2024-03-14T13:02:32.415 INFO:tasks.workunit.client.1.smithi137.stdout:3/923: write d4/de/d16/d1c/d5d/d85/d6b/de3/f118 [147226,82305] 0 2024-03-14T13:02:32.417 INFO:tasks.workunit.client.1.smithi137.stdout:3/924: write d4/de/d13/f21 [136317,8637] 0 2024-03-14T13:02:32.419 INFO:tasks.workunit.client.1.smithi137.stdout:3/925: symlink d4/de/d16/d1c/db8/l124 0 2024-03-14T13:02:32.420 INFO:tasks.workunit.client.1.smithi137.stdout:3/926: stat d4/de/d16/d1c/f41 0 2024-03-14T13:02:32.421 INFO:tasks.workunit.client.1.smithi137.stdout:3/927: write d4/de/d16/d1c/db8/dd1/fdd [4708467,110277] 0 2024-03-14T13:02:32.424 INFO:tasks.workunit.client.1.smithi137.stdout:3/928: creat d4/de/d16/d1c/d5d/d85/d6b/de3/d105/f125 x:0 0 0 2024-03-14T13:02:32.430 INFO:tasks.workunit.client.1.smithi137.stdout:3/929: write d4/de/f2a [1462457,85191] 0 2024-03-14T13:02:32.432 INFO:tasks.workunit.client.1.smithi137.stdout:3/930: symlink d4/de/d16/d74/l126 0 2024-03-14T13:02:32.437 INFO:tasks.workunit.client.1.smithi137.stdout:3/931: truncate d4/de/d16/d1c/d8d/dcd/f8e 4003267 0 2024-03-14T13:02:32.438 INFO:tasks.workunit.client.1.smithi137.stdout:3/932: fdatasync d4/de/d16/d1c/d8d/dcd/fbd 0 2024-03-14T13:02:32.444 INFO:tasks.workunit.client.1.smithi137.stdout:3/933: dwrite d4/de/d13/f61 [4194304,4194304] 0 2024-03-14T13:02:32.452 INFO:tasks.workunit.client.1.smithi137.stdout:3/934: dwrite d4/de/d16/d1c/d5d/f43 [4194304,4194304] 0 2024-03-14T13:02:32.456 INFO:tasks.workunit.client.1.smithi137.stdout:3/935: symlink d4/de/d16/d1c/d5d/d4b/df6/l127 0 2024-03-14T13:02:32.459 INFO:tasks.workunit.client.1.smithi137.stdout:3/936: mknod d4/de/d16/d1c/d5d/d4b/d119/c128 0 2024-03-14T13:02:32.461 INFO:tasks.workunit.client.1.smithi137.stdout:3/937: creat d4/de/d16/d1c/d5d/de0/f129 x:0 0 0 2024-03-14T13:02:32.474 INFO:tasks.workunit.client.1.smithi137.stdout:3/938: dwrite d4/de/d16/d1c/d8d/dcd/fc6 [0,4194304] 0 2024-03-14T13:02:32.483 INFO:tasks.workunit.client.1.smithi137.stdout:3/939: dwrite d4/de/d16/f1a [0,4194304] 0 2024-03-14T13:02:32.484 INFO:tasks.workunit.client.1.smithi137.stdout:3/940: write d4/d19/f70 [298600,107463] 0 2024-03-14T13:02:32.485 INFO:tasks.workunit.client.1.smithi137.stdout:3/941: chown d4/df/d82/f11a 0 1 2024-03-14T13:02:32.485 INFO:tasks.workunit.client.1.smithi137.stdout:3/942: readlink d4/df/le2 0 2024-03-14T13:02:32.489 INFO:tasks.workunit.client.1.smithi137.stdout:3/943: creat d4/de/f12a x:0 0 0 2024-03-14T13:02:32.491 INFO:tasks.workunit.client.1.smithi137.stdout:3/944: symlink d4/de/d16/d1c/d5d/d85/l12b 0 2024-03-14T13:02:32.493 INFO:tasks.workunit.client.1.smithi137.stdout:3/945: symlink d4/de/d16/d27/l12c 0 2024-03-14T13:02:32.495 INFO:tasks.workunit.client.1.smithi137.stdout:3/946: unlink d4/de/d16/c26 0 2024-03-14T13:02:32.497 INFO:tasks.workunit.client.1.smithi137.stdout:3/947: symlink d4/de/d16/d1c/l12d 0 2024-03-14T13:02:32.498 INFO:tasks.workunit.client.1.smithi137.stdout:3/948: stat d4/de/d16/d27/df9 0 2024-03-14T13:02:32.499 INFO:tasks.workunit.client.1.smithi137.stdout:3/949: creat d4/de/d16/d1c/d5d/de0/f12e x:0 0 0 2024-03-14T13:02:32.501 INFO:tasks.workunit.client.1.smithi137.stdout:3/950: mkdir d4/de/d16/d27/dd3/d11c/d12f 0 2024-03-14T13:02:32.512 INFO:tasks.workunit.client.1.smithi137.stdout:3/951: dwrite d4/de/f1f [0,4194304] 0 2024-03-14T13:02:32.516 INFO:tasks.workunit.client.1.smithi137.stdout:3/952: rename d4/de/d16/d27/fce to d4/de/f130 0 2024-03-14T13:02:32.517 INFO:tasks.workunit.client.1.smithi137.stdout:3/953: creat d4/df/d82/f131 x:0 0 0 2024-03-14T13:02:32.520 INFO:tasks.workunit.client.1.smithi137.stdout:3/954: rename d4/df/c76 to d4/de/d16/d1c/d5d/d85/d6b/c132 0 2024-03-14T13:02:32.522 INFO:tasks.workunit.client.1.smithi137.stdout:3/955: creat d4/de/d93/f133 x:0 0 0 2024-03-14T13:02:32.528 INFO:tasks.workunit.client.1.smithi137.stdout:3/956: dwrite d4/de/d16/d27/dd3/f11b [0,4194304] 0 2024-03-14T13:02:32.531 INFO:tasks.workunit.client.1.smithi137.stdout:3/957: getdents d4/de/d16/d1c/db8 0 2024-03-14T13:02:32.534 INFO:tasks.workunit.client.1.smithi137.stdout:3/958: truncate d4/de/d16/d1c/d8d/dcd/fba 1860362 0 2024-03-14T13:02:32.536 INFO:tasks.workunit.client.1.smithi137.stdout:3/959: dread d4/de/d16/fb9 [0,4194304] 0 2024-03-14T13:02:32.538 INFO:tasks.workunit.client.1.smithi137.stdout:3/960: mkdir d4/de/d16/d74/d134 0 2024-03-14T13:02:32.540 INFO:tasks.workunit.client.1.smithi137.stdout:3/961: symlink d4/de/d16/d27/dd3/d11c/d12f/l135 0 2024-03-14T13:02:32.542 INFO:tasks.workunit.client.1.smithi137.stdout:3/962: fsync d4/d19/f90 0 2024-03-14T13:02:32.549 INFO:tasks.workunit.client.1.smithi137.stdout:3/963: write d4/df/fbe [1927945,79150] 0 2024-03-14T13:02:32.552 INFO:tasks.workunit.client.1.smithi137.stdout:3/964: truncate d4/de/d16/d1c/d5d/d85/d6b/de3/f10b 950842 0 2024-03-14T13:02:32.552 INFO:tasks.workunit.client.1.smithi137.stdout:3/965: chown d4/d19/f66 2956443 1 2024-03-14T13:02:32.554 INFO:tasks.workunit.client.1.smithi137.stdout:3/966: chown d4/de/d16/d1c/d5d/l113 41828 1 2024-03-14T13:02:32.557 INFO:tasks.workunit.client.1.smithi137.stdout:3/967: write d4/de/d13/f9f [831012,27247] 0 2024-03-14T13:02:32.560 INFO:tasks.workunit.client.1.smithi137.stdout:3/968: symlink d4/de/d16/d1c/d5d/d85/d6b/df7/l136 0 2024-03-14T13:02:32.567 INFO:tasks.workunit.client.1.smithi137.stdout:3/969: dwrite d4/de/d16/fb9 [0,4194304] 0 2024-03-14T13:02:32.569 INFO:tasks.workunit.client.1.smithi137.stdout:3/970: write d4/de/d16/d1c/db8/dd1/fe4 [805591,85484] 0 2024-03-14T13:02:32.572 INFO:tasks.workunit.client.1.smithi137.stdout:3/971: creat d4/de/d16/d1c/d8d/f137 x:0 0 0 2024-03-14T13:02:32.577 INFO:tasks.workunit.client.1.smithi137.stdout:3/972: link d4/de/fcb d4/de/d16/f138 0 2024-03-14T13:02:32.579 INFO:tasks.workunit.client.1.smithi137.stdout:3/973: mknod d4/de/d16/d27/dd3/d11c/c139 0 2024-03-14T13:02:32.580 INFO:tasks.workunit.client.1.smithi137.stdout:3/974: dread - d4/de/d16/d1c/d5d/de0/f115 zero size 2024-03-14T13:02:32.580 INFO:tasks.workunit.client.1.smithi137.stdout:3/975: write d4/de/d13/f21 [1784484,58135] 0 2024-03-14T13:02:32.587 INFO:tasks.workunit.client.1.smithi137.stdout:3/976: dwrite d4/de/d16/d1c/d8d/dcd/f11e [0,4194304] 0 2024-03-14T13:02:32.590 INFO:tasks.workunit.client.1.smithi137.stdout:3/977: symlink d4/de/d16/d1c/d5d/d85/d6b/l13a 0 2024-03-14T13:02:32.591 INFO:tasks.workunit.client.1.smithi137.stdout:3/978: stat d4/de/d16/d1c/d8d 0 2024-03-14T13:02:32.592 INFO:tasks.workunit.client.1.smithi137.stdout:3/979: write d4/de/d13/f21 [2494820,11761] 0 2024-03-14T13:02:32.593 INFO:tasks.workunit.client.1.smithi137.stdout:3/980: truncate d4/df/f96 5115169 0 2024-03-14T13:02:32.600 INFO:tasks.workunit.client.1.smithi137.stdout:3/981: dwrite d4/de/d93/f133 [0,4194304] 0 2024-03-14T13:02:32.606 INFO:tasks.workunit.client.1.smithi137.stdout:3/982: link d4/f112 d4/de/d16/d1c/d5d/d85/d6b/de3/f13b 0 2024-03-14T13:02:32.607 INFO:tasks.workunit.client.1.smithi137.stdout:3/983: fdatasync d4/d19/f70 0 2024-03-14T13:02:32.609 INFO:tasks.workunit.client.1.smithi137.stdout:3/984: fdatasync d4/de/d16/d1c/d8d/dcd/fba 0 2024-03-14T13:02:32.612 INFO:tasks.workunit.client.1.smithi137.stdout:3/985: symlink d4/de/d16/d1c/d5d/d85/d6b/l13c 0 2024-03-14T13:02:32.613 INFO:tasks.workunit.client.1.smithi137.stdout:3/986: mknod d4/de/d16/d74/d134/c13d 0 2024-03-14T13:02:32.619 INFO:tasks.workunit.client.1.smithi137.stdout:3/987: dwrite d4/de/d16/d1c/d5d/f43 [8388608,4194304] 0 2024-03-14T13:02:32.624 INFO:tasks.workunit.client.1.smithi137.stdout:3/988: creat d4/de/d16/d1c/d8d/dcd/f13e x:0 0 0 2024-03-14T13:02:32.633 INFO:tasks.workunit.client.1.smithi137.stdout:3/989: rmdir d4/de/d16/d74 39 2024-03-14T13:02:32.634 INFO:tasks.workunit.client.1.smithi137.stdout:3/990: chown d4/de/d16/d1c/d5d/d85/cc5 1 1 2024-03-14T13:02:32.635 INFO:tasks.workunit.client.1.smithi137.stdout:3/991: fsync d4/de/d16/d1c/d8d/dcd/fba 0 2024-03-14T13:02:32.638 INFO:tasks.workunit.client.1.smithi137.stdout:3/992: unlink d4/de/f3c 0 2024-03-14T13:02:32.642 INFO:tasks.workunit.client.1.smithi137.stdout:3/993: rename d4/de/f130 to d4/de/d16/d1c/d8d/dcd/f13f 0 2024-03-14T13:02:32.646 INFO:tasks.workunit.client.1.smithi137.stdout:3/994: write d4/df/d82/fc3 [3945353,81136] 0 2024-03-14T13:02:32.648 INFO:tasks.workunit.client.1.smithi137.stdout:3/995: mkdir d4/de/d16/d1c/db8/dd1/d140 0 2024-03-14T13:02:32.651 INFO:tasks.workunit.client.1.smithi137.stdout:3/996: rename d4/de/d16/d1c/d5d/d85/d6b/f8a to d4/de/d16/d1c/d5d/d85/d6b/f141 0 2024-03-14T13:02:32.652 INFO:tasks.workunit.client.1.smithi137.stdout:3/997: write d4/de/d16/d27/dd3/fab [810265,116039] 0 2024-03-14T13:02:32.657 INFO:tasks.workunit.client.1.smithi137.stdout:3/998: write d4/de/d16/d1c/d5d/d85/d6b/de3/f13b [810512,76597] 0 2024-03-14T13:02:32.660 INFO:tasks.workunit.client.1.smithi137.stdout:3/999: creat d4/de/d16/d1c/f142 x:0 0 0 2024-03-14T13:02:32.663 INFO:tasks.workunit.client.1.smithi137.stderr:+ rm -rf -- ./tmp.CKF3lbLY2Q 2024-03-14T13:02:33.977 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: pgmap v31: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 124 MiB/s rd, 299 MiB/s wr, 641 op/s 2024-03-14T13:02:33.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: Upgrade: Updating mgr.smithi137.xbyzbp 2024-03-14T13:02:33.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:02:33.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:02:33.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:02:33.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:33 smithi137 ceph-mon[50336]: Deploying daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: pgmap v31: 65 pgs: 65 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 124 MiB/s rd, 299 MiB/s wr, 641 op/s 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: Upgrade: Updating mgr.smithi137.xbyzbp 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:02:34.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:33 smithi111 ceph-mon[39688]: Deploying daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:02:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:35 smithi137 ceph-mon[50336]: pgmap v32: 65 pgs: 65 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 138 MiB/s rd, 345 MiB/s wr, 730 op/s 2024-03-14T13:02:36.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:35 smithi111 ceph-mon[39688]: pgmap v32: 65 pgs: 65 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 138 MiB/s rd, 345 MiB/s wr, 730 op/s 2024-03-14T13:02:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:37 smithi111 ceph-mon[39688]: pgmap v33: 65 pgs: 65 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 90 MiB/s rd, 236 MiB/s wr, 480 op/s 2024-03-14T13:02:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:37.948 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:37.948 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:37.948 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:37.948 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:37 smithi137 ceph-mon[50336]: pgmap v33: 65 pgs: 65 active+clean; 4.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 90 MiB/s rd, 236 MiB/s wr, 480 op/s 2024-03-14T13:02:37.948 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:39.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:39 smithi111 ceph-mon[39688]: pgmap v34: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 91 MiB/s rd, 237 MiB/s wr, 594 op/s 2024-03-14T13:02:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:39 smithi137 ceph-mon[50336]: pgmap v34: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 91 MiB/s rd, 237 MiB/s wr, 594 op/s 2024-03-14T13:02:41.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:41 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:41 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:41 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:41 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:41 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:41 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:41 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:41.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:41 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:42.143 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:42 smithi137 ceph-mon[50336]: pgmap v35: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 37 MiB/s rd, 97 MiB/s wr, 313 op/s 2024-03-14T13:02:42.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:42 smithi111 ceph-mon[39688]: pgmap v35: 65 pgs: 65 active+clean; 3.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 37 MiB/s rd, 97 MiB/s wr, 313 op/s 2024-03-14T13:02:43.043 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-14T13:02:43.043 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-14T13:02:43.860 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:43.860 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:43 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:43.860 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:43 smithi137 ceph-mon[50336]: pgmap v36: 65 pgs: 65 active+clean; 2.7 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 37 MiB/s rd, 98 MiB/s wr, 351 op/s 2024-03-14T13:02:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:43 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:43 smithi111 ceph-mon[39688]: pgmap v36: 65 pgs: 65 active+clean; 2.7 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 37 MiB/s rd, 98 MiB/s wr, 351 op/s 2024-03-14T13:02:45.428 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-14T13:02:45.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: pgmap v37: 65 pgs: 65 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 74 MiB/s wr, 350 op/s 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.atysez"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.atysez"}]': finished 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.xbyzbp"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.xbyzbp"}]': finished 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:45 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: pgmap v37: 65 pgs: 65 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 26 MiB/s rd, 74 MiB/s wr, 350 op/s 2024-03-14T13:02:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:02:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.atysez"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi111.atysez"}]': finished 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.xbyzbp"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.xbyzbp"}]': finished 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:45 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.988 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all mgr 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all rgw 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all rbd-mirror 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all ceph-exporter 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all iscsi 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all nfs 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all nvmeof 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Standby manager daemon smithi137.xbyzbp restarted 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:02:46.989 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:02:46.990 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:02:46.990 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:46 smithi111 ceph-mon[39688]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:02:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all mgr 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all rgw 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all rbd-mirror 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all ceph-exporter 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all iscsi 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all nfs 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all nvmeof 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp restarted 2024-03-14T13:02:47.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:02:47.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:02:47.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:02:47.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:02:47.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:46 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1594252362' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:02:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:47 smithi111 ceph-mon[39688]: pgmap v38: 65 pgs: 65 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 222 op/s 2024-03-14T13:02:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:47 smithi111 ceph-mon[39688]: mgrmap e32: smithi111.atysez(active, since 70s), standbys: smithi137.xbyzbp 2024-03-14T13:02:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:47 smithi137 ceph-mon[50336]: pgmap v38: 65 pgs: 65 active+clean; 1.4 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 222 op/s 2024-03-14T13:02:48.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:47 smithi137 ceph-mon[50336]: mgrmap e32: smithi111.atysez(active, since 70s), standbys: smithi137.xbyzbp 2024-03-14T13:02:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:48 smithi137 ceph-mon[50336]: Upgrade: Updating node-exporter.smithi111 (1/2) 2024-03-14T13:02:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:48 smithi137 ceph-mon[50336]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-14T13:02:49.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:48 smithi111 ceph-mon[39688]: Upgrade: Updating node-exporter.smithi111 (1/2) 2024-03-14T13:02:49.102 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:48 smithi111 ceph-mon[39688]: Deploying daemon node-exporter.smithi111 on smithi111 2024-03-14T13:02:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:49 smithi137 ceph-mon[50336]: pgmap v39: 65 pgs: 65 active+clean; 280 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 377 op/s 2024-03-14T13:02:50.075 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:49 smithi111 ceph-mon[39688]: pgmap v39: 65 pgs: 65 active+clean; 280 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 377 op/s 2024-03-14T13:02:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:51 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:51 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:51 smithi137 ceph-mon[50336]: pgmap v40: 65 pgs: 65 active+clean; 280 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 262 op/s 2024-03-14T13:02:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:51 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:51 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:02:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:51 smithi111 ceph-mon[39688]: pgmap v40: 65 pgs: 65 active+clean; 280 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 262 op/s 2024-03-14T13:02:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:52 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:52 smithi137 ceph-mon[50336]: Upgrade: Updating node-exporter.smithi137 (2/2) 2024-03-14T13:02:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:52 smithi137 ceph-mon[50336]: Deploying daemon node-exporter.smithi137 on smithi137 2024-03-14T13:02:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:52 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:02:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:52 smithi111 ceph-mon[39688]: Upgrade: Updating node-exporter.smithi137 (2/2) 2024-03-14T13:02:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:52 smithi111 ceph-mon[39688]: Deploying daemon node-exporter.smithi137 on smithi137 2024-03-14T13:02:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:53 smithi137 ceph-mon[50336]: pgmap v41: 65 pgs: 65 active+clean; 279 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 307 op/s 2024-03-14T13:02:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:53 smithi111 ceph-mon[39688]: pgmap v41: 65 pgs: 65 active+clean; 279 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 307 op/s 2024-03-14T13:02:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:55 smithi137 ceph-mon[50336]: pgmap v42: 65 pgs: 65 active+clean; 284 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 451 op/s 2024-03-14T13:02:56.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:55 smithi111 ceph-mon[39688]: pgmap v42: 65 pgs: 65 active+clean; 284 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 451 op/s 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 -- 172.21.15.111:0/2112597696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 msgr2=0x7f927c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2112597696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c06e460 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9274009ce0 tx=0x7f9274004d10 comp rx=0 tx=0).stop 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 -- 172.21.15.111:0/2112597696 shutdown_connections 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2112597696 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 0x7f927c06ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2112597696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c06e460 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.317+0000 7f9280ca8700 1 -- 172.21.15.111:0/2112597696 >> 172.21.15.111:0/2112597696 conn(0x7f927c069de0 msgr2=0x7f927c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.320+0000 7f9280ca8700 1 -- 172.21.15.111:0/2112597696 shutdown_connections 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.320+0000 7f9280ca8700 1 -- 172.21.15.111:0/2112597696 wait complete. 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 Processor -- start 2024-03-14T13:02:56.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 -- start start 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 0x7f927c11c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c11d1c0 con 0x7f927c06f9d0 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.321+0000 7f9280ca8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c11d330 con 0x7f927c06e9a0 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55798/0 (socket says 172.21.15.111:55798) 2024-03-14T13:02:56.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 -- 172.21.15.111:0/2621933494 learned_addr learned my addr 172.21.15.111:0/2621933494 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f927a59c700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 0x7f927c11c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 -- 172.21.15.111:0/2621933494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 msgr2=0x7f927c11c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 0x7f927c11c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 -- 172.21.15.111:0/2621933494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9274009940 con 0x7f927c06f9d0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.322+0000 7f9279d9b700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f926c00eb40 tx=0x7f926c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.323+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926c007cc0 con 0x7f927c06f9d0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.323+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f926c00f040 con 0x7f927c06f9d0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.323+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c121cc0 con 0x7f927c06f9d0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.323+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f926c00c990 con 0x7f927c06f9d0 2024-03-14T13:02:56.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.323+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c122280 con 0x7f927c06f9d0 2024-03-14T13:02:56.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.324+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c04c710 con 0x7f927c06f9d0 2024-03-14T13:02:56.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.326+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 32) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f926c00caf0 con 0x7f927c06f9d0 2024-03-14T13:02:56.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.326+0000 7f926affd700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 0x7f926406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.327+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f926c08b2c0 con 0x7f927c06f9d0 2024-03-14T13:02:56.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.327+0000 7f927a59c700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 0x7f926406ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.327+0000 7f927a59c700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 0x7f926406ebf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92740094f0 tx=0x7f9274004d10 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:56.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.333+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f926c059360 con 0x7f927c06f9d0 2024-03-14T13:02:56.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.637+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f927c108200 con 0x7f926406c740 2024-03-14T13:02:56.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.639+0000 7f926affd700 1 -- 172.21.15.111:0/2621933494 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f927c108200 con 0x7f926406c740 2024-03-14T13:02:56.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 msgr2=0x7f926406ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 0x7f926406ebf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92740094f0 tx=0x7f9274004d10 comp rx=0 tx=0).stop 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 msgr2=0x7f927c11cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f926c00eb40 tx=0x7f926c00b040 comp rx=0 tx=0).stop 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 shutdown_connections 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f927c06e9a0 0x7f927c11c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f926406c740 0x7f926406ebf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 --2- 172.21.15.111:0/2621933494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f927c06f9d0 0x7f927c11cbc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 >> 172.21.15.111:0/2621933494 conn(0x7f927c069de0 msgr2=0x7f927c06b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 shutdown_connections 2024-03-14T13:02:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.641+0000 7f9280ca8700 1 -- 172.21.15.111:0/2621933494 wait complete. 2024-03-14T13:02:56.653 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:02:56.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 -- 172.21.15.111:0/875197850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fbd00 msgr2=0x7f7ff80fc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/875197850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fbd00 0x7f7ff80fc110 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8004940 tx=0x7f7fe8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:56.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 -- 172.21.15.111:0/875197850 shutdown_connections 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/875197850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/875197850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fbd00 0x7f7ff80fc110 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 -- 172.21.15.111:0/875197850 >> 172.21.15.111:0/875197850 conn(0x7f7ff80f7480 msgr2=0x7f7ff80f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 -- 172.21.15.111:0/875197850 shutdown_connections 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.751+0000 7f7ffccd1700 1 -- 172.21.15.111:0/875197850 wait complete. 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 Processor -- start 2024-03-14T13:02:56.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 -- start start 2024-03-14T13:02:56.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff810dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff810e380 con 0x7f7ff80fcfb0 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ffccd1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff810e4f0 con 0x7f7ff80fbd00 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ff659c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ff5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff810dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ff659c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:36570/0 (socket says 172.21.15.111:36570) 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ff5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff810dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55820/0 (socket says 172.21.15.111:55820) 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.752+0000 7f7ff659c700 1 -- 172.21.15.111:0/1971280008 learned_addr learned my addr 172.21.15.111:0/1971280008 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ff659c700 1 -- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 msgr2=0x7f7ff810dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ff659c700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff810dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ff659c700 1 -- 172.21.15.111:0/1971280008 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8005040 con 0x7f7ff80fbd00 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ff659c700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8007d40 tx=0x7f7fe8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fe800e030 con 0x7f7ff80fbd00 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff8112ea0 con 0x7f7ff80fbd00 2024-03-14T13:02:56.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fe801c400 con 0x7f7ff80fbd00 2024-03-14T13:02:56.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fe800b650 con 0x7f7ff80fbd00 2024-03-14T13:02:56.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.753+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff81133d0 con 0x7f7ff80fbd00 2024-03-14T13:02:56.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.754+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8101b40 con 0x7f7ff80fbd00 2024-03-14T13:02:56.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.755+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 32) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f7fe801c570 con 0x7f7ff80fbd00 2024-03-14T13:02:56.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.755+0000 7f7fe6ffd700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 0x7f7fe006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:56.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.756+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7fe808cbd0 con 0x7f7ff80fbd00 2024-03-14T13:02:56.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.756+0000 7f7ff5d9b700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 0x7f7fe006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:56.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.756+0000 7f7ff5d9b700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 0x7f7fe006ea20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009b80 tx=0x7f7fec00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:56.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:56.761+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f7fe805abe0 con 0x7f7ff80fbd00 2024-03-14T13:02:57.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.081+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ff802b4d0 con 0x7f7fe006c570 2024-03-14T13:02:57.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.082+0000 7f7fe6ffd700 1 -- 172.21.15.111:0/1971280008 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f7ff802b4d0 con 0x7f7fe006c570 2024-03-14T13:02:57.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 msgr2=0x7f7fe006ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 0x7f7fe006ea20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7fec009b80 tx=0x7f7fec00b040 comp rx=0 tx=0).stop 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 msgr2=0x7f7ff810d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8007d40 tx=0x7f7fe8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 shutdown_connections 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ff80fbd00 0x7f7ff810d840 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7fe006c570 0x7f7fe006ea20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 --2- 172.21.15.111:0/1971280008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ff80fcfb0 0x7f7ff810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 >> 172.21.15.111:0/1971280008 conn(0x7f7ff80f7480 msgr2=0x7f7ff80668e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 shutdown_connections 2024-03-14T13:02:57.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.085+0000 7f7ffccd1700 1 -- 172.21.15.111:0/1971280008 wait complete. 2024-03-14T13:02:57.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 -- 172.21.15.111:0/506332518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 msgr2=0x7f4cc81023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/506332518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc81023a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8004940 tx=0x7f4cb8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 -- 172.21.15.111:0/506332518 shutdown_connections 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/506332518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc81023a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/506332518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 -- 172.21.15.111:0/506332518 >> 172.21.15.111:0/506332518 conn(0x7f4cc80f7430 msgr2=0x7f4cc80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 -- 172.21.15.111:0/506332518 shutdown_connections 2024-03-14T13:02:57.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 -- 172.21.15.111:0/506332518 wait complete. 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.207+0000 7f4cd02a0700 1 Processor -- start 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cd02a0700 1 -- start start 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cd02a0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80fd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cd02a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cd02a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80fbf30 con 0x7f4cc80fffc0 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cd02a0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc80fc0a0 con 0x7f4cc8066220 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4ccd83b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cce03c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80fd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4ccd83b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55836/0 (socket says 172.21.15.111:55836) 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4ccd83b700 1 -- 172.21.15.111:0/3219368066 learned_addr learned my addr 172.21.15.111:0/3219368066 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cce03c700 1 -- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 msgr2=0x7f4cc80fb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cce03c700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cce03c700 1 -- 172.21.15.111:0/3219368066 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8005040 con 0x7f4cc8066220 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4ccd83b700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.208+0000 7f4cce03c700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80fd4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8065f60 tx=0x7f4cc400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.209+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc4004e90 con 0x7f4cc8066220 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.209+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc80fc320 con 0x7f4cc8066220 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.209+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cc400f040 con 0x7f4cc8066220 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.209+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cc400c890 con 0x7f4cc8066220 2024-03-14T13:02:57.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.209+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc8115920 con 0x7f4cc8066220 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.210+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc805f5f0 con 0x7f4cc8066220 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.210+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 32) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f4cc400cb00 con 0x7f4cc8066220 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.210+0000 7f4cbeffd700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 0x7f4cb406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.210+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f4cc408b230 con 0x7f4cc8066220 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.210+0000 7f4ccd83b700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 0x7f4cb406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.211+0000 7f4ccd83b700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 0x7f4cb406ea10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8004890 tx=0x7f4cb8009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:57.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.216+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f4cc4055740 con 0x7f4cc8066220 2024-03-14T13:02:57.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.516+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cc80f8f70 con 0x7f4cb406c560 2024-03-14T13:02:57.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.532+0000 7f4cbeffd700 1 -- 172.21.15.111:0/3219368066 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f4cc80f8f70 con 0x7f4cb406c560 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (2m) 15s ago 10m 17.3M - 0.25.0 c8568f914cd2 70fc8e769dee 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (11m) 15s ago 10m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (9m) 17s ago 9m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (2m) 15s ago 10m 82.7M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (4m) 15s ago 4m 79.2M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (4m) 15s ago 4m 146M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (4m) 17s ago 4m 91.8M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (4m) 17s ago 4m 151M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:02:57.532 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (105s) 15s ago 12m 490M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (21s) 17s ago 9m 48.8M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (12m) 15s ago 12m 54.9M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (9m) 17s ago 9m 42.5M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 starting - - - - 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (111s) 17s ago 9m 18.0M - 1.5.0 0da6a335fe13 37967b629a62 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (8m) 15s ago 8m 193M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (8m) 15s ago 8m 215M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (7m) 15s ago 7m 182M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (7m) 17s ago 7m 243M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (6m) 17s ago 6m 204M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (6m) 17s ago 6m 218M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:02:57.533 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (32s) 15s ago 9m 46.7M - 2.43.0 a07b618ecd1d 86c82831b856 2024-03-14T13:02:57.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 msgr2=0x7f4cb406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 0x7f4cb406ea10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8004890 tx=0x7f4cb8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:57.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 msgr2=0x7f4cc80fd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80fd4b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8065f60 tx=0x7f4cc400b040 comp rx=0 tx=0).stop 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 shutdown_connections 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4cc8066220 0x7f4cc80fd4b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f4cb406c560 0x7f4cb406ea10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 --2- 172.21.15.111:0/3219368066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc80fffc0 0x7f4cc80fb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 >> 172.21.15.111:0/3219368066 conn(0x7f4cc80f7430 msgr2=0x7f4cc80f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 shutdown_connections 2024-03-14T13:02:57.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.536+0000 7f4cd02a0700 1 -- 172.21.15.111:0/3219368066 wait complete. 2024-03-14T13:02:57.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 -- 172.21.15.111:0/2636937024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 msgr2=0x7f7e540fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/2636937024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e540fd3e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004940 tx=0x7f7e40009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:57.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 -- 172.21.15.111:0/2636937024 shutdown_connections 2024-03-14T13:02:57.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/2636937024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e540fd3e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/2636937024 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 0x7f7e540fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 -- 172.21.15.111:0/2636937024 >> 172.21.15.111:0/2636937024 conn(0x7f7e540f74e0 msgr2=0x7f7e540f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 -- 172.21.15.111:0/2636937024 shutdown_connections 2024-03-14T13:02:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.661+0000 7f7e5ad78700 1 -- 172.21.15.111:0/2636937024 wait complete. 2024-03-14T13:02:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 Processor -- start 2024-03-14T13:02:57.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 -- start start 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 0x7f7e5410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5410e430 con 0x7f7e540fcf70 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.662+0000 7f7e5ad78700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5410ec30 con 0x7f7e540fbe00 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55852/0 (socket says 172.21.15.111:55852) 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e58b14700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 0x7f7e5410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 -- 172.21.15.111:0/726482586 learned_addr learned my addr 172.21.15.111:0/726482586 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 -- 172.21.15.111:0/726482586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 msgr2=0x7f7e5410d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 0x7f7e5410d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 -- 172.21.15.111:0/726482586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40005040 con 0x7f7e540fcf70 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e53fff700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007d40 tx=0x7f7e40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e4000e030 con 0x7f7e540fcf70 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5410eeb0 con 0x7f7e540fcf70 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e4001c400 con 0x7f7e540fcf70 2024-03-14T13:02:57.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e4000b650 con 0x7f7e540fcf70 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.663+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5410f3f0 con 0x7f7e540fcf70 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.664+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e54101b00 con 0x7f7e540fcf70 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.665+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 32) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f7e4001c570 con 0x7f7e540fcf70 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.665+0000 7f7e517fa700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 0x7f7e4406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.665+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7e4008cbd0 con 0x7f7e540fcf70 2024-03-14T13:02:57.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.666+0000 7f7e58b14700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 0x7f7e4406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:57.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.666+0000 7f7e58b14700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 0x7f7e4406ea10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009b80 tx=0x7f7e4800b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:57.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:57.671+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f7e4005abe0 con 0x7f7e540fcf70 2024-03-14T13:02:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:57 smithi111 ceph-mon[39688]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:57 smithi111 ceph-mon[39688]: pgmap v43: 65 pgs: 65 active+clean; 284 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 381 op/s 2024-03-14T13:02:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:57 smithi111 ceph-mon[39688]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:57.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:57 smithi137 ceph-mon[50336]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:57.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:57 smithi137 ceph-mon[50336]: pgmap v43: 65 pgs: 65 active+clean; 284 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 381 op/s 2024-03-14T13:02:57.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:57 smithi137 ceph-mon[50336]: from='client.24481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:58.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.010+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7e5404bed0 con 0x7f7e540fcf70 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.010+0000 7f7e517fa700 1 -- 172.21.15.111:0/726482586 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f7e40004490 con 0x7f7e540fcf70 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:02:58.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:02:58.011 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 msgr2=0x7f7e4406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 0x7f7e4406ea10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009b80 tx=0x7f7e4800b040 comp rx=0 tx=0).stop 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 msgr2=0x7f7e5410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7e40007d40 tx=0x7f7e40009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 shutdown_connections 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e540fbe00 0x7f7e5410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7e4406c560 0x7f7e4406ea10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 --2- 172.21.15.111:0/726482586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e540fcf70 0x7f7e5410de30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 >> 172.21.15.111:0/726482586 conn(0x7f7e540f74e0 msgr2=0x7f7e54066470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:58.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 shutdown_connections 2024-03-14T13:02:58.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.012+0000 7f7e5ad78700 1 -- 172.21.15.111:0/726482586 wait complete. 2024-03-14T13:02:58.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 -- 172.21.15.111:0/76315621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fdbe0 msgr2=0x7f45040fc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 --2- 172.21.15.111:0/76315621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fdbe0 0x7f45040fc000 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f44f8004940 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 -- 172.21.15.111:0/76315621 shutdown_connections 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 --2- 172.21.15.111:0/76315621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fdbe0 0x7f45040fc000 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 --2- 172.21.15.111:0/76315621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fd290 0x7f45040fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.143+0000 7f450a11c700 1 -- 172.21.15.111:0/76315621 >> 172.21.15.111:0/76315621 conn(0x7f45040f6ba0 msgr2=0x7f45040f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 -- 172.21.15.111:0/76315621 shutdown_connections 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 -- 172.21.15.111:0/76315621 wait complete. 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 Processor -- start 2024-03-14T13:02:58.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 -- start start 2024-03-14T13:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 0x7f4504112af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:58.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.144+0000 7f450a11c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504113080 con 0x7f45040fd290 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f450a11c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45041131f0 con 0x7f45040fbb30 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55864/0 (socket says 172.21.15.111:55864) 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 -- 172.21.15.111:0/2230799347 learned_addr learned my addr 172.21.15.111:0/2230799347 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f4502ffd700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 0x7f4504112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 -- 172.21.15.111:0/2230799347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 msgr2=0x7f4504112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 0x7f4504112af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 -- 172.21.15.111:0/2230799347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8005040 con 0x7f45040fd290 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f4502ffd700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 0x7f4504112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f45037fe700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f44f400eb70 tx=0x7f44f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f4007cc0 con 0x7f45040fd290 2024-03-14T13:02:58.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4504113470 con 0x7f45040fd290 2024-03-14T13:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44f400f040 con 0x7f45040fd290 2024-03-14T13:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44f400c9d0 con 0x7f45040fd290 2024-03-14T13:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.145+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4504113a30 con 0x7f45040fd290 2024-03-14T13:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.146+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450405f5f0 con 0x7f45040fd290 2024-03-14T13:02:58.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.148+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 32) v1 ==== 90489+0+0 (secure 0 0 0) 0x7f44f400cc40 con 0x7f45040fd290 2024-03-14T13:02:58.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.148+0000 7f44ebfff700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 0x7f44ec06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:02:58.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.148+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f44f408b730 con 0x7f45040fd290 2024-03-14T13:02:58.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.148+0000 7f4502ffd700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 0x7f44ec06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:02:58.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.149+0000 7f4502ffd700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 0x7f44ec06ea10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44f80137d0 tx=0x7f44f8009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:02:58.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.153+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f44f4055cb0 con 0x7f45040fd290 2024-03-14T13:02:58.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.500+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f45040fdd50 con 0x7f44ec06c560 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.501+0000 7f44ebfff700 1 -- 172.21.15.111:0/2230799347 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+370 (secure 0 0 0) 0x7f45040fdd50 con 0x7f44ec06c560 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:02:58.501 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-14T13:02:58.502 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:02:58.502 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 msgr2=0x7f44ec06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 0x7f44ec06ea10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f44f80137d0 tx=0x7f44f8009e30 comp rx=0 tx=0).stop 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 msgr2=0x7f45041105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f44f400eb70 tx=0x7f44f400b040 comp rx=0 tx=0).stop 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 shutdown_connections 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44ec06c560 0x7f44ec06ea10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45040fd290 0x7f45041105a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 --2- 172.21.15.111:0/2230799347 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45040fbb30 0x7f4504112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 >> 172.21.15.111:0/2230799347 conn(0x7f45040f6ba0 msgr2=0x7f45040f8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 shutdown_connections 2024-03-14T13:02:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:02:58.504+0000 7f450a11c700 1 -- 172.21.15.111:0/2230799347 wait complete. 2024-03-14T13:02:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:58 smithi137 ceph-mon[50336]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:58 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/726482586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:02:59.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:58 smithi111 ceph-mon[39688]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:02:59.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:58 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/726482586' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:59 smithi137 ceph-mon[50336]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:00.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:02:59 smithi137 ceph-mon[50336]: pgmap v44: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 626 op/s 2024-03-14T13:03:00.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:59 smithi111 ceph-mon[39688]: from='client.14758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:00.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:02:59 smithi111 ceph-mon[39688]: pgmap v44: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 626 op/s 2024-03-14T13:03:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:01 smithi137 ceph-mon[50336]: pgmap v45: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 471 op/s 2024-03-14T13:03:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:01 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:01 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:01 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:02.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:01 smithi111 ceph-mon[39688]: pgmap v45: 65 pgs: 65 active+clean; 290 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 471 op/s 2024-03-14T13:03:02.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:01 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:02.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:01 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:02.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:01 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:03 smithi111 ceph-mon[39688]: pgmap v46: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 589 op/s 2024-03-14T13:03:04.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:03 smithi137 ceph-mon[50336]: pgmap v46: 65 pgs: 65 active+clean; 286 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 589 op/s 2024-03-14T13:03:05.267 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:05 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.267 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:05 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.267 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:05 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.267 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:05 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:05 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:05 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:05 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:05.456 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:05 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:06.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:06 smithi137 ceph-mon[50336]: pgmap v47: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 663 op/s 2024-03-14T13:03:06.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:06 smithi111 ceph-mon[39688]: pgmap v47: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 663 op/s 2024-03-14T13:03:07.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:07 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:07 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:08.312 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:08 smithi137 ceph-mon[50336]: pgmap v48: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 481 op/s 2024-03-14T13:03:08.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:08 smithi111 ceph-mon[39688]: pgmap v48: 65 pgs: 65 active+clean; 282 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 481 op/s 2024-03-14T13:03:09.863 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: pgmap v49: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 672 op/s 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: mgrmap e33: smithi111.atysez(active, since 92s), standbys: smithi137.xbyzbp 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:09.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: pgmap v49: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 672 op/s 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: mgrmap e33: smithi111.atysez(active, since 92s), standbys: smithi137.xbyzbp 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:11.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:11 smithi111 ceph-mon[39688]: pgmap v50: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 427 op/s 2024-03-14T13:03:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:11 smithi137 ceph-mon[50336]: pgmap v50: 65 pgs: 65 active+clean; 276 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 427 op/s 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 520 op/s 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:13 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: pgmap v51: 65 pgs: 65 active+clean; 275 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 520 op/s 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:14.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:14.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:13 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:15 smithi137 ceph-mon[50336]: pgmap v52: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 485 op/s 2024-03-14T13:03:16.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:15 smithi111 ceph-mon[39688]: pgmap v52: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 485 op/s 2024-03-14T13:03:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:16 smithi111 ceph-mon[39688]: Upgrade: Updating prometheus.smithi111 2024-03-14T13:03:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:16 smithi111 ceph-mon[39688]: Deploying daemon prometheus.smithi111 on smithi111 2024-03-14T13:03:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:16 smithi137 ceph-mon[50336]: Upgrade: Updating prometheus.smithi111 2024-03-14T13:03:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:16 smithi137 ceph-mon[50336]: Deploying daemon prometheus.smithi111 on smithi111 2024-03-14T13:03:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:17 smithi111 ceph-mon[39688]: pgmap v53: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 366 op/s 2024-03-14T13:03:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:17 smithi137 ceph-mon[50336]: pgmap v53: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 366 op/s 2024-03-14T13:03:18.971 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:18 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:03:19.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:18 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:03:19.820 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:19 smithi111 ceph-mon[39688]: pgmap v54: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 556 op/s 2024-03-14T13:03:20.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:19 smithi137 ceph-mon[50336]: pgmap v54: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 556 op/s 2024-03-14T13:03:21.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:21 smithi111 ceph-mon[39688]: pgmap v55: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 365 op/s 2024-03-14T13:03:21.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:21 smithi137 ceph-mon[50336]: pgmap v55: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 365 op/s 2024-03-14T13:03:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:22.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:22.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:22.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:22.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:22.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:23 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:23 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:23 smithi137 ceph-mon[50336]: pgmap v56: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 451 op/s 2024-03-14T13:03:23.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:23 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:23.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:23 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:23.762 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:23 smithi111 ceph-mon[39688]: pgmap v56: 65 pgs: 65 active+clean; 269 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 451 op/s 2024-03-14T13:03:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:25 smithi137 ceph-mon[50336]: pgmap v57: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 493 op/s 2024-03-14T13:03:26.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:25 smithi111 ceph-mon[39688]: pgmap v57: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 493 op/s 2024-03-14T13:03:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:27 smithi111 ceph-mon[39688]: pgmap v58: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 411 op/s 2024-03-14T13:03:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:27 smithi137 ceph-mon[50336]: pgmap v58: 65 pgs: 65 active+clean; 274 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 411 op/s 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.671+0000 7fcd428ae700 1 -- 172.21.15.111:0/2848449053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 msgr2=0x7fcd3c103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.671+0000 7fcd428ae700 1 --2- 172.21.15.111:0/2848449053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c103800 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009ce0 tx=0x7fcd34004d10 comp rx=0 tx=0).stop 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 -- 172.21.15.111:0/2848449053 shutdown_connections 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 --2- 172.21.15.111:0/2848449053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 0x7fcd3c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 --2- 172.21.15.111:0/2848449053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c103800 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 -- 172.21.15.111:0/2848449053 >> 172.21.15.111:0/2848449053 conn(0x7fcd3c072560 msgr2=0x7fcd3c0749b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 -- 172.21.15.111:0/2848449053 shutdown_connections 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 -- 172.21.15.111:0/2848449053 wait complete. 2024-03-14T13:03:28.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.672+0000 7fcd428ae700 1 Processor -- start 2024-03-14T13:03:28.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd428ae700 1 -- start start 2024-03-14T13:03:28.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd428ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd428ae700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 0x7fcd3c114570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd428ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c114c00 con 0x7fcd3c1033f0 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd428ae700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3c1153f0 con 0x7fcd3c1046a0 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd3bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd3bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57664/0 (socket says 172.21.15.111:57664) 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd3bfff700 1 -- 172.21.15.111:0/3888201923 learned_addr learned my addr 172.21.15.111:0/3888201923 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.673+0000 7fcd3b7fe700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 0x7fcd3c114570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd3bfff700 1 -- 172.21.15.111:0/3888201923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 msgr2=0x7fcd3c114570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd3bfff700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 0x7fcd3c114570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd3bfff700 1 -- 172.21.15.111:0/3888201923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd34009940 con 0x7fcd3c1033f0 2024-03-14T13:03:28.675 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd3bfff700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd34004e60 tx=0x7fcd34004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd3400e050 con 0x7fcd3c1033f0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3c115670 con 0x7fcd3c1033f0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3c115c30 con 0x7fcd3c1033f0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd3401d8f0 con 0x7fcd3c1033f0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.674+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd34012850 con 0x7fcd3c1033f0 2024-03-14T13:03:28.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.676+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd280052e0 con 0x7fcd3c1033f0 2024-03-14T13:03:28.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.676+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fcd34019040 con 0x7fcd3c1033f0 2024-03-14T13:03:28.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.677+0000 7fcd38ff9700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 0x7fcd2406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:28.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.677+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fcd3408d000 con 0x7fcd3c1033f0 2024-03-14T13:03:28.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.678+0000 7fcd3b7fe700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 0x7fcd2406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:28.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.678+0000 7fcd3b7fe700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 0x7fcd2406e9d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd30006fd0 tx=0x7fcd300090f0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:28.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:28.684+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fcd340c04d0 con 0x7fcd3c1033f0 2024-03-14T13:03:29.017 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.016+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd28000bf0 con 0x7fcd2406c520 2024-03-14T13:03:29.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.018+0000 7fcd38ff9700 1 -- 172.21.15.111:0/3888201923 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fcd28000bf0 con 0x7fcd2406c520 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 msgr2=0x7fcd2406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 0x7fcd2406e9d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcd30006fd0 tx=0x7fcd300090f0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 msgr2=0x7fcd3c114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcd34004e60 tx=0x7fcd34004d10 comp rx=0 tx=0).stop 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 shutdown_connections 2024-03-14T13:03:29.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcd2406c520 0x7fcd2406e9d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd3c1033f0 0x7fcd3c114030 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 --2- 172.21.15.111:0/3888201923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd3c1046a0 0x7fcd3c114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.020+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 >> 172.21.15.111:0/3888201923 conn(0x7fcd3c072560 msgr2=0x7fcd3c073da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.021+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 shutdown_connections 2024-03-14T13:03:29.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.021+0000 7fcd428ae700 1 -- 172.21.15.111:0/3888201923 wait complete. 2024-03-14T13:03:29.034 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:03:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 -- 172.21.15.111:0/2802959640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0102360 msgr2=0x7fd1c0104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/2802959640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0102360 0x7fd1c0104740 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4004940 tx=0x7fd1b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 -- 172.21.15.111:0/2802959640 shutdown_connections 2024-03-14T13:03:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/2802959640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0102360 0x7fd1c0104740 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/2802959640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0066280 0x7fd1c0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 -- 172.21.15.111:0/2802959640 >> 172.21.15.111:0/2802959640 conn(0x7fd1c00f7430 msgr2=0x7fd1c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:29.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.152+0000 7fd1c50d4700 1 -- 172.21.15.111:0/2802959640 shutdown_connections 2024-03-14T13:03:29.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.153+0000 7fd1c50d4700 1 -- 172.21.15.111:0/2802959640 wait complete. 2024-03-14T13:03:29.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.153+0000 7fd1c50d4700 1 Processor -- start 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.153+0000 7fd1c50d4700 1 -- start start 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1c50d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1c50d4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0102360 0x7fd1c0110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1c50d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0110640 con 0x7fd1c0066280 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1c50d4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0110e90 con 0x7fd1c0102360 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1be59c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0102360 0x7fd1c0110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57682/0 (socket says 172.21.15.111:57682) 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 -- 172.21.15.111:0/3268184606 learned_addr learned my addr 172.21.15.111:0/3268184606 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 -- 172.21.15.111:0/3268184606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0102360 msgr2=0x7fd1c0110040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0102360 0x7fd1c0110040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.154+0000 7fd1bed9d700 1 -- 172.21.15.111:0/3268184606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4005040 con 0x7fd1c0066280 2024-03-14T13:03:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1bed9d700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000eb70 tx=0x7fd1b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b0007cc0 con 0x7fd1c0066280 2024-03-14T13:03:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b000f040 con 0x7fd1c0066280 2024-03-14T13:03:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c0111110 con 0x7fd1c0066280 2024-03-14T13:03:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1b000c9d0 con 0x7fd1c0066280 2024-03-14T13:03:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.155+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c0111650 con 0x7fd1c0066280 2024-03-14T13:03:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.156+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c005f5f0 con 0x7fd1c0066280 2024-03-14T13:03:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.157+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fd1b000cb30 con 0x7fd1c0066280 2024-03-14T13:03:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.157+0000 7fd1a77fe700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 0x7fd1a806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.158+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd1b008aef0 con 0x7fd1c0066280 2024-03-14T13:03:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.158+0000 7fd1be59c700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 0x7fd1a806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.159+0000 7fd1be59c700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 0x7fd1a806e8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400b0d0 tx=0x7fd1b4009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:29.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.164+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fd1b00591d0 con 0x7fd1c0066280 2024-03-14T13:03:29.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.473+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1c01119f0 con 0x7fd1a806c440 2024-03-14T13:03:29.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.475+0000 7fd1a77fe700 1 -- 172.21.15.111:0/3268184606 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fd1c01119f0 con 0x7fd1a806c440 2024-03-14T13:03:29.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.477+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 msgr2=0x7fd1a806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.477+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 0x7fd1a806e8f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400b0d0 tx=0x7fd1b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:29.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.477+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 msgr2=0x7fd1c010fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.477+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000eb70 tx=0x7fd1b000b040 comp rx=0 tx=0).stop 2024-03-14T13:03:29.478 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 shutdown_connections 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd1a806c440 0x7fd1a806e8f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1c0066280 0x7fd1c010fb00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 --2- 172.21.15.111:0/3268184606 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1c0102360 0x7fd1c0110040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 >> 172.21.15.111:0/3268184606 conn(0x7fd1c00f7430 msgr2=0x7fd1c00f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 shutdown_connections 2024-03-14T13:03:29.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.478+0000 7fd1c50d4700 1 -- 172.21.15.111:0/3268184606 wait complete. 2024-03-14T13:03:29.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 -- 172.21.15.111:0/1745745866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fce70 msgr2=0x7f2bc40fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/1745745866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fce70 0x7f2bc40fd280 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4004940 tx=0x7f2bb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 -- 172.21.15.111:0/1745745866 shutdown_connections 2024-03-14T13:03:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/1745745866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc40fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/1745745866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fce70 0x7f2bc40fd280 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.626+0000 7f2bca4fc700 1 -- 172.21.15.111:0/1745745866 >> 172.21.15.111:0/1745745866 conn(0x7f2bc40f8550 msgr2=0x7f2bc40fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.627+0000 7f2bca4fc700 1 -- 172.21.15.111:0/1745745866 shutdown_connections 2024-03-14T13:03:29.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.628+0000 7f2bca4fc700 1 -- 172.21.15.111:0/1745745866 wait complete. 2024-03-14T13:03:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.628+0000 7f2bca4fc700 1 Processor -- start 2024-03-14T13:03:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bca4fc700 1 -- start start 2024-03-14T13:03:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bca4fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 0x7f2bc410fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bca4fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc410ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bca4fc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4110660 con 0x7f2bc40fdfe0 2024-03-14T13:03:29.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bca4fc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4114fb0 con 0x7f2bc40fce70 2024-03-14T13:03:29.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc410ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc410ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57702/0 (socket says 172.21.15.111:57702) 2024-03-14T13:03:29.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bc37fe700 1 -- 172.21.15.111:0/2133576730 learned_addr learned my addr 172.21.15.111:0/2133576730 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:03:29.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.629+0000 7f2bc3fff700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 0x7f2bc410fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.630+0000 7f2bc3fff700 1 -- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 msgr2=0x7f2bc410ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.630+0000 7f2bc3fff700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc410ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.630+0000 7f2bc3fff700 1 -- 172.21.15.111:0/2133576730 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb4005040 con 0x7f2bc40fce70 2024-03-14T13:03:29.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.631+0000 7f2bc3fff700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 0x7f2bc410fa90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400b900 tx=0x7f2bb4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:29.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.631+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb400e040 con 0x7f2bc40fce70 2024-03-14T13:03:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.632+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc41151b0 con 0x7f2bc40fce70 2024-03-14T13:03:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.632+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4115720 con 0x7f2bc40fce70 2024-03-14T13:03:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.633+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc404c710 con 0x7f2bc40fce70 2024-03-14T13:03:29.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.633+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2bb4004290 con 0x7f2bc40fce70 2024-03-14T13:03:29.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.633+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bb4013d60 con 0x7f2bc40fce70 2024-03-14T13:03:29.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.635+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f2bb40135b0 con 0x7f2bc40fce70 2024-03-14T13:03:29.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.636+0000 7f2bc0ff9700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 0x7f2bac06e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:29.635 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.636+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f2bb408c6f0 con 0x7f2bc40fce70 2024-03-14T13:03:29.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.636+0000 7f2bc37fe700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 0x7f2bac06e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:29.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.636+0000 7f2bc37fe700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 0x7f2bac06e9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009ca0 tx=0x7f2bb8009250 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 658 op/s 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:29 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.644+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f2bb405a900 con 0x7f2bc40fce70 2024-03-14T13:03:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: pgmap v59: 65 pgs: 65 active+clean; 270 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 658 op/s 2024-03-14T13:03:29.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: from='client.14762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:29.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:29 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:29.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.954+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc4115a00 con 0x7f2bac06c510 2024-03-14T13:03:29.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.970+0000 7f2bc0ff9700 1 -- 172.21.15.111:0/2133576730 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f2bc4115a00 con 0x7f2bac06c510 2024-03-14T13:03:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (2m) 1s ago 11m 24.9M - 0.25.0 c8568f914cd2 70fc8e769dee 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (11m) 1s ago 11m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (10m) 25s ago 10m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (2m) 1s ago 10m 82.8M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (5m) 1s ago 5m 80.0M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (5m) 1s ago 5m 141M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (5m) 25s ago 5m 90.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (5m) 25s ago 5m 157M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (2m) 1s ago 12m 494M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (54s) 25s ago 9m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (12m) 1s ago 12m 55.4M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (9m) 25s ago 9m 44.8M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:03:29.970 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (40s) 1s ago 11m 8887k - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (31s) 25s ago 9m 7088k - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (9m) 1s ago 9m 198M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (8m) 1s ago 8m 226M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (8m) 1s ago 8m 181M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (7m) 25s ago 7m 240M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (7m) 25s ago 7m 213M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (6m) 25s ago 6m 203M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:03:29.971 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (8s) 1s ago 10m 40.2M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 msgr2=0x7f2bac06e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 0x7f2bac06e9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009ca0 tx=0x7f2bb8009250 comp rx=0 tx=0).stop 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 msgr2=0x7f2bc410fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 0x7f2bc410fa90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400b900 tx=0x7f2bb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 shutdown_connections 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bc40fce70 0x7f2bc410fa90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f2bac06c510 0x7f2bac06e9c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 --2- 172.21.15.111:0/2133576730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2bc40fdfe0 0x7f2bc410ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 >> 172.21.15.111:0/2133576730 conn(0x7f2bc40f8550 msgr2=0x7f2bc40fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.972+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 shutdown_connections 2024-03-14T13:03:29.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:29.973+0000 7f2bca4fc700 1 -- 172.21.15.111:0/2133576730 wait complete. 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.098+0000 7f1b0ce87700 1 -- 172.21.15.111:0/72028590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fcf90 msgr2=0x7f1b080fd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.098+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/72028590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fcf90 0x7f1b080fd400 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1afc004940 tx=0x7f1afc009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- 172.21.15.111:0/72028590 shutdown_connections 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/72028590 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fcf90 0x7f1b080fd400 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/72028590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b080fbce0 0x7f1b080fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- 172.21.15.111:0/72028590 >> 172.21.15.111:0/72028590 conn(0x7f1b080f7440 msgr2=0x7f1b080f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- 172.21.15.111:0/72028590 shutdown_connections 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- 172.21.15.111:0/72028590 wait complete. 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 Processor -- start 2024-03-14T13:03:30.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- start start 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b080fcf90 0x7f1b0810ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b081105e0 con 0x7f1b080fbce0 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0ce87700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08110e30 con 0x7f1b080fcf90 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:57724/0 (socket says 172.21.15.111:57724) 2024-03-14T13:03:30.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.099+0000 7f1b0659c700 1 -- 172.21.15.111:0/2746484151 learned_addr learned my addr 172.21.15.111:0/2746484151 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0659c700 1 -- 172.21.15.111:0/2746484151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b080fcf90 msgr2=0x7f1b0810ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0659c700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b080fcf90 0x7f1b0810ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0659c700 1 -- 172.21.15.111:0/2746484151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc005040 con 0x7f1b080fbce0 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0659c700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1af800eb40 tx=0x7f1af800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1af8007cc0 con 0x7f1b080fbce0 2024-03-14T13:03:30.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1af800f040 con 0x7f1b080fbce0 2024-03-14T13:03:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b081110b0 con 0x7f1b080fbce0 2024-03-14T13:03:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1af8009570 con 0x7f1b080fbce0 2024-03-14T13:03:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.100+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b081115f0 con 0x7f1b080fbce0 2024-03-14T13:03:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.101+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08101920 con 0x7f1b080fbce0 2024-03-14T13:03:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.102+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f1af80096d0 con 0x7f1b080fbce0 2024-03-14T13:03:30.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.102+0000 7f1af6ffd700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 0x7f1af006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.102+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1af808b0d0 con 0x7f1b080fbce0 2024-03-14T13:03:30.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.102+0000 7f1b05d9b700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 0x7f1af006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:30.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.103+0000 7f1b05d9b700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 0x7f1af006e900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000c00 tx=0x7f1afc009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:30.108 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.109+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f1af80593b0 con 0x7f1b080fbce0 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.500+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1b08101340 con 0x7f1b080fbce0 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.500+0000 7f1af6ffd700 1 -- 172.21.15.111:0/2746484151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f1af8058f40 con 0x7f1b080fbce0 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:03:30.500 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:03:30.501 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 msgr2=0x7f1af006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 0x7f1af006e900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000c00 tx=0x7f1afc009e30 comp rx=0 tx=0).stop 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 msgr2=0x7f1b0810faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1af800eb40 tx=0x7f1af800b040 comp rx=0 tx=0).stop 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 shutdown_connections 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1af006c450 0x7f1af006e900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b080fbce0 0x7f1b0810faa0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 --2- 172.21.15.111:0/2746484151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b080fcf90 0x7f1b0810ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 >> 172.21.15.111:0/2746484151 conn(0x7f1b080f7440 msgr2=0x7f1b08066490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 shutdown_connections 2024-03-14T13:03:30.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.503+0000 7f1b0ce87700 1 -- 172.21.15.111:0/2746484151 wait complete. 2024-03-14T13:03:30.601 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:30 smithi111 ceph-mon[39688]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:30.601 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:30 smithi111 ceph-mon[39688]: from='client.24501 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 -- 172.21.15.111:0/2232749489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fbcd0 msgr2=0x7fdc300fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 --2- 172.21.15.111:0/2232749489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fbcd0 0x7fdc300fc0e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fdc28009ce0 tx=0x7fdc28004d10 comp rx=0 tx=0).stop 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 -- 172.21.15.111:0/2232749489 shutdown_connections 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 --2- 172.21.15.111:0/2232749489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fcf80 0x7fdc300fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 --2- 172.21.15.111:0/2232749489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fbcd0 0x7fdc300fc0e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 -- 172.21.15.111:0/2232749489 >> 172.21.15.111:0/2232749489 conn(0x7fdc300f7430 msgr2=0x7fdc300f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 -- 172.21.15.111:0/2232749489 shutdown_connections 2024-03-14T13:03:30.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.659+0000 7fdc364ae700 1 -- 172.21.15.111:0/2232749489 wait complete. 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 Processor -- start 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 -- start start 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fcf80 0x7fdc3010bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3010c200 con 0x7fdc300fcf80 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc364ae700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3010c370 con 0x7fdc300fbcd0 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc2ffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc2ffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:34800/0 (socket says 172.21.15.111:34800) 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc2ffff700 1 -- 172.21.15.111:0/3661421517 learned_addr learned my addr 172.21.15.111:0/3661421517 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:03:30.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.660+0000 7fdc2f7fe700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fcf80 0x7fdc3010bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:30.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.661+0000 7fdc2ffff700 1 -- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fcf80 msgr2=0x7fdc3010bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:30.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.661+0000 7fdc2ffff700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fcf80 0x7fdc3010bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:30.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.661+0000 7fdc2ffff700 1 -- 172.21.15.111:0/3661421517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc24005040 con 0x7fdc300fbcd0 2024-03-14T13:03:30.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.661+0000 7fdc2ffff700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc28004e60 tx=0x7fdc28004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:30.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.662+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc2800e040 con 0x7fdc300fbcd0 2024-03-14T13:03:30.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.663+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc2801cce0 con 0x7fdc300fbcd0 2024-03-14T13:03:30.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.663+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc28012b40 con 0x7fdc300fbcd0 2024-03-14T13:03:30.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.663+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc28009940 con 0x7fdc300fbcd0 2024-03-14T13:03:30.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.663+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc30111190 con 0x7fdc300fbcd0 2024-03-14T13:03:30.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.664+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3004c710 con 0x7fdc300fbcd0 2024-03-14T13:03:30.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.666+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fdc2801c830 con 0x7fdc300fbcd0 2024-03-14T13:03:30.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.666+0000 7fdc2cff9700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 0x7fdc1806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:03:30.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.666+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdc2808dbe0 con 0x7fdc300fbcd0 2024-03-14T13:03:30.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.667+0000 7fdc2f7fe700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 0x7fdc1806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:03:30.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.667+0000 7fdc2f7fe700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 0x7fdc1806ea80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc3010c760 tx=0x7fdc2400b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:03:30.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.672+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fdc28058360 con 0x7fdc300fbcd0 2024-03-14T13:03:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:30 smithi137 ceph-mon[50336]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:30 smithi137 ceph-mon[50336]: from='client.24501 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:30.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.996+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc301013f0 con 0x7fdc1806c5d0 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:30.997+0000 7fdc2cff9700 1 -- 172.21.15.111:0/3661421517 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fdc301013f0 con 0x7fdc1806c5d0 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:03:30.997 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:03:30.998 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-14T13:03:30.998 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:03:30.998 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:03:30.999 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 msgr2=0x7fdc1806ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 0x7fdc1806ea80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdc3010c760 tx=0x7fdc2400b040 comp rx=0 tx=0).stop 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 msgr2=0x7fdc3010b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc28004e60 tx=0x7fdc28004d10 comp rx=0 tx=0).stop 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 shutdown_connections 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc300fbcd0 0x7fdc3010b670 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fdc1806c5d0 0x7fdc1806ea80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 --2- 172.21.15.111:0/3661421517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc300fcf80 0x7fdc3010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 >> 172.21.15.111:0/3661421517 conn(0x7fdc300f7430 msgr2=0x7fdc300f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 shutdown_connections 2024-03-14T13:03:31.000 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:03:31.000+0000 7fdc364ae700 1 -- 172.21.15.111:0/3661421517 wait complete. 2024-03-14T13:03:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:31 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/2746484151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:31 smithi111 ceph-mon[39688]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 468 op/s 2024-03-14T13:03:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:31 smithi111 ceph-mon[39688]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:31 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/2746484151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:31 smithi137 ceph-mon[50336]: pgmap v60: 65 pgs: 65 active+clean; 270 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 468 op/s 2024-03-14T13:03:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:31 smithi137 ceph-mon[50336]: from='client.24507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:03:33.332 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-14T13:03:33.333 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-14T13:03:33.721 DEBUG:teuthology.parallel:result is None 2024-03-14T13:03:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:33 smithi137 ceph-mon[50336]: pgmap v61: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 592 op/s 2024-03-14T13:03:34.089 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:33 smithi111 ceph-mon[39688]: pgmap v61: 65 pgs: 65 active+clean; 272 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 592 op/s 2024-03-14T13:03:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:03:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:35 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:35.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:35 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:03:36.231 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-14T13:03:36.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-03-14T13:03:36.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:36 smithi137 ceph-mon[50336]: pgmap v62: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 599 op/s 2024-03-14T13:03:36.500 DEBUG:teuthology.parallel:result is None 2024-03-14T13:03:36.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-14T13:03:36.527 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:36 smithi111 ceph-mon[39688]: pgmap v62: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 599 op/s 2024-03-14T13:03:36.538 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-14T13:03:36.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-03-14T13:03:36.601 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-03-14T13:03:36.602 DEBUG:teuthology.parallel:result is None 2024-03-14T13:03:37.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:37 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:37.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:37 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:38.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:38 smithi111 ceph-mon[39688]: pgmap v63: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 464 op/s 2024-03-14T13:03:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:38 smithi111 ceph-mon[39688]: Upgrade: Updating alertmanager.smithi111 2024-03-14T13:03:38.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:38 smithi111 ceph-mon[39688]: Deploying daemon alertmanager.smithi111 on smithi111 2024-03-14T13:03:38.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:38 smithi137 ceph-mon[50336]: pgmap v63: 65 pgs: 65 active+clean; 265 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 464 op/s 2024-03-14T13:03:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:38 smithi137 ceph-mon[50336]: Upgrade: Updating alertmanager.smithi111 2024-03-14T13:03:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:38 smithi137 ceph-mon[50336]: Deploying daemon alertmanager.smithi111 on smithi111 2024-03-14T13:03:40.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:40 smithi137 ceph-mon[50336]: pgmap v64: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 649 op/s 2024-03-14T13:03:40.620 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:40 smithi111 ceph-mon[39688]: pgmap v64: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 649 op/s 2024-03-14T13:03:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:41 smithi111 ceph-mon[39688]: pgmap v65: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 402 op/s 2024-03-14T13:03:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:41 smithi137 ceph-mon[50336]: pgmap v65: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 402 op/s 2024-03-14T13:03:43.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:43 smithi111 ceph-mon[39688]: pgmap v66: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 404 op/s 2024-03-14T13:03:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:43 smithi137 ceph-mon[50336]: pgmap v66: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 404 op/s 2024-03-14T13:03:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:45 smithi137 ceph-mon[50336]: pgmap v67: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 675 KiB/s rd, 624 KiB/s wr, 282 op/s 2024-03-14T13:03:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:45 smithi111 ceph-mon[39688]: pgmap v67: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 675 KiB/s rd, 624 KiB/s wr, 282 op/s 2024-03-14T13:03:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:47 smithi111 ceph-mon[39688]: pgmap v68: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 KiB/s rd, 433 KiB/s wr, 189 op/s 2024-03-14T13:03:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:47 smithi137 ceph-mon[50336]: pgmap v68: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 511 KiB/s rd, 433 KiB/s wr, 189 op/s 2024-03-14T13:03:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:49 smithi137 ceph-mon[50336]: pgmap v69: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 512 KiB/s rd, 433 KiB/s wr, 192 op/s 2024-03-14T13:03:50.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:49 smithi111 ceph-mon[39688]: pgmap v69: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 512 KiB/s rd, 433 KiB/s wr, 192 op/s 2024-03-14T13:03:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:51 smithi111 ceph-mon[39688]: pgmap v70: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-14T13:03:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:51 smithi137 ceph-mon[50336]: pgmap v70: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-14T13:03:52.488 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:52 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:52 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:03:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:53 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:53 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:53 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:53.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:53 smithi111 ceph-mon[39688]: pgmap v71: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-14T13:03:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:53 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:53 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:53 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:03:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:53 smithi137 ceph-mon[50336]: pgmap v71: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2024-03-14T13:03:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:55 smithi137 ceph-mon[50336]: pgmap v72: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-14T13:03:56.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:55 smithi111 ceph-mon[39688]: pgmap v72: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2024-03-14T13:03:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:57 smithi111 ceph-mon[39688]: pgmap v73: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-14T13:03:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:57 smithi137 ceph-mon[50336]: pgmap v73: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-14T13:03:59.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:59 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:59 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:59 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:03:59 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:59 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:59 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:59 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:03:59.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:03:59 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:00.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:00 smithi111 ceph-mon[39688]: pgmap v74: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T13:04:00.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:00 smithi137 ceph-mon[50336]: pgmap v74: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-14T13:04:01.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 -- 172.21.15.111:0/2041413036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec066f50 msgr2=0x7f98ec0ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 --2- 172.21.15.111:0/2041413036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec066f50 0x7f98ec0ffa00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f98e8004940 tx=0x7f98e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 -- 172.21.15.111:0/2041413036 shutdown_connections 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 --2- 172.21.15.111:0/2041413036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec0fff40 0x7f98ec102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 --2- 172.21.15.111:0/2041413036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec066f50 0x7f98ec0ffa00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 -- 172.21.15.111:0/2041413036 >> 172.21.15.111:0/2041413036 conn(0x7f98ec0f7490 msgr2=0x7f98ec0f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 -- 172.21.15.111:0/2041413036 shutdown_connections 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.112+0000 7f98f3612700 1 -- 172.21.15.111:0/2041413036 wait complete. 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 Processor -- start 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 -- start start 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec066f50 0x7f98ec10d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10e3d0 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f3612700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10e540 con 0x7f98ec066f50 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54294/0 (socket says 172.21.15.111:54294) 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 -- 172.21.15.111:0/3244168577 learned_addr learned my addr 172.21.15.111:0/3244168577 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f13ae700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec066f50 0x7f98ec10d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 -- 172.21.15.111:0/3244168577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec066f50 msgr2=0x7f98ec10d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec066f50 0x7f98ec10d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 -- 172.21.15.111:0/3244168577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8005040 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.113+0000 7f98f0bad700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c830 tx=0x7f98dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.114+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98dc0106f0 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.114+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98dc010d00 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.114+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec112fa0 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.114+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98dc012e90 con 0x7f98ec0fff40 2024-03-14T13:04:01.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.114+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec113530 con 0x7f98ec0fff40 2024-03-14T13:04:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.115+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec04c710 con 0x7f98ec0fff40 2024-03-14T13:04:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.115+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f98dc010850 con 0x7f98ec0fff40 2024-03-14T13:04:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.116+0000 7f98e1ffb700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 0x7f98d806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.116+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f98dc08b480 con 0x7f98ec0fff40 2024-03-14T13:04:01.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.116+0000 7f98f13ae700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 0x7f98d806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.116+0000 7f98f13ae700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 0x7f98d806e8f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f98e8004890 tx=0x7f98e8009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.121+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f98dc0596b0 con 0x7f98ec0fff40 2024-03-14T13:04:01.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.340+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f98ec0f8f80 con 0x7f98d806c440 2024-03-14T13:04:01.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.341+0000 7f98e1ffb700 1 -- 172.21.15.111:0/3244168577 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7f98ec0f8f80 con 0x7f98d806c440 2024-03-14T13:04:01.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 msgr2=0x7f98d806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 0x7f98d806e8f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f98e8004890 tx=0x7f98e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 msgr2=0x7f98ec10ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f98dc00c830 tx=0x7f98dc004470 comp rx=0 tx=0).stop 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 shutdown_connections 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f98ec066f50 0x7f98ec10d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f98d806c440 0x7f98d806e8f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 --2- 172.21.15.111:0/3244168577 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98ec0fff40 0x7f98ec10ddd0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 >> 172.21.15.111:0/3244168577 conn(0x7f98ec0f7490 msgr2=0x7f98ec0f7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 shutdown_connections 2024-03-14T13:04:01.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.344+0000 7f98f3612700 1 -- 172.21.15.111:0/3244168577 wait complete. 2024-03-14T13:04:01.355 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:04:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.452+0000 7faec05ba700 1 -- 172.21.15.111:0/1390186753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 msgr2=0x7faeb80fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.452+0000 7faec05ba700 1 --2- 172.21.15.111:0/1390186753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb80fe8d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faeb4004940 tx=0x7faeb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:01.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 -- 172.21.15.111:0/1390186753 shutdown_connections 2024-03-14T13:04:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 --2- 172.21.15.111:0/1390186753 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 0x7faeb80faf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 --2- 172.21.15.111:0/1390186753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb80fe8d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 -- 172.21.15.111:0/1390186753 >> 172.21.15.111:0/1390186753 conn(0x7faeb80f6ba0 msgr2=0x7faeb80f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 -- 172.21.15.111:0/1390186753 shutdown_connections 2024-03-14T13:04:01.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 -- 172.21.15.111:0/1390186753 wait complete. 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 Processor -- start 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 -- start start 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.453+0000 7faec05ba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faec05ba700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 0x7faeb810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faec05ba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb810e4f0 con 0x7faeb80fe4c0 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faec05ba700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb810ece0 con 0x7faeb80fee10 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54314/0 (socket says 172.21.15.111:54314) 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 -- 172.21.15.111:0/3985254743 learned_addr learned my addr 172.21.15.111:0/3985254743 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebdb55700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 0x7faeb810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 -- 172.21.15.111:0/3985254743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 msgr2=0x7faeb810de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 0x7faeb810de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 -- 172.21.15.111:0/3985254743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb4005040 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faebe356700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009380 tx=0x7faeb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb400e040 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb810ef60 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeb400bd20 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeb401c630 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.454+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb810f520 con 0x7faeb80fe4c0 2024-03-14T13:04:01.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.455+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb805f5f0 con 0x7faeb80fe4c0 2024-03-14T13:04:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.456+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7faeb4013420 con 0x7faeb80fe4c0 2024-03-14T13:04:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.456+0000 7faeaeffd700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 0x7faea406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.456+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faeb408c1c0 con 0x7faeb80fe4c0 2024-03-14T13:04:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.456+0000 7faebdb55700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 0x7faea406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.457+0000 7faebdb55700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 0x7faea406e8f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faea8006fd0 tx=0x7faea8008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.461+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7faeb405a3f0 con 0x7faeb80fe4c0 2024-03-14T13:04:01.606 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:01 smithi111 ceph-mon[39688]: pgmap v75: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:01.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.680+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faeb810f7b0 con 0x7faea406c440 2024-03-14T13:04:01.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.682+0000 7faeaeffd700 1 -- 172.21.15.111:0/3985254743 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+369 (secure 0 0 0) 0x7faeb810f7b0 con 0x7faea406c440 2024-03-14T13:04:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 msgr2=0x7faea406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 0x7faea406e8f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faea8006fd0 tx=0x7faea8008040 comp rx=0 tx=0).stop 2024-03-14T13:04:01.683 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 msgr2=0x7faeb810d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009380 tx=0x7faeb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 shutdown_connections 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7faea406c440 0x7faea406e8f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb80fe4c0 0x7faeb810d920 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 --2- 172.21.15.111:0/3985254743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeb80fee10 0x7faeb810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 >> 172.21.15.111:0/3985254743 conn(0x7faeb80f6ba0 msgr2=0x7faeb80f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 shutdown_connections 2024-03-14T13:04:01.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.684+0000 7faec05ba700 1 -- 172.21.15.111:0/3985254743 wait complete. 2024-03-14T13:04:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:01 smithi137 ceph-mon[50336]: pgmap v75: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:01.785 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- 172.21.15.111:0/1840367381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 msgr2=0x7f44d80fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/1840367381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d80fd430 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f44cc004940 tx=0x7f44cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- 172.21.15.111:0/1840367381 shutdown_connections 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/1840367381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d80fd430 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/1840367381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 0x7f44d80fc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- 172.21.15.111:0/1840367381 >> 172.21.15.111:0/1840367381 conn(0x7f44d80f7490 msgr2=0x7f44d80f98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- 172.21.15.111:0/1840367381 shutdown_connections 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- 172.21.15.111:0/1840367381 wait complete. 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 Processor -- start 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- start start 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 0x7f44d810d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810e3f0 con 0x7f44d80fcfc0 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44dc8a6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810e560 con 0x7f44d80fbd10 2024-03-14T13:04:01.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.785+0000 7f44d5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54324/0 (socket says 172.21.15.111:54324) 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 -- 172.21.15.111:0/3459421161 learned_addr learned my addr 172.21.15.111:0/3459421161 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d659c700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 0x7f44d810d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 -- 172.21.15.111:0/3459421161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 msgr2=0x7f44d810d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 0x7f44d810d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 -- 172.21.15.111:0/3459421161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc005040 con 0x7f44d80fcfc0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44d5d9b700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0137b0 tx=0x7f44cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44cc00e070 con 0x7f44d80fcfc0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44cc01c400 con 0x7f44d80fcfc0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d8112ec0 con 0x7f44d80fcfc0 2024-03-14T13:04:01.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f44cc00b650 con 0x7f44d80fcfc0 2024-03-14T13:04:01.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.786+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81133f0 con 0x7f44d80fcfc0 2024-03-14T13:04:01.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.787+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8101b50 con 0x7f44d80fcfc0 2024-03-14T13:04:01.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.788+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f44cc01c9e0 con 0x7f44d80fcfc0 2024-03-14T13:04:01.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.788+0000 7f44c6ffd700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 0x7f44c006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:01.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.788+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f44cc08caf0 con 0x7f44d80fcfc0 2024-03-14T13:04:01.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.788+0000 7f44d659c700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 0x7f44c006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:01.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.788+0000 7f44d659c700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 0x7f44c006e8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009b80 tx=0x7f44c800b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:01.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:01.792+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f44cc05ad20 con 0x7f44d80fcfc0 2024-03-14T13:04:02.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.002+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d802b3e0 con 0x7f44c006c440 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.011+0000 7f44c6ffd700 1 -- 172.21.15.111:0/3459421161 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f44d802b3e0 con 0x7f44c006c440 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (14s) 3s ago 11m 16.5M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (12m) 3s ago 12m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (10m) 57s ago 10m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (3m) 3s ago 11m 84.7M - 9.4.7 954c08fa6188 8e31a4322099 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (5m) 3s ago 5m 64.0M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (5m) 3s ago 5m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (5m) 57s ago 5m 90.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (5m) 57s ago 5m 157M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:04:02.011 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (2m) 3s ago 13m 497M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (86s) 57s ago 10m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (13m) 3s ago 13m 55.8M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (10m) 57s ago 10m 44.8M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (72s) 3s ago 11m 9483k - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (63s) 57s ago 10m 7088k - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (9m) 3s ago 9m 209M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (9m) 3s ago 9m 235M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (8m) 3s ago 8m 196M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (8m) 57s ago 8m 240M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (7m) 57s ago 7m 213M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (7m) 57s ago 7m 203M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:04:02.012 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (40s) 3s ago 10m 49.2M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 msgr2=0x7f44c006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 0x7f44c006e8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009b80 tx=0x7f44c800b040 comp rx=0 tx=0).stop 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 msgr2=0x7f44d810ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f44cc0137b0 tx=0x7f44cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 shutdown_connections 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44d80fbd10 0x7f44d810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f44c006c440 0x7f44c006e8f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 --2- 172.21.15.111:0/3459421161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44d80fcfc0 0x7f44d810ddf0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 >> 172.21.15.111:0/3459421161 conn(0x7f44d80f7490 msgr2=0x7f44d8066860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:02.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 shutdown_connections 2024-03-14T13:04:02.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.013+0000 7f44dc8a6700 1 -- 172.21.15.111:0/3459421161 wait complete. 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- 172.21.15.111:0/505660839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fcf80 msgr2=0x7f24c00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 --2- 172.21.15.111:0/505660839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fcf80 0x7f24c00fd3f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f24b0004940 tx=0x7f24b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- 172.21.15.111:0/505660839 shutdown_connections 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 --2- 172.21.15.111:0/505660839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fcf80 0x7f24c00fd3f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 --2- 172.21.15.111:0/505660839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fbcd0 0x7f24c00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- 172.21.15.111:0/505660839 >> 172.21.15.111:0/505660839 conn(0x7f24c00f7430 msgr2=0x7f24c00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:02.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- 172.21.15.111:0/505660839 shutdown_connections 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- 172.21.15.111:0/505660839 wait complete. 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 Processor -- start 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.113+0000 7f24c59a3700 1 -- start start 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fcf80 0x7f24c010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c010d390 con 0x7f24c00fbcd0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c010dbe0 con 0x7f24c00fcf80 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54342/0 (socket says 172.21.15.111:54342) 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 -- 172.21.15.111:0/3149595174 learned_addr learned my addr 172.21.15.111:0/3149595174 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24be7fc700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fcf80 0x7f24c010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 -- 172.21.15.111:0/3149595174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fcf80 msgr2=0x7f24c010cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fcf80 0x7f24c010cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 -- 172.21.15.111:0/3149595174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0005040 con 0x7f24c00fbcd0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24beffd700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24a800fce0 tx=0x7f24a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24a8007ea0 con 0x7f24c00fbcd0 2024-03-14T13:04:02.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24a800dd80 con 0x7f24c00fbcd0 2024-03-14T13:04:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c010de60 con 0x7f24c00fbcd0 2024-03-14T13:04:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24a8012460 con 0x7f24c00fbcd0 2024-03-14T13:04:02.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.114+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c010e420 con 0x7f24c00fbcd0 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.115+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c005f5f0 con 0x7f24c00fbcd0 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.116+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f24a8009360 con 0x7f24c00fbcd0 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.116+0000 7f24b77fe700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 0x7f24ac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.116+0000 7f24be7fc700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 0x7f24ac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.116+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f24a808db00 con 0x7f24c00fbcd0 2024-03-14T13:04:02.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.117+0000 7f24be7fc700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 0x7f24ac06e8f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24b000b3b0 tx=0x7f24b0009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:02.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.121+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f24a805bd30 con 0x7f24c00fbcd0 2024-03-14T13:04:02.348 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:02 smithi111 ceph-mon[39688]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.348 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:02 smithi111 ceph-mon[39688]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.348 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:02 smithi111 ceph-mon[39688]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.403+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f24c010e820 con 0x7f24c00fbcd0 2024-03-14T13:04:02.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.405+0000 7f24b77fe700 1 -- 172.21.15.111:0/3149595174 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f24a805b8c0 con 0x7f24c00fbcd0 2024-03-14T13:04:02.404 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:04:02.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:04:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:04:02.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:04:02.406 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 msgr2=0x7f24ac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 0x7f24ac06e8f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24b000b3b0 tx=0x7f24b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 msgr2=0x7f24c010c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24a800fce0 tx=0x7f24a80046b0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 shutdown_connections 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f24ac06c440 0x7f24ac06e8f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c00fbcd0 0x7f24c010c850 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 --2- 172.21.15.111:0/3149595174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c00fcf80 0x7f24c010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 >> 172.21.15.111:0/3149595174 conn(0x7f24c00f7430 msgr2=0x7f24c00f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 shutdown_connections 2024-03-14T13:04:02.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.407+0000 7f24c59a3700 1 -- 172.21.15.111:0/3149595174 wait complete. 2024-03-14T13:04:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.519+0000 7f30f0d95700 1 -- 172.21.15.111:0/3201485887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 msgr2=0x7f30ec0ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.519+0000 7f30f0d95700 1 --2- 172.21.15.111:0/3201485887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec0ff9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f30dc004940 tx=0x7f30dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- 172.21.15.111:0/3201485887 shutdown_connections 2024-03-14T13:04:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 --2- 172.21.15.111:0/3201485887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30ec0fff00 0x7f30ec102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 --2- 172.21.15.111:0/3201485887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec0ff9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- 172.21.15.111:0/3201485887 >> 172.21.15.111:0/3201485887 conn(0x7f30ec0f7430 msgr2=0x7f30ec0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- 172.21.15.111:0/3201485887 shutdown_connections 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- 172.21.15.111:0/3201485887 wait complete. 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 Processor -- start 2024-03-14T13:04:02.520 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- start start 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30ec0fff00 0x7f30ec10de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec10e510 con 0x7f30ec0662e0 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30f0d95700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec10ed00 con 0x7f30ec0fff00 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30ea59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54350/0 (socket says 172.21.15.111:54350) 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.520+0000 7f30ea59c700 1 -- 172.21.15.111:0/1847371707 learned_addr learned my addr 172.21.15.111:0/1847371707 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:02.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30ea59c700 1 -- 172.21.15.111:0/1847371707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30ec0fff00 msgr2=0x7f30ec10de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30ea59c700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30ec0fff00 0x7f30ec10de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30ea59c700 1 -- 172.21.15.111:0/1847371707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc005040 con 0x7f30ec0662e0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30ea59c700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00b3b0 tx=0x7f30dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30dc00e040 con 0x7f30ec0662e0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30dc00bd20 con 0x7f30ec0662e0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30dc01c630 con 0x7f30ec0662e0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec10ef80 con 0x7f30ec0662e0 2024-03-14T13:04:02.522 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.521+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec10f540 con 0x7f30ec0662e0 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.522+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec05f5f0 con 0x7f30ec0662e0 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.523+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f30dc013960 con 0x7f30ec0662e0 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.523+0000 7f30daffd700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 0x7f30d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.523+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f30dc08c0c0 con 0x7f30ec0662e0 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.523+0000 7f30e9d9b700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 0x7f30d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:02.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.523+0000 7f30e9d9b700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 0x7f30d406e8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30e0006fd0 tx=0x7f30e0006db0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:02.527 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.528+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f30dc05a390 con 0x7f30ec0662e0 2024-03-14T13:04:02.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.753+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f30ec10f930 con 0x7f30d406c440 2024-03-14T13:04:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.754+0000 7f30daffd700 1 -- 172.21.15.111:0/1847371707 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f30ec10f930 con 0x7f30d406c440 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/2 daemons upgraded", 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading grafana daemons", 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:04:02.754 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:04:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:02 smithi137 ceph-mon[50336]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:02 smithi137 ceph-mon[50336]: from='client.14786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:02 smithi137 ceph-mon[50336]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 msgr2=0x7f30d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 0x7f30d406e8f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30e0006fd0 tx=0x7f30e0006db0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 msgr2=0x7f30ec10d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00b3b0 tx=0x7f30dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 shutdown_connections 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f30d406c440 0x7f30d406e8f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30ec0662e0 0x7f30ec10d940 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 --2- 172.21.15.111:0/1847371707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30ec0fff00 0x7f30ec10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 >> 172.21.15.111:0/1847371707 conn(0x7f30ec0f7430 msgr2=0x7f30ec0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 shutdown_connections 2024-03-14T13:04:02.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:02.756+0000 7f30f0d95700 1 -- 172.21.15.111:0/1847371707 wait complete. 2024-03-14T13:04:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3149595174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: pgmap v76: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:03.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:03 smithi111 ceph-mon[39688]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3149595174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: pgmap v76: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:03.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:03 smithi137 ceph-mon[50336]: from='client.14798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:04.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:04 smithi111 ceph-mon[39688]: Upgrade: Updating grafana.smithi111 2024-03-14T13:04:04.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:04 smithi111 ceph-mon[39688]: Deploying daemon grafana.smithi111 on smithi111 2024-03-14T13:04:04.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:04 smithi137 ceph-mon[50336]: Upgrade: Updating grafana.smithi111 2024-03-14T13:04:04.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:04 smithi137 ceph-mon[50336]: Deploying daemon grafana.smithi111 on smithi111 2024-03-14T13:04:05.628 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:05 smithi111 ceph-mon[39688]: pgmap v77: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:05 smithi137 ceph-mon[50336]: pgmap v77: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:07.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:07.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:07 smithi111 ceph-mon[39688]: pgmap v78: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:08.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:07 smithi137 ceph-mon[50336]: pgmap v78: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:10 smithi137 ceph-mon[50336]: pgmap v79: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:10 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:10 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:10 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:10 smithi111 ceph-mon[39688]: pgmap v79: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:10 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:10 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:10.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:10 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:11.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:11 smithi111 ceph-mon[39688]: pgmap v80: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:11 smithi137 ceph-mon[50336]: pgmap v80: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:13 smithi137 ceph-mon[50336]: pgmap v81: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:14.017 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:13 smithi111 ceph-mon[39688]: pgmap v81: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:15.974 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:15 smithi111 ceph-mon[39688]: pgmap v82: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:15.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:15 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:15.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:15 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:15.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:15 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:15.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:15 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:15 smithi137 ceph-mon[50336]: pgmap v82: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:15 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:15 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:15 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:16.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:15 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:18.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:17 smithi137 ceph-mon[50336]: pgmap v83: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:18.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:17 smithi111 ceph-mon[39688]: pgmap v83: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: pgmap v84: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.220 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: pgmap v84: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-14T13:04:20.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:20 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-14T13:04:20.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-14T13:04:20.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-14T13:04:20.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:20.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:20 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:21.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:21 smithi137 ceph-mon[50336]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:04:21.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:21 smithi137 ceph-mon[50336]: Upgrade: Finalizing container_image settings 2024-03-14T13:04:21.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:21 smithi137 ceph-mon[50336]: Upgrade: Complete! 2024-03-14T13:04:21.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:21 smithi111 ceph-mon[39688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-14T13:04:21.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:21 smithi111 ceph-mon[39688]: Upgrade: Finalizing container_image settings 2024-03-14T13:04:21.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:21 smithi111 ceph-mon[39688]: Upgrade: Complete! 2024-03-14T13:04:22.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:22 smithi137 ceph-mon[50336]: pgmap v85: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:22.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:22 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:22 smithi111 ceph-mon[39688]: pgmap v85: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:22.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:22 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:24.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:24 smithi137 ceph-mon[50336]: pgmap v86: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:24 smithi111 ceph-mon[39688]: pgmap v86: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:26.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:26 smithi137 ceph-mon[50336]: pgmap v87: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:26.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:26 smithi111 ceph-mon[39688]: pgmap v87: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:28.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:28 smithi137 ceph-mon[50336]: pgmap v88: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:28 smithi111 ceph-mon[39688]: pgmap v88: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:30.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:30 smithi137 ceph-mon[50336]: pgmap v89: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:30 smithi111 ceph-mon[39688]: pgmap v89: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:31.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:31 smithi111 ceph-mon[39688]: pgmap v90: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:31.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:31 smithi137 ceph-mon[50336]: pgmap v90: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:32.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/1211500547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 msgr2=0x7fb8e80fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:32.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/1211500547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80fc0e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004940 tx=0x7fb8d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:32.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/1211500547 shutdown_connections 2024-03-14T13:04:32.873 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/1211500547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 0x7fb8e80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/1211500547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80fc0e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/1211500547 >> 172.21.15.111:0/1211500547 conn(0x7fb8e80f7430 msgr2=0x7fb8e80f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/1211500547 shutdown_connections 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/1211500547 wait complete. 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 Processor -- start 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.872+0000 7fb8ee8d0700 1 -- start start 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 0x7fb8e80fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80fe850 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80fe990 con 0x7fb8e80fcf80 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42616/0 (socket says 172.21.15.111:42616) 2024-03-14T13:04:32.874 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 -- 172.21.15.111:0/3184691198 learned_addr learned my addr 172.21.15.111:0/3184691198 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e77fe700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 0x7fb8e80fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 -- 172.21.15.111:0/3184691198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 msgr2=0x7fb8e80fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 0x7fb8e80fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 -- 172.21.15.111:0/3184691198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8005040 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e7fff700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800b3b0 tx=0x7fb8d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d800e040 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8d800bc50 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80fec10 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d801c630 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.873+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e81159c0 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.874+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8101b10 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.875+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fb8d8013420 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.875+0000 7fb8e4ff9700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 0x7fb8d006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.876+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fb8d808c0d0 con 0x7fb8e80fbcd0 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.876+0000 7fb8e77fe700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 0x7fb8d006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:32.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.876+0000 7fb8e77fe700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 0x7fb8d006e8f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8100380 tx=0x7fb8dc00b040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:32.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:32.880+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fb8d8056870 con 0x7fb8e80fbcd0 2024-03-14T13:04:33.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.097+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8e8104800 con 0x7fb8d006c440 2024-03-14T13:04:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.098+0000 7fb8e4ff9700 1 -- 172.21.15.111:0/3184691198 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fb8e8104800 con 0x7fb8d006c440 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 msgr2=0x7fb8d006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 0x7fb8d006e8f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8100380 tx=0x7fb8dc00b040 comp rx=0 tx=0).stop 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 msgr2=0x7fb8e80ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800b3b0 tx=0x7fb8d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 shutdown_connections 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fb8d006c440 0x7fb8d006e8f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e80fbcd0 0x7fb8e80ffda0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 --2- 172.21.15.111:0/3184691198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e80fcf80 0x7fb8e80fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 >> 172.21.15.111:0/3184691198 conn(0x7fb8e80f7430 msgr2=0x7fb8e8065c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 shutdown_connections 2024-03-14T13:04:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:33.100+0000 7fb8ee8d0700 1 -- 172.21.15.111:0/3184691198 wait complete. 2024-03-14T13:04:33.850 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-14T13:04:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:33 smithi137 ceph-mon[50336]: pgmap v91: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:33 smithi137 ceph-mon[50336]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:34.121 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:34.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:33 smithi111 ceph-mon[39688]: pgmap v91: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:34.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:33 smithi111 ceph-mon[39688]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:35 smithi137 ceph-mon[50336]: pgmap v92: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:36.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- 172.21.15.111:0/184714492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c101d70 msgr2=0x7fd66c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:36.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 --2- 172.21.15.111:0/184714492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c101d70 0x7fd66c102140 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd65c004940 tx=0x7fd65c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:36.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- 172.21.15.111:0/184714492 shutdown_connections 2024-03-14T13:04:36.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 --2- 172.21.15.111:0/184714492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c102710 0x7fd66c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 --2- 172.21.15.111:0/184714492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c101d70 0x7fd66c102140 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- 172.21.15.111:0/184714492 >> 172.21.15.111:0/184714492 conn(0x7fd66c072bc0 msgr2=0x7fd66c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:36.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- 172.21.15.111:0/184714492 shutdown_connections 2024-03-14T13:04:36.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- 172.21.15.111:0/184714492 wait complete. 2024-03-14T13:04:36.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 Processor -- start 2024-03-14T13:04:36.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.007+0000 7fd6727f3700 1 -- start start 2024-03-14T13:04:36.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c101d70 0x7fd66c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:36.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c103730 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c103870 con 0x7fd66c101d70 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42632/0 (socket says 172.21.15.111:42632) 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 -- 172.21.15.111:0/1472526686 learned_addr learned my addr 172.21.15.111:0/1472526686 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66bfff700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c101d70 0x7fd66c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 -- 172.21.15.111:0/1472526686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c101d70 msgr2=0x7fd66c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c101d70 0x7fd66c102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 -- 172.21.15.111:0/1472526686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c005040 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd66b7fe700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd66000b790 tx=0x7fd660009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd66000f6f0 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd66000fd00 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c119d70 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6600103a0 con 0x7fd66c102710 2024-03-14T13:04:36.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.008+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c11a280 con 0x7fd66c102710 2024-03-14T13:04:36.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.009+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0fd4c0 con 0x7fd66c102710 2024-03-14T13:04:36.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.010+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fd66000f850 con 0x7fd66c102710 2024-03-14T13:04:36.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.010+0000 7fd668ff9700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 0x7fd65406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:36.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.010+0000 7fd66bfff700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 0x7fd65406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:36.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.010+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd66008b2d0 con 0x7fd66c102710 2024-03-14T13:04:36.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.011+0000 7fd66bfff700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 0x7fd65406e8f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd65c004f90 tx=0x7fd65c009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:36.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.015+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fd660055a70 con 0x7fd66c102710 2024-03-14T13:04:36.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:35 smithi111 ceph-mon[39688]: pgmap v92: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:36.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.286+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd66c108e00 con 0x7fd66c102710 2024-03-14T13:04:36.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.286+0000 7fd668ff9700 1 -- 172.21.15.111:0/1472526686 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7fd660059090 con 0x7fd66c102710 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 msgr2=0x7fd65406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 0x7fd65406e8f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd65c004f90 tx=0x7fd65c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 msgr2=0x7fd66c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd66000b790 tx=0x7fd660009220 comp rx=0 tx=0).stop 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 shutdown_connections 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66c101d70 0x7fd66c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd65406c440 0x7fd65406e8f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 --2- 172.21.15.111:0/1472526686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66c102710 0x7fd66c103110 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:36.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 >> 172.21.15.111:0/1472526686 conn(0x7fd66c072bc0 msgr2=0x7fd66c065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:36.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 shutdown_connections 2024-03-14T13:04:36.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:36.288+0000 7fd6727f3700 1 -- 172.21.15.111:0/1472526686 wait complete. 2024-03-14T13:04:36.298 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:04:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:36 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1472526686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:36 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:37.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-14T13:04:37.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:36 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1472526686' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:37.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:36 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:37.443 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:37 smithi111 ceph-mon[39688]: pgmap v93: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:38.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:37 smithi137 ceph-mon[50336]: pgmap v93: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.345+0000 7f7a7c83f700 1 -- 172.21.15.111:0/2847264033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 msgr2=0x7f7a78103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.345+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/2847264033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a78103240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004940 tx=0x7f7a60009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- 172.21.15.111:0/2847264033 shutdown_connections 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/2847264033 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a780fb4a0 0x7f7a780fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/2847264033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a78103240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- 172.21.15.111:0/2847264033 >> 172.21.15.111:0/2847264033 conn(0x7f7a780714c0 msgr2=0x7f7a780718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- 172.21.15.111:0/2847264033 shutdown_connections 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- 172.21.15.111:0/2847264033 wait complete. 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 Processor -- start 2024-03-14T13:04:39.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- start start 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a780fb4a0 0x7f7a780fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a780fd150 con 0x7f7a78102e70 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a7c83f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a780fd290 con 0x7f7a780fb4a0 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.346+0000 7f7a75d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42642/0 (socket says 172.21.15.111:42642) 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 -- 172.21.15.111:0/1589261303 learned_addr learned my addr 172.21.15.111:0/1589261303 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 -- 172.21.15.111:0/1589261303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a780fb4a0 msgr2=0x7f7a780fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a780fb4a0 0x7f7a780fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 -- 172.21.15.111:0/1589261303 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60005040 con 0x7f7a78102e70 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a75d9b700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c830 tx=0x7f7a68004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a6800a760 con 0x7f7a78102e70 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a6800ad70 con 0x7f7a78102e70 2024-03-14T13:04:39.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a78119d70 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a680113a0 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.347+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7811a1f0 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.348+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a78100110 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.349+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f7a68012400 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.349+0000 7f7a6effd700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 0x7f7a6406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.349+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f7a6808b540 con 0x7f7a78102e70 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.349+0000 7f7a7659c700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 0x7f7a6406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.349+0000 7f7a7659c700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 0x7f7a6406e8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004f90 tx=0x7f7a60009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.354+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f7a68055ce0 con 0x7f7a78102e70 2024-03-14T13:04:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.623+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7a78109e00 con 0x7f7a78102e70 2024-03-14T13:04:39.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.624+0000 7f7a6effd700 1 -- 172.21.15.111:0/1589261303 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f7a68059300 con 0x7f7a78102e70 2024-03-14T13:04:39.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 msgr2=0x7f7a6406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 0x7f7a6406e8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7a60004f90 tx=0x7f7a60009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 msgr2=0x7f7a780fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800c830 tx=0x7f7a68004470 comp rx=0 tx=0).stop 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 shutdown_connections 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a780fb4a0 0x7f7a780fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f7a6406c440 0x7f7a6406e8f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 --2- 172.21.15.111:0/1589261303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a78102e70 0x7f7a780fcbc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 >> 172.21.15.111:0/1589261303 conn(0x7f7a780714c0 msgr2=0x7f7a780fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 shutdown_connections 2024-03-14T13:04:39.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:39.626+0000 7f7a7c83f700 1 -- 172.21.15.111:0/1589261303 wait complete. 2024-03-14T13:04:39.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)" 2024-03-14T13:04:39.980 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:39 smithi111 ceph-mon[39688]: pgmap v94: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:39.980 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:39 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/1589261303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:40.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:39 smithi137 ceph-mon[50336]: pgmap v94: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:39 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/1589261303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:40.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-14T13:04:40.602 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:41.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:41 smithi111 ceph-mon[39688]: pgmap v95: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:41 smithi137 ceph-mon[50336]: pgmap v95: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/395083218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24102710 msgr2=0x7f1f240fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/395083218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24102710 0x7f1f240fbd80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004940 tx=0x7f1f20009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/395083218 shutdown_connections 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/395083218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24102710 0x7f1f240fbd80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/395083218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24101d70 0x7f1f24102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/395083218 >> 172.21.15.111:0/395083218 conn(0x7f1f24072bc0 msgr2=0x7f1f24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.647+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/395083218 shutdown_connections 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/395083218 wait complete. 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 Processor -- start 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 -- start start 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24102710 0x7f1f2410cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2410d390 con 0x7f1f24101d70 2024-03-14T13:04:42.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f2b9c5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2410d500 con 0x7f1f24102710 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f29761700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f29761700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58242/0 (socket says 172.21.15.111:58242) 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f29761700 1 -- 172.21.15.111:0/3692662363 learned_addr learned my addr 172.21.15.111:0/3692662363 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.648+0000 7f1f28f60700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24102710 0x7f1f2410cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f29761700 1 -- 172.21.15.111:0/3692662363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24102710 msgr2=0x7f1f2410cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f29761700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24102710 0x7f1f2410cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f29761700 1 -- 172.21.15.111:0/3692662363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20005040 con 0x7f1f24101d70 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f29761700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800b790 tx=0x7f1f18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f1800f6f0 con 0x7f1f24101d70 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f1800fd00 con 0x7f1f24101d70 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f24119ef0 con 0x7f1f24101d70 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f180103a0 con 0x7f1f24101d70 2024-03-14T13:04:42.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.649+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2411a2c0 con 0x7f1f24101d70 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.650+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f240ff080 con 0x7f1f24101d70 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.650+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f1f1800f850 con 0x7f1f24101d70 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.651+0000 7f1f15ffb700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 0x7f1f1006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.651+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f1f1808b2e0 con 0x7f1f24101d70 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.651+0000 7f1f28f60700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 0x7f1f1006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:42.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.651+0000 7f1f28f60700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 0x7f1f1006e8f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004890 tx=0x7f1f20009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:42.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.656+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f1f18059510 con 0x7f1f24101d70 2024-03-14T13:04:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.932+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1f2404bed0 con 0x7f1f24101d70 2024-03-14T13:04:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.933+0000 7f1f15ffb700 1 -- 172.21.15.111:0/3692662363 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f1f180590a0 con 0x7f1f24101d70 2024-03-14T13:04:42.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 msgr2=0x7f1f1006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:42.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 0x7f1f1006e8f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1f20004890 tx=0x7f1f20009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 msgr2=0x7f1f2410c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800b790 tx=0x7f1f18009250 comp rx=0 tx=0).stop 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 shutdown_connections 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f1f1006c440 0x7f1f1006e8f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f24101d70 0x7f1f2410c880 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 --2- 172.21.15.111:0/3692662363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f24102710 0x7f1f2410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 >> 172.21.15.111:0/3692662363 conn(0x7f1f24072bc0 msgr2=0x7f1f240fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 shutdown_connections 2024-03-14T13:04:42.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:42.935+0000 7f1f2b9c5700 1 -- 172.21.15.111:0/3692662363 wait complete. 2024-03-14T13:04:42.944 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:04:43.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-14T13:04:43.997 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:43 smithi137 ceph-mon[50336]: pgmap v96: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:43 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3692662363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:43 smithi111 ceph-mon[39688]: pgmap v96: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:44.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:43 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3692662363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:04:45.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 -- 172.21.15.111:0/1999105311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac102710 msgr2=0x7f76ac0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 --2- 172.21.15.111:0/1999105311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac102710 0x7f76ac0fbd80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f76a0004940 tx=0x7f76a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 -- 172.21.15.111:0/1999105311 shutdown_connections 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 --2- 172.21.15.111:0/1999105311 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac102710 0x7f76ac0fbd80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 --2- 172.21.15.111:0/1999105311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac101d70 0x7f76ac102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 -- 172.21.15.111:0/1999105311 >> 172.21.15.111:0/1999105311 conn(0x7f76ac072bc0 msgr2=0x7f76ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 -- 172.21.15.111:0/1999105311 shutdown_connections 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.859+0000 7f76b2265700 1 -- 172.21.15.111:0/1999105311 wait complete. 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 Processor -- start 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 -- start start 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac102710 0x7f76ac10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac10d310 con 0x7f76ac101d70 2024-03-14T13:04:45.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76b2265700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac10d480 con 0x7f76ac102710 2024-03-14T13:04:45.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:45.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58272/0 (socket says 172.21.15.111:58272) 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.860+0000 7f76abfff700 1 -- 172.21.15.111:0/2381014893 learned_addr learned my addr 172.21.15.111:0/2381014893 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76abfff700 1 -- 172.21.15.111:0/2381014893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac102710 msgr2=0x7f76ac10cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76abfff700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac102710 0x7f76ac10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76abfff700 1 -- 172.21.15.111:0/2381014893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a0005040 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76abfff700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f769c00b790 tx=0x7f769c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f769c00f770 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f769c00fd80 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac119e70 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f769c00d310 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.861+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac11a2a0 con 0x7f76ac101d70 2024-03-14T13:04:45.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.862+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac0ff080 con 0x7f76ac101d70 2024-03-14T13:04:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.862+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f769c010420 con 0x7f76ac101d70 2024-03-14T13:04:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.863+0000 7f76a8ff9700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 0x7f769406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.863+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f769c08b4b0 con 0x7f76ac101d70 2024-03-14T13:04:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.863+0000 7f76ab7fe700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 0x7f769406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:45.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.863+0000 7f76ab7fe700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 0x7f769406e8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f76a000b3b0 tx=0x7f76a0009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:45.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:45.868+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f769c0596e0 con 0x7f76ac101d70 2024-03-14T13:04:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:45 smithi111 ceph-mon[39688]: pgmap v97: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:45 smithi137 ceph-mon[50336]: pgmap v97: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:46.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:46.077+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}) v1 -- 0x7f76ac108d80 con 0x7f769406c440 2024-03-14T13:04:46.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:46 smithi111 ceph-mon[39688]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:46 smithi137 ceph-mon[50336]: from='client.14818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:47.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.746+0000 7f76a8ff9700 1 -- 172.21.15.111:0/2381014893 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+5093 (secure 0 0 0) 0x7f76ac108d80 con 0x7f769406c440 2024-03-14T13:04:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 msgr2=0x7f769406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 0x7f769406e8f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f76a000b3b0 tx=0x7f76a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:47.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 msgr2=0x7f76ac10c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f769c00b790 tx=0x7f769c009250 comp rx=0 tx=0).stop 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 shutdown_connections 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f769406c440 0x7f769406e8f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76ac101d70 0x7f76ac10c800 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 --2- 172.21.15.111:0/2381014893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac102710 0x7f76ac10cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 >> 172.21.15.111:0/2381014893 conn(0x7f76ac072bc0 msgr2=0x7f76ac0fafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 shutdown_connections 2024-03-14T13:04:47.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:47.749+0000 7f76b2265700 1 -- 172.21.15.111:0/2381014893 wait complete. 2024-03-14T13:04:47.759 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:04:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:47 smithi111 ceph-mon[39688]: pgmap v98: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:47 smithi137 ceph-mon[50336]: pgmap v98: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:48.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch ps' 2024-03-14T13:04:48.768 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:50.086 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:49 smithi111 ceph-mon[39688]: pgmap v99: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:50.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:49 smithi137 ceph-mon[50336]: pgmap v99: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 -- 172.21.15.111:0/2430048514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 msgr2=0x7fd05c0fb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 --2- 172.21.15.111:0/2430048514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fb870 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd04c004940 tx=0x7fd04c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 -- 172.21.15.111:0/2430048514 shutdown_connections 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 --2- 172.21.15.111:0/2430048514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fb870 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 --2- 172.21.15.111:0/2430048514 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 0x7fd05c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 -- 172.21.15.111:0/2430048514 >> 172.21.15.111:0/2430048514 conn(0x7fd05c072380 msgr2=0x7fd05c072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 -- 172.21.15.111:0/2430048514 shutdown_connections 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.702+0000 7fd060803700 1 -- 172.21.15.111:0/2430048514 wait complete. 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 Processor -- start 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 -- start start 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 0x7fd05c0fcb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0fe530 con 0x7fd05c0fb420 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd060803700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0fd080 con 0x7fd05c102f80 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51474/0 (socket says 172.21.15.111:51474) 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 -- 172.21.15.111:0/3896562888 learned_addr learned my addr 172.21.15.111:0/3896562888 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd059d9b700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 0x7fd05c0fcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 -- 172.21.15.111:0/3896562888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 msgr2=0x7fd05c0fcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 0x7fd05c0fcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 -- 172.21.15.111:0/3896562888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c005040 con 0x7fd05c0fb420 2024-03-14T13:04:50.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd05a59c700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd050002a50 tx=0x7fd0500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd050007cc0 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd05000f040 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.703+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd050009540 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.704+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c0fd360 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.704+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c11a020 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.704+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c1053b0 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.705+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fd0500096a0 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.705+0000 7fd04affd700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 0x7fd04406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.705+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd05008b220 con 0x7fd05c0fb420 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.705+0000 7fd059d9b700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 0x7fd04406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.706+0000 7fd059d9b700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 0x7fd04406e8f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd04c004f90 tx=0x7fd04c009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:50.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.710+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fd0500559c0 con 0x7fd05c0fb420 2024-03-14T13:04:50.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.914+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd05c0fe6a0 con 0x7fd04406c440 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.924+0000 7fd04affd700 1 -- 172.21.15.111:0/3896562888 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fd05c0fe6a0 con 0x7fd04406c440 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (63s) 36s ago 12m 16.5M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (12m) 36s ago 12m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (11m) 106s ago 11m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:04:50.923 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (42s) 36s ago 11m 47.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (6m) 36s ago 6m 64.0M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (6m) 36s ago 6m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (6m) 106s ago 6m 90.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (6m) 106s ago 6m 157M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (3m) 36s ago 14m 498M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (2m) 106s ago 11m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (14m) 36s ago 14m 55.9M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (11m) 106s ago 11m 44.8M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (2m) 36s ago 12m 16.2M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (112s) 106s ago 11m 7088k - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (10m) 36s ago 10m 209M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (9m) 36s ago 9m 235M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (9m) 36s ago 9m 196M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (9m) 106s ago 9m 240M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (8m) 106s ago 8m 213M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:04:50.924 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (8m) 106s ago 8m 203M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:04:50.925 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (89s) 36s ago 11m 49.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:04:50.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 msgr2=0x7fd04406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:50.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 0x7fd04406e8f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd04c004f90 tx=0x7fd04c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:50.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 msgr2=0x7fd05c0fc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd050002a50 tx=0x7fd0500046b0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 shutdown_connections 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd04406c440 0x7fd04406e8f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd05c0fb420 0x7fd05c0fc600 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 --2- 172.21.15.111:0/3896562888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd05c102f80 0x7fd05c0fcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 >> 172.21.15.111:0/3896562888 conn(0x7fd05c072380 msgr2=0x7fd05c0f9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 shutdown_connections 2024-03-14T13:04:50.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:50.926+0000 7fd060803700 1 -- 172.21.15.111:0/3896562888 wait complete. 2024-03-14T13:04:51.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:50 smithi111 ceph-mon[39688]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:04:51.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:50 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:04:51.640 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-14T13:04:51.641 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T13:04:51.641 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs true' 2024-03-14T13:04:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:51 smithi111 ceph-mon[39688]: pgmap v100: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:51 smithi111 ceph-mon[39688]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:51 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:51.912 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:52.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:51 smithi137 ceph-mon[50336]: pgmap v100: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:52.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:51 smithi137 ceph-mon[50336]: from='client.14822 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:04:52.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:51 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 -- 172.21.15.111:0/574048176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 msgr2=0x7f6538102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 --2- 172.21.15.111:0/574048176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538102140 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f652c004940 tx=0x7f652c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 -- 172.21.15.111:0/574048176 shutdown_connections 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 --2- 172.21.15.111:0/574048176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f65380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 --2- 172.21.15.111:0/574048176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538102140 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 -- 172.21.15.111:0/574048176 >> 172.21.15.111:0/574048176 conn(0x7f6538072bc0 msgr2=0x7f6538072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 -- 172.21.15.111:0/574048176 shutdown_connections 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.696+0000 7f653cef4700 1 -- 172.21.15.111:0/574048176 wait complete. 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 Processor -- start 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 -- start start 2024-03-14T13:04:53.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f6538112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6538113590 con 0x7f6538101d70 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653cef4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653810c8f0 con 0x7f6538102710 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51486/0 (socket says 172.21.15.111:51486) 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 -- 172.21.15.111:0/3013943114 learned_addr learned my addr 172.21.15.111:0/3013943114 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f6535d9b700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f6538112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 -- 172.21.15.111:0/3013943114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 msgr2=0x7f6538112eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f6538112eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 -- 172.21.15.111:0/3013943114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c005040 con 0x7f6538101d70 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f6535d9b700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f6538112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f653659c700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b3b0 tx=0x7f652c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.697+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652c00e040 con 0x7f6538101d70 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.698+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653810cb70 con 0x7f6538101d70 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.698+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f652c013420 con 0x7f6538101d70 2024-03-14T13:04:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.698+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652c01c630 con 0x7f6538101d70 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.698+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653810d130 con 0x7f6538101d70 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.698+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65380ff080 con 0x7f6538101d70 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.699+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f652c013970 con 0x7f6538101d70 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.699+0000 7f6526ffd700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 0x7f652006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.699+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f652c08c740 con 0x7f6538101d70 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.699+0000 7f6535d9b700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 0x7f652006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.700+0000 7f6535d9b700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 0x7f652006e8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6528006fd0 tx=0x7f6528008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.704+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f652c05a970 con 0x7f6538101d70 2024-03-14T13:04:53.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.898+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f6538108da0 con 0x7f6538101d70 2024-03-14T13:04:53.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.905+0000 7f6526ffd700 1 -- 172.21.15.111:0/3013943114 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=0 v48)=0 v48) v1 ==== 125+0+0 (secure 0 0 0) 0x7f652c05a500 con 0x7f6538101d70 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 msgr2=0x7f652006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 0x7f652006e8f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6528006fd0 tx=0x7f6528008040 comp rx=0 tx=0).stop 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 msgr2=0x7f6538112970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b3b0 tx=0x7f652c009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 shutdown_connections 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f652006c440 0x7f652006e8f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6538101d70 0x7f6538112970 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 --2- 172.21.15.111:0/3013943114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6538102710 0x7f6538112eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 >> 172.21.15.111:0/3013943114 conn(0x7f6538072bc0 msgr2=0x7f65380fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 shutdown_connections 2024-03-14T13:04:53.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:53.907+0000 7f653cef4700 1 -- 172.21.15.111:0/3013943114 wait complete. 2024-03-14T13:04:54.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:53 smithi111 ceph-mon[39688]: pgmap v101: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:54.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:53 smithi137 ceph-mon[50336]: pgmap v101: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:04:54.704 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-14T13:04:54.704 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T13:04:54.705 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-14T13:04:54.977 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:54 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3013943114' entity='client.admin' 2024-03-14T13:04:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:54 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:54 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:54 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:55.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:54 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:54 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3013943114' entity='client.admin' 2024-03-14T13:04:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:54 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:04:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:54 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:04:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:54 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:04:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:54 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:04:56.242 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:55 smithi111 ceph-mon[39688]: pgmap v102: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:56.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:55 smithi137 ceph-mon[50336]: pgmap v102: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:04:56.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 -- 172.21.15.111:0/699869002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 msgr2=0x7f0594106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 --2- 172.21.15.111:0/699869002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594106f40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0590004940 tx=0x7f0590009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 -- 172.21.15.111:0/699869002 shutdown_connections 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 --2- 172.21.15.111:0/699869002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594106f40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 --2- 172.21.15.111:0/699869002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05940fded0 0x7f05940fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 -- 172.21.15.111:0/699869002 >> 172.21.15.111:0/699869002 conn(0x7f0594072bc0 msgr2=0x7f0594072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 -- 172.21.15.111:0/699869002 shutdown_connections 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.788+0000 7f059bcce700 1 -- 172.21.15.111:0/699869002 wait complete. 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 Processor -- start 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 -- start start 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05940fded0 0x7f0594113a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0594114640 con 0x7f05940fe870 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f059bcce700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059410d9a0 con 0x7f05940fded0 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:56.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51504/0 (socket says 172.21.15.111:51504) 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 -- 172.21.15.111:0/2323825879 learned_addr learned my addr 172.21.15.111:0/2323825879 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 -- 172.21.15.111:0/2323825879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05940fded0 msgr2=0x7f0594113a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05940fded0 0x7f0594113a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.789+0000 7f0599269700 1 -- 172.21.15.111:0/2323825879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0590005040 con 0x7f05940fe870 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f0599269700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0590004890 tx=0x7f0590009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059000e040 con 0x7f05940fe870 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0590009af0 con 0x7f05940fe870 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059410dc80 con 0x7f05940fe870 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f059001c3f0 con 0x7f05940fe870 2024-03-14T13:04:56.791 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.790+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059410e1c0 con 0x7f05940fe870 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.791+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0594065df0 con 0x7f05940fe870 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.791+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f0590013420 con 0x7f05940fe870 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.792+0000 7f058a7fc700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 0x7f058006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.792+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f059008c7a0 con 0x7f05940fe870 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.792+0000 7f0599a6a700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 0x7f058006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:56.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.792+0000 7f0599a6a700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 0x7f058006e8f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0584006fd0 tx=0x7f0584006db0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:56.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.796+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f059005aa70 con 0x7f05940fe870 2024-03-14T13:04:56.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.988+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f059404bed0 con 0x7f05940fe870 2024-03-14T13:04:56.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.988+0000 7f058a7fc700 1 -- 172.21.15.111:0/2323825879 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v48)=0 v48) v1 ==== 155+0+0 (secure 0 0 0) 0x7f059005a600 con 0x7f05940fe870 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 msgr2=0x7f058006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 0x7f058006e8f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0584006fd0 tx=0x7f0584006db0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 msgr2=0x7f0594113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0590004890 tx=0x7f0590009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 shutdown_connections 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05940fded0 0x7f0594113a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f058006c440 0x7f058006e8f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 --2- 172.21.15.111:0/2323825879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05940fe870 0x7f0594113f60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 >> 172.21.15.111:0/2323825879 conn(0x7f0594072bc0 msgr2=0x7f0594103800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 shutdown_connections 2024-03-14T13:04:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:56.990+0000 7f059bcce700 1 -- 172.21.15.111:0/2323825879 wait complete. 2024-03-14T13:04:57.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-14T13:04:57.957 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:04:58.204 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:57 smithi111 ceph-mon[39688]: pgmap v103: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:58.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:57 smithi137 ceph-mon[50336]: pgmap v103: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:04:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 -- 172.21.15.111:0/3762359430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc101d20 msgr2=0x7fd9cc1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 --2- 172.21.15.111:0/3762359430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc101d20 0x7fd9cc1020f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd9c0004940 tx=0x7fd9c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 -- 172.21.15.111:0/3762359430 shutdown_connections 2024-03-14T13:04:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 --2- 172.21.15.111:0/3762359430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 --2- 172.21.15.111:0/3762359430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc101d20 0x7fd9cc1020f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 -- 172.21.15.111:0/3762359430 >> 172.21.15.111:0/3762359430 conn(0x7fd9cc0f78d0 msgr2=0x7fd9cc0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 -- 172.21.15.111:0/3762359430 shutdown_connections 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.686+0000 7fd9d1608700 1 -- 172.21.15.111:0/3762359430 wait complete. 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 Processor -- start 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 -- start start 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc101d20 0x7fd9cc113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc1146a0 con 0x7fd9cc1026c0 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9d1608700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc10da90 con 0x7fd9cc101d20 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51524/0 (socket says 172.21.15.111:51524) 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 -- 172.21.15.111:0/2258842704 learned_addr learned my addr 172.21.15.111:0/2258842704 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:04:59.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9caffd700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc101d20 0x7fd9cc113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 -- 172.21.15.111:0/2258842704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc101d20 msgr2=0x7fd9cc113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc101d20 0x7fd9cc113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.687+0000 7fd9ca7fc700 1 -- 172.21.15.111:0/2258842704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0005040 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9ca7fc700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c800 tx=0x7fd9bc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9bc0106f0 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9bc010d00 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9bc0113a0 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc10dd70 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.688+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc10e330 con 0x7fd9cc1026c0 2024-03-14T13:04:59.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.689+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc0fd470 con 0x7fd9cc1026c0 2024-03-14T13:04:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.689+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fd9bc012400 con 0x7fd9cc1026c0 2024-03-14T13:04:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.689+0000 7fd9b37fe700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 0x7fd9b406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:04:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.690+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd9bc08b380 con 0x7fd9cc1026c0 2024-03-14T13:04:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.690+0000 7fd9caffd700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 0x7fd9b406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:04:59.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.690+0000 7fd9caffd700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 0x7fd9b406e900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000b840 tx=0x7fd9c0009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:04:59.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.694+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fd9bc059650 con 0x7fd9cc1026c0 2024-03-14T13:04:59.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.886+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fd9cc108ce0 con 0x7fd9cc1026c0 2024-03-14T13:04:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.886+0000 7fd9b37fe700 1 -- 172.21.15.111:0/2258842704 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v48)=0 v48) v1 ==== 163+0+0 (secure 0 0 0) 0x7fd9bc0591e0 con 0x7fd9cc1026c0 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 msgr2=0x7fd9b406e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 0x7fd9b406e900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000b840 tx=0x7fd9c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 msgr2=0x7fd9cc114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c800 tx=0x7fd9bc004470 comp rx=0 tx=0).stop 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 shutdown_connections 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9cc101d20 0x7fd9cc113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fd9b406c450 0x7fd9b406e900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 --2- 172.21.15.111:0/2258842704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9cc1026c0 0x7fd9cc114050 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:04:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 >> 172.21.15.111:0/2258842704 conn(0x7fd9cc0f78d0 msgr2=0x7fd9cc0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:04:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 shutdown_connections 2024-03-14T13:04:59.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:04:59.888+0000 7fd9d1608700 1 -- 172.21.15.111:0/2258842704 wait complete. 2024-03-14T13:05:00.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:04:59 smithi137 ceph-mon[50336]: pgmap v104: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:05:00.274 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:04:59 smithi111 ceph-mon[39688]: pgmap v104: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:05:00.609 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-14T13:05:00.878 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:05:01.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:01 smithi111 ceph-mon[39688]: pgmap v105: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:01.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:01 smithi137 ceph-mon[50336]: pgmap v105: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 -- 172.21.15.111:0/2459344932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101d70 msgr2=0x7f09ec102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 --2- 172.21.15.111:0/2459344932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101d70 0x7f09ec102140 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f09e0004940 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 -- 172.21.15.111:0/2459344932 shutdown_connections 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 --2- 172.21.15.111:0/2459344932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec102710 0x7f09ec0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 --2- 172.21.15.111:0/2459344932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec101d70 0x7f09ec102140 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 -- 172.21.15.111:0/2459344932 >> 172.21.15.111:0/2459344932 conn(0x7f09ec072bc0 msgr2=0x7f09ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 -- 172.21.15.111:0/2459344932 shutdown_connections 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.745+0000 7f09f0c70700 1 -- 172.21.15.111:0/2459344932 wait complete. 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 Processor -- start 2024-03-14T13:05:02.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 -- start start 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 0x7f09ec1129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1135d0 con 0x7f09ec102710 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09f0c70700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec10c930 con 0x7f09ec101d70 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45212/0 (socket says 172.21.15.111:45212) 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 -- 172.21.15.111:0/3099559421 learned_addr learned my addr 172.21.15.111:0/3099559421 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09ea59c700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 0x7f09ec1129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 -- 172.21.15.111:0/3099559421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 msgr2=0x7f09ec1129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 0x7f09ec1129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09e9d9b700 1 -- 172.21.15.111:0/3099559421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e0005040 con 0x7f09ec102710 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.746+0000 7f09ea59c700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 0x7f09ec1129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09e9d9b700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00daf0 tx=0x7f09dc00b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:02.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc00ed70 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09dc017920 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec10cbb0 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09dc016970 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.747+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec10d170 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.748+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec0fd4c0 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.748+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f09dc014070 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.748+0000 7f09daffd700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 0x7f09d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.749+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f09dc08d6e0 con 0x7f09ec102710 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.749+0000 7f09ea59c700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 0x7f09d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.749+0000 7f09ea59c700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 0x7f09d406e8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09e0004f90 tx=0x7f09e0009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:02.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.753+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f09dc05b910 con 0x7f09ec102710 2024-03-14T13:05:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.951+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f09ec108e10 con 0x7f09ec102710 2024-03-14T13:05:02.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.952+0000 7f09daffd700 1 -- 172.21.15.111:0/3099559421 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v48)=0 v48) v1 ==== 135+0+0 (secure 0 0 0) 0x7f09dc05b4a0 con 0x7f09ec102710 2024-03-14T13:05:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.953+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 msgr2=0x7f09d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.953+0000 7f09f0c70700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 0x7f09d406e8f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f09e0004f90 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.953+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 msgr2=0x7f09ec112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:02.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.953+0000 7f09f0c70700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00daf0 tx=0x7f09dc00b4c0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 shutdown_connections 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09ec101d70 0x7f09ec1129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f09d406c440 0x7f09d406e8f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 --2- 172.21.15.111:0/3099559421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec102710 0x7f09ec112ef0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 >> 172.21.15.111:0/3099559421 conn(0x7f09ec072bc0 msgr2=0x7f09ec0fafd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 shutdown_connections 2024-03-14T13:05:02.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:02.954+0000 7f09f0c70700 1 -- 172.21.15.111:0/3099559421 wait complete. 2024-03-14T13:05:03.664 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-14T13:05:03.937 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:05:04.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:03 smithi111 ceph-mon[39688]: pgmap v106: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:05:04.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:03 smithi137 ceph-mon[50336]: pgmap v106: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-14T13:05:05.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 -- 172.21.15.111:0/945805911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 msgr2=0x7f726c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 --2- 172.21.15.111:0/945805911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c0fce20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7258004940 tx=0x7f7258009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 -- 172.21.15.111:0/945805911 shutdown_connections 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 --2- 172.21.15.111:0/945805911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c0fce20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 --2- 172.21.15.111:0/945805911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f726c102e10 0x7f726c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.720+0000 7f7272be0700 1 -- 172.21.15.111:0/945805911 >> 172.21.15.111:0/945805911 conn(0x7f726c0f89a0 msgr2=0x7f726c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 -- 172.21.15.111:0/945805911 shutdown_connections 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 -- 172.21.15.111:0/945805911 wait complete. 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 Processor -- start 2024-03-14T13:05:05.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 -- start start 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f726c102e10 0x7f726c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1146f0 con 0x7f726c1037b0 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f7272be0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c10da50 con 0x7f726c102e10 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.721+0000 7f726bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45222/0 (socket says 172.21.15.111:45222) 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 -- 172.21.15.111:0/1248740246 learned_addr learned my addr 172.21.15.111:0/1248740246 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 -- 172.21.15.111:0/1248740246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f726c102e10 msgr2=0x7f726c113ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f726c102e10 0x7f726c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 -- 172.21.15.111:0/1248740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7258005040 con 0x7f726c1037b0 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f726bfff700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7258009380 tx=0x7f7258009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f725800e040 con 0x7f726c1037b0 2024-03-14T13:05:05.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c10dcd0 con 0x7f726c1037b0 2024-03-14T13:05:05.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7258013420 con 0x7f726c1037b0 2024-03-14T13:05:05.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f725801c630 con 0x7f726c1037b0 2024-03-14T13:05:05.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.722+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c10e290 con 0x7f726c1037b0 2024-03-14T13:05:05.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.723+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c0fe560 con 0x7f726c1037b0 2024-03-14T13:05:05.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.724+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f72580139e0 con 0x7f726c1037b0 2024-03-14T13:05:05.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.724+0000 7f72697fa700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 0x7f725c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:05.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.724+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f725808c6c0 con 0x7f726c1037b0 2024-03-14T13:05:05.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.724+0000 7f727097c700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 0x7f725c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:05.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.724+0000 7f727097c700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 0x7f725c06e8f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7260006fd0 tx=0x7f7260006db0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:05.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.729+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f725805a8f0 con 0x7f726c1037b0 2024-03-14T13:05:05.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.938+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}) v1 -- 0x7f726c04bed0 con 0x7f725c06c440 2024-03-14T13:05:05.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.943+0000 7f72697fa700 1 -- 172.21.15.111:0/1248740246 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f726c04bed0 con 0x7f725c06c440 2024-03-14T13:05:05.944 INFO:teuthology.orchestra.run.smithi111.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 msgr2=0x7f725c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 0x7f725c06e8f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7260006fd0 tx=0x7f7260006db0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 msgr2=0x7f726c114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7258009380 tx=0x7f7258009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 shutdown_connections 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f726c102e10 0x7f726c113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f725c06c440 0x7f725c06e8f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 --2- 172.21.15.111:0/1248740246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f726c1037b0 0x7f726c114010 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:05.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.945+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 >> 172.21.15.111:0/1248740246 conn(0x7f726c0f89a0 msgr2=0x7f726c0f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:05.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.946+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 shutdown_connections 2024-03-14T13:05:05.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:05.946+0000 7f7272be0700 1 -- 172.21.15.111:0/1248740246 wait complete. 2024-03-14T13:05:06.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:05 smithi111 ceph-mon[39688]: pgmap v107: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:05:06.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:05 smithi137 ceph-mon[50336]: pgmap v107: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:05:06.720 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-14T13:05:06.720 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T13:05:06.720 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-14T13:05:07.014 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:07.014 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T13:05:07.015 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:06 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:07.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='client.14842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2f465319e3c83d48f999325459bcdfd0346449c0 2024-03-14T13:05:07.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:06 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:08.037 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:05:08.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:07 smithi137 ceph-mon[50336]: pgmap v108: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:08.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:07 smithi111 ceph-mon[39688]: pgmap v108: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:09.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: Upgrade: Target is version 18.2.2-825-g2f465319 (reef) 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de, digests ['quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de'] 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: Upgrade: Setting container_image for all mgr 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:09 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: Upgrade: Target is version 18.2.2-825-g2f465319 (reef) 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de, digests ['quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de'] 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: Upgrade: Setting container_image for all mgr 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:09.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:09 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-14T13:05:10.270 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:10 smithi111 ceph-mon[39688]: pgmap v109: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:05:10.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:10 smithi137 ceph-mon[50336]: pgmap v109: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 6 op/s 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 -- 172.21.15.111:0/3053330131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 msgr2=0x7fd008102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/3053330131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008102140 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcffc004940 tx=0x7fcffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 -- 172.21.15.111:0/3053330131 shutdown_connections 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/3053330131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 0x7fd0080fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/3053330131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008102140 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 -- 172.21.15.111:0/3053330131 >> 172.21.15.111:0/3053330131 conn(0x7fd008072bc0 msgr2=0x7fd008072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 -- 172.21.15.111:0/3053330131 shutdown_connections 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.822+0000 7fd00d5f1700 1 -- 172.21.15.111:0/3053330131 wait complete. 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 Processor -- start 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 -- start start 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 0x7fd008114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008114720 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd00d5f1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00810da80 con 0x7fd008102710 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46592/0 (socket says 172.21.15.111:46592) 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 -- 172.21.15.111:0/2818299254 learned_addr learned my addr 172.21.15.111:0/2818299254 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd0067fc700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 0x7fd008114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 -- 172.21.15.111:0/2818299254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 msgr2=0x7fd008114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 0x7fd008114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.823+0000 7fd006ffd700 1 -- 172.21.15.111:0/2818299254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc005040 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fd006ffd700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009180 tx=0x7fcffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc00e040 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcffc004290 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00810dd00 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcffc01c3f0 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.824+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00810e240 con 0x7fd008101d70 2024-03-14T13:05:10.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.825+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080fd4c0 con 0x7fd008101d70 2024-03-14T13:05:10.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.825+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fcffc013790 con 0x7fd008101d70 2024-03-14T13:05:10.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.826+0000 7fcfef7fe700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 0x7fcff006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:10.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.826+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fcffc08c6b0 con 0x7fd008101d70 2024-03-14T13:05:10.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.826+0000 7fd0067fc700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 0x7fcff006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:10.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.826+0000 7fd0067fc700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 0x7fcff006e8f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcff8006fd0 tx=0x7fcff8008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:10.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:10.830+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fcffc05a980 con 0x7fd008101d70 2024-03-14T13:05:11.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.050+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd008108e10 con 0x7fcff006c440 2024-03-14T13:05:11.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.051+0000 7fcfef7fe700 1 -- 172.21.15.111:0/2818299254 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fd008108e10 con 0x7fcff006c440 2024-03-14T13:05:11.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.053+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 msgr2=0x7fcff006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.053+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 0x7fcff006e8f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcff8006fd0 tx=0x7fcff8008040 comp rx=0 tx=0).stop 2024-03-14T13:05:11.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.053+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 msgr2=0x7fd008113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.053+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009180 tx=0x7fcffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 shutdown_connections 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fcff006c440 0x7fcff006e8f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd008101d70 0x7fd008113b00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 --2- 172.21.15.111:0/2818299254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd008102710 0x7fd008114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 >> 172.21.15.111:0/2818299254 conn(0x7fd008072bc0 msgr2=0x7fd0080658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 shutdown_connections 2024-03-14T13:05:11.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.054+0000 7fd00d5f1700 1 -- 172.21.15.111:0/2818299254 wait complete. 2024-03-14T13:05:11.064 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:05:11.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 -- 172.21.15.111:0/1998895448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 msgr2=0x7fde680fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1998895448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde680fe260 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fde60009ce0 tx=0x7fde60004d10 comp rx=0 tx=0).stop 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 -- 172.21.15.111:0/1998895448 shutdown_connections 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1998895448 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 0x7fde68106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1998895448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde680fe260 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 -- 172.21.15.111:0/1998895448 >> 172.21.15.111:0/1998895448 conn(0x7fde68072bc0 msgr2=0x7fde68072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 -- 172.21.15.111:0/1998895448 shutdown_connections 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.160+0000 7fde6fc32700 1 -- 172.21.15.111:0/1998895448 wait complete. 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 Processor -- start 2024-03-14T13:05:11.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 -- start start 2024-03-14T13:05:11.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 0x7fde68113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68114600 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.161+0000 7fde6fc32700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6810d9f0 con 0x7fde680fe830 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46596/0 (socket says 172.21.15.111:46596) 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 -- 172.21.15.111:0/1521635494 learned_addr learned my addr 172.21.15.111:0/1521635494 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d1cd700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 0x7fde68113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 -- 172.21.15.111:0/1521635494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 msgr2=0x7fde68113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 0x7fde68113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 -- 172.21.15.111:0/1521635494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde60009940 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6d9ce700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fde60009740 tx=0x7fde60004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6000e070 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6810dc70 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde600128c0 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde6001b7d0 con 0x7fde680fde90 2024-03-14T13:05:11.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.162+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6810e1b0 con 0x7fde680fde90 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.163+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68065e30 con 0x7fde680fde90 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.164+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fde60019040 con 0x7fde680fde90 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.164+0000 7fde667fc700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 0x7fde5006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.164+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fde6008cb00 con 0x7fde680fde90 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.164+0000 7fde6d1cd700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 0x7fde5006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.164+0000 7fde6d1cd700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 0x7fde5006e8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde6810ebb0 tx=0x7fde5c009040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.169+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fde6005acb0 con 0x7fde680fde90 2024-03-14T13:05:11.398 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.397+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fde68109dc0 con 0x7fde5006c440 2024-03-14T13:05:11.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.399+0000 7fde667fc700 1 -- 172.21.15.111:0/1521635494 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fde68109dc0 con 0x7fde5006c440 2024-03-14T13:05:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 msgr2=0x7fde5006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 0x7fde5006e8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde6810ebb0 tx=0x7fde5c009040 comp rx=0 tx=0).stop 2024-03-14T13:05:11.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 msgr2=0x7fde68113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fde60009740 tx=0x7fde60004d10 comp rx=0 tx=0).stop 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 shutdown_connections 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fde5006c440 0x7fde5006e8f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde680fde90 0x7fde68113a70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 --2- 172.21.15.111:0/1521635494 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde680fe830 0x7fde68113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 >> 172.21.15.111:0/1521635494 conn(0x7fde68072bc0 msgr2=0x7fde681037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 shutdown_connections 2024-03-14T13:05:11.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.401+0000 7fde6fc32700 1 -- 172.21.15.111:0/1521635494 wait complete. 2024-03-14T13:05:11.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 -- 172.21.15.111:0/2102866705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 msgr2=0x7f8ae00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 --2- 172.21.15.111:0/2102866705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae00fce10 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4004940 tx=0x7f8ad4009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 -- 172.21.15.111:0/2102866705 shutdown_connections 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 --2- 172.21.15.111:0/2102866705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae00fce10 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 --2- 172.21.15.111:0/2102866705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 -- 172.21.15.111:0/2102866705 >> 172.21.15.111:0/2102866705 conn(0x7f8ae00f89b0 msgr2=0x7f8ae00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 -- 172.21.15.111:0/2102866705 shutdown_connections 2024-03-14T13:05:11.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.504+0000 7f8ae5975700 1 -- 172.21.15.111:0/2102866705 wait complete. 2024-03-14T13:05:11.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 Processor -- start 2024-03-14T13:05:11.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 -- start start 2024-03-14T13:05:11.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0114700 con 0x7f8ae01037a0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ae5975700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae010da60 con 0x7f8ae0102e00 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46622/0 (socket says 172.21.15.111:46622) 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 -- 172.21.15.111:0/4027596213 learned_addr learned my addr 172.21.15.111:0/4027596213 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8adeffd700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 -- 172.21.15.111:0/4027596213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 msgr2=0x7f8ae0113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae0113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8ade7fc700 1 -- 172.21.15.111:0/4027596213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad4005040 con 0x7f8ae01037a0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.505+0000 7f8adeffd700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ade7fc700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009380 tx=0x7f8ad4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad400e040 con 0x7f8ae01037a0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ad4013420 con 0x7f8ae01037a0 2024-03-14T13:05:11.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae010dce0 con 0x7f8ae01037a0 2024-03-14T13:05:11.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad401c630 con 0x7f8ae01037a0 2024-03-14T13:05:11.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.506+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae010e2a0 con 0x7f8ae01037a0 2024-03-14T13:05:11.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.507+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae00fe550 con 0x7f8ae01037a0 2024-03-14T13:05:11.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.507+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f8ad40139e0 con 0x7f8ae01037a0 2024-03-14T13:05:11.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.508+0000 7f8ac77fe700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 0x7f8ac806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.508+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f8ad408c6c0 con 0x7f8ae01037a0 2024-03-14T13:05:11.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.508+0000 7f8adeffd700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 0x7f8ac806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.508+0000 7f8adeffd700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 0x7f8ac806e8f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006fd0 tx=0x7f8ad0008040 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.512+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f8ad405a8f0 con 0x7f8ae01037a0 2024-03-14T13:05:11.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.717+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae0109e80 con 0x7f8ac806c440 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.726+0000 7f8ac77fe700 1 -- 172.21.15.111:0/4027596213 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f8ae0109e80 con 0x7f8ac806c440 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (84s) 57s ago 12m 16.5M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (13m) 57s ago 13m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (11m) 2m ago 11m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (63s) 57s ago 12m 47.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (6m) 57s ago 6m 64.0M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (6m) 57s ago 6m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (6m) 2m ago 6m 90.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (6m) 2m ago 6m 157M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (4m) 57s ago 14m 498M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:05:11.727 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (2m) 2m ago 11m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (14m) 57s ago 14m 55.9M 2048M 17.2.7-646-gb92acd1f a07fb176315d 90683c439e1e 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (11m) 2m ago 11m 44.8M 2048M 17.2.7-646-gb92acd1f a07fb176315d dba54c072a9f 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (2m) 57s ago 13m 16.2M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (2m) 2m ago 11m 7088k - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (10m) 57s ago 10m 209M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (10m) 57s ago 10m 235M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (9m) 57s ago 9m 196M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (9m) 2m ago 9m 240M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (8m) 2m ago 8m 213M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (8m) 2m ago 8m 203M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:05:11.728 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (110s) 57s ago 12m 49.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 msgr2=0x7f8ac806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 0x7f8ac806e8f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0006fd0 tx=0x7f8ad0008040 comp rx=0 tx=0).stop 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 msgr2=0x7f8ae0114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009380 tx=0x7f8ad4009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 shutdown_connections 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ae0102e00 0x7f8ae0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f8ac806c440 0x7f8ac806e8f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 --2- 172.21.15.111:0/4027596213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ae01037a0 0x7f8ae0114020 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.728+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 >> 172.21.15.111:0/4027596213 conn(0x7f8ae00f89b0 msgr2=0x7f8ae00f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.729+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 shutdown_connections 2024-03-14T13:05:11.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.729+0000 7f8ae5975700 1 -- 172.21.15.111:0/4027596213 wait complete. 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 -- 172.21.15.111:0/1814445105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 msgr2=0x7f0ef0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 --2- 172.21.15.111:0/1814445105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0102140 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0eec004940 tx=0x7f0eec009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 -- 172.21.15.111:0/1814445105 shutdown_connections 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 --2- 172.21.15.111:0/1814445105 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 0x7f0ef00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 --2- 172.21.15.111:0/1814445105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0102140 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 -- 172.21.15.111:0/1814445105 >> 172.21.15.111:0/1814445105 conn(0x7f0ef0072bc0 msgr2=0x7f0ef0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 -- 172.21.15.111:0/1814445105 shutdown_connections 2024-03-14T13:05:11.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.832+0000 7f0ef8483700 1 -- 172.21.15.111:0/1814445105 wait complete. 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 Processor -- start 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 -- start start 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 0x7f0ef0113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef0114640 con 0x7f0ef0101d70 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef8483700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef010da30 con 0x7f0ef0102710 2024-03-14T13:05:11.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46642/0 (socket says 172.21.15.111:46642) 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 -- 172.21.15.111:0/3463188208 learned_addr learned my addr 172.21.15.111:0/3463188208 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef5a1e700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 0x7f0ef0113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 -- 172.21.15.111:0/3463188208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 msgr2=0x7f0ef0113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 0x7f0ef0113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.833+0000 7f0ef621f700 1 -- 172.21.15.111:0/3463188208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec005040 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ef621f700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0eec000c00 tx=0x7f0eec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eec00e040 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0eec013420 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef010dcb0 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0eec01c630 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.834+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef010e270 con 0x7f0ef0101d70 2024-03-14T13:05:11.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.835+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef00ff080 con 0x7f0ef0101d70 2024-03-14T13:05:11.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.835+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f0eec013590 con 0x7f0ef0101d70 2024-03-14T13:05:11.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.836+0000 7f0ee6ffd700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 0x7f0edc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:11.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.836+0000 7f0ef5a1e700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 0x7f0edc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:11.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.836+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f0eec08c640 con 0x7f0ef0101d70 2024-03-14T13:05:11.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.836+0000 7f0ef5a1e700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 0x7f0edc06e8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ef010ebf0 tx=0x7f0ee0009ee0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:11.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:11.840+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f0eec05a910 con 0x7f0ef0101d70 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: Upgrade: Updating mon.smithi111 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: Deploying daemon mon.smithi111 on smithi111 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: pgmap v110: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:11.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:11 smithi111 ceph-mon[39688]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: Upgrade: Updating mon.smithi111 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: Deploying daemon mon.smithi111 on smithi111 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: pgmap v110: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:11 smithi137 ceph-mon[50336]: from='client.14846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.117+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0ef010f130 con 0x7f0ef0101d70 2024-03-14T13:05:12.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.117+0000 7f0ee6ffd700 1 -- 172.21.15.111:0/3463188208 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+741 (secure 0 0 0) 0x7f0eec05a4a0 con 0x7f0ef0101d70 2024-03-14T13:05:12.118 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:05:12.118 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:05:12.119 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 12, 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 msgr2=0x7f0edc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 0x7f0edc06e8f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0ef010ebf0 tx=0x7f0ee0009ee0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 msgr2=0x7f0ef0113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0eec000c00 tx=0x7f0eec009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 shutdown_connections 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f0edc06c440 0x7f0edc06e8f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ef0101d70 0x7f0ef0113ab0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 --2- 172.21.15.111:0/3463188208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ef0102710 0x7f0ef0113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 >> 172.21.15.111:0/3463188208 conn(0x7f0ef0072bc0 msgr2=0x7f0ef00fb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 shutdown_connections 2024-03-14T13:05:12.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.120+0000 7f0ef8483700 1 -- 172.21.15.111:0/3463188208 wait complete. 2024-03-14T13:05:12.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.220+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/1041680883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 msgr2=0x7fbb880fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.220+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/1041680883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb880fbd80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c004940 tx=0x7fbb7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/1041680883 shutdown_connections 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/1041680883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb880fbd80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/1041680883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb88101d70 0x7fbb88102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/1041680883 >> 172.21.15.111:0/1041680883 conn(0x7fbb88072bc0 msgr2=0x7fbb88072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/1041680883 shutdown_connections 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/1041680883 wait complete. 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 Processor -- start 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- start start 2024-03-14T13:05:12.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb88101d70 0x7fbb8810ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8810f680 con 0x7fbb88102710 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb8c8bc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8810f7c0 con 0x7fbb88101d70 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.221+0000 7fbb85d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46658/0 (socket says 172.21.15.111:46658) 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 -- 172.21.15.111:0/4202208342 learned_addr learned my addr 172.21.15.111:0/4202208342 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 -- 172.21.15.111:0/4202208342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb88101d70 msgr2=0x7fbb8810ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb88101d70 0x7fbb8810ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 -- 172.21.15.111:0/4202208342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c005040 con 0x7fbb88102710 2024-03-14T13:05:12.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb85d9b700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009390 tx=0x7fbb7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7c00e040 con 0x7fbb88102710 2024-03-14T13:05:12.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb7c013420 con 0x7fbb88102710 2024-03-14T13:05:12.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb88117ab0 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb7c01c630 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.222+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb88118040 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.223+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb880ff080 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.224+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7fbb7c013a00 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.224+0000 7fbb76ffd700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 0x7fbb7006e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.224+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fbb7c08c7b0 con 0x7fbb88102710 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.224+0000 7fbb8659c700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 0x7fbb7006e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.224+0000 7fbb8659c700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 0x7fbb7006e900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb78004d00 tx=0x7fbb78009ce0 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.229+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7fbb7c056f50 con 0x7fbb88102710 2024-03-14T13:05:12.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.481+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbb88065dd0 con 0x7fbb88102710 2024-03-14T13:05:12.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.481+0000 7fbb76ffd700 1 -- 172.21.15.111:0/4202208342 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1854 (secure 0 0 0) 0x7fbb7c05a570 con 0x7fbb88102710 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:e21 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 21 2024-03-14T13:05:12.483 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:04:49.764363+0000 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:05:12.484 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:12.485 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:12.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 msgr2=0x7fbb7006e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 0x7fbb7006e900 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb78004d00 tx=0x7fbb78009ce0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 msgr2=0x7fbb8810f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c009390 tx=0x7fbb7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 shutdown_connections 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb88101d70 0x7fbb8810ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7fbb7006c450 0x7fbb7006e900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 --2- 172.21.15.111:0/4202208342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb88102710 0x7fbb8810f140 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.483+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 >> 172.21.15.111:0/4202208342 conn(0x7fbb88072bc0 msgr2=0x7fbb880fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.484+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 shutdown_connections 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.484+0000 7fbb8c8bc700 1 -- 172.21.15.111:0/4202208342 wait complete. 2024-03-14T13:05:12.487 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 21 2024-03-14T13:05:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.596+0000 7f46cbf91700 1 -- 172.21.15.111:0/2585030171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4101d70 msgr2=0x7f46c4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.596+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2585030171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4101d70 0x7f46c4102140 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f46c0004940 tx=0x7f46c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.596+0000 7f46cbf91700 1 -- 172.21.15.111:0/2585030171 shutdown_connections 2024-03-14T13:05:12.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2585030171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4102710 0x7f46c40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2585030171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4101d70 0x7f46c4102140 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- 172.21.15.111:0/2585030171 >> 172.21.15.111:0/2585030171 conn(0x7f46c4072bc0 msgr2=0x7f46c4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- 172.21.15.111:0/2585030171 shutdown_connections 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- 172.21.15.111:0/2585030171 wait complete. 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 Processor -- start 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- start start 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46c4103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4103db0 con 0x7f46c4102710 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.597+0000 7f46cbf91700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4103ef0 con 0x7f46c4101d70 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46684/0 (socket says 172.21.15.111:46684) 2024-03-14T13:05:12.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 learned_addr learned my addr 172.21.15.111:0/2046765771 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c9d2d700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46c4103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 msgr2=0x7f46c4103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46c4103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0005040 con 0x7f46c4102710 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46c952c700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f46b400b790 tx=0x7f46b4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46b400f6f0 con 0x7f46c4102710 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c4119dd0 con 0x7f46c4102710 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46b400fd00 con 0x7f46c4102710 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46b40103a0 con 0x7f46c4102710 2024-03-14T13:05:12.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.598+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c411a2e0 con 0x7f46c4102710 2024-03-14T13:05:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.599+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c40fd4c0 con 0x7f46c4102710 2024-03-14T13:05:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.600+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f46b400f850 con 0x7f46c4102710 2024-03-14T13:05:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.600+0000 7f46ba7fc700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 0x7f46b006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.600+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f46b408b2b0 con 0x7f46c4102710 2024-03-14T13:05:12.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.600+0000 7f46c9d2d700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 0x7f46b006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.601+0000 7f46c9d2d700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 0x7f46b006e8f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46c0004f90 tx=0x7f46c0009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.605+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f46b4055a50 con 0x7f46c4102710 2024-03-14T13:05:12.613 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.780+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 msgr2=0x7f46c4103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.780+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 msgr2=0x7f46c4103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.780+0000 7f46c952c700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f46b400b790 tx=0x7f46b4009220 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.780+0000 7f46c952c700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f46b400b790 tx=0x7f46b4009220 comp rx=0 tx=0).stop 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 msgr2=0x7f46c4103790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46c4102710 0x7f46c4103790 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 0x7f46b0076460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46b0078cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b0079630 con 0x7f46b0074080 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b0079770 con 0x7f46c4101d70 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46c952c700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 msgr2=0x7f46b0076460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46c952c700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 0x7f46b0076460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:12.783 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.781+0000 7f46ca52e700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46b0078cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.783+0000 7f46ca52e700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 msgr2=0x7f46b0076460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.783+0000 7f46ca52e700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 0x7f46b0076460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.783+0000 7f46ca52e700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7f46b0082110 con 0x7f46c4101d70 2024-03-14T13:05:12.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.783+0000 7f46ca52e700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46b0078cb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009f20 tx=0x7f46bc0095d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.786+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46bc00be20 con 0x7f46c4101d70 2024-03-14T13:05:12.788 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.788+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46bc0148c0 con 0x7f46c4101d70 2024-03-14T13:05:12.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.789+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f46bc0133c0 con 0x7f46c4101d70 2024-03-14T13:05:12.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.831+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 --> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c404bed0 con 0x7f46b006c440 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.832+0000 7f46ba7fc700 1 -- 172.21.15.111:0/2046765771 <== mgr.14658 v2:172.21.15.111:6800/3203466385 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f46c404bed0 con 0x7f46b006c440 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:05:12.833 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "2/21 daemons upgraded", 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mon daemons", 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:05:12.834 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:05:12.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.834+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 msgr2=0x7f46b006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.834+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 0x7f46b006e8f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f46c0004f90 tx=0x7f46c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.834+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 msgr2=0x7f46b0078cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.834+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46b0078cb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009f20 tx=0x7f46bc0095d0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 shutdown_connections 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46c4101d70 0x7f46b0078cb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b0074080 0x7f46b0076460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 --2- 172.21.15.111:0/2046765771 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f46b006c440 0x7f46b006e8f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 >> 172.21.15.111:0/2046765771 conn(0x7f46c4072bc0 msgr2=0x7f46c41056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 shutdown_connections 2024-03-14T13:05:12.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.835+0000 7f46cbf91700 1 -- 172.21.15.111:0/2046765771 wait complete. 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-mon[39688]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-mon[39688]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/3463188208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-mon[39688]: from='client.? 172.21.15.111:0/4202208342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[39665]: 2024-03-14T13:05:12.779+0000 7f6496022700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:05:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:12 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[39665]: 2024-03-14T13:05:12.779+0000 7f6496022700 -1 mon.smithi111@0(leader) e2 *** Got Signal Terminated *** 2024-03-14T13:05:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 -- 172.21.15.111:0/3739746750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fe830 msgr2=0x7f0280106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 --2- 172.21.15.111:0/3739746750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fe830 0x7f0280106f00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0274004940 tx=0x7f0274009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:12.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 -- 172.21.15.111:0/3739746750 shutdown_connections 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 --2- 172.21.15.111:0/3739746750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fe830 0x7f0280106f00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 --2- 172.21.15.111:0/3739746750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fde90 0x7f02800fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 -- 172.21.15.111:0/3739746750 >> 172.21.15.111:0/3739746750 conn(0x7f0280072bc0 msgr2=0x7f0280072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 -- 172.21.15.111:0/3739746750 shutdown_connections 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.940+0000 7f0284d85700 1 -- 172.21.15.111:0/3739746750 wait complete. 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 Processor -- start 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 -- start start 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 0x7f0280114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02801146e0 con 0x7f02800fe830 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f0284d85700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028010da90 con 0x7f02800fde90 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027dd9b700 1 -- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 msgr2=0x7f0280114000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:3300/0 2024-03-14T13:05:12.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 0x7f0280114000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.111:44844/0 (socket says 172.21.15.111:44844) 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 -- 172.21.15.111:0/484110269 learned_addr learned my addr 172.21.15.111:0/484110269 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 -- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 msgr2=0x7f0280114000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 0x7f0280114000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 -- 172.21.15.111:0/484110269 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274005040 con 0x7f02800fde90 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.941+0000 7f027e59c700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f027000b760 tx=0x7f0270009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.942+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0270010790 con 0x7f02800fde90 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.942+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028010dd70 con 0x7f02800fde90 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.942+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0270010da0 con 0x7f02800fde90 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.942+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f027000f300 con 0x7f02800fde90 2024-03-14T13:05:12.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.942+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028010e2b0 con 0x7f02800fde90 2024-03-14T13:05:12.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.943+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0280065e30 con 0x7f02800fde90 2024-03-14T13:05:12.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.943+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 33) v1 ==== 90387+0+0 (secure 0 0 0) 0x7f02700116d0 con 0x7f02800fde90 2024-03-14T13:05:12.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.944+0000 7f026effd700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 0x7f026806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:12.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.944+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f027008b840 con 0x7f02800fde90 2024-03-14T13:05:12.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.944+0000 7f027dd9b700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 0x7f026806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:12.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.944+0000 7f027dd9b700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 0x7f026806e840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f027400b020 tx=0x7f0274009e30 comp rx=0 tx=0).ready entity=mgr.14658 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:12.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:12.948+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181436 (secure 0 0 0) 0x7f0270059740 con 0x7f02800fde90 2024-03-14T13:05:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:12 smithi137 ceph-mon[50336]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:12 smithi137 ceph-mon[50336]: from='client.14854 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:12 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/3463188208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:13.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:12 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/4202208342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:05:13.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.233+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0280109e90 con 0x7f02800fde90 2024-03-14T13:05:13.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.234+0000 7f026effd700 1 -- 172.21.15.111:0/484110269 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f0270059560 con 0x7f02800fde90 2024-03-14T13:05:13.235 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 msgr2=0x7f026806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 0x7f026806e840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f027400b020 tx=0x7f0274009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 msgr2=0x7f0280113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f027000b760 tx=0x7f0270009220 comp rx=0 tx=0).stop 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 shutdown_connections 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02800fde90 0x7f0280113ac0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:6800/3203466385,v1:172.21.15.111:6801/3203466385] conn(0x7f026806c390 0x7f026806e840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 --2- 172.21.15.111:0/484110269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800fe830 0x7f0280114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 >> 172.21.15.111:0/484110269 conn(0x7f0280072bc0 msgr2=0x7f0280103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 shutdown_connections 2024-03-14T13:05:13.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:13.236+0000 7f0284d85700 1 -- 172.21.15.111:0/484110269 wait complete. 2024-03-14T13:05:13.579 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:13 smithi111 bash[145084]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111 2024-03-14T13:05:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:13 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service: Succeeded. 2024-03-14T13:05:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:13 smithi111 systemd[1]: Stopped Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:05:13.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:13 smithi111 systemd[1]: Starting Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: set uid:gid to 167:167 (ceph:ceph) 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable), process ceph-mon, pid 7 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: pidfile_write: ignore empty --pid-file 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: load: jerasure load: lrc 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: RocksDB version: 7.9.2 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Git sha 0 2024-03-14T13:05:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Compile date 2024-03-13 02:27:37 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: DB SUMMARY 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: DB Session ID: W1EJP11S51LTYLI0ANGJ 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: CURRENT file: CURRENT 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: IDENTITY file: IDENTITY 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1170 Bytes 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 1, files: 000033.sst 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 000031.log size: 3934431 ; 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.error_if_exists: 0 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.create_if_missing: 0 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.paranoid_checks: 1 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-14T13:05:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.env: 0x559f89a91720 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.fs: PosixFileSystem 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.info_log: 0x559f8b849380 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_file_opening_threads: 16 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.statistics: (nil) 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.use_fsync: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_log_file_size: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.keep_log_file_num: 1000 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.recycle_log_file_num: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_fallocate: 1 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_mmap_reads: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_mmap_writes: 0 2024-03-14T13:05:14.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.use_direct_reads: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.create_missing_column_families: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.db_log_dir: 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.wal_dir: 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.advise_random_on_open: 1 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.db_write_buffer_size: 0 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.write_buffer_manager: 0x559f8aac2140 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-14T13:05:14.644 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.rate_limiter: (nil) 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.wal_recovery_mode: 2 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enable_thread_tracking: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enable_pipelined_write: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.unordered_write: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.row_cache: None 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.wal_filter: None 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_ingest_behind: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.two_write_queues: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T13:05:14.645 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.wal_compression: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.atomic_flush: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.log_readahead_size: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_background_jobs: 2 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_background_compactions: -1 2024-03-14T13:05:14.646 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_subcompactions: 1 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_open_files: -1 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_background_flushes: -1 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Compression algorithms supported: 2024-03-14T13:05:14.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kZSTD supported: 0 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kXpressCompression supported: 0 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kZlibCompression supported: 1 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kSnappyCompression supported: 1 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kLZ4Compression supported: 1 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: kBZip2Compression supported: 0 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000015 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.merge_operator: 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_filter: None 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T13:05:14.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559f8b8494a0) 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_type: 0 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_index_type: 0 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_shortening: 1 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: checksum: 4 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: no_block_cache: 0 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache: 0x559f8ab63350 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-03-14T13:05:14.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_options: 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: capacity : 536870912 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_shard_bits : 4 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: strict_capacity_limit : 0 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_compressed: (nil) 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: persistent_cache: (nil) 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size: 4096 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size_deviation: 10 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_restart_interval: 16 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_block_restart_interval: 1 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_block_size: 4096 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: partition_filters: 0 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: use_delta_encoding: 1 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: filter_policy: bloomfilter 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: whole_key_filtering: 1 2024-03-14T13:05:14.650 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: verify_compression: 0 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: format_version: 5 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_index_compression: 1 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_align: 0 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_auto_readahead_size: 262144 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: prepopulate_block_cache: 0 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: initial_auto_readahead_size: 8192 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression: NoCompression 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.num_levels: 7 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T13:05:14.651 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T13:05:14.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T13:05:14.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.arena_block_size: 1048576 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T13:05:14.654 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T13:05:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T13:05:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T13:05:14.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.table_properties_collectors: 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.inplace_update_support: 0 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.bloom_locality: 0 2024-03-14T13:05:14.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.max_successive_merges: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.ttl: 2592000 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enable_blob_files: false 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.min_blob_size: 0 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T13:05:14.657 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.blob_file_starting_level: 0 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12090, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1f5f6b4a-8e77-42f1-9bab-b608194c8bf2 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421514335214, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421514346411, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 3465602, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12084, "largest_seqno": 15600, "table_properties": {"data_size": 3452290, "index_size": 8387, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3909, "raw_key_size": 34083, "raw_average_key_size": 22, "raw_value_size": 3421844, "raw_average_value_size": 2216, "num_data_blocks": 381, "num_entries": 1544, "num_filter_entries": 1544, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710421514, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1f5f6b4a-8e77-42f1-9bab-b608194c8bf2", "db_session_id": "W1EJP11S51LTYLI0ANGJ", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421514346508, "job": 1, "event": "recovery_finished"} 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-03-14T13:05:14.658 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi111/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559f8ac00000 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: DB pointer 0x559f8abee000 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** DB Stats ** 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T13:05:14.659 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 1/0 3.31 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L6 1/0 5.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 8.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 692.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 692.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.660 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 239.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 239.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Block cache BinnedLRUCache@0x559f8ab63350#7 capacity: 512.00 MB usage: 13.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.03 KB,0.0007689%) IndexBlock(1,9.08 KB,0.00173151%) Misc(1,0.00 KB,0%) 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T13:05:14.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: starting mon.smithi111 rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi111 fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???) e2 preinit fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).mds e21 new map 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).mds e21 print_map 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: e21 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: legacy client fscid: 1 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Filesystem 'cephfs' (1) 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: fs_name cephfs 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: epoch 21 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: modified 2024-03-14T13:04:49.764363+0000 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: tableserver 0 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: root 0 2024-03-14T13:05:14.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: session_timeout 60 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: session_autoclose 300 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_file_size 1099511627776 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_xattr_size 65536 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: required_client_features {} 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: last_failure 0 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: last_failure_osd_epoch 0 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: max_mds 2 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: in 0,1 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: up {0=14482,1=24261} 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: failed 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: damaged 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: stopped 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_pools [3] 2024-03-14T13:05:14.663 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_pool 2 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: inline_data disabled 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: balancer 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: bal_rank_mask -1 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: standby_count_wanted 1 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: [mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 ceph-mon[145314]: mon.smithi111@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 bash[145276]: ba28dff3939fd81b4823d29b7669f489086743787f4f97b8463e2cc1d5329413 2024-03-14T13:05:14.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:14 smithi111 systemd[1]: Started Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: from='client.? 172.21.15.111:0/484110269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: pgmap v112: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:05:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: pgmap v113: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: mon.smithi111 calling monitor election 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: osdmap e49: 6 total, 6 up, 6 in 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: mgrmap e33: smithi111.atysez(active, since 3m), standbys: smithi137.xbyzbp 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: mgrmap e34: smithi111.atysez(active, since 3m), standbys: smithi137.xbyzbp 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: overall HEALTH_OK 2024-03-14T13:05:17.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:17 smithi137 ceph-mon[50336]: from='mgr.14658 ' entity='' 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: from='mgr.14658 172.21.15.111:0/644987979' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: from='client.14866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/484110269' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: pgmap v112: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:05:17.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: pgmap v113: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: mon.smithi111 calling monitor election 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: osdmap e49: 6 total, 6 up, 6 in 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: mgrmap e33: smithi111.atysez(active, since 3m), standbys: smithi137.xbyzbp 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: mgrmap e34: smithi111.atysez(active, since 3m), standbys: smithi137.xbyzbp 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: overall HEALTH_OK 2024-03-14T13:05:17.749 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:17 smithi111 ceph-mon[145314]: from='mgr.14658 ' entity='' 2024-03-14T13:05:24.311 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: Active manager daemon smithi111.atysez restarted 2024-03-14T13:05:24.311 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: Activating manager daemon smithi111.atysez 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: osdmap e50: 6 total, 6 up, 6 in 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: mgrmap e35: smithi111.atysez(active, starting, since 0.00960714s), standbys: smithi137.xbyzbp 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: Standby manager daemon smithi137.xbyzbp restarted 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:05:24.312 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:05:24.313 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:24 smithi111 ceph-mon[145314]: Manager daemon smithi111.atysez is now available 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: Active manager daemon smithi111.atysez restarted 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: Activating manager daemon smithi111.atysez 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: osdmap e50: 6 total, 6 up, 6 in 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: mgrmap e35: smithi111.atysez(active, starting, since 0.00960714s), standbys: smithi137.xbyzbp 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/crt"}]: dispatch 2024-03-14T13:05:24.379 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp restarted 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: Standby manager daemon smithi137.xbyzbp started 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi111.atysez", "id": "smithi111.atysez"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.xbyzbp/key"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr metadata", "who": "smithi137.xbyzbp", "id": "smithi137.xbyzbp"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.? 172.21.15.137:0/1980201499' entity='mgr.smithi137.xbyzbp' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:05:24.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:05:24.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:05:24.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-14T13:05:24.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-14T13:05:24.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-14T13:05:24.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:24 smithi137 ceph-mon[50336]: Manager daemon smithi111.atysez is now available 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T13:05:25.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:25 smithi137 ceph-mon[50336]: mgrmap e36: smithi111.atysez(active, since 1.01004s), standbys: smithi137.xbyzbp 2024-03-14T13:05:25.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/mirror_snapshot_schedule"}]: dispatch 2024-03-14T13:05:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.atysez/trash_purge_schedule"}]: dispatch 2024-03-14T13:05:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:25 smithi111 ceph-mon[145314]: mgrmap e36: smithi111.atysez(active, since 1.01004s), standbys: smithi137.xbyzbp 2024-03-14T13:05:26.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:26 smithi111 ceph-mon[145314]: [14/Mar/2024:13:05:24] ENGINE Bus STARTING 2024-03-14T13:05:26.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:26 smithi111 ceph-mon[145314]: [14/Mar/2024:13:05:25] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T13:05:26.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:26 smithi111 ceph-mon[145314]: [14/Mar/2024:13:05:25] ENGINE Serving on http://172.21.15.111:8765 2024-03-14T13:05:26.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:26 smithi111 ceph-mon[145314]: [14/Mar/2024:13:05:25] ENGINE Bus STARTED 2024-03-14T13:05:26.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:26 smithi111 ceph-mon[145314]: pgmap v3: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:26.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:05:24] ENGINE Bus STARTING 2024-03-14T13:05:26.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:05:25] ENGINE Serving on https://172.21.15.111:7150 2024-03-14T13:05:26.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:05:25] ENGINE Serving on http://172.21.15.111:8765 2024-03-14T13:05:26.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:26 smithi137 ceph-mon[50336]: [14/Mar/2024:13:05:25] ENGINE Bus STARTED 2024-03-14T13:05:26.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:26 smithi137 ceph-mon[50336]: pgmap v3: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:27 smithi137 ceph-mon[50336]: pgmap v4: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:27 smithi137 ceph-mon[50336]: mgrmap e37: smithi111.atysez(active, since 3s), standbys: smithi137.xbyzbp 2024-03-14T13:05:27.606 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:27 smithi111 ceph-mon[145314]: pgmap v4: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:27.607 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:27 smithi111 ceph-mon[145314]: mgrmap e37: smithi111.atysez(active, since 3s), standbys: smithi137.xbyzbp 2024-03-14T13:05:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:28 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:28 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:28 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:28 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:28.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:28 smithi111 ceph-mon[145314]: pgmap v5: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:28 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:28 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:28 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:28 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:29.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:28 smithi137 ceph-mon[50336]: pgmap v5: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:05:30.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:30 smithi137 ceph-mon[50336]: mgrmap e38: smithi111.atysez(active, since 4s), standbys: smithi137.xbyzbp 2024-03-14T13:05:30.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:30 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:30.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:30 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:30 smithi111 ceph-mon[145314]: mgrmap e38: smithi111.atysez(active, since 4s), standbys: smithi137.xbyzbp 2024-03-14T13:05:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:30.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:31 smithi137 ceph-mon[50336]: pgmap v6: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-14T13:05:31.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:31 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:31.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:31 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:31.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:31 smithi111 ceph-mon[145314]: pgmap v6: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 43 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-14T13:05:31.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:31 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:31.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:31 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:33.752 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:33 smithi111 ceph-mon[145314]: pgmap v7: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-14T13:05:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:33 smithi137 ceph-mon[50336]: pgmap v7: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 31 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-14T13:05:35.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Detected new or changed devices on smithi111 2024-03-14T13:05:35.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:35.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: pgmap v8: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:05:35.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:34 smithi111 ceph-mon[145314]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:05:35.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Detected new or changed devices on smithi111 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.conf 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.conf 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: pgmap v8: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 24 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:05:35.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:34 smithi137 ceph-mon[50336]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:36.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:35 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-14T13:05:36.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: Updating smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:05:36.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: Updating smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:36.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-14T13:05:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:36 smithi111 ceph-mon[145314]: pgmap v9: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:05:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:36 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:36 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:37.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:36 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:37.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:36 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:37.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:36 smithi137 ceph-mon[50336]: pgmap v9: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 21 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-14T13:05:37.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:36 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:37.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:36 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:37.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:36 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:37.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:36 smithi137 ceph-mon[50336]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:37.946 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:37 smithi137 ceph-mon[50336]: Upgrade: Updating mon.smithi137 2024-03-14T13:05:37.946 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:37 smithi137 ceph-mon[50336]: Deploying daemon mon.smithi137 on smithi137 2024-03-14T13:05:38.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:37 smithi111 ceph-mon[145314]: Upgrade: Updating mon.smithi137 2024-03-14T13:05:38.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:37 smithi111 ceph-mon[145314]: Deploying daemon mon.smithi137 on smithi137 2024-03-14T13:05:38.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:38 smithi137 systemd[1]: Stopping Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:05:38.728 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:38 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137[50293]: 2024-03-14T13:05:38.400+0000 7fb435f66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi137 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:05:38.729 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:38 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137[50293]: 2024-03-14T13:05:38.400+0000 7fb435f66700 -1 mon.smithi137@1(peon) e2 *** Got Signal Terminated *** 2024-03-14T13:05:39.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:38 smithi137 bash[107950]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137 2024-03-14T13:05:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi137.service: Succeeded. 2024-03-14T13:05:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 systemd[1]: Stopped Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:05:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 systemd[1]: Starting Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: set uid:gid to 167:167 (ceph:ceph) 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable), process ceph-mon, pid 7 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: pidfile_write: ignore empty --pid-file 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: load: jerasure load: lrc 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: RocksDB version: 7.9.2 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Git sha 0 2024-03-14T13:05:40.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Compile date 2024-03-13 02:27:37 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: DB SUMMARY 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: DB Session ID: 67O61J4OR3Y4WF238UAY 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: CURRENT file: CURRENT 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: IDENTITY file: IDENTITY 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1095 Bytes 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi137/store.db dir, Total Num: 1, files: 000027.sst 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi137/store.db: 000025.log size: 8438536 ; 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.error_if_exists: 0 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.create_if_missing: 0 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.paranoid_checks: 1 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-14T13:05:40.257 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.env: 0x55c886a41720 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.fs: PosixFileSystem 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.info_log: 0x55c8890dd380 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_file_opening_threads: 16 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.statistics: (nil) 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.use_fsync: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_log_file_size: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.keep_log_file_num: 1000 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.recycle_log_file_num: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_fallocate: 1 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_mmap_reads: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_mmap_writes: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.use_direct_reads: 0 2024-03-14T13:05:40.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-14T13:05:40.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.create_missing_column_families: 0 2024-03-14T13:05:40.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.db_log_dir: 2024-03-14T13:05:40.259 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.wal_dir: 2024-03-14T13:05:40.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-14T13:05:40.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-14T13:05:40.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-14T13:05:40.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-14T13:05:40.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.advise_random_on_open: 1 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.db_write_buffer_size: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.write_buffer_manager: 0x55c888356140 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.rate_limiter: (nil) 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.wal_recovery_mode: 2 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enable_thread_tracking: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enable_pipelined_write: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.unordered_write: 0 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-14T13:05:40.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.row_cache: None 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.wal_filter: None 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_ingest_behind: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.two_write_queues: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.manual_wal_flush: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.wal_compression: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.atomic_flush: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-14T13:05:40.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.log_readahead_size: 0 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.best_efforts_recovery: 0 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.allow_data_in_errors: 0 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.db_host_id: __hostname__ 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_background_jobs: 2 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_background_compactions: -1 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_subcompactions: 1 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-14T13:05:40.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_total_wal_size: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_open_files: -1 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bytes_per_sync: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_readahead_size: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_background_flushes: -1 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Compression algorithms supported: 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-14T13:05:40.264 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kZSTD supported: 0 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kXpressCompression supported: 0 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kLZ4HCCompression supported: 1 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kZlibCompression supported: 1 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kSnappyCompression supported: 1 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kLZ4Compression supported: 1 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: kBZip2Compression supported: 0 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000009 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.merge_operator: 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_filter: None 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_filter_factory: None 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.sst_partitioner_factory: None 2024-03-14T13:05:40.265 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8890dd4a0) 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_type: 0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_index_type: 0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_shortening: 1 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: checksum: 4 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: no_block_cache: 0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache: 0x55c8883f71f0 2024-03-14T13:05:40.266 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_options: 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: capacity : 536870912 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_shard_bits : 4 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: strict_capacity_limit : 0 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_compressed: (nil) 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: persistent_cache: (nil) 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size: 4096 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size_deviation: 10 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_restart_interval: 16 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_block_restart_interval: 1 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_block_size: 4096 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: partition_filters: 0 2024-03-14T13:05:40.267 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: use_delta_encoding: 1 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: filter_policy: bloomfilter 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: whole_key_filtering: 1 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: verify_compression: 0 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: format_version: 5 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_index_compression: 1 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_align: 0 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_auto_readahead_size: 262144 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: prepopulate_block_cache: 0 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: initial_auto_readahead_size: 8192 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.write_buffer_size: 33554432 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_write_buffer_number: 2 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression: NoCompression 2024-03-14T13:05:40.268 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression: Disabled 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.prefix_extractor: nullptr 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.num_levels: 7 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-14T13:05:40.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.level: 32767 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.strategy: 0 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.enabled: false 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.target_file_size_base: 67108864 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-14T13:05:40.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.arena_block_size: 1048576 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-14T13:05:40.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.disable_auto_compactions: 0 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.table_properties_collectors: 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.inplace_update_support: 0 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-14T13:05:40.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.bloom_locality: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.max_successive_merges: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.paranoid_file_checks: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.force_consistency_checks: 1 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.report_bg_io_stats: 0 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.ttl: 2592000 2024-03-14T13:05:40.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enable_blob_files: false 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.min_blob_size: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_file_size: 268435456 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.blob_file_starting_level: 0 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-03-14T13:05:40.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12073, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3f5dd895-740b-4fc1-afdd-6443962073ef 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421539830004, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421539849517, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 5055127, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12070, "largest_seqno": 16482, "table_properties": {"data_size": 5039836, "index_size": 9854, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4421, "raw_key_size": 38830, "raw_average_key_size": 22, "raw_value_size": 5005146, "raw_average_value_size": 2863, "num_data_blocks": 445, "num_entries": 1748, "num_filter_entries": 1748, "num_deletions": 260, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710421539, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3f5dd895-740b-4fc1-afdd-6443962073ef", "db_session_id": "67O61J4OR3Y4WF238UAY", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710421539849634, "job": 1, "event": "recovery_finished"} 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi137/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c888494000 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: DB pointer 0x55c888482000 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-14T13:05:40.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** DB Stats ** 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 1/0 4.82 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.2 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L6 1/0 5.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 2/0 10.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.2 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-14T13:05:40.276 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 625.2 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 625.2 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.005, interval 0.005 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-14T13:05:40.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 206.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 206.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Block cache BinnedLRUCache@0x55c8883f71f0#7 capacity: 512.00 MB usage: 14.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,10.08 KB,0.00192225%) Misc(1,0.00 KB,0%) 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: starting mon.smithi137 rank 1 at public addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] at bind addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi137 fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???) e2 preinit fsid 591fecbc-e201-11ee-95c9-87774f69a715 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).mds e21 new map 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).mds e21 print_map 2024-03-14T13:05:40.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: e21 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: legacy client fscid: 1 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Filesystem 'cephfs' (1) 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: fs_name cephfs 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: epoch 21 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: modified 2024-03-14T13:04:49.764363+0000 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: tableserver 0 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: root 0 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: session_timeout 60 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: session_autoclose 300 2024-03-14T13:05:40.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_file_size 1099511627776 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_xattr_size 65536 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: required_client_features {} 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: last_failure 0 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: last_failure_osd_epoch 0 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_mds 2 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: in 0,1 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: up {0=14482,1=24261} 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: failed 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: damaged 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: stopped 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_pools [3] 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_pool 2 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: inline_data disabled 2024-03-14T13:05:40.280 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: balancer 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: bal_rank_mask -1 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: standby_count_wanted 1 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 export targets 0 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).osd e50 crush map has features 3314933000852226048, adjusting msgr requires 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).osd e50 crush map has features 288514051259236352, adjusting msgr requires 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 ceph-mon[108145]: mon.smithi137@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-03-14T13:05:40.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 bash[108108]: 512b2064ceaeedb689649944e449950c0a56f07a0f53751716245d7c1c1e6b9e 2024-03-14T13:05:40.282 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:39 smithi137 systemd[1]: Started Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:05:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: pgmap v11: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-14T13:05:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: overall HEALTH_OK 2024-03-14T13:05:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: mon.smithi111 calling monitor election 2024-03-14T13:05:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: mon.smithi137 calling monitor election 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: monmap e3: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: osdmap e50: 6 total, 6 up, 6 in 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: mgrmap e38: smithi111.atysez(active, since 16s), standbys: smithi137.xbyzbp 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: overall HEALTH_OK 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:41.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:41 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: pgmap v11: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 20 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-14T13:05:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: overall HEALTH_OK 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: mon.smithi111 calling monitor election 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: mon.smithi137 calling monitor election 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: mon.smithi111 is new leader, mons smithi111,smithi137 in quorum (ranks 0,1) 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: monmap e3: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: osdmap e50: 6 total, 6 up, 6 in 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: mgrmap e38: smithi111.atysez(active, since 16s), standbys: smithi137.xbyzbp 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: overall HEALTH_OK 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:41.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:41 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:05:43.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- 172.21.15.111:0/996285111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 msgr2=0x7f3a841020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/996285111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a841020d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3a74004940 tx=0x7f3a74009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- 172.21.15.111:0/996285111 shutdown_connections 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/996285111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a840fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/996285111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a841020d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- 172.21.15.111:0/996285111 >> 172.21.15.111:0/996285111 conn(0x7f3a840f78b0 msgr2=0x7f3a840f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- 172.21.15.111:0/996285111 shutdown_connections 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- 172.21.15.111:0/996285111 wait complete. 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 Processor -- start 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.351+0000 7f3a8b46b700 1 -- start start 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a8b46b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a8b46b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a8b46b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a84116670 con 0x7f3a84101d00 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a8b46b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8410fc50 con 0x7f3a841026a0 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48358/0 (socket says 172.21.15.111:48358) 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 -- 172.21.15.111:0/1635604098 learned_addr learned my addr 172.21.15.111:0/1635604098 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a88a06700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a88a06700 1 -- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 msgr2=0x7f3a84116020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a88a06700 1 -- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 msgr2=0x7f3a84116020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a88a06700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a88a06700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 -- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 msgr2=0x7f3a84116020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 -- 172.21.15.111:0/1635604098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a74005040 con 0x7f3a84101d00 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a89207700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b3b0 tx=0x7f3a74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a7400e040 con 0x7f3a84101d00 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a74013420 con 0x7f3a84101d00 2024-03-14T13:05:43.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8410fed0 con 0x7f3a84101d00 2024-03-14T13:05:43.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.352+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a7401c630 con 0x7f3a84101d00 2024-03-14T13:05:43.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.353+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a84110490 con 0x7f3a84101d00 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.354+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a840fd450 con 0x7f3a84101d00 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.354+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f3a74013970 con 0x7f3a84101d00 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.355+0000 7f3a81ffb700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 0x7f3a6c06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.355+0000 7f3a88a06700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 0x7f3a6c06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.355+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3a7408cf70 con 0x7f3a84101d00 2024-03-14T13:05:43.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.355+0000 7f3a88a06700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 0x7f3a6c06eda0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3a84111380 tx=0x7f3a78009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:43.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.360+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3a7405aa60 con 0x7f3a84101d00 2024-03-14T13:05:43.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.573+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a84108cc0 con 0x7f3a6c06c8f0 2024-03-14T13:05:43.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.574+0000 7f3a81ffb700 1 -- 172.21.15.111:0/1635604098 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f3a84108cc0 con 0x7f3a6c06c8f0 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 msgr2=0x7f3a6c06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 0x7f3a6c06eda0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3a84111380 tx=0x7f3a78009ee0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 msgr2=0x7f3a84115ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400b3b0 tx=0x7f3a74009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 shutdown_connections 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a6c06c8f0 0x7f3a6c06eda0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a84101d00 0x7f3a84115ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 --2- 172.21.15.111:0/1635604098 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a841026a0 0x7f3a84116020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 >> 172.21.15.111:0/1635604098 conn(0x7f3a840f78b0 msgr2=0x7f3a840f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:43.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 shutdown_connections 2024-03-14T13:05:43.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.576+0000 7f3a8b46b700 1 -- 172.21.15.111:0/1635604098 wait complete. 2024-03-14T13:05:43.586 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:05:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:43 smithi111 ceph-mon[145314]: pgmap v12: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:43 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 -- 172.21.15.111:0/2386664996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0101d70 msgr2=0x7f67f0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2386664996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0101d70 0x7f67f0102140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f67e4004940 tx=0x7f67e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 -- 172.21.15.111:0/2386664996 shutdown_connections 2024-03-14T13:05:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2386664996 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0102710 0x7f67f00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.677 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2386664996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0101d70 0x7f67f0102140 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 -- 172.21.15.111:0/2386664996 >> 172.21.15.111:0/2386664996 conn(0x7f67f0072bc0 msgr2=0x7f67f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 -- 172.21.15.111:0/2386664996 shutdown_connections 2024-03-14T13:05:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.677+0000 7f67f6d33700 1 -- 172.21.15.111:0/2386664996 wait complete. 2024-03-14T13:05:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 Processor -- start 2024-03-14T13:05:43.678 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 -- start start 2024-03-14T13:05:43.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.680 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0113530 con 0x7f67f0102710 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f6d33700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f010c920 con 0x7f67f0101d70 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48368/0 (socket says 172.21.15.111:48368) 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 -- 172.21.15.111:0/2018866252 learned_addr learned my addr 172.21.15.111:0/2018866252 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f4acf700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f4acf700 1 -- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 msgr2=0x7f67f01129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f4acf700 1 -- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 msgr2=0x7f67f01129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f4acf700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67f4acf700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 -- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 msgr2=0x7f67f01129a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.678+0000 7f67effff700 1 -- 172.21.15.111:0/2018866252 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e4005040 con 0x7f67f0102710 2024-03-14T13:05:43.681 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67effff700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67dc002a50 tx=0x7f67dc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67dc007cc0 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f010cc00 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67dc00f040 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67dc00d990 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.679+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f010d1c0 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.680+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f00ff080 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.680+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f67dc009710 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.680+0000 7f67ed7fa700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 0x7f67e006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.681+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f67dc08ba80 con 0x7f67f0102710 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.681+0000 7f67f4acf700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 0x7f67e006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:43.682 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.681+0000 7f67f4acf700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 0x7f67e006ed90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67e4004f90 tx=0x7f67e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:43.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.685+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f67dc059590 con 0x7f67f0102710 2024-03-14T13:05:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:43 smithi137 ceph-mon[108145]: pgmap v12: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:43 smithi137 ceph-mon[108145]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2 up:standby-replay 2024-03-14T13:05:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:43.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.901+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67f0112670 con 0x7f67e006c8e0 2024-03-14T13:05:43.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.902+0000 7f67ed7fa700 1 -- 172.21.15.111:0/2018866252 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f67f0112670 con 0x7f67e006c8e0 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.904+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 msgr2=0x7f67e006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.904+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 0x7f67e006ed90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f67e4004f90 tx=0x7f67e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.904+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 msgr2=0x7f67f0112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.904+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f67dc002a50 tx=0x7f67dc004600 comp rx=0 tx=0).stop 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 shutdown_connections 2024-03-14T13:05:43.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f67e006c8e0 0x7f67e006ed90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f67f0101d70 0x7f67f01129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 --2- 172.21.15.111:0/2018866252 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f67f0102710 0x7f67f0112ee0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 >> 172.21.15.111:0/2018866252 conn(0x7f67f0072bc0 msgr2=0x7f67f00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 shutdown_connections 2024-03-14T13:05:43.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:43.905+0000 7f67f6d33700 1 -- 172.21.15.111:0/2018866252 wait complete. 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 -- 172.21.15.111:0/1977106992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 msgr2=0x7f1e3c1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 --2- 172.21.15.111:0/1977106992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1020d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c004940 tx=0x7f1e2c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 -- 172.21.15.111:0/1977106992 shutdown_connections 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 --2- 172.21.15.111:0/1977106992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c0fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 --2- 172.21.15.111:0/1977106992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1020d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 -- 172.21.15.111:0/1977106992 >> 172.21.15.111:0/1977106992 conn(0x7f1e3c0f78d0 msgr2=0x7f1e3c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 -- 172.21.15.111:0/1977106992 shutdown_connections 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.005+0000 7f1e4424b700 1 -- 172.21.15.111:0/1977106992 wait complete. 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 Processor -- start 2024-03-14T13:05:44.007 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 -- start start 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c113570 con 0x7f1e3c101d00 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e4424b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c10c960 con 0x7f1e3c1026a0 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48384/0 (socket says 172.21.15.111:48384) 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 -- 172.21.15.111:0/4040155041 learned_addr learned my addr 172.21.15.111:0/4040155041 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e417e6700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e417e6700 1 -- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 msgr2=0x7f1e3c112f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e417e6700 1 -- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 msgr2=0x7f1e3c112f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e417e6700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e417e6700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:44.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 -- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 msgr2=0x7f1e3c112f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.006+0000 7f1e41fe7700 1 -- 172.21.15.111:0/4040155041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c005040 con 0x7f1e3c101d00 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e41fe7700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c00b3b0 tx=0x7f1e2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2c00e040 con 0x7f1e3c101d00 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e2c013420 con 0x7f1e3c101d00 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c10cbe0 con 0x7f1e3c101d00 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2c01c630 con 0x7f1e3c101d00 2024-03-14T13:05:44.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.007+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c10d1a0 con 0x7f1e3c101d00 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.008+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c0fd450 con 0x7f1e3c101d00 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.008+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f1e2c013970 con 0x7f1e3c101d00 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.009+0000 7f1e327fc700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 0x7f1e2806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.009+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1e2c08cdc0 con 0x7f1e3c101d00 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.009+0000 7f1e417e6700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 0x7f1e2806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.009+0000 7f1e417e6700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 0x7f1e2806ed90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c10db20 tx=0x7f1e38009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.013+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1e2c05a9f0 con 0x7f1e3c101d00 2024-03-14T13:05:44.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.212+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e3c108d50 con 0x7f1e2806c8e0 2024-03-14T13:05:44.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.222+0000 7f1e327fc700 1 -- 172.21.15.111:0/4040155041 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f1e3c108d50 con 0x7f1e2806c8e0 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (116s) 14s ago 13m 16.9M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (13m) 14s ago 13m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (12m) 1s ago 12m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (95s) 14s ago 12m 82.1M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (7m) 14s ago 7m 64.3M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (7m) 14s ago 7m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (7m) 1s ago 7m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (7m) 1s ago 7m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (4m) 14s ago 14m 467M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:05:44.222 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (3m) 1s ago 12m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (30s) 14s ago 15m 28.7M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (4s) 1s ago 12m 26.4M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (2m) 14s ago 13m 18.6M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (2m) 1s ago 12m 17.9M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (11m) 14s ago 11m 210M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (10m) 14s ago 10m 237M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (10m) 14s ago 10m 197M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (9m) 1s ago 9m 265M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (9m) 1s ago 9m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (9m) 1s ago 9m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:05:44.223 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (2m) 14s ago 12m 57.9M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 msgr2=0x7f1e2806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 0x7f1e2806ed90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c10db20 tx=0x7f1e38009f70 comp rx=0 tx=0).stop 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 msgr2=0x7f1e3c1129e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c00b3b0 tx=0x7f1e2c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 shutdown_connections 2024-03-14T13:05:44.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2806c8e0 0x7f1e2806ed90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e3c101d00 0x7f1e3c1129e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 --2- 172.21.15.111:0/4040155041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e3c1026a0 0x7f1e3c112f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 >> 172.21.15.111:0/4040155041 conn(0x7f1e3c0f78d0 msgr2=0x7f1e3c0f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 shutdown_connections 2024-03-14T13:05:44.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.224+0000 7f1e4424b700 1 -- 172.21.15.111:0/4040155041 wait complete. 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- 172.21.15.111:0/1001228142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 msgr2=0x7feb700fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 --2- 172.21.15.111:0/1001228142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb700fbd80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb6c004940 tx=0x7feb6c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- 172.21.15.111:0/1001228142 shutdown_connections 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 --2- 172.21.15.111:0/1001228142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb700fbd80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 --2- 172.21.15.111:0/1001228142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- 172.21.15.111:0/1001228142 >> 172.21.15.111:0/1001228142 conn(0x7feb70072bc0 msgr2=0x7feb70072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- 172.21.15.111:0/1001228142 shutdown_connections 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- 172.21.15.111:0/1001228142 wait complete. 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 Processor -- start 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.324+0000 7feb7737d700 1 -- start start 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb7737d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb7737d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb7737d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb70114690 con 0x7feb70102710 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb7737d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7010da80 con 0x7feb70101d70 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48406/0 (socket says 172.21.15.111:48406) 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 -- 172.21.15.111:0/3205258179 learned_addr learned my addr 172.21.15.111:0/3205258179 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb75119700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb75119700 1 -- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 msgr2=0x7feb70113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb75119700 1 -- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 msgr2=0x7feb70113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb75119700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb75119700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 -- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 msgr2=0x7feb70113b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 -- 172.21.15.111:0/3205258179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c005040 con 0x7feb70102710 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb74918700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009390 tx=0x7feb6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c00e040 con 0x7feb70102710 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c013420 con 0x7feb70102710 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7010dd00 con 0x7feb70102710 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.325+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb6c01c630 con 0x7feb70102710 2024-03-14T13:05:44.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.326+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7010e2c0 con 0x7feb70102710 2024-03-14T13:05:44.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.327+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb700fd4c0 con 0x7feb70102710 2024-03-14T13:05:44.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.327+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7feb6c0139e0 con 0x7feb70102710 2024-03-14T13:05:44.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.327+0000 7feb65ffb700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 0x7feb5c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.327+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7feb6c08cf10 con 0x7feb70102710 2024-03-14T13:05:44.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.328+0000 7feb75119700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 0x7feb5c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.328+0000 7feb75119700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 0x7feb5c06ed90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb60009ce0 tx=0x7feb60009f70 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.332+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7feb6c05aaa0 con 0x7feb70102710 2024-03-14T13:05:44.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.605+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feb70108d30 con 0x7feb70102710 2024-03-14T13:05:44.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.606+0000 7feb65ffb700 1 -- 172.21.15.111:0/3205258179 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7feb6c05a630 con 0x7feb70102710 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:05:44.606 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 10, 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 msgr2=0x7feb5c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 0x7feb5c06ed90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7feb60009ce0 tx=0x7feb60009f70 comp rx=0 tx=0).stop 2024-03-14T13:05:44.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 msgr2=0x7feb70114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009390 tx=0x7feb6c009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 shutdown_connections 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb5c06c8e0 0x7feb5c06ed90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb70101d70 0x7feb70113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 --2- 172.21.15.111:0/3205258179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb70102710 0x7feb70114040 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 >> 172.21.15.111:0/3205258179 conn(0x7feb70072bc0 msgr2=0x7feb70065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 shutdown_connections 2024-03-14T13:05:44.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.608+0000 7feb7737d700 1 -- 172.21.15.111:0/3205258179 wait complete. 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- 172.21.15.111:0/768333453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 msgr2=0x7fa6840fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 --2- 172.21.15.111:0/768333453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa6840fce30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa674004940 tx=0x7fa674009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- 172.21.15.111:0/768333453 shutdown_connections 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 --2- 172.21.15.111:0/768333453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa6840fce30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 --2- 172.21.15.111:0/768333453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6841031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- 172.21.15.111:0/768333453 >> 172.21.15.111:0/768333453 conn(0x7fa6840f89d0 msgr2=0x7fa6840fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- 172.21.15.111:0/768333453 shutdown_connections 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- 172.21.15.111:0/768333453 wait complete. 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 Processor -- start 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.707+0000 7fa688a9b700 1 -- start start 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa684070e20 con 0x7fa6841037c0 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa684070f60 con 0x7fa684102e20 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48422/0 (socket says 172.21.15.111:48422) 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa68259c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 -- 172.21.15.111:0/3712597461 learned_addr learned my addr 172.21.15.111:0/3712597461 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa68259c700 1 -- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 msgr2=0x7fa6840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa68259c700 1 -- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 msgr2=0x7fa6840702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa68259c700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa68259c700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-14T13:05:44.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 -- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 msgr2=0x7fa6840702c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 -- 172.21.15.111:0/3712597461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674005040 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa681d9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa674009380 tx=0x7fa674009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67400e040 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa674013420 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa67401c630 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa684119da0 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.708+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68411a280 con 0x7fa6841037c0 2024-03-14T13:05:44.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.709+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6840fe570 con 0x7fa6841037c0 2024-03-14T13:05:44.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.710+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7fa6740139e0 con 0x7fa6841037c0 2024-03-14T13:05:44.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.710+0000 7fa672ffd700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 0x7fa66c06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:44.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.710+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa67408ce30 con 0x7fa6841037c0 2024-03-14T13:05:44.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.710+0000 7fa68259c700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 0x7fa66c06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:44.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.710+0000 7fa68259c700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 0x7fa66c06ed90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa678006fd0 tx=0x7fa678008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:44.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.715+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa674056f30 con 0x7fa6841037c0 2024-03-14T13:05:44.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.952+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa684109e30 con 0x7fa6841037c0 2024-03-14T13:05:44.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.953+0000 7fa672ffd700 1 -- 172.21.15.111:0/3712597461 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fa67405a550 con 0x7fa6841037c0 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:05:44.953 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:05:44.954 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:05:44.955 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:44.956 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:05:44.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 msgr2=0x7fa66c06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 0x7fa66c06ed90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa678006fd0 tx=0x7fa678008040 comp rx=0 tx=0).stop 2024-03-14T13:05:44.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 msgr2=0x7fa684070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa674009380 tx=0x7fa674009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 shutdown_connections 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa66c06c8e0 0x7fa66c06ed90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa684102e20 0x7fa6840702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 --2- 172.21.15.111:0/3712597461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6841037c0 0x7fa684070800 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 >> 172.21.15.111:0/3712597461 conn(0x7fa6840f89d0 msgr2=0x7fa6840f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 shutdown_connections 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:44.955+0000 7fa688a9b700 1 -- 172.21.15.111:0/3712597461 wait complete. 2024-03-14T13:05:44.957 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:05:45.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 -- 172.21.15.111:0/3033744464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0101d70 msgr2=0x7f35a0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 --2- 172.21.15.111:0/3033744464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0101d70 0x7f35a0102140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3594004940 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:45.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 -- 172.21.15.111:0/3033744464 shutdown_connections 2024-03-14T13:05:45.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 --2- 172.21.15.111:0/3033744464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0102710 0x7f35a00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 --2- 172.21.15.111:0/3033744464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0101d70 0x7f35a0102140 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 -- 172.21.15.111:0/3033744464 >> 172.21.15.111:0/3033744464 conn(0x7f35a0072bc0 msgr2=0x7f35a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:45.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 -- 172.21.15.111:0/3033744464 shutdown_connections 2024-03-14T13:05:45.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.055+0000 7f35a7132700 1 -- 172.21.15.111:0/3033744464 wait complete. 2024-03-14T13:05:45.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.056+0000 7f35a7132700 1 Processor -- start 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.056+0000 7f35a7132700 1 -- start start 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0101d70 0x7f35a01129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0113530 con 0x7f35a0102710 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a010c920 con 0x7f35a0101d70 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48452/0 (socket says 172.21.15.111:48452) 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 -- 172.21.15.111:0/2337392569 learned_addr learned my addr 172.21.15.111:0/2337392569 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a4ece700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0101d70 0x7f35a01129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 -- 172.21.15.111:0/2337392569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0101d70 msgr2=0x7f35a01129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0101d70 0x7f35a01129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 -- 172.21.15.111:0/2337392569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594005040 con 0x7f35a0102710 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359ffff700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c800 tx=0x7f358c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358c0106f0 con 0x7f35a0102710 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f358c010d00 con 0x7f35a0102710 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a010cc00 con 0x7f35a0102710 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358c0113a0 con 0x7f35a0102710 2024-03-14T13:05:45.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.057+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a010d1c0 con 0x7f35a0102710 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.059+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00ff080 con 0x7f35a0102710 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.059+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f358c012400 con 0x7f35a0102710 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.059+0000 7f359d7fa700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 0x7f359006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.059+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f358c08baf0 con 0x7f35a0102710 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.059+0000 7f35a4ece700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 0x7f359006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.060+0000 7f35a4ece700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 0x7f359006ed90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3594004f90 tx=0x7f3594009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:45.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.064+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f358c059790 con 0x7f35a0102710 2024-03-14T13:05:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: pgmap v13: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:44 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3205258179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:45.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='client.34130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: pgmap v13: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:45.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:44 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3205258179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:05:45.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.278+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f35a010e060 con 0x7f359006c8e0 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.279+0000 7f359d7fa700 1 -- 172.21.15.111:0/2337392569 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f35a010e060 con 0x7f359006c8e0 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:05:45.279 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "4/21 daemons upgraded", 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mon daemons", 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:05:45.280 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.281+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 msgr2=0x7f359006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.281+0000 7f35a7132700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 0x7f359006ed90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3594004f90 tx=0x7f3594009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.281+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 msgr2=0x7f35a0112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.281+0000 7f35a7132700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c800 tx=0x7f358c004470 comp rx=0 tx=0).stop 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 shutdown_connections 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c8e0 0x7f359006ed90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a0101d70 0x7f35a01129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 --2- 172.21.15.111:0/2337392569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a0102710 0x7f35a0112ee0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 >> 172.21.15.111:0/2337392569 conn(0x7f35a0072bc0 msgr2=0x7f35a00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:45.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 shutdown_connections 2024-03-14T13:05:45.282 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.282+0000 7f35a7132700 1 -- 172.21.15.111:0/2337392569 wait complete. 2024-03-14T13:05:45.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 -- 172.21.15.111:0/1453285052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20102710 msgr2=0x7f4f200fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 --2- 172.21.15.111:0/1453285052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20102710 0x7f4f200fbd80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4f14004940 tx=0x7f4f14009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 -- 172.21.15.111:0/1453285052 shutdown_connections 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 --2- 172.21.15.111:0/1453285052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20102710 0x7f4f200fbd80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 --2- 172.21.15.111:0/1453285052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20101d70 0x7f4f20102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.381+0000 7f4f25b06700 1 -- 172.21.15.111:0/1453285052 >> 172.21.15.111:0/1453285052 conn(0x7f4f20072bc0 msgr2=0x7f4f20072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 -- 172.21.15.111:0/1453285052 shutdown_connections 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 -- 172.21.15.111:0/1453285052 wait complete. 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 Processor -- start 2024-03-14T13:05:45.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 -- start start 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20102710 0x7f4f2010f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2010f680 con 0x7f4f20101d70 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.382+0000 7f4f25b06700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2010f7c0 con 0x7f4f20102710 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48470/0 (socket says 172.21.15.111:48470) 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 -- 172.21.15.111:0/862499011 learned_addr learned my addr 172.21.15.111:0/862499011 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1effd700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20102710 0x7f4f2010f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 -- 172.21.15.111:0/862499011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20102710 msgr2=0x7f4f2010f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20102710 0x7f4f2010f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 -- 172.21.15.111:0/862499011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14005040 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f1f7fe700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f10002a50 tx=0x7f4f100046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f10007cc0 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f20117ab0 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f1000f040 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f1000d9c0 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.383+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f20118040 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.384+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f200ff080 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.385+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f4f1000db80 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.385+0000 7f4f07fff700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 0x7f4f0806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.385+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4f1008b9c0 con 0x7f4f20101d70 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.385+0000 7f4f1effd700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 0x7f4f0806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:05:45.386 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.386+0000 7f4f1effd700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 0x7f4f0806ed90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f20066020 tx=0x7f4f14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:05:45.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.390+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4f10055a40 con 0x7f4f20101d70 2024-03-14T13:05:45.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.662+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4f20108d70 con 0x7f4f20101d70 2024-03-14T13:05:45.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.662+0000 7f4f07fff700 1 -- 172.21.15.111:0/862499011 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f4f10059060 con 0x7f4f20101d70 2024-03-14T13:05:45.662 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.664+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 msgr2=0x7f4f0806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.664+0000 7f4f25b06700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 0x7f4f0806ed90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f20066020 tx=0x7f4f14009e30 comp rx=0 tx=0).stop 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.664+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 msgr2=0x7f4f2010ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.664+0000 7f4f25b06700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4f10002a50 tx=0x7f4f100046b0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 shutdown_connections 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0806c8e0 0x7f4f0806ed90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f20101d70 0x7f4f2010ec00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 --2- 172.21.15.111:0/862499011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f20102710 0x7f4f2010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:05:45.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 >> 172.21.15.111:0/862499011 conn(0x7f4f20072bc0 msgr2=0x7f4f200fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:05:45.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 shutdown_connections 2024-03-14T13:05:45.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:05:45.665+0000 7f4f25b06700 1 -- 172.21.15.111:0/862499011 wait complete. 2024-03-14T13:05:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:45 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3712597461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:05:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:45 smithi137 ceph-mon[108145]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:45 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/862499011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:05:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:45 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3712597461' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:05:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:45 smithi111 ceph-mon[145314]: from='client.34150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:05:46.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:45 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/862499011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:05:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:46 smithi137 ceph-mon[108145]: pgmap v14: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:05:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:46 smithi111 ceph-mon[145314]: pgmap v14: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:05:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: Detected new or changed devices on smithi137 2024-03-14T13:05:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:48.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:48.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:48.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:48.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:48.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: Detected new or changed devices on smithi137 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:05:48.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:49.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:48 smithi111 ceph-mon[145314]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-14T13:05:49.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:48 smithi111 ceph-mon[145314]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-14T13:05:49.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:48 smithi111 ceph-mon[145314]: pgmap v15: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:49.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:48 smithi137 ceph-mon[108145]: Reconfiguring mon.smithi111 (monmap changed)... 2024-03-14T13:05:49.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:48 smithi137 ceph-mon[108145]: Reconfiguring daemon mon.smithi111 on smithi111 2024-03-14T13:05:49.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:48 smithi137 ceph-mon[108145]: pgmap v15: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:05:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: Reconfiguring mgr.smithi111.atysez (monmap changed)... 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:50 smithi111 ceph-mon[145314]: Reconfiguring daemon mgr.smithi111.atysez on smithi111 2024-03-14T13:05:50.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: Reconfiguring mgr.smithi111.atysez (monmap changed)... 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.atysez", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:50 smithi137 ceph-mon[108145]: Reconfiguring daemon mgr.smithi111.atysez on smithi111 2024-03-14T13:05:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:51 smithi111 ceph-mon[145314]: pgmap v16: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:05:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:05:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:51 smithi137 ceph-mon[108145]: pgmap v16: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:05:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:05:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:52.484 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:52 smithi111 ceph-mon[145314]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-14T13:05:52.484 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:52 smithi111 ceph-mon[145314]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-14T13:05:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:52 smithi137 ceph-mon[108145]: Reconfiguring crash.smithi111 (monmap changed)... 2024-03-14T13:05:52.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:52 smithi137 ceph-mon[108145]: Reconfiguring daemon crash.smithi111 on smithi111 2024-03-14T13:05:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:53 smithi111 ceph-mon[145314]: pgmap v17: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T13:05:53.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:53 smithi137 ceph-mon[108145]: pgmap v17: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:05:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T13:05:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:54 smithi111 ceph-mon[145314]: Reconfiguring osd.0 (monmap changed)... 2024-03-14T13:05:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:54 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.0 on smithi111 2024-03-14T13:05:54.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:54 smithi137 ceph-mon[108145]: Reconfiguring osd.0 (monmap changed)... 2024-03-14T13:05:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:54 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.0 on smithi111 2024-03-14T13:05:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:05:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:55 smithi137 ceph-mon[108145]: pgmap v18: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:05:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T13:05:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:55.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:55 smithi111 ceph-mon[145314]: pgmap v18: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:05:55.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:55.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:55.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T13:05:55.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:56.731 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: Reconfiguring osd.1 (monmap changed)... 2024-03-14T13:05:56.731 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.1 on smithi111 2024-03-14T13:05:56.731 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:56.731 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:56.731 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T13:05:56.732 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:56.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: Reconfiguring osd.1 (monmap changed)... 2024-03-14T13:05:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.1 on smithi111 2024-03-14T13:05:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T13:05:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:57.540 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:57 smithi111 ceph-mon[145314]: Reconfiguring osd.2 (monmap changed)... 2024-03-14T13:05:57.541 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:57 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.2 on smithi111 2024-03-14T13:05:57.541 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:57 smithi111 ceph-mon[145314]: pgmap v19: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:05:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:57 smithi137 ceph-mon[108145]: Reconfiguring osd.2 (monmap changed)... 2024-03-14T13:05:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:57 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.2 on smithi111 2024-03-14T13:05:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:57 smithi137 ceph-mon[108145]: pgmap v19: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:05:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: Reconfiguring mds.cephfs.smithi111.zmslha (monmap changed)... 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: Reconfiguring daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T13:05:59.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:05:58 smithi137 ceph-mon[108145]: pgmap v20: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: Reconfiguring mds.cephfs.smithi111.zmslha (monmap changed)... 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: Reconfiguring daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T13:05:59.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:05:58 smithi111 ceph-mon[145314]: pgmap v20: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: Reconfiguring mds.cephfs.smithi111.yzwmcx (monmap changed)... 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:00.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:00 smithi137 ceph-mon[108145]: Reconfiguring daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T13:06:00.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: Reconfiguring mds.cephfs.smithi111.yzwmcx (monmap changed)... 2024-03-14T13:06:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:00.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:00 smithi111 ceph-mon[145314]: Reconfiguring daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T13:06:01.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:01 smithi137 ceph-mon[108145]: pgmap v21: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:01.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:01 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:01.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:01 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:01.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:01 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:01.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:01 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:01 smithi111 ceph-mon[145314]: pgmap v21: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:01 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:01 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:01 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:01.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:01 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:02 smithi111 ceph-mon[145314]: Reconfiguring crash.smithi137 (monmap changed)... 2024-03-14T13:06:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:02 smithi111 ceph-mon[145314]: Reconfiguring daemon crash.smithi137 on smithi137 2024-03-14T13:06:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:02 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:02.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:02 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:02 smithi137 ceph-mon[108145]: Reconfiguring crash.smithi137 (monmap changed)... 2024-03-14T13:06:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:02 smithi137 ceph-mon[108145]: Reconfiguring daemon crash.smithi137 on smithi137 2024-03-14T13:06:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:02 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:02 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: pgmap v22: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:03.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: Reconfiguring mgr.smithi137.xbyzbp (monmap changed)... 2024-03-14T13:06:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:06:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:06:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:03.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:03 smithi111 ceph-mon[145314]: Reconfiguring daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: pgmap v22: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: Reconfiguring mgr.smithi137.xbyzbp (monmap changed)... 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.xbyzbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:03 smithi137 ceph-mon[108145]: Reconfiguring daemon mgr.smithi137.xbyzbp on smithi137 2024-03-14T13:06:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:05.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: Reconfiguring mon.smithi137 (monmap changed)... 2024-03-14T13:06:05.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:06:05.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:06:05.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:05.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: Reconfiguring daemon mon.smithi137 on smithi137 2024-03-14T13:06:05.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:04 smithi111 ceph-mon[145314]: pgmap v23: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:05.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:05.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: Reconfiguring mon.smithi137 (monmap changed)... 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: Reconfiguring daemon mon.smithi137 on smithi137 2024-03-14T13:06:05.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:04 smithi137 ceph-mon[108145]: pgmap v23: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:06.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: Reconfiguring osd.3 (monmap changed)... 2024-03-14T13:06:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T13:06:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:06.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:06 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.3 on smithi137 2024-03-14T13:06:06.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: Reconfiguring osd.3 (monmap changed)... 2024-03-14T13:06:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T13:06:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:06.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:06 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.3 on smithi137 2024-03-14T13:06:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:07 smithi137 ceph-mon[108145]: pgmap v24: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T13:06:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:07 smithi111 ceph-mon[145314]: pgmap v24: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T13:06:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:08.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: Reconfiguring osd.4 (monmap changed)... 2024-03-14T13:06:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.4 on smithi137 2024-03-14T13:06:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T13:06:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:08 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: Reconfiguring osd.4 (monmap changed)... 2024-03-14T13:06:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.4 on smithi137 2024-03-14T13:06:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T13:06:08.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:08 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:09.746 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:09 smithi137 ceph-mon[108145]: pgmap v25: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:09.747 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:09 smithi137 ceph-mon[108145]: Reconfiguring osd.5 (monmap changed)... 2024-03-14T13:06:09.747 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:09 smithi137 ceph-mon[108145]: Reconfiguring daemon osd.5 on smithi137 2024-03-14T13:06:09.747 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:09 smithi111 ceph-mon[145314]: pgmap v25: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:09 smithi111 ceph-mon[145314]: Reconfiguring osd.5 (monmap changed)... 2024-03-14T13:06:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:09 smithi111 ceph-mon[145314]: Reconfiguring daemon osd.5 on smithi137 2024-03-14T13:06:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:11.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:11.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:11.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: Reconfiguring mds.cephfs.smithi137.ryucox (monmap changed)... 2024-03-14T13:06:11.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:11.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:11.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: Reconfiguring daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T13:06:11.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:10 smithi111 ceph-mon[145314]: pgmap v26: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:11.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: Reconfiguring mds.cephfs.smithi137.ryucox (monmap changed)... 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: Reconfiguring daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T13:06:11.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:10 smithi137 ceph-mon[108145]: pgmap v26: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: Reconfiguring mds.cephfs.smithi137.iiqzcp (monmap changed)... 2024-03-14T13:06:12.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:12.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:12.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:12 smithi111 ceph-mon[145314]: Reconfiguring daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: Reconfiguring mds.cephfs.smithi137.iiqzcp (monmap changed)... 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:12.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:12 smithi137 ceph-mon[108145]: Reconfiguring daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: pgmap v27: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]: dispatch 2024-03-14T13:06:13.626 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]': finished 2024-03-14T13:06:13.627 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]: dispatch 2024-03-14T13:06:13.627 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:13 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]': finished 2024-03-14T13:06:13.763 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: pgmap v27: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:13.763 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]: dispatch 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi111"}]': finished 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]: dispatch 2024-03-14T13:06:13.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:13 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]': finished 2024-03-14T13:06:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:14 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all mon 2024-03-14T13:06:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:14 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all mon 2024-03-14T13:06:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: pgmap v28: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: Upgrade: Updating crash.smithi111 (1/2) 2024-03-14T13:06:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:15 smithi137 ceph-mon[108145]: Deploying daemon crash.smithi111 on smithi111 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 -- 172.21.15.111:0/2472460580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff318102e00 msgr2=0x7ff3181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 --2- 172.21.15.111:0/2472460580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff318102e00 0x7ff3181031d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff30c004940 tx=0x7ff30c009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 -- 172.21.15.111:0/2472460580 shutdown_connections 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 --2- 172.21.15.111:0/2472460580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3181037a0 0x7ff3180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 --2- 172.21.15.111:0/2472460580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff318102e00 0x7ff3181031d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:15.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 -- 172.21.15.111:0/2472460580 >> 172.21.15.111:0/2472460580 conn(0x7ff3180f89b0 msgr2=0x7ff3180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 -- 172.21.15.111:0/2472460580 shutdown_connections 2024-03-14T13:06:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.767+0000 7ff31e123700 1 -- 172.21.15.111:0/2472460580 wait complete. 2024-03-14T13:06:15.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 Processor -- start 2024-03-14T13:06:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 -- start start 2024-03-14T13:06:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff318102e00 0x7ff318113b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3181146b0 con 0x7ff3181037a0 2024-03-14T13:06:15.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff31e123700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31810daa0 con 0x7ff318102e00 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff316ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff316ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40660/0 (socket says 172.21.15.111:40660) 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.768+0000 7ff316ffd700 1 -- 172.21.15.111:0/3407936766 learned_addr learned my addr 172.21.15.111:0/3407936766 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff3177fe700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff318102e00 0x7ff318113b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff316ffd700 1 -- 172.21.15.111:0/3407936766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff318102e00 msgr2=0x7ff318113b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff316ffd700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff318102e00 0x7ff318113b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff316ffd700 1 -- 172.21.15.111:0/3407936766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c005040 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff316ffd700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff308002a50 tx=0x7ff308004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff308007cc0 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff30800f040 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31810dd80 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30800d990 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.769+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31810e340 con 0x7ff3181037a0 2024-03-14T13:06:15.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.770+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3180fe550 con 0x7ff3181037a0 2024-03-14T13:06:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.771+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7ff308009710 con 0x7ff3181037a0 2024-03-14T13:06:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.771+0000 7ff2fffff700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 0x7ff30006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.771+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7ff30808bb20 con 0x7ff3181037a0 2024-03-14T13:06:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.771+0000 7ff3177fe700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 0x7ff30006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:15.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.771+0000 7ff3177fe700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 0x7ff30006ede0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff30c004560 tx=0x7ff30c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: pgmap v28: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: Upgrade: Updating crash.smithi111 (1/2) 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:15.777 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:15 smithi111 ceph-mon[145314]: Deploying daemon crash.smithi111 on smithi111 2024-03-14T13:06:15.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:15.778+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff308059590 con 0x7ff3181037a0 2024-03-14T13:06:16.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.018+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3181137f0 con 0x7ff30006c930 2024-03-14T13:06:16.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.020+0000 7ff2fffff700 1 -- 172.21.15.111:0/3407936766 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7ff3181137f0 con 0x7ff30006c930 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 msgr2=0x7ff30006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 0x7ff30006ede0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff30c004560 tx=0x7ff30c009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 msgr2=0x7ff318114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff308002a50 tx=0x7ff308004600 comp rx=0 tx=0).stop 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 shutdown_connections 2024-03-14T13:06:16.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff30006c930 0x7ff30006ede0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff318102e00 0x7ff318113b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.022+0000 7ff31e123700 1 --2- 172.21.15.111:0/3407936766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3181037a0 0x7ff318114060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.023+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 >> 172.21.15.111:0/3407936766 conn(0x7ff3180f89b0 msgr2=0x7ff3180f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.023+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 shutdown_connections 2024-03-14T13:06:16.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.023+0000 7ff31e123700 1 -- 172.21.15.111:0/3407936766 wait complete. 2024-03-14T13:06:16.034 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:06:16.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/110083992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc0101d10 msgr2=0x7f6cc01020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/110083992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc0101d10 0x7f6cc01020e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0004940 tx=0x7f6cb0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/110083992 shutdown_connections 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/110083992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc01026b0 0x7f6cc00fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/110083992 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc0101d10 0x7f6cc01020e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/110083992 >> 172.21.15.111:0/110083992 conn(0x7f6cc00f7880 msgr2=0x7f6cc00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/110083992 shutdown_connections 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.134+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/110083992 wait complete. 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 Processor -- start 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 -- start start 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc0101d10 0x7f6cc010eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc010f9e0 con 0x7f6cc01026b0 2024-03-14T13:06:16.137 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cc5fbb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc010fb20 con 0x7f6cc0101d10 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40688/0 (socket says 172.21.15.111:40688) 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 -- 172.21.15.111:0/2953153484 learned_addr learned my addr 172.21.15.111:0/2953153484 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cbf7fe700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc0101d10 0x7f6cc010eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 -- 172.21.15.111:0/2953153484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc0101d10 msgr2=0x7f6cc010eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc0101d10 0x7f6cc010eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 -- 172.21.15.111:0/2953153484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0005040 con 0x7f6cc01026b0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cb7fff700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800b790 tx=0x7f6ca8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.135+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca800f6f0 con 0x7f6cc01026b0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.136+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ca800fd00 con 0x7f6cc01026b0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.136+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc0117e40 con 0x7f6cc01026b0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.136+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca80103a0 con 0x7f6cc01026b0 2024-03-14T13:06:16.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.136+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc01183d0 con 0x7f6cc01026b0 2024-03-14T13:06:16.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.136+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc00fd460 con 0x7f6cc01026b0 2024-03-14T13:06:16.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.137+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f6ca800f850 con 0x7f6cc01026b0 2024-03-14T13:06:16.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.137+0000 7f6cbcff9700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 0x7f6cac06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.137+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6ca808b890 con 0x7f6cc01026b0 2024-03-14T13:06:16.139 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.137+0000 7f6cbf7fe700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 0x7f6cac06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.138+0000 7f6cbf7fe700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 0x7f6cac06ed90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0004f90 tx=0x7f6cb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.142+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ca8055b40 con 0x7f6cc01026b0 2024-03-14T13:06:16.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.361+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cc0108de0 con 0x7f6cac06c8e0 2024-03-14T13:06:16.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.362+0000 7f6cbcff9700 1 -- 172.21.15.111:0/2953153484 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f6cb0007e00 con 0x7f6cac06c8e0 2024-03-14T13:06:16.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 msgr2=0x7f6cac06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 0x7f6cac06ed90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0004f90 tx=0x7f6cb0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 msgr2=0x7f6cc010f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800b790 tx=0x7f6ca8009220 comp rx=0 tx=0).stop 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 shutdown_connections 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6cac06c8e0 0x7f6cac06ed90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cc0101d10 0x7f6cc010eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 --2- 172.21.15.111:0/2953153484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cc01026b0 0x7f6cc010f410 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 >> 172.21.15.111:0/2953153484 conn(0x7f6cc00f7880 msgr2=0x7f6cc00f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 shutdown_connections 2024-03-14T13:06:16.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.365+0000 7f6cc5fbb700 1 -- 172.21.15.111:0/2953153484 wait complete. 2024-03-14T13:06:16.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 -- 172.21.15.111:0/3340145856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 msgr2=0x7faba0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 --2- 172.21.15.111:0/3340145856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0106e60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab94004940 tx=0x7fab94009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 -- 172.21.15.111:0/3340145856 shutdown_connections 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 --2- 172.21.15.111:0/3340145856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0106e60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 --2- 172.21.15.111:0/3340145856 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 0x7faba00fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 -- 172.21.15.111:0/3340145856 >> 172.21.15.111:0/3340145856 conn(0x7faba00f78d0 msgr2=0x7faba00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 -- 172.21.15.111:0/3340145856 shutdown_connections 2024-03-14T13:06:16.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 -- 172.21.15.111:0/3340145856 wait complete. 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.467+0000 7faba64b3700 1 Processor -- start 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7faba64b3700 1 -- start start 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7faba64b3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 0x7faba0113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7faba64b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7faba64b3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba01145f0 con 0x7faba00fe820 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7faba64b3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba010d9e0 con 0x7faba00fde80 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40704/0 (socket says 172.21.15.111:40704) 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 -- 172.21.15.111:0/509601345 learned_addr learned my addr 172.21.15.111:0/509601345 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9ffff700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 0x7faba0113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 -- 172.21.15.111:0/509601345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 msgr2=0x7faba0113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 0x7faba0113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 -- 172.21.15.111:0/509601345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab94005040 con 0x7faba00fe820 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.468+0000 7fab9f7fe700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab9400b3b0 tx=0x7fab94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.469+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab9400e040 con 0x7faba00fe820 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.469+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba010dcc0 con 0x7faba00fe820 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.469+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab94013420 con 0x7faba00fe820 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.469+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba010e200 con 0x7faba00fe820 2024-03-14T13:06:16.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.469+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab9401c630 con 0x7faba00fe820 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.470+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba0066710 con 0x7faba00fe820 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.470+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7fab94013590 con 0x7faba00fe820 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.470+0000 7fab9cff9700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 0x7fab8806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.470+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fab9408cf20 con 0x7faba00fe820 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.470+0000 7fab9ffff700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 0x7fab8806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.471+0000 7fab9ffff700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 0x7fab8806ed90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab90007fd0 tx=0x7fab90009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.475+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fab940c0960 con 0x7faba00fe820 2024-03-14T13:06:16.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.684+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faba010f0b0 con 0x7fab8806c8e0 2024-03-14T13:06:16.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.694+0000 7fab9cff9700 1 -- 172.21.15.111:0/509601345 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7faba010f0b0 con 0x7fab8806c8e0 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (2m) 46s ago 14m 16.9M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (14m) 46s ago 14m 6647k - 17.2.7-646-gb92acd1f a07fb176315d 1823d2ad4cfb 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (12m) 33s ago 12m 6651k - 17.2.7-646-gb92acd1f a07fb176315d 32753aa27df0 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (2m) 46s ago 13m 82.1M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (7m) 46s ago 7m 64.3M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (7m) 46s ago 7m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (7m) 33s ago 7m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:06:16.695 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (7m) 33s ago 7m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (5m) 46s ago 15m 467M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (3m) 33s ago 12m 448M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (62s) 46s ago 15m 28.7M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (37s) 33s ago 12m 26.4M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (3m) 46s ago 14m 18.6M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (3m) 33s ago 12m 17.9M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (11m) 46s ago 11m 210M 1568M 17.2.7-646-gb92acd1f a07fb176315d 27393ada5226 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (11m) 46s ago 11m 237M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (10m) 46s ago 10m 197M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (10m) 33s ago 10m 265M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (9m) 33s ago 9m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (9m) 33s ago 9m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:06:16.696 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (2m) 46s ago 13m 57.9M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:06:16.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 msgr2=0x7fab8806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 0x7fab8806ed90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab90007fd0 tx=0x7fab90009040 comp rx=0 tx=0).stop 2024-03-14T13:06:16.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 msgr2=0x7faba0113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab9400b3b0 tx=0x7fab94009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 shutdown_connections 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab8806c8e0 0x7fab8806ed90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faba00fde80 0x7faba0113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 --2- 172.21.15.111:0/509601345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faba00fe820 0x7faba0113fa0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 >> 172.21.15.111:0/509601345 conn(0x7faba00f78d0 msgr2=0x7faba00f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 shutdown_connections 2024-03-14T13:06:16.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.696+0000 7faba64b3700 1 -- 172.21.15.111:0/509601345 wait complete. 2024-03-14T13:06:16.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 -- 172.21.15.111:0/2509942344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fe7c0 msgr2=0x7fefec106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 --2- 172.21.15.111:0/2509942344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fe7c0 0x7fefec106e00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fefe0004940 tx=0x7fefe0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 -- 172.21.15.111:0/2509942344 shutdown_connections 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 --2- 172.21.15.111:0/2509942344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fe7c0 0x7fefec106e00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 --2- 172.21.15.111:0/2509942344 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fde20 0x7fefec0fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 -- 172.21.15.111:0/2509942344 >> 172.21.15.111:0/2509942344 conn(0x7fefec0f7870 msgr2=0x7fefec0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 -- 172.21.15.111:0/2509942344 shutdown_connections 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.802+0000 7feff16d8700 1 -- 172.21.15.111:0/2509942344 wait complete. 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 Processor -- start 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 -- start start 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fe7c0 0x7fefec113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec114510 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7feff16d8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec10db10 con 0x7fefec0fe7c0 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7fefeaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40724/0 (socket says 172.21.15.111:40724) 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.803+0000 7fefeaffd700 1 -- 172.21.15.111:0/2615556034 learned_addr learned my addr 172.21.15.111:0/2615556034 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefea7fc700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fe7c0 0x7fefec113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefeaffd700 1 -- 172.21.15.111:0/2615556034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fe7c0 msgr2=0x7fefec113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefeaffd700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fe7c0 0x7fefec113ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefeaffd700 1 -- 172.21.15.111:0/2615556034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe0005040 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefeaffd700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c830 tx=0x7fefdc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefdc00a760 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefdc00ad70 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec10ddf0 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefdc0113a0 con 0x7fefec0fde20 2024-03-14T13:06:16.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.804+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec10e330 con 0x7fefec0fde20 2024-03-14T13:06:16.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.805+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec066b50 con 0x7fefec0fde20 2024-03-14T13:06:16.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.805+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7fefdc011500 con 0x7fefec0fde20 2024-03-14T13:06:16.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.806+0000 7fefd37fe700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 0x7fefd406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:16.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.806+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fefdc08bbf0 con 0x7fefec0fde20 2024-03-14T13:06:16.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.806+0000 7fefea7fc700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 0x7fefd406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:16.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.806+0000 7fefea7fc700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 0x7fefd406eda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefec10ecd0 tx=0x7fefe0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:16.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:16.810+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fefdc059890 con 0x7fefec0fde20 2024-03-14T13:06:17.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.114+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fefec109d70 con 0x7fefec0fde20 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.115+0000 7fefd37fe700 1 -- 172.21.15.111:0/2615556034 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fefdc059420 con 0x7fefec0fde20 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:06:17.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 10, 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:06:17.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 msgr2=0x7fefd406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 0x7fefd406eda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fefec10ecd0 tx=0x7fefe0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 msgr2=0x7fefec113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00c830 tx=0x7fefdc004470 comp rx=0 tx=0).stop 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 shutdown_connections 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefd406c8f0 0x7fefd406eda0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefec0fde20 0x7fefec113980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 --2- 172.21.15.111:0/2615556034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefec0fe7c0 0x7fefec113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 >> 172.21.15.111:0/2615556034 conn(0x7fefec0f7870 msgr2=0x7fefec0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 shutdown_connections 2024-03-14T13:06:17.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.118+0000 7feff16d8700 1 -- 172.21.15.111:0/2615556034 wait complete. 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 -- 172.21.15.111:0/1960690194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fde90 msgr2=0x7f8f8c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/1960690194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fe260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c004940 tx=0x7f8f7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 -- 172.21.15.111:0/1960690194 shutdown_connections 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/1960690194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/1960690194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fe260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 -- 172.21.15.111:0/1960690194 >> 172.21.15.111:0/1960690194 conn(0x7f8f8c072bc0 msgr2=0x7f8f8c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 -- 172.21.15.111:0/1960690194 shutdown_connections 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.222+0000 7f8f91ed6700 1 -- 172.21.15.111:0/1960690194 wait complete. 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.223+0000 7f8f91ed6700 1 Processor -- start 2024-03-14T13:06:17.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.223+0000 7f8f91ed6700 1 -- start start 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c0fc820 con 0x7f8f8c0fe830 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c0fc960 con 0x7f8f8c0fde90 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40734/0 (socket says 172.21.15.111:40734) 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 -- 172.21.15.111:0/490442899 learned_addr learned my addr 172.21.15.111:0/490442899 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8b7fe700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 -- 172.21.15.111:0/490442899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 msgr2=0x7f8f8c0fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 -- 172.21.15.111:0/490442899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c005040 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8b7fe700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f8affd700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f80002a50 tx=0x7f8f800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f80007cc0 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c119e00 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f8000f040 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f8000da70 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.224+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c11a200 con 0x7f8f8c0fe830 2024-03-14T13:06:17.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.225+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8c065e30 con 0x7f8f8c0fe830 2024-03-14T13:06:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.226+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f8f8000dc10 con 0x7f8f8c0fe830 2024-03-14T13:06:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.227+0000 7f8f73fff700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 0x7f8f7406ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.227+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f8f8008b3f0 con 0x7f8f8c0fe830 2024-03-14T13:06:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.227+0000 7f8f8b7fe700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 0x7f8f7406ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.227+0000 7f8f8b7fe700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 0x7f8f7406ed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00b3b0 tx=0x7f8f7c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.231+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8f80055470 con 0x7f8f8c0fe830 2024-03-14T13:06:17.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.485+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8f8c109e50 con 0x7f8f8c0fe830 2024-03-14T13:06:17.486 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:06:17.487 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:06:17.488 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:17.489 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.485+0000 7f8f73fff700 1 -- 172.21.15.111:0/490442899 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f8f80058a90 con 0x7f8f8c0fe830 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 msgr2=0x7f8f7406ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 0x7f8f7406ed90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00b3b0 tx=0x7f8f7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 msgr2=0x7f8f8c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8f80002a50 tx=0x7f8f800046b0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 shutdown_connections 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8f7406c8e0 0x7f8f7406ed90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f8c0fde90 0x7f8f8c0fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 --2- 172.21.15.111:0/490442899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f8c0fe830 0x7f8f8c0fc290 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.487+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 >> 172.21.15.111:0/490442899 conn(0x7f8f8c072bc0 msgr2=0x7f8f8c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.488+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 shutdown_connections 2024-03-14T13:06:17.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.488+0000 7f8f91ed6700 1 -- 172.21.15.111:0/490442899 wait complete. 2024-03-14T13:06:17.491 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 -- 172.21.15.111:0/1947492121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 msgr2=0x7f22f8109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 --2- 172.21.15.111:0/1947492121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8109190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22e8004970 tx=0x7f22e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 -- 172.21.15.111:0/1947492121 shutdown_connections 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 --2- 172.21.15.111:0/1947492121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8109190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 --2- 172.21.15.111:0/1947492121 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 0x7f22f8070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 -- 172.21.15.111:0/1947492121 >> 172.21.15.111:0/1947492121 conn(0x7f22f80f89b0 msgr2=0x7f22f80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.594 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 -- 172.21.15.111:0/1947492121 shutdown_connections 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.591+0000 7f22ffedd700 1 -- 172.21.15.111:0/1947492121 wait complete. 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 Processor -- start 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 -- start start 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 0x7f22f8115a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f8116610 con 0x7f22f8070cc0 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22ffedd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f810fbf0 con 0x7f22f80703b0 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22fd478700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22fd478700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40756/0 (socket says 172.21.15.111:40756) 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22fd478700 1 -- 172.21.15.111:0/57685075 learned_addr learned my addr 172.21.15.111:0/57685075 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.592+0000 7f22fdc79700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 0x7f22f8115a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22fd478700 1 -- 172.21.15.111:0/57685075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 msgr2=0x7f22f8115a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22fd478700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 0x7f22f8115a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22fd478700 1 -- 172.21.15.111:0/57685075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e8005040 con 0x7f22f8070cc0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22fd478700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f22e8004f50 tx=0x7f22e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22e800e040 con 0x7f22f8070cc0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22e800be90 con 0x7f22f8070cc0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f810fe70 con 0x7f22f8070cc0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22e801c3f0 con 0x7f22f8070cc0 2024-03-14T13:06:17.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.593+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f81103b0 con 0x7f22f8070cc0 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.594+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f8071a40 con 0x7f22f8070cc0 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.595+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f22e8013800 con 0x7f22f8070cc0 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.596+0000 7f22ee7fc700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 0x7f22e406edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.596+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f22e808da80 con 0x7f22f8070cc0 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.596+0000 7f22fdc79700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 0x7f22e406edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.596+0000 7f22fdc79700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 0x7f22e406edf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009fd0 tx=0x7f22f4009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.600+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f22e805b570 con 0x7f22f8070cc0 2024-03-14T13:06:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:17 smithi137 ceph-mon[108145]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:17.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:17 smithi137 ceph-mon[108145]: pgmap v29: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:17.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:17 smithi137 ceph-mon[108145]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:17.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:17 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2615556034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:17.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.817+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f22f8111140 con 0x7f22e406c940 2024-03-14T13:06:17.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.818+0000 7f22ee7fc700 1 -- 172.21.15.111:0/57685075 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+385 (secure 0 0 0) 0x7f22f8111140 con 0x7f22e406c940 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "4/21 daemons upgraded", 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading crash daemons", 2024-03-14T13:06:17.819 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:06:17.820 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:06:17.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 msgr2=0x7f22e406edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 0x7f22e406edf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009fd0 tx=0x7f22f4009f60 comp rx=0 tx=0).stop 2024-03-14T13:06:17.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 msgr2=0x7f22f8115fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f22e8004f50 tx=0x7f22e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 shutdown_connections 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f22e406c940 0x7f22e406edf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22f80703b0 0x7f22f8115a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 --2- 172.21.15.111:0/57685075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22f8070cc0 0x7f22f8115fc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.820+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 >> 172.21.15.111:0/57685075 conn(0x7f22f80f89b0 msgr2=0x7f22f8105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.821+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 shutdown_connections 2024-03-14T13:06:17.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.821+0000 7f22ffedd700 1 -- 172.21.15.111:0/57685075 wait complete. 2024-03-14T13:06:17.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:17 smithi111 ceph-mon[145314]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:17.853 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:17 smithi111 ceph-mon[145314]: pgmap v29: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:17.853 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:17 smithi111 ceph-mon[145314]: from='client.34162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:17.853 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:17 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2615556034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 -- 172.21.15.111:0/2574527414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 msgr2=0x7f6ac4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/2574527414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4103240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0004940 tx=0x7f6ac0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 -- 172.21.15.111:0/2574527414 shutdown_connections 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/2574527414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 0x7f6ac40fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/2574527414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4103240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 -- 172.21.15.111:0/2574527414 >> 172.21.15.111:0/2574527414 conn(0x7f6ac40714c0 msgr2=0x7f6ac40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 -- 172.21.15.111:0/2574527414 shutdown_connections 2024-03-14T13:06:17.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.926+0000 7f6acb2aa700 1 -- 172.21.15.111:0/2574527414 wait complete. 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 Processor -- start 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 -- start start 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 0x7f6ac41139c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac4114550 con 0x7f6ac4102e70 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6acb2aa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac410d990 con 0x7f6ac40fb4a0 2024-03-14T13:06:17.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6ac8845700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6ac8845700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40768/0 (socket says 172.21.15.111:40768) 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6ac8845700 1 -- 172.21.15.111:0/4055737054 learned_addr learned my addr 172.21.15.111:0/4055737054 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.927+0000 7f6ac9046700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 0x7f6ac41139c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ac8845700 1 -- 172.21.15.111:0/4055737054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 msgr2=0x7f6ac41139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ac8845700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 0x7f6ac41139c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ac8845700 1 -- 172.21.15.111:0/4055737054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0005040 con 0x7f6ac4102e70 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ac8845700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400eb40 tx=0x7f6ab400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab4007cc0 con 0x7f6ac4102e70 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ab400f040 con 0x7f6ac4102e70 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac410dc10 con 0x7f6ac4102e70 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ab400c990 con 0x7f6ac4102e70 2024-03-14T13:06:17.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.928+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac410e1d0 con 0x7f6ac4102e70 2024-03-14T13:06:17.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.929+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac4100110 con 0x7f6ac4102e70 2024-03-14T13:06:17.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.930+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f6ab400cc00 con 0x7f6ac4102e70 2024-03-14T13:06:17.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.930+0000 7f6ab9ffb700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 0x7f6ab006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:17.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.930+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6ab408b880 con 0x7f6ac4102e70 2024-03-14T13:06:17.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.930+0000 7f6ac9046700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 0x7f6ab006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:17.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.930+0000 7f6ac9046700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 0x7f6ab006ed90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b0d0 tx=0x7f6ac0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:17.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:17.935+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ab4059440 con 0x7f6ac4102e70 2024-03-14T13:06:18.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.212+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6ac410f010 con 0x7f6ac4102e70 2024-03-14T13:06:18.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.212+0000 7f6ab9ffb700 1 -- 172.21.15.111:0/4055737054 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f6ab4058fd0 con 0x7f6ac4102e70 2024-03-14T13:06:18.213 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-03-14T13:06:18.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 msgr2=0x7f6ab006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 0x7f6ab006ed90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b0d0 tx=0x7f6ac0009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 msgr2=0x7f6ac4113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400eb40 tx=0x7f6ab400b040 comp rx=0 tx=0).stop 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 shutdown_connections 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ab006c8e0 0x7f6ab006ed90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ac40fb4a0 0x7f6ac41139c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 --2- 172.21.15.111:0/4055737054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ac4102e70 0x7f6ac4113f00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 >> 172.21.15.111:0/4055737054 conn(0x7f6ac40714c0 msgr2=0x7f6ac40fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 shutdown_connections 2024-03-14T13:06:18.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:18.214+0000 7f6acb2aa700 1 -- 172.21.15.111:0/4055737054 wait complete. 2024-03-14T13:06:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:18 smithi137 ceph-mon[108145]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:18 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/490442899' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:06:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:18.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:18 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4055737054' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:06:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:18 smithi111 ceph-mon[145314]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:18 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/490442899' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:06:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:18.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:18 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4055737054' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:06:19.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:19 smithi137 ceph-mon[108145]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:19.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:19 smithi137 ceph-mon[108145]: pgmap v30: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:19 smithi111 ceph-mon[145314]: from='client.34178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:19 smithi111 ceph-mon[145314]: pgmap v30: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: Upgrade: Updating crash.smithi137 (2/2) 2024-03-14T13:06:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:20.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: Deploying daemon crash.smithi137 on smithi137 2024-03-14T13:06:20.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:20 smithi111 ceph-mon[145314]: pgmap v31: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: Upgrade: Updating crash.smithi137 (2/2) 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: Deploying daemon crash.smithi137 on smithi137 2024-03-14T13:06:20.989 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:20 smithi137 ceph-mon[108145]: pgmap v31: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:23.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:23 smithi111 ceph-mon[145314]: pgmap v32: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:23.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:23.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:23.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:23 smithi137 ceph-mon[108145]: pgmap v32: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:24.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:24.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:25.623 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:25 smithi111 ceph-mon[145314]: pgmap v33: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:25.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:25 smithi137 ceph-mon[108145]: pgmap v33: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:26 smithi111 ceph-mon[145314]: pgmap v34: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:27.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:27.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:27.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:26 smithi137 ceph-mon[108145]: pgmap v34: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:27.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:27.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:29 smithi137 ceph-mon[108145]: pgmap v35: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:29 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:29 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:29 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:29 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.863 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:29 smithi111 ceph-mon[145314]: pgmap v35: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:06:29.863 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:29 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:29 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:29 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:29.864 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:29 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:31.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:30 smithi137 ceph-mon[108145]: pgmap v36: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:30 smithi111 ceph-mon[145314]: pgmap v36: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:06:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: pgmap v37: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]: dispatch 2024-03-14T13:06:33.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]': finished 2024-03-14T13:06:33.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]: dispatch 2024-03-14T13:06:33.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]': finished 2024-03-14T13:06:33.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: pgmap v37: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi111"}]': finished 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]: dispatch 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]': finished 2024-03-14T13:06:33.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-14T13:06:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:34 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all crash 2024-03-14T13:06:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:34 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-14T13:06:34.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:34 smithi111 ceph-mon[145314]: Upgrade: osd.0 is safe to restart 2024-03-14T13:06:34.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:34 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all crash 2024-03-14T13:06:34.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:34 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-14T13:06:34.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:34 smithi137 ceph-mon[108145]: Upgrade: osd.0 is safe to restart 2024-03-14T13:06:35.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:35 smithi111 ceph-mon[145314]: pgmap v38: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:35.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:35.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T13:06:35.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:35.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:35 smithi137 ceph-mon[108145]: pgmap v38: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:35.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:35.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-14T13:06:35.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:36 smithi111 ceph-mon[145314]: Upgrade: Updating osd.0 2024-03-14T13:06:36.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:36 smithi111 ceph-mon[145314]: Deploying daemon osd.0 on smithi111 2024-03-14T13:06:36.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:36 smithi137 ceph-mon[108145]: Upgrade: Updating osd.0 2024-03-14T13:06:36.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:36 smithi137 ceph-mon[108145]: Deploying daemon osd.0 on smithi111 2024-03-14T13:06:37.469 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:37 smithi111 ceph-mon[145314]: pgmap v39: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:37.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:37 smithi137 ceph-mon[108145]: pgmap v39: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:06:37.786 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:37 smithi111 systemd[1]: Stopping Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:06:38.140 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:38 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[63261]: 2024-03-14T13:06:38.016+0000 7f9b1b2a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:06:38.140 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:38 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[63261]: 2024-03-14T13:06:38.016+0000 7f9b1b2a7700 -1 osd.0 50 *** Got signal Terminated *** 2024-03-14T13:06:38.140 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:38 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[63261]: 2024-03-14T13:06:38.016+0000 7f9b1b2a7700 -1 osd.0 50 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:06:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:38 smithi111 ceph-mon[145314]: osd.0 marked itself down and dead 2024-03-14T13:06:38.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:38 smithi137 ceph-mon[108145]: osd.0 marked itself down and dead 2024-03-14T13:06:39.382 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:39 smithi111 bash[159238]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0 2024-03-14T13:06:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:39 smithi111 ceph-mon[145314]: pgmap v40: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:39 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:06:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:39 smithi111 ceph-mon[145314]: osdmap e51: 6 total, 5 up, 6 in 2024-03-14T13:06:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:39 smithi137 ceph-mon[108145]: pgmap v40: 65 pgs: 65 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:06:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:39 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:06:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:39 smithi137 ceph-mon[108145]: osdmap e51: 6 total, 5 up, 6 in 2024-03-14T13:06:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:40 smithi111 ceph-mon[145314]: osdmap e52: 6 total, 5 up, 6 in 2024-03-14T13:06:40.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:40 smithi137 ceph-mon[108145]: osdmap e52: 6 total, 5 up, 6 in 2024-03-14T13:06:41.390 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:40 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.0.service: Succeeded. 2024-03-14T13:06:41.390 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:40 smithi111 systemd[1]: Stopped Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:06:41.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:41 smithi111 systemd[1]: Starting Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:06:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:41 smithi137 ceph-mon[108145]: pgmap v43: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:41.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:41 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded (PG_DEGRADED) 2024-03-14T13:06:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:41 smithi111 ceph-mon[145314]: pgmap v43: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:41 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded (PG_DEGRADED) 2024-03-14T13:06:42.390 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-14T13:06:42.390 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-14T13:06:42.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-14T13:06:42.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-14T13:06:42.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-14T13:06:42.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-14T13:06:42.391 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:42 smithi111 bash[159509]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-14T13:06:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:43 smithi111 ceph-mon[145314]: pgmap v44: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:43.641 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:43 smithi111 ceph-osd[159827]: -- 172.21.15.111:0/2344960469 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cbdd2731e0 con 0x55cbdd242c00 2024-03-14T13:06:43.641 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:43 smithi111 bash[159789]: 02bbe6f32bcc221a28362046ab243fed1e182191328805799ffbe1c773a8675b 2024-03-14T13:06:43.641 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:43 smithi111 systemd[1]: Started Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:06:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:43 smithi137 ceph-mon[108145]: pgmap v44: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:44.080 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:43 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:06:43.756+0000 7f756ac80640 -1 Falling back to public interface 2024-03-14T13:06:44.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:44 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:44 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:45.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:45 smithi111 ceph-mon[145314]: pgmap v45: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/321 objects degraded (19.626%) 2024-03-14T13:06:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:45 smithi137 ceph-mon[108145]: pgmap v45: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 63/321 objects degraded (19.626%) 2024-03-14T13:06:47.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:47 smithi111 ceph-mon[145314]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:47 smithi137 ceph-mon[108145]: pgmap v46: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:48.240 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:47 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:06:47.959+0000 7f756ac80640 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 -- 172.21.15.111:0/4062628072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 msgr2=0x7f84a41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 --2- 172.21.15.111:0/4062628072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41092f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8494004940 tx=0x7f8494009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 -- 172.21.15.111:0/4062628072 shutdown_connections 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 --2- 172.21.15.111:0/4062628072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41092f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 --2- 172.21.15.111:0/4062628072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 0x7f84a40fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 -- 172.21.15.111:0/4062628072 >> 172.21.15.111:0/4062628072 conn(0x7f84a4072bc0 msgr2=0x7f84a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 -- 172.21.15.111:0/4062628072 shutdown_connections 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.326+0000 7f84ac6f8700 1 -- 172.21.15.111:0/4062628072 wait complete. 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 Processor -- start 2024-03-14T13:06:48.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 -- start start 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 0x7f84a4115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4116800 con 0x7f84a40fc590 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84ac6f8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a410fbd0 con 0x7f84a40fbc80 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84a9c93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84a9c93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59218/0 (socket says 172.21.15.111:59218) 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84a9c93700 1 -- 172.21.15.111:0/600705640 learned_addr learned my addr 172.21.15.111:0/600705640 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.327+0000 7f84aa494700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 0x7f84a4115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84a9c93700 1 -- 172.21.15.111:0/600705640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 msgr2=0x7f84a4115c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84a9c93700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 0x7f84a4115c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84a9c93700 1 -- 172.21.15.111:0/600705640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8494005040 con 0x7f84a40fc590 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84a9c93700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f849400b3b0 tx=0x7f8494009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:48.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f849400e040 con 0x7f84a40fc590 2024-03-14T13:06:48.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84ac6f8700 1 -- 172.21.15.111:0/600705640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a410fe50 con 0x7f84a40fc590 2024-03-14T13:06:48.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8494013420 con 0x7f84a40fc590 2024-03-14T13:06:48.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f84ac6f8700 1 -- 172.21.15.111:0/600705640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4110410 con 0x7f84a40fc590 2024-03-14T13:06:48.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.328+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f849401c630 con 0x7f84a40fc590 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.329+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848c0052b0 con 0x7f84a40fc590 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.330+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f849400bb50 con 0x7f84a40fc590 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.330+0000 7f849affd700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 0x7f849006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.330+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f849408ced0 con 0x7f84a40fc590 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.330+0000 7f84aa494700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 0x7f849006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.330+0000 7f84aa494700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 0x7f849006ed90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84a0004990 tx=0x7f84a0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:48.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.337+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f849405aa60 con 0x7f84a40fc590 2024-03-14T13:06:48.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.579+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f848c000bc0 con 0x7f849006c8e0 2024-03-14T13:06:48.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.580+0000 7f849affd700 1 -- 172.21.15.111:0/600705640 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f848c000bc0 con 0x7f849006c8e0 2024-03-14T13:06:48.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.582+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 msgr2=0x7f849006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.582+0000 7f8498ff9700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 0x7f849006ed90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f84a0004990 tx=0x7f84a0009ee0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 msgr2=0x7f84a41161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f849400b3b0 tx=0x7f8494009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:48.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 shutdown_connections 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f849006c8e0 0x7f849006ed90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a40fbc80 0x7f84a4115c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 --2- 172.21.15.111:0/600705640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84a40fc590 0x7f84a41161b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 >> 172.21.15.111:0/600705640 conn(0x7f84a4072bc0 msgr2=0x7f84a4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 shutdown_connections 2024-03-14T13:06:48.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.583+0000 7f8498ff9700 1 -- 172.21.15.111:0/600705640 wait complete. 2024-03-14T13:06:48.594 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:06:48.640 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:48 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:06:48.418+0000 7f756ac80640 -1 osd.0 50 log_to_monitors true 2024-03-14T13:06:48.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.691+0000 7fd4663ae700 1 -- 172.21.15.111:0/3417801765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 msgr2=0x7fd460102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.691+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3417801765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd460102140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd450004940 tx=0x7fd450009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 -- 172.21.15.111:0/3417801765 shutdown_connections 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3417801765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd4600fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3417801765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd460102140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 -- 172.21.15.111:0/3417801765 >> 172.21.15.111:0/3417801765 conn(0x7fd460072bc0 msgr2=0x7fd460072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 -- 172.21.15.111:0/3417801765 shutdown_connections 2024-03-14T13:06:48.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 -- 172.21.15.111:0/3417801765 wait complete. 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.692+0000 7fd4663ae700 1 Processor -- start 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd4663ae700 1 -- start start 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd4663ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd4663ae700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd460112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd4663ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd460113530 con 0x7fd460101d70 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd4663ae700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46010c920 con 0x7fd460102710 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd45ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd45ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59252/0 (socket says 172.21.15.111:59252) 2024-03-14T13:06:48.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd45ffff700 1 -- 172.21.15.111:0/3906609803 learned_addr learned my addr 172.21.15.111:0/3906609803 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.693+0000 7fd45f7fe700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd460112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45ffff700 1 -- 172.21.15.111:0/3906609803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 msgr2=0x7fd460112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45ffff700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd460112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45ffff700 1 -- 172.21.15.111:0/3906609803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd450005040 con 0x7fd460101d70 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45f7fe700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd460112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45ffff700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd450009360 tx=0x7fd450009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd45000e040 con 0x7fd460101d70 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd450013420 con 0x7fd460101d70 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46010cba0 con 0x7fd460101d70 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd45001c5f0 con 0x7fd460101d70 2024-03-14T13:06:48.695 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.694+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46010d160 con 0x7fd460101d70 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.695+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4600ff080 con 0x7fd460101d70 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.695+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7fd450013590 con 0x7fd460101d70 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.696+0000 7fd45cff9700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 0x7fd44806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.696+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd45008ced0 con 0x7fd460101d70 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.696+0000 7fd45f7fe700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 0x7fd44806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:48.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.696+0000 7fd45f7fe700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 0x7fd44806ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd454009fd0 tx=0x7fd454009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:48.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.701+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd45005ab10 con 0x7fd460101d70 2024-03-14T13:06:48.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.931+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd46010dff0 con 0x7fd44806c8e0 2024-03-14T13:06:48.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.932+0000 7fd45cff9700 1 -- 172.21.15.111:0/3906609803 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd46010dff0 con 0x7fd44806c8e0 2024-03-14T13:06:48.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.934+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 msgr2=0x7fd44806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.934+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 0x7fd44806ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd454009fd0 tx=0x7fd454009ee0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 msgr2=0x7fd4601129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd450009360 tx=0x7fd450009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 shutdown_connections 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd44806c8e0 0x7fd44806ed90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd460101d70 0x7fd4601129a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 --2- 172.21.15.111:0/3906609803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd460102710 0x7fd460112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 >> 172.21.15.111:0/3906609803 conn(0x7fd460072bc0 msgr2=0x7fd4600fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 shutdown_connections 2024-03-14T13:06:48.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:48.935+0000 7fd4663ae700 1 -- 172.21.15.111:0/3906609803 wait complete. 2024-03-14T13:06:49.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 -- 172.21.15.111:0/1747354938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 msgr2=0x7f7150102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 --2- 172.21.15.111:0/1747354938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f7150102140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7140004940 tx=0x7f7140009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 -- 172.21.15.111:0/1747354938 shutdown_connections 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 --2- 172.21.15.111:0/1747354938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f71500fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 --2- 172.21.15.111:0/1747354938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f7150102140 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.039+0000 7f7156945700 1 -- 172.21.15.111:0/1747354938 >> 172.21.15.111:0/1747354938 conn(0x7f7150072bc0 msgr2=0x7f7150072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 -- 172.21.15.111:0/1747354938 shutdown_connections 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 -- 172.21.15.111:0/1747354938 wait complete. 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 Processor -- start 2024-03-14T13:06:49.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 -- start start 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.040+0000 7f7156945700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f715010f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f7156945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715010f6c0 con 0x7f7150101d70 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f7156945700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715010f800 con 0x7f7150102710 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59266/0 (socket says 172.21.15.111:59266) 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 -- 172.21.15.111:0/3017991195 learned_addr learned my addr 172.21.15.111:0/3017991195 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:49.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714f7fe700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f715010f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 -- 172.21.15.111:0/3017991195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 msgr2=0x7f715010f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f715010f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 -- 172.21.15.111:0/3017991195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140005040 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714f7fe700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f715010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714ffff700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7140009180 tx=0x7f7140009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714000e040 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7140004290 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.041+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7150117ac0 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.042+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f714001c3f0 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.042+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150118050 con 0x7f7150101d70 2024-03-14T13:06:49.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.043+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71500ff080 con 0x7f7150101d70 2024-03-14T13:06:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.043+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f714001c660 con 0x7f7150101d70 2024-03-14T13:06:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.044+0000 7f714cff9700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 0x7f713806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.044+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f714008cf50 con 0x7f7150101d70 2024-03-14T13:06:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.044+0000 7f714f7fe700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 0x7f713806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.044+0000 7f714f7fe700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 0x7f713806ed90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7150066060 tx=0x7f7144008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.049+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f71400c0960 con 0x7f7150101d70 2024-03-14T13:06:49.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.278+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f715004bed0 con 0x7f713806c8e0 2024-03-14T13:06:49.289 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.289+0000 7f714cff9700 1 -- 172.21.15.111:0/3017991195 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f715004bed0 con 0x7f713806c8e0 2024-03-14T13:06:49.289 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:06:49.289 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (3m) 0s ago 14m 27.7M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:06:49.289 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (31s) 0s ago 14m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (26s) 23s ago 13m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (2m) 0s ago 13m 83.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (8m) 0s ago 8m 64.3M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (8m) 0s ago 8m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (8m) 23s ago 8m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (8m) 23s ago 8m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (5m) 0s ago 16m 494M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (4m) 23s ago 13m 449M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (95s) 0s ago 16m 44.3M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (69s) 23s ago 13m 37.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (3m) 0s ago 14m 18.7M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (3m) 23s ago 13m 18.1M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:06:49.290 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6s) 0s ago 12m 11.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (11m) 0s ago 11m 237M 1568M 17.2.7-646-gb92acd1f a07fb176315d e5e61b592118 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (11m) 0s ago 11m 197M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (10m) 23s ago 10m 267M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (10m) 23s ago 10m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (10m) 23s ago 10m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (3m) 0s ago 13m 62.2M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:06:49.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 msgr2=0x7f713806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 0x7f713806ed90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7150066060 tx=0x7f7144008040 comp rx=0 tx=0).stop 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 msgr2=0x7f715010ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7140009180 tx=0x7f7140009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 shutdown_connections 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f713806c8e0 0x7f713806ed90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7150101d70 0x7f715010ebb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 --2- 172.21.15.111:0/3017991195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7150102710 0x7f715010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.291+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 >> 172.21.15.111:0/3017991195 conn(0x7f7150072bc0 msgr2=0x7f71500faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.292+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 shutdown_connections 2024-03-14T13:06:49.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.292+0000 7f7156945700 1 -- 172.21.15.111:0/3017991195 wait complete. 2024-03-14T13:06:49.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.404+0000 7f93a3673700 1 -- 172.21.15.111:0/57474662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 msgr2=0x7f939c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.404+0000 7f93a3673700 1 --2- 172.21.15.111:0/57474662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c0fe260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9398004940 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- 172.21.15.111:0/57474662 shutdown_connections 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 --2- 172.21.15.111:0/57474662 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 --2- 172.21.15.111:0/57474662 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c0fe260 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- 172.21.15.111:0/57474662 >> 172.21.15.111:0/57474662 conn(0x7f939c072bc0 msgr2=0x7f939c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- 172.21.15.111:0/57474662 shutdown_connections 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- 172.21.15.111:0/57474662 wait complete. 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 Processor -- start 2024-03-14T13:06:49.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- start start 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c114680 con 0x7f939c0fde90 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.405+0000 7f93a3673700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c10da70 con 0x7f939c0fe830 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59286/0 (socket says 172.21.15.111:59286) 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 -- 172.21.15.111:0/2072168610 learned_addr learned my addr 172.21.15.111:0/2072168610 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a0c0e700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 -- 172.21.15.111:0/2072168610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 msgr2=0x7f939c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 -- 172.21.15.111:0/2072168610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398005040 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a0c0e700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a140f700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9398009360 tx=0x7f9398009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939800e040 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9398013420 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c10dcf0 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939801c630 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.406+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c10e230 con 0x7f939c0fde90 2024-03-14T13:06:49.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.408+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c1019b0 con 0x7f939c0fde90 2024-03-14T13:06:49.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.408+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f939801c8a0 con 0x7f939c0fde90 2024-03-14T13:06:49.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.409+0000 7f9391ffb700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 0x7f938806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.409+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f939808cdc0 con 0x7f939c0fde90 2024-03-14T13:06:49.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.409+0000 7f93a0c0e700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 0x7f938806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.409+0000 7f93a0c0e700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 0x7f938806ed90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f938c009fd0 tx=0x7f938c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.413+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f939805a9f0 con 0x7f939c0fde90 2024-03-14T13:06:49.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.719+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f939c10f140 con 0x7f939c0fde90 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.719+0000 7f9391ffb700 1 -- 172.21.15.111:0/2072168610 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+738 (secure 0 0 0) 0x7f939805a580 con 0x7f939c0fde90 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:49.719 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 5 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 9, 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:06:49.720 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:06:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.721+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 msgr2=0x7f938806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.721+0000 7f93a3673700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 0x7f938806ed90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f938c009fd0 tx=0x7f938c00b040 comp rx=0 tx=0).stop 2024-03-14T13:06:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.721+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 msgr2=0x7f939c113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.721+0000 7f93a3673700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9398009360 tx=0x7f9398009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:49.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 shutdown_connections 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f938806c8e0 0x7f938806ed90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f939c0fde90 0x7f939c113af0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 --2- 172.21.15.111:0/2072168610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c0fe830 0x7f939c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 >> 172.21.15.111:0/2072168610 conn(0x7f939c072bc0 msgr2=0x7f939c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 shutdown_connections 2024-03-14T13:06:49.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.722+0000 7f93a3673700 1 -- 172.21.15.111:0/2072168610 wait complete. 2024-03-14T13:06:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:49 smithi137 ceph-mon[108145]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:49 smithi137 ceph-mon[108145]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T13:06:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:49.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3826455840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fe7d0 msgr2=0x7fe2f0106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3826455840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f0106e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2e4004940 tx=0x7fe2e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:49.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3826455840 shutdown_connections 2024-03-14T13:06:49.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3826455840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f0106e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3826455840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fde30 0x7fe2f00fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3826455840 >> 172.21.15.111:0/3826455840 conn(0x7fe2f00f78c0 msgr2=0x7fe2f00f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3826455840 shutdown_connections 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.839+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3826455840 wait complete. 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 Processor -- start 2024-03-14T13:06:49.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 -- start start 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f010bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f010c440 con 0x7fe2f00fde30 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2f59bf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f010c5b0 con 0x7fe2f00fe7d0 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59300/0 (socket says 172.21.15.111:59300) 2024-03-14T13:06:49.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 -- 172.21.15.111:0/3347989838 learned_addr learned my addr 172.21.15.111:0/3347989838 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:49.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2ee7fc700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f010bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 -- 172.21.15.111:0/3347989838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fe7d0 msgr2=0x7fe2f010bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:49.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f010bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:49.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 -- 172.21.15.111:0/3347989838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e4005040 con 0x7fe2f00fde30 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.840+0000 7fe2eeffd700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000eb40 tx=0x7fe2e000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.841+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e0007cc0 con 0x7fe2f00fde30 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.841+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f0126660 con 0x7fe2f00fde30 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.841+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2e000f040 con 0x7fe2f00fde30 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.841+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2e000c990 con 0x7fe2f00fde30 2024-03-14T13:06:49.847 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.841+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f0126b70 con 0x7fe2f00fde30 2024-03-14T13:06:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.842+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7fe2e000cc00 con 0x7fe2f00fde30 2024-03-14T13:06:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.842+0000 7fe2d77fe700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 0x7fe2d806eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.842+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fe2e008b8c0 con 0x7fe2f00fde30 2024-03-14T13:06:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.842+0000 7fe2ee7fc700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 0x7fe2d806eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:49.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.843+0000 7fe2ee7fc700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 0x7fe2d806eda0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe2f010cf30 tx=0x7fe2e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:49.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.846+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f0066b90 con 0x7fe2f00fde30 2024-03-14T13:06:49.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:49.851+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe2e00559f0 con 0x7fe2f00fde30 2024-03-14T13:06:49.890 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:06:49 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:06:49.442+0000 7f75580d1700 -1 osd.0 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:06:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:49 smithi111 ceph-mon[145314]: pgmap v47: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:49.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:49 smithi111 ceph-mon[145314]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-14T13:06:49.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:49.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:50.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.094+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe2f010cce0 con 0x7fe2f00fde30 2024-03-14T13:06:50.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.095+0000 7fe2d77fe700 1 -- 172.21.15.111:0/3347989838 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fe2e0059010 con 0x7fe2f00fde30 2024-03-14T13:06:50.095 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:06:50.095 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:06:50.095 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:06:50.096 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:06:50.097 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:50.098 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 msgr2=0x7fe2d806eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 0x7fe2d806eda0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe2f010cf30 tx=0x7fe2e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 msgr2=0x7fe2f010b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000eb40 tx=0x7fe2e000b040 comp rx=0 tx=0).stop 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 shutdown_connections 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe2d806c8f0 0x7fe2d806eda0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe2f00fde30 0x7fe2f010b9c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 --2- 172.21.15.111:0/3347989838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f00fe7d0 0x7fe2f010bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 >> 172.21.15.111:0/3347989838 conn(0x7fe2f00f78c0 msgr2=0x7fe2f00fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 shutdown_connections 2024-03-14T13:06:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.097+0000 7fe2f59bf700 1 -- 172.21.15.111:0/3347989838 wait complete. 2024-03-14T13:06:50.100 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 -- 172.21.15.111:0/2845437478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fe7d0 msgr2=0x7f99bc106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 --2- 172.21.15.111:0/2845437478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc106e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f99ac004940 tx=0x7f99ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 -- 172.21.15.111:0/2845437478 shutdown_connections 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 --2- 172.21.15.111:0/2845437478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc106e10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 --2- 172.21.15.111:0/2845437478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fde30 0x7f99bc0fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 -- 172.21.15.111:0/2845437478 >> 172.21.15.111:0/2845437478 conn(0x7f99bc0f78c0 msgr2=0x7f99bc0f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 -- 172.21.15.111:0/2845437478 shutdown_connections 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.201+0000 7f99c0fef700 1 -- 172.21.15.111:0/2845437478 wait complete. 2024-03-14T13:06:50.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 Processor -- start 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 -- start start 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc114440 con 0x7f99bc0fde30 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99c0fef700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99bc10da40 con 0x7f99bc0fe7d0 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59320/0 (socket says 172.21.15.111:59320) 2024-03-14T13:06:50.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 -- 172.21.15.111:0/1656865810 learned_addr learned my addr 172.21.15.111:0/1656865810 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99b9d9b700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 -- 172.21.15.111:0/1656865810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fe7d0 msgr2=0x7f99bc113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 -- 172.21.15.111:0/1656865810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac005040 con 0x7f99bc0fde30 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.202+0000 7f99ba59c700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c8c0 tx=0x7f99b00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.203+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b000a760 con 0x7f99bc0fde30 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.203+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b000ad70 con 0x7f99bc0fde30 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.203+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99bc10dd20 con 0x7f99bc0fde30 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.203+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99b00113a0 con 0x7f99bc0fde30 2024-03-14T13:06:50.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.203+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99bc10e230 con 0x7f99bc0fde30 2024-03-14T13:06:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.204+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99bc066b90 con 0x7f99bc0fde30 2024-03-14T13:06:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.204+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f99b0012400 con 0x7f99bc0fde30 2024-03-14T13:06:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.204+0000 7f99aaffd700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 0x7f99a406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.205+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f99b008bba0 con 0x7f99bc0fde30 2024-03-14T13:06:50.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.205+0000 7f99b9d9b700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 0x7f99a406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.205+0000 7f99b9d9b700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 0x7f99a406eda0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99bc10ed20 tx=0x7f99ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:50.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.209+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f99b0059840 con 0x7f99bc0fde30 2024-03-14T13:06:50.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.425+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f99bc109d80 con 0x7f99a406c8f0 2024-03-14T13:06:50.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.426+0000 7f99aaffd700 1 -- 172.21.15.111:0/1656865810 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f99bc109d80 con 0x7f99a406c8f0 2024-03-14T13:06:50.425 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "7/21 daemons upgraded", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:06:50.426 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:06:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.435+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 msgr2=0x7f99a406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.435+0000 7f99c0fef700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 0x7f99a406eda0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f99bc10ed20 tx=0x7f99ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.435+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 msgr2=0x7f99bc1138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.435+0000 7f99c0fef700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f99b000c8c0 tx=0x7f99b00044c0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.436+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 shutdown_connections 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.436+0000 7f99c0fef700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99a406c8f0 0x7f99a406eda0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.436+0000 7f99c0fef700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99bc0fde30 0x7f99bc1138b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.436+0000 7f99c0fef700 1 --2- 172.21.15.111:0/1656865810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99bc0fe7d0 0x7f99bc113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.436+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 >> 172.21.15.111:0/1656865810 conn(0x7f99bc0f78c0 msgr2=0x7f99bc0f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:50.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.437+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 shutdown_connections 2024-03-14T13:06:50.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.437+0000 7f99c0fef700 1 -- 172.21.15.111:0/1656865810 wait complete. 2024-03-14T13:06:50.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.549+0000 7f143418a700 1 -- 172.21.15.111:0/3527122899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 msgr2=0x7f142c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.549+0000 7f143418a700 1 --2- 172.21.15.111:0/3527122899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c0fe260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1428004940 tx=0x7f1428009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:50.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 -- 172.21.15.111:0/3527122899 shutdown_connections 2024-03-14T13:06:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 --2- 172.21.15.111:0/3527122899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 0x7f142c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 --2- 172.21.15.111:0/3527122899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c0fe260 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 -- 172.21.15.111:0/3527122899 >> 172.21.15.111:0/3527122899 conn(0x7f142c072bc0 msgr2=0x7f142c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 -- 172.21.15.111:0/3527122899 shutdown_connections 2024-03-14T13:06:50.550 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 -- 172.21.15.111:0/3527122899 wait complete. 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 Processor -- start 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.550+0000 7f143418a700 1 -- start start 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f143418a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f143418a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 0x7f142c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f143418a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c114680 con 0x7f142c0fde90 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f143418a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c10da70 con 0x7f142c0fe830 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34838/0 (socket says 172.21.15.111:34838) 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 -- 172.21.15.111:0/448214403 learned_addr learned my addr 172.21.15.111:0/448214403 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:06:50.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431725700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 0x7f142c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 -- 172.21.15.111:0/448214403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 msgr2=0x7f142c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 0x7f142c114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 -- 172.21.15.111:0/448214403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428005040 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f1431f26700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f14280093b0 tx=0x7f1428009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f142800e040 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f142800bed0 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f142801c3f0 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.551+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c10dcf0 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.552+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c10e230 con 0x7f142c0fde90 2024-03-14T13:06:50.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.553+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c1019b0 con 0x7f142c0fde90 2024-03-14T13:06:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.553+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 38) v1 ==== 90628+0+0 (secure 0 0 0) 0x7f1428013790 con 0x7f142c0fde90 2024-03-14T13:06:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.553+0000 7f14227fc700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 0x7f141806ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:06:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.553+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f142808cf80 con 0x7f142c0fde90 2024-03-14T13:06:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.553+0000 7f1431725700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 0x7f141806ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:06:50.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.554+0000 7f1431725700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 0x7f141806ed90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f142c10ec30 tx=0x7f141c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:06:50.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.558+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f142805ab10 con 0x7f142c0fde90 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: osdmap e53: 6 total, 5 up, 6 in 2024-03-14T13:06:50.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:06:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2072168610' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:50.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:50 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3347989838' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='client.34194 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='client.34198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: osdmap e53: 6 total, 5 up, 6 in 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2072168610' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:50.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:50 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3347989838' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:06:50.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.836+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f142c10f140 con 0x7f142c0fde90 2024-03-14T13:06:50.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.836+0000 7f14227fc700 1 -- 172.21.15.111:0/448214403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1239 (secure 0 0 0) 0x7f1428013420 con 0x7f142c0fde90 2024-03-14T13:06:50.836 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded 2024-03-14T13:06:50.836 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded 2024-03-14T13:06:50.836 INFO:teuthology.orchestra.run.smithi111.stdout: pg 1.0 is active+undersized+degraded, acting [3,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.0 is active+undersized+degraded, acting [3,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1 is active+undersized+degraded, acting [2,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.4 is active+undersized+degraded, acting [1,4] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.5 is active+undersized+degraded, acting [3,4] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.8 is active+undersized+degraded, acting [3,5] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.9 is active+undersized+degraded, acting [1,4] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.f is active+undersized+degraded, acting [4,5] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.10 is active+undersized+degraded, acting [2,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.12 is active+undersized+degraded, acting [3,1] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.13 is active+undersized+degraded, acting [4,2] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.15 is active+undersized+degraded, acting [1,3] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.19 is active+undersized+degraded, acting [4,2] 2024-03-14T13:06:50.837 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1b is active+undersized+degraded, acting [1,5] 2024-03-14T13:06:50.838 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1d is active+undersized+degraded, acting [3,5] 2024-03-14T13:06:50.838 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1e is active+undersized+degraded, acting [2,5] 2024-03-14T13:06:50.838 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1f is active+undersized+degraded, acting [3,4] 2024-03-14T13:06:50.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.838+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 msgr2=0x7f141806ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 0x7f141806ed90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f142c10ec30 tx=0x7f141c00b040 comp rx=0 tx=0).stop 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 msgr2=0x7f142c113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f14280093b0 tx=0x7f1428009e30 comp rx=0 tx=0).stop 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 shutdown_connections 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f141806c8e0 0x7f141806ed90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f142c0fde90 0x7f142c113af0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 --2- 172.21.15.111:0/448214403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0fe830 0x7f142c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 >> 172.21.15.111:0/448214403 conn(0x7f142c072bc0 msgr2=0x7f142c0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 shutdown_connections 2024-03-14T13:06:50.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:06:50.839+0000 7f143418a700 1 -- 172.21.15.111:0/448214403 wait complete. 2024-03-14T13:06:51.545 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659] boot 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: osdmap e54: 6 total, 6 up, 6 in 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/448214403' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:06:51.546 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:51 smithi111 ceph-mon[145314]: osdmap e55: 6 total, 6 up, 6 in 2024-03-14T13:06:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: pgmap v49: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: from='client.34210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: osd.0 [v2:172.21.15.111:6802/2253644659,v1:172.21.15.111:6803/2253644659] boot 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: osdmap e54: 6 total, 6 up, 6 in 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/448214403' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:06:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:51 smithi137 ceph-mon[108145]: osdmap e55: 6 total, 6 up, 6 in 2024-03-14T13:06:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:06:53.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: pgmap v52: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:06:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:06:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:54.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:06:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:06:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:06:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:06:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-03-14T13:06:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:06:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:06:55.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:55 smithi137 ceph-mon[108145]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:55 smithi111 ceph-mon[145314]: pgmap v53: 65 pgs: 19 active+undersized+degraded, 15 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 63/321 objects degraded (19.626%) 2024-03-14T13:06:56.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:56 smithi137 ceph-mon[108145]: mgrmap e39: smithi111.atysez(active, since 91s), standbys: smithi137.xbyzbp 2024-03-14T13:06:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:56 smithi111 ceph-mon[145314]: mgrmap e39: smithi111.atysez(active, since 91s), standbys: smithi137.xbyzbp 2024-03-14T13:06:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:57 smithi137 ceph-mon[108145]: pgmap v54: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.1 KiB/s rd, 903 B/s wr, 6 op/s 2024-03-14T13:06:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:57 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded) 2024-03-14T13:06:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:57 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:06:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:57 smithi111 ceph-mon[145314]: pgmap v54: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.1 KiB/s rd, 903 B/s wr, 6 op/s 2024-03-14T13:06:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:57 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 63/321 objects degraded (19.626%), 19 pgs degraded) 2024-03-14T13:06:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:57 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:06:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:06:59 smithi137 ceph-mon[108145]: pgmap v55: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-14T13:06:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:06:59 smithi111 ceph-mon[145314]: pgmap v55: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-14T13:07:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:01 smithi137 ceph-mon[108145]: pgmap v56: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 627 B/s wr, 4 op/s 2024-03-14T13:07:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:01 smithi111 ceph-mon[145314]: pgmap v56: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 627 B/s wr, 4 op/s 2024-03-14T13:07:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:03 smithi137 ceph-mon[108145]: pgmap v57: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 568 B/s wr, 4 op/s 2024-03-14T13:07:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:03 smithi111 ceph-mon[145314]: pgmap v57: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 568 B/s wr, 4 op/s 2024-03-14T13:07:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:05 smithi111 ceph-mon[145314]: pgmap v58: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-14T13:07:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:05 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:05 smithi137 ceph-mon[108145]: pgmap v58: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-14T13:07:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:05 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:07.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:07 smithi111 ceph-mon[145314]: pgmap v59: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-14T13:07:07.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:07 smithi137 ceph-mon[108145]: pgmap v59: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.6 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-14T13:07:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:09 smithi111 ceph-mon[145314]: pgmap v60: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:07:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:09 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:07:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:09.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:09 smithi137 ceph-mon[108145]: pgmap v60: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:09.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:07:09.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:09 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-14T13:07:09.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:09.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:10.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:10 smithi111 ceph-mon[145314]: Upgrade: osd.1 is safe to restart 2024-03-14T13:07:10.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:10.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T13:07:10.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:10 smithi137 ceph-mon[108145]: Upgrade: osd.1 is safe to restart 2024-03-14T13:07:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-14T13:07:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:11.588 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:11 smithi111 ceph-mon[145314]: Upgrade: Updating osd.1 2024-03-14T13:07:11.589 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:11 smithi111 ceph-mon[145314]: Deploying daemon osd.1 on smithi111 2024-03-14T13:07:11.589 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:11 smithi111 ceph-mon[145314]: pgmap v61: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:11 smithi137 ceph-mon[108145]: Upgrade: Updating osd.1 2024-03-14T13:07:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:11 smithi137 ceph-mon[108145]: Deploying daemon osd.1 on smithi111 2024-03-14T13:07:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:11 smithi137 ceph-mon[108145]: pgmap v61: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:13.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:13 smithi111 ceph-mon[145314]: pgmap v62: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:13.641 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:13 smithi111 systemd[1]: Stopping Ceph osd.1 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:07:13.641 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:13 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[72179]: 2024-03-14T13:07:13.418+0000 7f67e2dac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:07:13.641 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:13 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[72179]: 2024-03-14T13:07:13.418+0000 7f67e2dac700 -1 osd.1 55 *** Got signal Terminated *** 2024-03-14T13:07:13.641 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:13 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[72179]: 2024-03-14T13:07:13.418+0000 7f67e2dac700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:07:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:13 smithi137 ceph-mon[108145]: pgmap v62: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:14.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:14 smithi111 ceph-mon[145314]: osd.1 marked itself down and dead 2024-03-14T13:07:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:14 smithi137 ceph-mon[108145]: osd.1 marked itself down and dead 2024-03-14T13:07:15.384 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:15 smithi111 bash[165907]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1 2024-03-14T13:07:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:15 smithi111 ceph-mon[145314]: pgmap v63: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:15 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:07:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:15 smithi111 ceph-mon[145314]: osdmap e56: 6 total, 5 up, 6 in 2024-03-14T13:07:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:15 smithi137 ceph-mon[108145]: pgmap v63: 65 pgs: 65 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:15 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:07:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:15 smithi137 ceph-mon[108145]: osdmap e56: 6 total, 5 up, 6 in 2024-03-14T13:07:16.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:16 smithi137 ceph-mon[108145]: osdmap e57: 6 total, 5 up, 6 in 2024-03-14T13:07:16.825 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:16 smithi111 ceph-mon[145314]: osdmap e57: 6 total, 5 up, 6 in 2024-03-14T13:07:17.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:17 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.1.service: Succeeded. 2024-03-14T13:07:17.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:17 smithi111 systemd[1]: Stopped Ceph osd.1 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:07:17.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:17 smithi111 systemd[1]: Starting Ceph osd.1 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:07:17.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:17 smithi111 ceph-mon[145314]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-14T13:07:17.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:17 smithi111 ceph-mon[145314]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-03-14T13:07:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:17 smithi137 ceph-mon[108145]: pgmap v66: 65 pgs: 34 peering, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s 2024-03-14T13:07:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:17 smithi137 ceph-mon[108145]: Health check failed: Reduced data availability: 12 pgs peering (PG_AVAILABILITY) 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-14T13:07:18.582 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:18 smithi111 bash[166180]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-14T13:07:19.669 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:19 smithi111 ceph-mon[145314]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:19.670 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:19 smithi111 ceph-osd[166504]: -- 172.21.15.111:0/3115424647 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a40eb91e0 con 0x560a40e88c00 2024-03-14T13:07:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:19 smithi137 ceph-mon[108145]: pgmap v67: 65 pgs: 34 peering, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:20.115 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:19 smithi111 bash[166466]: 09dcfc43c596066ad812456dd3567bc428395febbb25132717c725768b57aaaf 2024-03-14T13:07:20.115 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:19 smithi111 systemd[1]: Started Ceph osd.1 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:07:20.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:20 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:07:20.114+0000 7f4fad535640 -1 Falling back to public interface 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- 172.21.15.111:0/3585525519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c101d20 msgr2=0x7f469c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 --2- 172.21.15.111:0/3585525519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c101d20 0x7f469c1020f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f468c004940 tx=0x7f468c009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- 172.21.15.111:0/3585525519 shutdown_connections 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 --2- 172.21.15.111:0/3585525519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c1026c0 0x7f469c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 --2- 172.21.15.111:0/3585525519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c101d20 0x7f469c1020f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- 172.21.15.111:0/3585525519 >> 172.21.15.111:0/3585525519 conn(0x7f469c0f7850 msgr2=0x7f469c0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- 172.21.15.111:0/3585525519 shutdown_connections 2024-03-14T13:07:20.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- 172.21.15.111:0/3585525519 wait complete. 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 Processor -- start 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.955+0000 7f46a18fe700 1 -- start start 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f46a18fe700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c101d20 0x7f469c113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f46a18fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f46a18fe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c1143e0 con 0x7f469c1026c0 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f46a18fe700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c10d9e0 con 0x7f469c101d20 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51802/0 (socket says 172.21.15.111:51802) 2024-03-14T13:07:20.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 -- 172.21.15.111:0/1622837857 learned_addr learned my addr 172.21.15.111:0/1622837857 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469affd700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c101d20 0x7f469c113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 -- 172.21.15.111:0/1622837857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c101d20 msgr2=0x7f469c113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c101d20 0x7f469c113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 -- 172.21.15.111:0/1622837857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c005040 con 0x7f469c1026c0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.956+0000 7f469a7fc700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f469000c890 tx=0x7f46900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.957+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f469000a700 con 0x7f469c1026c0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.957+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f469000ad10 con 0x7f469c1026c0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.957+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4690012e70 con 0x7f469c1026c0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.957+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c10dcc0 con 0x7f469c1026c0 2024-03-14T13:07:20.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.957+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c10e230 con 0x7f469c1026c0 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.958+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c0fd470 con 0x7f469c1026c0 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.958+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f469000a860 con 0x7f469c1026c0 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.959+0000 7f46837fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 0x7f468406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.959+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f469008b9f0 con 0x7f469c1026c0 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.959+0000 7f469affd700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 0x7f468406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:20.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.959+0000 7f469affd700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 0x7f468406ec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f468c004f90 tx=0x7f468c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:20.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:20.963+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f469008de60 con 0x7f469c1026c0 2024-03-14T13:07:21.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:20 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:20 smithi137 ceph-mon[108145]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:21.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:21.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:21.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:20 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:21.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:20 smithi111 ceph-mon[145314]: pgmap v68: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:21.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.183+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f469c108ce0 con 0x7f468406c7c0 2024-03-14T13:07:21.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.184+0000 7f46837fe700 1 -- 172.21.15.111:0/1622837857 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f469c108ce0 con 0x7f468406c7c0 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 msgr2=0x7f468406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 0x7f468406ec70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f468c004f90 tx=0x7f468c009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 msgr2=0x7f469c113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f469000c890 tx=0x7f46900044c0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 shutdown_connections 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f468406c7c0 0x7f468406ec70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f469c101d20 0x7f469c113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 --2- 172.21.15.111:0/1622837857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f469c1026c0 0x7f469c113d90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 >> 172.21.15.111:0/1622837857 conn(0x7f469c0f7850 msgr2=0x7f469c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 shutdown_connections 2024-03-14T13:07:21.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.186+0000 7f46a18fe700 1 -- 172.21.15.111:0/1622837857 wait complete. 2024-03-14T13:07:21.197 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 -- 172.21.15.111:0/1304087670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 msgr2=0x7fca200fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 --2- 172.21.15.111:0/1304087670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca200fe260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca10004940 tx=0x7fca10009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 -- 172.21.15.111:0/1304087670 shutdown_connections 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 --2- 172.21.15.111:0/1304087670 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 0x7fca20106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 --2- 172.21.15.111:0/1304087670 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca200fe260 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 -- 172.21.15.111:0/1304087670 >> 172.21.15.111:0/1304087670 conn(0x7fca20072bc0 msgr2=0x7fca20072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 -- 172.21.15.111:0/1304087670 shutdown_connections 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.307+0000 7fca281cd700 1 -- 172.21.15.111:0/1304087670 wait complete. 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 Processor -- start 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 -- start start 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 0x7fca20114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca20114710 con 0x7fca200fde90 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2010da70 con 0x7fca200fe830 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51820/0 (socket says 172.21.15.111:51820) 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 -- 172.21.15.111:0/3231677170 learned_addr learned my addr 172.21.15.111:0/3231677170 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25768700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 0x7fca20114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 -- 172.21.15.111:0/3231677170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 msgr2=0x7fca20114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 0x7fca20114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 -- 172.21.15.111:0/3231677170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10005040 con 0x7fca200fde90 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca25f69700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fca10009360 tx=0x7fca10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1000e040 con 0x7fca200fde90 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca10013420 con 0x7fca200fde90 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2010dcf0 con 0x7fca200fde90 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.308+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1001c630 con 0x7fca200fde90 2024-03-14T13:07:21.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.309+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2010e230 con 0x7fca200fde90 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.309+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca20065e30 con 0x7fca200fde90 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.310+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fca100139e0 con 0x7fca200fde90 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.310+0000 7fca167fc700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 0x7fca0c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.310+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fca1008cc50 con 0x7fca200fde90 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.310+0000 7fca25768700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 0x7fca0c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.311+0000 7fca25768700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 0x7fca0c06ec70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca1c006fd0 tx=0x7fca1c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.316+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fca1005aa00 con 0x7fca200fde90 2024-03-14T13:07:21.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.531+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fca20109eb0 con 0x7fca0c06c7c0 2024-03-14T13:07:21.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.532+0000 7fca167fc700 1 -- 172.21.15.111:0/3231677170 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fca20109eb0 con 0x7fca0c06c7c0 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 msgr2=0x7fca0c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 0x7fca0c06ec70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fca1c006fd0 tx=0x7fca1c008040 comp rx=0 tx=0).stop 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 msgr2=0x7fca20113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fca10009360 tx=0x7fca10009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 shutdown_connections 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca0c06c7c0 0x7fca0c06ec70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca200fde90 0x7fca20113af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 --2- 172.21.15.111:0/3231677170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca200fe830 0x7fca20114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 >> 172.21.15.111:0/3231677170 conn(0x7fca20072bc0 msgr2=0x7fca201037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 shutdown_connections 2024-03-14T13:07:21.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.534+0000 7fca281cd700 1 -- 172.21.15.111:0/3231677170 wait complete. 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 -- 172.21.15.111:0/1437268586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 msgr2=0x7f6378103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 --2- 172.21.15.111:0/1437268586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378103240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6368004940 tx=0x7f6368009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 -- 172.21.15.111:0/1437268586 shutdown_connections 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 --2- 172.21.15.111:0/1437268586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 0x7f63780fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 --2- 172.21.15.111:0/1437268586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378103240 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 -- 172.21.15.111:0/1437268586 >> 172.21.15.111:0/1437268586 conn(0x7f63780714c0 msgr2=0x7f63780718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 -- 172.21.15.111:0/1437268586 shutdown_connections 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 -- 172.21.15.111:0/1437268586 wait complete. 2024-03-14T13:07:21.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.635+0000 7f637e58a700 1 Processor -- start 2024-03-14T13:07:21.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f637e58a700 1 -- start start 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f637e58a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 0x7f6378113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f637e58a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f637e58a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6378114600 con 0x7f6378102e70 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f637e58a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637810d9f0 con 0x7f63780fb4a0 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51828/0 (socket says 172.21.15.111:51828) 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 -- 172.21.15.111:0/3733430092 learned_addr learned my addr 172.21.15.111:0/3733430092 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:21.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f6377fff700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 0x7f6378113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 -- 172.21.15.111:0/3733430092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 msgr2=0x7f6378113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 0x7f6378113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 -- 172.21.15.111:0/3733430092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6368005040 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.636+0000 7f63777fe700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c830 tx=0x7f636c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.637+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636c0106f0 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.637+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636c010d00 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.637+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637810dcd0 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.637+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636c012e90 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.637+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637810e290 con 0x7f6378102e70 2024-03-14T13:07:21.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.638+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6378100110 con 0x7f6378102e70 2024-03-14T13:07:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.638+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f636c010850 con 0x7f6378102e70 2024-03-14T13:07:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.638+0000 7f6374ff9700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 0x7f636006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.638+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f636c08b990 con 0x7f6378102e70 2024-03-14T13:07:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.639+0000 7f6377fff700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 0x7f636006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.639+0000 7f6377fff700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 0x7f636006ec80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f636800b840 tx=0x7f6368009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.644+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f636c059740 con 0x7f6378102e70 2024-03-14T13:07:21.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.855+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f637810f0c0 con 0x7f636006c7d0 2024-03-14T13:07:21.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.864+0000 7f6374ff9700 1 -- 172.21.15.111:0/3733430092 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f637810f0c0 con 0x7f636006c7d0 2024-03-14T13:07:21.865 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:07:21.865 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (3m) 33s ago 15m 27.7M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:07:21.865 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (64s) 33s ago 15m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (59s) 56s ago 13m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (3m) 33s ago 14m 83.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (8m) 33s ago 8m 64.3M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (9m) 33s ago 9m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (8m) 56s ago 8m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (9m) 56s ago 9m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (6m) 33s ago 16m 494M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (4m) 56s ago 13m 449M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (2m) 33s ago 16m 44.3M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (102s) 56s ago 13m 37.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (4m) 33s ago 15m 18.7M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 56s ago 13m 18.1M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (38s) 33s ago 12m 11.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:07:21.866 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 starting - - - 1568M 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (11m) 33s ago 11m 197M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (11m) 56s ago 11m 267M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (11m) 56s ago 11m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (10m) 56s ago 10m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (4m) 33s ago 14m 62.2M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:07:21.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.866+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 msgr2=0x7f636006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.866+0000 7f637e58a700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 0x7f636006ec80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f636800b840 tx=0x7f6368009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.866+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 msgr2=0x7f6378113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.866+0000 7f637e58a700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c830 tx=0x7f636c004470 comp rx=0 tx=0).stop 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 shutdown_connections 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f636006c7d0 0x7f636006ec80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63780fb4a0 0x7f6378113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 --2- 172.21.15.111:0/3733430092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6378102e70 0x7f6378113fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 >> 172.21.15.111:0/3733430092 conn(0x7f63780714c0 msgr2=0x7f63780fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 shutdown_connections 2024-03-14T13:07:21.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.867+0000 7f637e58a700 1 -- 172.21.15.111:0/3733430092 wait complete. 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 -- 172.21.15.111:0/2283175014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fe830 msgr2=0x7f62fc106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 --2- 172.21.15.111:0/2283175014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fe830 0x7f62fc106f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f62e4004940 tx=0x7f62e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 -- 172.21.15.111:0/2283175014 shutdown_connections 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 --2- 172.21.15.111:0/2283175014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fe830 0x7f62fc106f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 --2- 172.21.15.111:0/2283175014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fde90 0x7f62fc0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 -- 172.21.15.111:0/2283175014 >> 172.21.15.111:0/2283175014 conn(0x7f62fc072bc0 msgr2=0x7f62fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 -- 172.21.15.111:0/2283175014 shutdown_connections 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.966+0000 7f6300e02700 1 -- 172.21.15.111:0/2283175014 wait complete. 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 Processor -- start 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 -- start start 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 0x7f62fc114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc114670 con 0x7f62fc0fde90 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f6300e02700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc10da60 con 0x7f62fc0fe830 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51842/0 (socket says 172.21.15.111:51842) 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 -- 172.21.15.111:0/3848037832 learned_addr learned my addr 172.21.15.111:0/3848037832 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62f9d9b700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 0x7f62fc114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 -- 172.21.15.111:0/3848037832 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 msgr2=0x7f62fc114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 0x7f62fc114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 -- 172.21.15.111:0/3848037832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4005040 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62f9d9b700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 0x7f62fc114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.967+0000 7f62fa59c700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f62ec002a50 tx=0x7f62ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.968+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62ec007cc0 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.968+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62ec00f040 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.968+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc10dd40 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.968+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62ec009540 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.968+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc10e280 con 0x7f62fc0fde90 2024-03-14T13:07:21.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.969+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc065e30 con 0x7f62fc0fde90 2024-03-14T13:07:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.969+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f62ec0096a0 con 0x7f62fc0fde90 2024-03-14T13:07:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.969+0000 7f62f2ffd700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 0x7f62e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.970+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f62ec08b700 con 0x7f62fc0fde90 2024-03-14T13:07:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.970+0000 7f62f9d9b700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 0x7f62e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:21.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.970+0000 7f62f9d9b700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 0x7f62e806ec70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f62fc10ec20 tx=0x7f62e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:21.974 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:21.975+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f62ec0594b0 con 0x7f62fc0fde90 2024-03-14T13:07:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:21 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 50/321 objects degraded (15.576%), 16 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:21 smithi137 ceph-mon[108145]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs peering) 2024-03-14T13:07:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:21 smithi137 ceph-mon[108145]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:22.079 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:21 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 50/321 objects degraded (15.576%), 16 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:22.079 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:21 smithi111 ceph-mon[145314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs peering) 2024-03-14T13:07:22.079 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:21 smithi111 ceph-mon[145314]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:22.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.266+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f62fc109dd0 con 0x7f62fc0fde90 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.267+0000 7f62f2ffd700 1 -- 172.21.15.111:0/3848037832 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f62ec059040 con 0x7f62fc0fde90 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4, 2024-03-14T13:07:22.267 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 8, 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 5 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:07:22.268 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 msgr2=0x7f62e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 0x7f62e806ec70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f62fc10ec20 tx=0x7f62e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 msgr2=0x7f62fc113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f62ec002a50 tx=0x7f62ec0046b0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 shutdown_connections 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62e806c7c0 0x7f62e806ec70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62fc0fde90 0x7f62fc113ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 --2- 172.21.15.111:0/3848037832 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc0fe830 0x7f62fc114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 >> 172.21.15.111:0/3848037832 conn(0x7f62fc072bc0 msgr2=0x7f62fc103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 shutdown_connections 2024-03-14T13:07:22.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.269+0000 7f6300e02700 1 -- 172.21.15.111:0/3848037832 wait complete. 2024-03-14T13:07:22.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/891751466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 msgr2=0x7f4df80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/891751466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df80fbd30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4dec004940 tx=0x7f4dec009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/891751466 shutdown_connections 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/891751466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df80fbd30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/891751466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 0x7f4df81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.374+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/891751466 >> 172.21.15.111:0/891751466 conn(0x7f4df80f7870 msgr2=0x7f4df80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.375+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/891751466 shutdown_connections 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.375+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/891751466 wait complete. 2024-03-14T13:07:22.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.375+0000 7f4dfe1d0700 1 Processor -- start 2024-03-14T13:07:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.375+0000 7f4dfe1d0700 1 -- start start 2024-03-14T13:07:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4dfe1d0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 0x7f4df81127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4dfe1d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4dfe1d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df8113400 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4dfe1d0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df810c970 con 0x7f4df8101d20 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51868/0 (socket says 172.21.15.111:51868) 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 -- 172.21.15.111:0/1441819621 learned_addr learned my addr 172.21.15.111:0/1441819621 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df77fe700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 0x7f4df81127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 -- 172.21.15.111:0/1441819621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 msgr2=0x7f4df81127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 0x7f4df81127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 -- 172.21.15.111:0/1441819621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec005040 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4df6ffd700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0093b0 tx=0x7f4dec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dec00e040 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dec013420 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df810cbf0 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.376+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dec01c630 con 0x7f4df81026c0 2024-03-14T13:07:22.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.377+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df810d1b0 con 0x7f4df81026c0 2024-03-14T13:07:22.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.377+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df80fd470 con 0x7f4df81026c0 2024-03-14T13:07:22.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.378+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4dec0139e0 con 0x7f4df81026c0 2024-03-14T13:07:22.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.378+0000 7f4ddffff700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 0x7f4de006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.378+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f4dec08cb80 con 0x7f4df81026c0 2024-03-14T13:07:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.378+0000 7f4df77fe700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 0x7f4de006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.379+0000 7f4df77fe700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 0x7f4de006ec70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4de8006fd0 tx=0x7f4de8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:22.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.383+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4dec05a9c0 con 0x7f4df81026c0 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.660+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4df8108d30 con 0x7f4df81026c0 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.660+0000 7f4ddffff700 1 -- 172.21.15.111:0/1441819621 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f4dec05a550 con 0x7f4df81026c0 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:07:22.660 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:07:22.661 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:07:22.662 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:22.663 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 msgr2=0x7f4de006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 0x7f4de006ec70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4de8006fd0 tx=0x7f4de8008040 comp rx=0 tx=0).stop 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 msgr2=0x7f4df8112d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0093b0 tx=0x7f4dec009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 shutdown_connections 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4de006c7c0 0x7f4de006ec70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4df8101d20 0x7f4df81127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 --2- 172.21.15.111:0/1441819621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df81026c0 0x7f4df8112d20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 >> 172.21.15.111:0/1441819621 conn(0x7f4df80f7870 msgr2=0x7f4df80f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 shutdown_connections 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.663+0000 7f4dfe1d0700 1 -- 172.21.15.111:0/1441819621 wait complete. 2024-03-14T13:07:22.664 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:07:22.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 -- 172.21.15.111:0/1885347359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c070bf0 msgr2=0x7f313c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 --2- 172.21.15.111:0/1885347359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c070bf0 0x7f313c1091a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f312c004940 tx=0x7f312c009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:22.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 -- 172.21.15.111:0/1885347359 shutdown_connections 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 --2- 172.21.15.111:0/1885347359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c070bf0 0x7f313c1091a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 --2- 172.21.15.111:0/1885347359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c0702e0 0x7f313c0706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 -- 172.21.15.111:0/1885347359 >> 172.21.15.111:0/1885347359 conn(0x7f313c0f89a0 msgr2=0x7f313c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 -- 172.21.15.111:0/1885347359 shutdown_connections 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.770+0000 7f31443fd700 1 -- 172.21.15.111:0/1885347359 wait complete. 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 Processor -- start 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 -- start start 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c070bf0 0x7f313c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c116890 con 0x7f313c0702e0 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f31443fd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313c10fc60 con 0x7f313c070bf0 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51888/0 (socket says 172.21.15.111:51888) 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 -- 172.21.15.111:0/815902077 learned_addr learned my addr 172.21.15.111:0/815902077 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:22.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3141998700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c070bf0 0x7f313c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 -- 172.21.15.111:0/815902077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c070bf0 msgr2=0x7f313c116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c070bf0 0x7f313c116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.771+0000 7f3142199700 1 -- 172.21.15.111:0/815902077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c005040 con 0x7f313c0702e0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f3142199700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f313800c8c0 tx=0x7f31380044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f313800a760 con 0x7f313c0702e0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f313800ad70 con 0x7f313c0702e0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313c10ff40 con 0x7f313c0702e0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31380113a0 con 0x7f313c0702e0 2024-03-14T13:07:22.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.772+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313c110480 con 0x7f313c0702e0 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.773+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313c103c20 con 0x7f313c0702e0 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.773+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3138011500 con 0x7f313c0702e0 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.773+0000 7f3132ffd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 0x7f312806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.774+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f313808ba80 con 0x7f313c0702e0 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.774+0000 7f3141998700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 0x7f312806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:22.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.774+0000 7f3141998700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 0x7f312806ec70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f312c004f90 tx=0x7f312c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:22.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:22.778+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3138059830 con 0x7f313c0702e0 2024-03-14T13:07:23.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:22 smithi137 ceph-mon[108145]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:22 smithi137 ceph-mon[108145]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:22 smithi137 ceph-mon[108145]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:22 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3848037832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:22 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1441819621' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:07:23.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.008+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f313c10bfb0 con 0x7f312806c7c0 2024-03-14T13:07:23.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.010+0000 7f3132ffd700 1 -- 172.21.15.111:0/815902077 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f313c10bfb0 con 0x7f312806c7c0 2024-03-14T13:07:23.009 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:07:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:07:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:07:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "7/21 daemons upgraded", 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:07:23.010 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:07:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 msgr2=0x7f312806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 0x7f312806ec70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f312c004f90 tx=0x7f312c009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:23.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 msgr2=0x7f313c115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f313800c8c0 tx=0x7f31380044c0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 shutdown_connections 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f312806c7c0 0x7f312806ec70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f313c0702e0 0x7f313c115d00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 --2- 172.21.15.111:0/815902077 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f313c070bf0 0x7f313c116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 >> 172.21.15.111:0/815902077 conn(0x7f313c0f89a0 msgr2=0x7f313c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 shutdown_connections 2024-03-14T13:07:23.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.012+0000 7f31443fd700 1 -- 172.21.15.111:0/815902077 wait complete. 2024-03-14T13:07:23.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:22 smithi111 ceph-mon[145314]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:23.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:22 smithi111 ceph-mon[145314]: from='client.34228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:23.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:22 smithi111 ceph-mon[145314]: pgmap v69: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:23.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:22 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3848037832' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:23.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:22 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1441819621' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:07:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 -- 172.21.15.111:0/859564894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe01037c0 msgr2=0x7fcfe00fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/859564894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe01037c0 0x7fcfe00fce30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc004940 tx=0x7fcfcc009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 -- 172.21.15.111:0/859564894 shutdown_connections 2024-03-14T13:07:23.127 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/859564894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe01037c0 0x7fcfe00fce30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/859564894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe0102e20 0x7fcfe01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 -- 172.21.15.111:0/859564894 >> 172.21.15.111:0/859564894 conn(0x7fcfe00f89d0 msgr2=0x7fcfe00fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 -- 172.21.15.111:0/859564894 shutdown_connections 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.125+0000 7fcfe6e68700 1 -- 172.21.15.111:0/859564894 wait complete. 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 Processor -- start 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 -- start start 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01037c0 0x7fcfe0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0070d90 con 0x7fcfe0102e20 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe6e68700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0070ed0 con 0x7fcfe01037c0 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe4c04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:23.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe4c04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51904/0 (socket says 172.21.15.111:51904) 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfe4c04700 1 -- 172.21.15.111:0/2362092588 learned_addr learned my addr 172.21.15.111:0/2362092588 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.126+0000 7fcfdffff700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01037c0 0x7fcfe0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe4c04700 1 -- 172.21.15.111:0/2362092588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01037c0 msgr2=0x7fcfe0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe4c04700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01037c0 0x7fcfe0070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe4c04700 1 -- 172.21.15.111:0/2362092588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc005040 con 0x7fcfe0102e20 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe4c04700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4002a50 tx=0x7fcfd40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd4007cc0 con 0x7fcfe0102e20 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfd400f040 con 0x7fcfe0102e20 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe0119dd0 con 0x7fcfe0102e20 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfd400d9c0 con 0x7fcfe0102e20 2024-03-14T13:07:23.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.127+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe011a250 con 0x7fcfe0102e20 2024-03-14T13:07:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.129+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe00fe570 con 0x7fcfe0102e20 2024-03-14T13:07:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.129+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcfd400db20 con 0x7fcfe0102e20 2024-03-14T13:07:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.129+0000 7fcfdd7fa700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 0x7fcfd006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.129+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fcfd408b740 con 0x7fcfe0102e20 2024-03-14T13:07:23.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.129+0000 7fcfdffff700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 0x7fcfd006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:23.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.130+0000 7fcfdffff700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 0x7fcfd006ec80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0071c00 tx=0x7fcfcc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:23.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.134+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcfd40559e0 con 0x7fcfe0102e20 2024-03-14T13:07:23.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.442+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcfe005f5f0 con 0x7fcfe0102e20 2024-03-14T13:07:23.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.443+0000 7fcfdd7fa700 1 -- 172.21.15.111:0/2362092588 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+1160 (secure 0 0 0) 0x7fcfd4059000 con 0x7fcfe0102e20 2024-03-14T13:07:23.442 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 50/321 objects degraded (15.576%), 16 pgs degraded 2024-03-14T13:07:23.442 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: osd.1 (root=default,host=smithi111) is down 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 50/321 objects degraded (15.576%), 16 pgs degraded 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 1.0 is active+undersized+degraded, acting [3,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.0 is active+undersized+degraded, acting [3,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.2 is active+undersized+degraded, acting [5,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.4 is active+undersized+degraded, acting [0,4] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.6 is active+undersized+degraded, acting [3,4] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.9 is active+undersized+degraded, acting [4,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.a is active+undersized+degraded, acting [4,3] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.d is active+undersized+degraded, acting [3,2] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.10 is active+undersized+degraded, acting [2,0] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.11 is active+undersized+degraded, acting [4,3] 2024-03-14T13:07:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.12 is active+undersized+degraded, acting [3,0] 2024-03-14T13:07:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.15 is active+undersized+degraded, acting [3,0] 2024-03-14T13:07:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-03-14T13:07:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1b is active+undersized+degraded, acting [0,5] 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 msgr2=0x7fcfd006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 0x7fcfd006ec80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0071c00 tx=0x7fcfcc009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 msgr2=0x7fcfe00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4002a50 tx=0x7fcfd40046b0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 shutdown_connections 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcfd006c7d0 0x7fcfd006ec80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcfe0102e20 0x7fcfe00702c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 --2- 172.21.15.111:0/2362092588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfe01037c0 0x7fcfe0070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 >> 172.21.15.111:0/2362092588 conn(0x7fcfe00f89d0 msgr2=0x7fcfe00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:23.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 shutdown_connections 2024-03-14T13:07:23.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:23.445+0000 7fcfe6e68700 1 -- 172.21.15.111:0/2362092588 wait complete. 2024-03-14T13:07:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:23 smithi137 ceph-mon[108145]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:24.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:23 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2362092588' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:07:24.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:23 smithi111 ceph-mon[145314]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:24.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:23 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2362092588' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:07:24.890 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:24 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:07:24.640+0000 7f4fad535640 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-14T13:07:25.390 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:07:25.136+0000 7f4fad535640 -1 osd.1 55 log_to_monitors true 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:25 smithi111 ceph-mon[145314]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: pgmap v70: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:25 smithi137 ceph-mon[108145]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-14T13:07:26.522 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:07:26.144+0000 7f4f9d18b700 -1 osd.1 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: osdmap e58: 6 total, 5 up, 6 in 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:07:27.142 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:26 smithi111 ceph-mon[145314]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: osdmap e58: 6 total, 5 up, 6 in 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: from='osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:07:27.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:26 smithi137 ceph-mon[108145]: pgmap v72: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:27 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:07:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:27 smithi111 ceph-mon[145314]: osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126] boot 2024-03-14T13:07:27.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:27 smithi111 ceph-mon[145314]: osdmap e59: 6 total, 6 up, 6 in 2024-03-14T13:07:27.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:27 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:07:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:27 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:07:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:27 smithi137 ceph-mon[108145]: osd.1 [v2:172.21.15.111:6810/2339106126,v1:172.21.15.111:6811/2339106126] boot 2024-03-14T13:07:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:27 smithi137 ceph-mon[108145]: osdmap e59: 6 total, 6 up, 6 in 2024-03-14T13:07:28.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:27 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-14T13:07:29.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:29 smithi111 ceph-mon[145314]: osdmap e60: 6 total, 6 up, 6 in 2024-03-14T13:07:29.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:29 smithi111 ceph-mon[145314]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:29 smithi137 ceph-mon[108145]: osdmap e60: 6 total, 6 up, 6 in 2024-03-14T13:07:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:29 smithi137 ceph-mon[108145]: pgmap v75: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 50/321 objects degraded (15.576%) 2024-03-14T13:07:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:30.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-14T13:07:30.892 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:30 smithi111 ceph-mon[145314]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 21/321 objects degraded (6.542%) 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:31.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-14T13:07:31.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:30 smithi137 ceph-mon[108145]: pgmap v76: 65 pgs: 7 active+undersized+degraded, 5 active+undersized, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 3.2 KiB/s rd, 6 op/s; 21/321 objects degraded (6.542%) 2024-03-14T13:07:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:31 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:32.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:31 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:32 smithi111 ceph-mon[145314]: pgmap v77: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-14T13:07:33.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:32 smithi137 ceph-mon[108145]: pgmap v77: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-14T13:07:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:33 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded) 2024-03-14T13:07:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:33 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:07:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:33 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/321 objects degraded (6.542%), 7 pgs degraded) 2024-03-14T13:07:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:33 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:07:35.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:35 smithi111 ceph-mon[145314]: pgmap v78: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-14T13:07:35.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:35 smithi137 ceph-mon[108145]: pgmap v78: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 2 op/s 2024-03-14T13:07:37.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:37 smithi111 ceph-mon[145314]: pgmap v79: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:07:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:37 smithi137 ceph-mon[108145]: pgmap v79: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:07:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:39 smithi111 ceph-mon[145314]: pgmap v80: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:07:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:39 smithi137 ceph-mon[108145]: pgmap v80: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:07:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:41 smithi137 ceph-mon[108145]: pgmap v81: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:07:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:41 smithi111 ceph-mon[145314]: pgmap v81: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:07:43.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:43 smithi137 ceph-mon[108145]: pgmap v82: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:43 smithi111 ceph-mon[145314]: pgmap v82: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:45.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:45 smithi111 ceph-mon[145314]: pgmap v83: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:45.560 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:45 smithi137 ceph-mon[108145]: pgmap v83: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:45.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:46 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:46 smithi111 ceph-mon[145314]: Upgrade: osd.2 is safe to restart 2024-03-14T13:07:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T13:07:46.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:46 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-14T13:07:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:46 smithi137 ceph-mon[108145]: Upgrade: osd.2 is safe to restart 2024-03-14T13:07:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-14T13:07:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:07:47.750 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:47 smithi111 ceph-mon[145314]: pgmap v84: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:47.750 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:47 smithi111 ceph-mon[145314]: Upgrade: Updating osd.2 2024-03-14T13:07:47.750 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:47 smithi111 ceph-mon[145314]: Deploying daemon osd.2 on smithi111 2024-03-14T13:07:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:47 smithi137 ceph-mon[108145]: pgmap v84: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:47 smithi137 ceph-mon[108145]: Upgrade: Updating osd.2 2024-03-14T13:07:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:47 smithi137 ceph-mon[108145]: Deploying daemon osd.2 on smithi111 2024-03-14T13:07:49.486 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:49 smithi111 ceph-mon[145314]: pgmap v85: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:49 smithi137 ceph-mon[108145]: pgmap v85: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:07:49.890 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:49 smithi111 systemd[1]: Stopping Ceph osd.2 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:07:49.890 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:49 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[81165]: 2024-03-14T13:07:49.713+0000 7ff12d878700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:07:49.891 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:49 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[81165]: 2024-03-14T13:07:49.713+0000 7ff12d878700 -1 osd.2 60 *** Got signal Terminated *** 2024-03-14T13:07:49.891 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:49 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[81165]: 2024-03-14T13:07:49.713+0000 7ff12d878700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:07:50.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:50 smithi137 ceph-mon[108145]: osd.2 marked itself down and dead 2024-03-14T13:07:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:50 smithi111 ceph-mon[145314]: osd.2 marked itself down and dead 2024-03-14T13:07:51.640 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:51 smithi111 bash[173316]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2 2024-03-14T13:07:51.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:51 smithi111 ceph-mon[145314]: pgmap v86: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:51.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:51 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:07:51.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:51 smithi111 ceph-mon[145314]: osdmap e61: 6 total, 5 up, 6 in 2024-03-14T13:07:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:51 smithi137 ceph-mon[108145]: pgmap v86: 65 pgs: 65 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:07:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:51 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:07:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:51 smithi137 ceph-mon[108145]: osdmap e61: 6 total, 5 up, 6 in 2024-03-14T13:07:52.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:52 smithi111 ceph-mon[145314]: osdmap e62: 6 total, 5 up, 6 in 2024-03-14T13:07:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:52 smithi137 ceph-mon[108145]: osdmap e62: 6 total, 5 up, 6 in 2024-03-14T13:07:53.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/3828925867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 msgr2=0x7f0e540fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/3828925867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e540fe260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0e50004940 tx=0x7f0e50009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/3828925867 shutdown_connections 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/3828925867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 0x7f0e54106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/3828925867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e540fe260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/3828925867 >> 172.21.15.111:0/3828925867 conn(0x7f0e54072bc0 msgr2=0x7f0e54072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/3828925867 shutdown_connections 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.572+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/3828925867 wait complete. 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 Processor -- start 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 -- start start 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 0x7f0e54114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e54114680 con 0x7f0e540fde90 2024-03-14T13:07:53.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e5bf7b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5410da70 con 0x7f0e540fe830 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e59d17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e59d17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55400/0 (socket says 172.21.15.111:55400) 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.573+0000 7f0e59d17700 1 -- 172.21.15.111:0/2852219231 learned_addr learned my addr 172.21.15.111:0/2852219231 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e59516700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 0x7f0e54114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e59d17700 1 -- 172.21.15.111:0/2852219231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 msgr2=0x7f0e54114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e59d17700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 0x7f0e54114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e59d17700 1 -- 172.21.15.111:0/2852219231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50005040 con 0x7f0e540fde90 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e59d17700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009360 tx=0x7f0e50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e5000e040 con 0x7f0e540fde90 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e50013420 con 0x7f0e540fde90 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e5001c630 con 0x7f0e540fde90 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5410dcf0 con 0x7f0e540fde90 2024-03-14T13:07:53.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.574+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5410e230 con 0x7f0e540fde90 2024-03-14T13:07:53.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.575+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e541019b0 con 0x7f0e540fde90 2024-03-14T13:07:53.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.576+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0e500139e0 con 0x7f0e540fde90 2024-03-14T13:07:53.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.576+0000 7f0e4a7fc700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 0x7f0e4006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.577+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0e5008d6c0 con 0x7f0e540fde90 2024-03-14T13:07:53.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.577+0000 7f0e59516700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 0x7f0e4006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.577+0000 7f0e59516700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 0x7f0e4006ec70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007fd0 tx=0x7f0e44009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:53.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.584+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0e5005b510 con 0x7f0e540fde90 2024-03-14T13:07:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:53 smithi137 ceph-mon[108145]: pgmap v89: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:53 smithi137 ceph-mon[108145]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-14T13:07:53.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.822+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e5410f1b0 con 0x7f0e4006c7c0 2024-03-14T13:07:53.823 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.824+0000 7f0e4a7fc700 1 -- 172.21.15.111:0/2852219231 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f0e5410f1b0 con 0x7f0e4006c7c0 2024-03-14T13:07:53.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 msgr2=0x7f0e4006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 0x7f0e4006ec70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0e44007fd0 tx=0x7f0e44009040 comp rx=0 tx=0).stop 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 msgr2=0x7f0e54113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009360 tx=0x7f0e50009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 shutdown_connections 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e4006c7c0 0x7f0e4006ec70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e540fde90 0x7f0e54113af0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 --2- 172.21.15.111:0/2852219231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e540fe830 0x7f0e54114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 >> 172.21.15.111:0/2852219231 conn(0x7f0e54072bc0 msgr2=0x7f0e540ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 shutdown_connections 2024-03-14T13:07:53.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.826+0000 7f0e5bf7b700 1 -- 172.21.15.111:0/2852219231 wait complete. 2024-03-14T13:07:53.836 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:07:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:53 smithi111 ceph-mon[145314]: pgmap v89: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:53 smithi111 ceph-mon[145314]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-14T13:07:53.891 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:53 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.2.service: Succeeded. 2024-03-14T13:07:53.891 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:53 smithi111 systemd[1]: Stopped Ceph osd.2 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:07:53.891 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:53 smithi111 systemd[1]: Starting Ceph osd.2 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:07:53.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.930+0000 7fd984ecd700 1 -- 172.21.15.111:0/3718954568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9800fb4a0 msgr2=0x7fd9800fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.930+0000 7fd984ecd700 1 --2- 172.21.15.111:0/3718954568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9800fb4a0 0x7fd9800fb910 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd974004940 tx=0x7fd974009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:53.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- 172.21.15.111:0/3718954568 shutdown_connections 2024-03-14T13:07:53.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 --2- 172.21.15.111:0/3718954568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9800fb4a0 0x7fd9800fb910 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 --2- 172.21.15.111:0/3718954568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd980102e70 0x7fd980103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- 172.21.15.111:0/3718954568 >> 172.21.15.111:0/3718954568 conn(0x7fd9800714c0 msgr2=0x7fd9800718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- 172.21.15.111:0/3718954568 shutdown_connections 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- 172.21.15.111:0/3718954568 wait complete. 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 Processor -- start 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- start start 2024-03-14T13:07:53.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9800fb4a0 0x7fd980113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd980114600 con 0x7fd980102e70 2024-03-14T13:07:53.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.931+0000 7fd984ecd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98010d9f0 con 0x7fd9800fb4a0 2024-03-14T13:07:53.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55416/0 (socket says 172.21.15.111:55416) 2024-03-14T13:07:53.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 -- 172.21.15.111:0/8280909 learned_addr learned my addr 172.21.15.111:0/8280909 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:53.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97e59c700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9800fb4a0 0x7fd980113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 -- 172.21.15.111:0/8280909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9800fb4a0 msgr2=0x7fd980113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9800fb4a0 0x7fd980113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 -- 172.21.15.111:0/8280909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974005040 con 0x7fd980102e70 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd97dd9b700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd97400b3b0 tx=0x7fd974009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd97400e040 con 0x7fd980102e70 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd974013420 con 0x7fd980102e70 2024-03-14T13:07:53.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd97401c610 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98010dc70 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.932+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98010e230 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.933+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd980100110 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.934+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd974013a00 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.934+0000 7fd96effd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 0x7fd96806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.934+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd97408cb50 con 0x7fd980102e70 2024-03-14T13:07:53.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.934+0000 7fd97e59c700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 0x7fd96806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:53.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.934+0000 7fd97e59c700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 0x7fd96806ec70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd970004f40 tx=0x7fd97000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:53.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:53.938+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd97405a900 con 0x7fd980102e70 2024-03-14T13:07:54.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.163+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd980113740 con 0x7fd96806c7c0 2024-03-14T13:07:54.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.165+0000 7fd96effd700 1 -- 172.21.15.111:0/8280909 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fd980113740 con 0x7fd96806c7c0 2024-03-14T13:07:54.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 msgr2=0x7fd96806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 0x7fd96806ec70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd970004f40 tx=0x7fd97000b040 comp rx=0 tx=0).stop 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 msgr2=0x7fd980113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd97400b3b0 tx=0x7fd974009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 shutdown_connections 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd96806c7c0 0x7fd96806ec70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9800fb4a0 0x7fd980113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 --2- 172.21.15.111:0/8280909 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd980102e70 0x7fd980113fb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 >> 172.21.15.111:0/8280909 conn(0x7fd9800714c0 msgr2=0x7fd9800fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:54.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 shutdown_connections 2024-03-14T13:07:54.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.167+0000 7fd984ecd700 1 -- 172.21.15.111:0/8280909 wait complete. 2024-03-14T13:07:54.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/3229956887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 msgr2=0x7fd1d8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/3229956887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d8106f00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc004940 tx=0x7fd1cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/3229956887 shutdown_connections 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/3229956887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d8106f00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/3229956887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 0x7fd1d80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/3229956887 >> 172.21.15.111:0/3229956887 conn(0x7fd1d8072bc0 msgr2=0x7fd1d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/3229956887 shutdown_connections 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/3229956887 wait complete. 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.273+0000 7fd1dd0fc700 1 Processor -- start 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1dd0fc700 1 -- start start 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1dd0fc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 0x7fd1d80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1dd0fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1dd0fc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80fc820 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1dd0fc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80fc960 con 0x7fd1d80fde90 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55432/0 (socket says 172.21.15.111:55432) 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 -- 172.21.15.111:0/2647544442 learned_addr learned my addr 172.21.15.111:0/2647544442 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d6d9d700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 0x7fd1d80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 -- 172.21.15.111:0/2647544442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 msgr2=0x7fd1d80fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 0x7fd1d80fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 -- 172.21.15.111:0/2647544442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc005040 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.274+0000 7fd1d659c700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009360 tx=0x7fd1cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.275+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1cc00e040 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.275+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80fcbb0 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.275+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1cc00bed0 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.275+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1cc01c3f0 con 0x7fd1d80fe830 2024-03-14T13:07:54.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.275+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d811a190 con 0x7fd1d80fe830 2024-03-14T13:07:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.276+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d8065e30 con 0x7fd1d80fe830 2024-03-14T13:07:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.276+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd1cc013790 con 0x7fd1d80fe830 2024-03-14T13:07:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.277+0000 7fd1bf7fe700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 0x7fd1c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.277+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fd1cc08cd80 con 0x7fd1d80fe830 2024-03-14T13:07:54.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.281+0000 7fd1d6d9d700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 0x7fd1c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.281+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd1cc057020 con 0x7fd1d80fe830 2024-03-14T13:07:54.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.282+0000 7fd1d6d9d700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 0x7fd1c006ec70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8004f40 tx=0x7fd1c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:54.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.513+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1d8109dd0 con 0x7fd1c006c7c0 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.523+0000 7fd1bf7fe700 1 -- 172.21.15.111:0/2647544442 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd1d8109dd0 con 0x7fd1c006c7c0 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (4m) 29s ago 15m 27.7M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (97s) 29s ago 15m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (92s) 88s ago 14m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (3m) 29s ago 15m 85.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (9m) 29s ago 9m 64.2M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:07:54.523 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (9m) 29s ago 9m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (9m) 88s ago 9m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (9m) 88s ago 9m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (6m) 29s ago 17m 496M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (5m) 88s ago 14m 449M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (2m) 29s ago 17m 47.0M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (2m) 88s ago 14m 37.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (5m) 29s ago 15m 18.8M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 88s ago 14m 18.1M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (71s) 29s ago 13m 84.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (35s) 29s ago 12m 10.8M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (12m) 29s ago 12m 196M 1568M 17.2.7-646-gb92acd1f a07fb176315d 7a493ab21eb0 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (12m) 88s ago 12m 267M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (11m) 88s ago 11m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:07:54.524 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (11m) 88s ago 11m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:07:54.525 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (4m) 29s ago 14m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:07:54.525 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 msgr2=0x7fd1c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.525 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 0x7fd1c006ec70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8004f40 tx=0x7fd1c800b040 comp rx=0 tx=0).stop 2024-03-14T13:07:54.525 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 msgr2=0x7fd1d80fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.525 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc009360 tx=0x7fd1cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 shutdown_connections 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1c006c7c0 0x7fd1c006ec70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1d80fde90 0x7fd1d80fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.525+0000 7fd1dd0fc700 1 --2- 172.21.15.111:0/2647544442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1d80fe830 0x7fd1d80fc290 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.526+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 >> 172.21.15.111:0/2647544442 conn(0x7fd1d8072bc0 msgr2=0x7fd1d8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.526+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 shutdown_connections 2024-03-14T13:07:54.526 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.526+0000 7fd1dd0fc700 1 -- 172.21.15.111:0/2647544442 wait complete. 2024-03-14T13:07:54.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 -- 172.21.15.111:0/1128455175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fbc80 msgr2=0x7f711c0fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 --2- 172.21.15.111:0/1128455175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fbc80 0x7f711c0fc050 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7118004940 tx=0x7f7118009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 -- 172.21.15.111:0/1128455175 shutdown_connections 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 --2- 172.21.15.111:0/1128455175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fc590 0x7f711c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 --2- 172.21.15.111:0/1128455175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fbc80 0x7f711c0fc050 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 -- 172.21.15.111:0/1128455175 >> 172.21.15.111:0/1128455175 conn(0x7f711c072bc0 msgr2=0x7f711c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 -- 172.21.15.111:0/1128455175 shutdown_connections 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.638+0000 7f7124424700 1 -- 172.21.15.111:0/1128455175 wait complete. 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 Processor -- start 2024-03-14T13:07:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 -- start start 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fbc80 0x7f711c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711c1168c0 con 0x7f711c0fc590 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.639+0000 7f7124424700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711c10fc90 con 0x7f711c0fbc80 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55444/0 (socket says 172.21.15.111:55444) 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 -- 172.21.15.111:0/1239778733 learned_addr learned my addr 172.21.15.111:0/1239778733 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71221c0700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fbc80 0x7f711c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 -- 172.21.15.111:0/1239778733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fbc80 msgr2=0x7f711c115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fbc80 0x7f711c115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 -- 172.21.15.111:0/1239778733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7118005040 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f71219bf700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f710c00c8c0 tx=0x7f710c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f710c00a760 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f711c10ff70 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f710c00ad70 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f710c0113a0 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.640+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711c1104b0 con 0x7f711c0fc590 2024-03-14T13:07:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.641+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711c065e30 con 0x7f711c0fc590 2024-03-14T13:07:54.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.642+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f710c012400 con 0x7f711c0fc590 2024-03-14T13:07:54.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.642+0000 7f7112ffd700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 0x7f710806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:54.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.643+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f710c08c4f0 con 0x7f711c0fc590 2024-03-14T13:07:54.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.643+0000 7f71221c0700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 0x7f710806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:54.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.643+0000 7f71221c0700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 0x7f710806ec70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7118004f90 tx=0x7f7118009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:54.646 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.647+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f710c05a400 con 0x7f711c0fc590 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.947+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f711c10c020 con 0x7f711c0fc590 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.947+0000 7f7112ffd700 1 -- 172.21.15.111:0/1239778733 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f710c059f90 con 0x7f711c0fc590 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:54.947 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 3, 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 7, 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:07:54.948 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 msgr2=0x7f710806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 0x7f710806ec70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7118004f90 tx=0x7f7118009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 msgr2=0x7f711c116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f710c00c8c0 tx=0x7f710c0044c0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 shutdown_connections 2024-03-14T13:07:54.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f710806c7c0 0x7f710806ec70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f711c0fbc80 0x7f711c115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 --2- 172.21.15.111:0/1239778733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f711c0fc590 0x7f711c116270 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 >> 172.21.15.111:0/1239778733 conn(0x7f711c072bc0 msgr2=0x7f711c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 shutdown_connections 2024-03-14T13:07:54.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:54.949+0000 7f7124424700 1 -- 172.21.15.111:0/1239778733 wait complete. 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- 172.21.15.111:0/392545345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 msgr2=0x7f6218103200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 --2- 172.21.15.111:0/392545345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218103200 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6208009ce0 tx=0x7f6208004d10 comp rx=0 tx=0).stop 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- 172.21.15.111:0/392545345 shutdown_connections 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 --2- 172.21.15.111:0/392545345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 0x7f62180fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 --2- 172.21.15.111:0/392545345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218103200 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- 172.21.15.111:0/392545345 >> 172.21.15.111:0/392545345 conn(0x7f62180f8a20 msgr2=0x7f62180fae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- 172.21.15.111:0/392545345 shutdown_connections 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- 172.21.15.111:0/392545345 wait complete. 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 Processor -- start 2024-03-14T13:07:55.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.053+0000 7f621db66700 1 -- start start 2024-03-14T13:07:55.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 0x7f6218113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218114610 con 0x7f6218102e30 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f621810dbf0 con 0x7f62181037d0 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55466/0 (socket says 172.21.15.111:55466) 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 -- 172.21.15.111:0/3622771632 learned_addr learned my addr 172.21.15.111:0/3622771632 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f6216ffd700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 0x7f6218113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 -- 172.21.15.111:0/3622771632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 msgr2=0x7f6218113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 0x7f6218113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 -- 172.21.15.111:0/3622771632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6208009940 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f62177fe700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6208005010 tx=0x7f6208004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f620800e040 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6208007d10 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f621810de70 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6208012650 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.054+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f621810e430 con 0x7f6218102e30 2024-03-14T13:07:55.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.055+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62180fe580 con 0x7f6218102e30 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.056+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f620801c440 con 0x7f6218102e30 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.056+0000 7f620ffff700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 0x7f620406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.056+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f620808d6b0 con 0x7f6218102e30 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.056+0000 7f6216ffd700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 0x7f620406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.057+0000 7f6216ffd700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 0x7f620406ec70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f621810f320 tx=0x7f620000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.061+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f620805b3e0 con 0x7f6218102e30 2024-03-14T13:07:55.123 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-14T13:07:55.123 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-14T13:07:55.123 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-14T13:07:55.123 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-14T13:07:55.124 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-14T13:07:55.124 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-14T13:07:55.124 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:54 smithi111 bash[173615]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-14T13:07:55.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.304+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f621810f0d0 con 0x7f6218102e30 2024-03-14T13:07:55.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.305+0000 7f620ffff700 1 -- 172.21.15.111:0/3622771632 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f6208004e10 con 0x7f6218102e30 2024-03-14T13:07:55.304 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:07:55.304 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:07:55.305 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:07:55.306 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:55.307 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 msgr2=0x7f620406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 0x7f620406ec70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f621810f320 tx=0x7f620000b040 comp rx=0 tx=0).stop 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 msgr2=0x7f6218113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6208005010 tx=0x7f6208004d10 comp rx=0 tx=0).stop 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 shutdown_connections 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f620406c7c0 0x7f620406ec70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6218102e30 0x7f6218113a80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 --2- 172.21.15.111:0/3622771632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62181037d0 0x7f6218113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 >> 172.21.15.111:0/3622771632 conn(0x7f62180f8a20 msgr2=0x7f62180f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 shutdown_connections 2024-03-14T13:07:55.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.307+0000 7f621db66700 1 -- 172.21.15.111:0/3622771632 wait complete. 2024-03-14T13:07:55.309 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:07:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: pgmap v90: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1239778733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- 172.21.15.111:0/795105195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4101d70 msgr2=0x7f90a4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 --2- 172.21.15.111:0/795105195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4101d70 0x7f90a4102140 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9098004940 tx=0x7f9098009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- 172.21.15.111:0/795105195 shutdown_connections 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 --2- 172.21.15.111:0/795105195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4102710 0x7f90a40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 --2- 172.21.15.111:0/795105195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4101d70 0x7f90a4102140 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- 172.21.15.111:0/795105195 >> 172.21.15.111:0/795105195 conn(0x7f90a4072bc0 msgr2=0x7f90a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- 172.21.15.111:0/795105195 shutdown_connections 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- 172.21.15.111:0/795105195 wait complete. 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 Processor -- start 2024-03-14T13:07:55.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.415+0000 7f90aae32700 1 -- start start 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90aae32700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4101d70 0x7f90a410ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90aae32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90aae32700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a410f670 con 0x7f90a4102710 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90aae32700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a410f7b0 con 0x7f90a4101d70 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55480/0 (socket says 172.21.15.111:55480) 2024-03-14T13:07:55.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 -- 172.21.15.111:0/1416151363 learned_addr learned my addr 172.21.15.111:0/1416151363 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a8bce700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4101d70 0x7f90a410ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 -- 172.21.15.111:0/1416151363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4101d70 msgr2=0x7f90a410ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4101d70 0x7f90a410ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 -- 172.21.15.111:0/1416151363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098005040 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.416+0000 7f90a3fff700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f909000b790 tx=0x7f9090009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.417+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909000f6f0 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.417+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f909000fd00 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.417+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a4117b20 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.417+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90900103a0 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.417+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a41180b0 con 0x7f90a4102710 2024-03-14T13:07:55.418 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.418+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a40ff080 con 0x7f90a4102710 2024-03-14T13:07:55.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.418+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f909000f850 con 0x7f90a4102710 2024-03-14T13:07:55.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.419+0000 7f90a17fa700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 0x7f909406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.419+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f909008b810 con 0x7f90a4102710 2024-03-14T13:07:55.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.419+0000 7f90a8bce700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 0x7f909406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.419 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.419+0000 7f90a8bce700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 0x7f909406ec70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9098004f90 tx=0x7f9098009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.423+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9090055b30 con 0x7f90a4102710 2024-03-14T13:07:55.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.653+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90a4108da0 con 0x7f909406c7c0 2024-03-14T13:07:55.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.654+0000 7f90a17fa700 1 -- 172.21.15.111:0/1416151363 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f90a4108da0 con 0x7f909406c7c0 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:07:55.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:07:55.655 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:07:55.655 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "8/21 daemons upgraded", 2024-03-14T13:07:55.655 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:07:55.655 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:07:55.655 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 msgr2=0x7f909406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 0x7f909406ec70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9098004f90 tx=0x7f9098009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 msgr2=0x7f90a410ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f909000b790 tx=0x7f9090009220 comp rx=0 tx=0).stop 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 shutdown_connections 2024-03-14T13:07:55.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f909406c7c0 0x7f909406ec70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90a4101d70 0x7f90a410ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 --2- 172.21.15.111:0/1416151363 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90a4102710 0x7f90a410ef90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 >> 172.21.15.111:0/1416151363 conn(0x7f90a4072bc0 msgr2=0x7f90a40faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 shutdown_connections 2024-03-14T13:07:55.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.656+0000 7f90aae32700 1 -- 172.21.15.111:0/1416151363 wait complete. 2024-03-14T13:07:55.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: from='client.34250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: from='client.34254 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: pgmap v90: 65 pgs: 7 peering, 5 stale+active+clean, 53 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail 2024-03-14T13:07:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:07:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:55 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1239778733' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:07:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 -- 172.21.15.111:0/2626285181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7564101d20 msgr2=0x7f75641020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 --2- 172.21.15.111:0/2626285181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7564101d20 0x7f75641020f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7558004940 tx=0x7f7558009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 -- 172.21.15.111:0/2626285181 shutdown_connections 2024-03-14T13:07:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 --2- 172.21.15.111:0/2626285181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75641026c0 0x7f75640fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 --2- 172.21.15.111:0/2626285181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7564101d20 0x7f75641020f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 -- 172.21.15.111:0/2626285181 >> 172.21.15.111:0/2626285181 conn(0x7f75640f7870 msgr2=0x7f75640f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 -- 172.21.15.111:0/2626285181 shutdown_connections 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.758+0000 7f7568acf700 1 -- 172.21.15.111:0/2626285181 wait complete. 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 Processor -- start 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 -- start start 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7564101d20 0x7f7564102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564104b20 con 0x7f75641026c0 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7568acf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564103600 con 0x7f7564101d20 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7561d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7561d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55500/0 (socket says 172.21.15.111:55500) 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f7561d9b700 1 -- 172.21.15.111:0/3161856866 learned_addr learned my addr 172.21.15.111:0/3161856866 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:07:55.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.759+0000 7f756259c700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7564101d20 0x7f7564102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7561d9b700 1 -- 172.21.15.111:0/3161856866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7564101d20 msgr2=0x7f7564102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7561d9b700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7564101d20 0x7f7564102b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7561d9b700 1 -- 172.21.15.111:0/3161856866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558005040 con 0x7f75641026c0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7561d9b700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7554002a50 tx=0x7f7554004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7554007cc0 con 0x7f75641026c0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f755400f040 con 0x7f75641026c0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755400d920 con 0x7f75641026c0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75641038b0 con 0x7f75641026c0 2024-03-14T13:07:55.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.760+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756411a000 con 0x7f75641026c0 2024-03-14T13:07:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.761+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75640fd470 con 0x7f75641026c0 2024-03-14T13:07:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.761+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7554009680 con 0x7f75641026c0 2024-03-14T13:07:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.762+0000 7f7552ffd700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 0x7f754c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:07:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.762+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f755408b680 con 0x7f75641026c0 2024-03-14T13:07:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.762+0000 7f756259c700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 0x7f754c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:07:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.763+0000 7f756259c700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 0x7f754c06ec70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7558004f90 tx=0x7f7558009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:07:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:55.767+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f75540559a0 con 0x7f75641026c0 2024-03-14T13:07:56.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.045+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f756411a240 con 0x7f75641026c0 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.046+0000 7f7552ffd700 1 -- 172.21.15.111:0/3161856866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+437 (secure 0 0 0) 0x7f7554058fc0 con 0x7f75641026c0 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 3 pgs peering 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: osd.2 (root=default,host=smithi111) is down 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs peering 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1 is stuck peering for 4m, current state peering, last acting [1,0] 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: pg 3.7 is stuck peering for 4m, current state peering, last acting [1,4] 2024-03-14T13:07:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: pg 3.8 is stuck peering for 4m, current state peering, last acting [1,5] 2024-03-14T13:07:56.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 msgr2=0x7f754c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 0x7f754c06ec70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7558004f90 tx=0x7f7558009e30 comp rx=0 tx=0).stop 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 msgr2=0x7f75641030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7554002a50 tx=0x7f7554004600 comp rx=0 tx=0).stop 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 shutdown_connections 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f754c06c7c0 0x7f754c06ec70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7564101d20 0x7f7564102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 --2- 172.21.15.111:0/3161856866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75641026c0 0x7f75641030c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 >> 172.21.15.111:0/3161856866 conn(0x7f75640f7870 msgr2=0x7f7564066200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 shutdown_connections 2024-03-14T13:07:56.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:07:56.048+0000 7f7568acf700 1 -- 172.21.15.111:0/3161856866 wait complete. 2024-03-14T13:07:56.140 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:55 smithi111 ceph-osd[174075]: -- 172.21.15.111:0/945971644 <== mon.0 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5623a4e171e0 con 0x5623a4de6c00 2024-03-14T13:07:56.501 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-mon[145314]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:56.502 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3622771632' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:07:56.502 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3161856866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:07:56.502 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:56.502 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:56.502 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:56 smithi111 bash[174015]: 206403cb01bafef07c596711783df51327226efea8127918be3cbef76a63ecdf 2024-03-14T13:07:56.502 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:56 smithi111 systemd[1]: Started Ceph osd.2 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:07:56.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:56 smithi137 ceph-mon[108145]: from='client.34258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:56 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3622771632' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:07:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:56 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3161856866' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:07:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:56.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:07:56.890 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:07:56 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:07:56.501+0000 7f8f0b236640 -1 Falling back to public interface 2024-03-14T13:07:57.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:57 smithi111 ceph-mon[145314]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:57.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:57 smithi111 ceph-mon[145314]: pgmap v91: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:07:57.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:57 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:57.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:57 smithi111 ceph-mon[145314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-14T13:07:57.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:57 smithi137 ceph-mon[108145]: from='client.34270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:07:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:57 smithi137 ceph-mon[108145]: pgmap v91: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:07:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:57 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 39/321 objects degraded (12.150%), 12 pgs degraded (PG_DEGRADED) 2024-03-14T13:07:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:57 smithi137 ceph-mon[108145]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-14T13:07:57.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:07:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:07:59 smithi111 ceph-mon[145314]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:07:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:07:59 smithi137 ceph-mon[108145]: pgmap v92: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:08:00.883 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:08:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:08:00.416+0000 7f8f0b236640 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-14T13:08:01.140 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:08:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:08:00.882+0000 7f8f0b236640 -1 osd.2 60 log_to_monitors true 2024-03-14T13:08:01.640 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:08:01 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:08:01.413+0000 7f8efa68b700 -1 osd.2 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:08:01.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:01 smithi111 ceph-mon[145314]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:08:01.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:01 smithi111 ceph-mon[145314]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T13:08:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:01 smithi137 ceph-mon[108145]: pgmap v93: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:08:01.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:01 smithi137 ceph-mon[108145]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-14T13:08:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:02 smithi137 ceph-mon[108145]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T13:08:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:02 smithi137 ceph-mon[108145]: osdmap e63: 6 total, 5 up, 6 in 2024-03-14T13:08:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:02 smithi137 ceph-mon[108145]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:08:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:02 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:02.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:02 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:02 smithi111 ceph-mon[145314]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-14T13:08:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:02 smithi111 ceph-mon[145314]: osdmap e63: 6 total, 5 up, 6 in 2024-03-14T13:08:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:02 smithi111 ceph-mon[145314]: from='osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-03-14T13:08:02.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:02 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:02.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:02 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:03.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:08:03.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:08:03.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195] boot 2024-03-14T13:08:03.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: osdmap e64: 6 total, 6 up, 6 in 2024-03-14T13:08:03.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:08:03.673 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:03.673 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:03.673 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:03 smithi111 ceph-mon[145314]: osdmap e65: 6 total, 6 up, 6 in 2024-03-14T13:08:03.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: pgmap v95: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 39/321 objects degraded (12.150%) 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: osd.2 [v2:172.21.15.111:6818/3541679195,v1:172.21.15.111:6819/3541679195] boot 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: osdmap e64: 6 total, 6 up, 6 in 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:03 smithi137 ceph-mon[108145]: osdmap e65: 6 total, 6 up, 6 in 2024-03-14T13:08:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:05 smithi111 ceph-mon[145314]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 39/321 objects degraded (12.150%) 2024-03-14T13:08:05.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:05 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:05 smithi137 ceph-mon[108145]: pgmap v98: 65 pgs: 12 active+undersized+degraded, 15 active+undersized, 38 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s; 39/321 objects degraded (12.150%) 2024-03-14T13:08:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:05 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-14T13:08:07.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:07 smithi111 ceph-mon[145314]: pgmap v99: 65 pgs: 4 active+undersized+degraded, 5 active+undersized, 56 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 14/321 objects degraded (4.361%) 2024-03-14T13:08:07.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:07.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-14T13:08:07.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:07 smithi137 ceph-mon[108145]: pgmap v99: 65 pgs: 4 active+undersized+degraded, 5 active+undersized, 56 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 14/321 objects degraded (4.361%) 2024-03-14T13:08:08.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:08 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 14/321 objects degraded (4.361%), 4 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:08.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:08 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 14/321 objects degraded (4.361%), 4 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:09.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:09 smithi111 ceph-mon[145314]: pgmap v100: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-14T13:08:09.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:09 smithi137 ceph-mon[108145]: pgmap v100: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 op/s 2024-03-14T13:08:10.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:10 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/321 objects degraded (4.361%), 4 pgs degraded) 2024-03-14T13:08:10.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:10 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:08:10.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:10.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:10 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:10.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:10 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/321 objects degraded (4.361%), 4 pgs degraded) 2024-03-14T13:08:10.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:10 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:08:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:10.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:10 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:11.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:11 smithi111 ceph-mon[145314]: pgmap v101: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-14T13:08:11.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:11 smithi137 ceph-mon[108145]: pgmap v101: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1 op/s 2024-03-14T13:08:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:13 smithi111 ceph-mon[145314]: pgmap v102: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-14T13:08:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:13 smithi137 ceph-mon[108145]: pgmap v102: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-14T13:08:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:15 smithi111 ceph-mon[145314]: pgmap v103: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-14T13:08:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:15 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:15 smithi137 ceph-mon[108145]: pgmap v103: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-14T13:08:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:15 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:17.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:17 smithi111 ceph-mon[145314]: pgmap v104: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-14T13:08:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:17 smithi137 ceph-mon[108145]: pgmap v104: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-14T13:08:19.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:19 smithi111 ceph-mon[145314]: pgmap v105: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:08:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:19 smithi137 ceph-mon[108145]: pgmap v105: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:08:21.726 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:21 smithi137 ceph-mon[108145]: pgmap v106: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:08:21.726 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:21 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:21 smithi111 ceph-mon[145314]: pgmap v106: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:08:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:21 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:22.686 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:22 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:22.686 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:22 smithi137 ceph-mon[108145]: Upgrade: osd.3 is safe to restart 2024-03-14T13:08:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:22 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-14T13:08:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:22 smithi111 ceph-mon[145314]: Upgrade: osd.3 is safe to restart 2024-03-14T13:08:23.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:23 smithi137 ceph-mon[108145]: pgmap v107: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:08:23.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:23.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T13:08:23.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:23 smithi111 ceph-mon[145314]: pgmap v107: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:08:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-14T13:08:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:24 smithi137 ceph-mon[108145]: Upgrade: Updating osd.3 2024-03-14T13:08:24.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:24 smithi137 ceph-mon[108145]: Deploying daemon osd.3 on smithi137 2024-03-14T13:08:24.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:24 smithi111 ceph-mon[145314]: Upgrade: Updating osd.3 2024-03-14T13:08:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:24 smithi111 ceph-mon[145314]: Deploying daemon osd.3 on smithi137 2024-03-14T13:08:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:25.441 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:25 smithi137 ceph-mon[108145]: pgmap v108: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:08:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:25 smithi111 ceph-mon[145314]: pgmap v108: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:08:26.006 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:25 smithi137 systemd[1]: Stopping Ceph osd.3 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:08:26.006 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:25 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[56679]: 2024-03-14T13:08:25.951+0000 7fdd859e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:08:26.006 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:25 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[56679]: 2024-03-14T13:08:25.951+0000 7fdd859e0700 -1 osd.3 65 *** Got signal Terminated *** 2024-03-14T13:08:26.006 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:25 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[56679]: 2024-03-14T13:08:25.951+0000 7fdd859e0700 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:08:26.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/1526961194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fde90 msgr2=0x7f2f080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/1526961194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fde90 0x7f2f080fe260 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009ce0 tx=0x7f2ef8004d10 comp rx=0 tx=0).stop 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/1526961194 shutdown_connections 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/1526961194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fe830 0x7f2f08106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/1526961194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fde90 0x7f2f080fe260 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/1526961194 >> 172.21.15.111:0/1526961194 conn(0x7f2f08072bc0 msgr2=0x7f2f08072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/1526961194 shutdown_connections 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.162+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/1526961194 wait complete. 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 Processor -- start 2024-03-14T13:08:26.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 -- start start 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fde90 0x7f2f08113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08114600 con 0x7f2f080fe830 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.163+0000 7f2f0d1b1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0810d9f0 con 0x7f2f080fde90 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59464/0 (socket says 172.21.15.111:59464) 2024-03-14T13:08:26.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 -- 172.21.15.111:0/2930597228 learned_addr learned my addr 172.21.15.111:0/2930597228 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f06d9d700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fde90 0x7f2f08113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 -- 172.21.15.111:0/2930597228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fde90 msgr2=0x7f2f08113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fde90 0x7f2f08113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 -- 172.21.15.111:0/2930597228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8009940 con 0x7f2f080fe830 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0659c700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0002a50 tx=0x7f2ef00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef0007cc0 con 0x7f2f080fe830 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0810dcd0 con 0x7f2f080fe830 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ef000f040 con 0x7f2f080fe830 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ef000d9c0 con 0x7f2f080fe830 2024-03-14T13:08:26.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.164+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0810e210 con 0x7f2f080fe830 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.165+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f08065e30 con 0x7f2f080fe830 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.166+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2ef000db20 con 0x7f2f080fe830 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.166+0000 7f2eff7fe700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 0x7f2ef406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.166+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2ef008b700 con 0x7f2f080fe830 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.166+0000 7f2f06d9d700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 0x7f2ef406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.167+0000 7f2f06d9d700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 0x7f2ef406ec70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8015040 tx=0x7f2ef8004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.171+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2ef0059430 con 0x7f2f080fe830 2024-03-14T13:08:26.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.388+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f08109dc0 con 0x7f2ef406c7c0 2024-03-14T13:08:26.389 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.390+0000 7f2eff7fe700 1 -- 172.21.15.111:0/2930597228 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2f08109dc0 con 0x7f2ef406c7c0 2024-03-14T13:08:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.391+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 msgr2=0x7f2ef406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.391+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 0x7f2ef406ec70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8015040 tx=0x7f2ef8004d10 comp rx=0 tx=0).stop 2024-03-14T13:08:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.391+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 msgr2=0x7f2f08113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.391+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2ef0002a50 tx=0x7f2ef00046b0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 shutdown_connections 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2ef406c7c0 0x7f2ef406ec70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f080fde90 0x7f2f08113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 --2- 172.21.15.111:0/2930597228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2f080fe830 0x7f2f08113fb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 >> 172.21.15.111:0/2930597228 conn(0x7f2f08072bc0 msgr2=0x7f2f081037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 shutdown_connections 2024-03-14T13:08:26.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.392+0000 7f2f0d1b1700 1 -- 172.21.15.111:0/2930597228 wait complete. 2024-03-14T13:08:26.401 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:08:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 -- 172.21.15.111:0/1923154416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e4101d20 msgr2=0x7f39e41020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 --2- 172.21.15.111:0/1923154416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e4101d20 0x7f39e41020f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39d4004940 tx=0x7f39d4009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:26.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 -- 172.21.15.111:0/1923154416 shutdown_connections 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 --2- 172.21.15.111:0/1923154416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e41026c0 0x7f39e40fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 --2- 172.21.15.111:0/1923154416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e4101d20 0x7f39e41020f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 -- 172.21.15.111:0/1923154416 >> 172.21.15.111:0/1923154416 conn(0x7f39e40f7870 msgr2=0x7f39e40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 -- 172.21.15.111:0/1923154416 shutdown_connections 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.500+0000 7f39e97e5700 1 -- 172.21.15.111:0/1923154416 wait complete. 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 Processor -- start 2024-03-14T13:08:26.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 -- start start 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e4101d20 0x7f39e410ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e410f480 con 0x7f39e41026c0 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e97e5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e410f5c0 con 0x7f39e4101d20 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59488/0 (socket says 172.21.15.111:59488) 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.501+0000 7f39e27fc700 1 -- 172.21.15.111:0/3958335733 learned_addr learned my addr 172.21.15.111:0/3958335733 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e2ffd700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e4101d20 0x7f39e410ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e27fc700 1 -- 172.21.15.111:0/3958335733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e4101d20 msgr2=0x7f39e410ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e27fc700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e4101d20 0x7f39e410ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e27fc700 1 -- 172.21.15.111:0/3958335733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4005040 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e27fc700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39d800b790 tx=0x7f39d8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d800f6f0 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39d800fd00 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e4117a70 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39d80103a0 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.502+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e4117ef0 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.503+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e40fd470 con 0x7f39e41026c0 2024-03-14T13:08:26.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.503+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f39d800f850 con 0x7f39e41026c0 2024-03-14T13:08:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.504+0000 7f39cb7fe700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 0x7f39cc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.504+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f39d808b7e0 con 0x7f39e41026c0 2024-03-14T13:08:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.504+0000 7f39e2ffd700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 0x7f39cc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.504+0000 7f39e2ffd700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 0x7f39cc06ec70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39d4004f90 tx=0x7f39d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.508+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f39d8055b00 con 0x7f39e41026c0 2024-03-14T13:08:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.719+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f39e40f8ff0 con 0x7f39cc06c7c0 2024-03-14T13:08:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.721+0000 7f39cb7fe700 1 -- 172.21.15.111:0/3958335733 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f39e40f8ff0 con 0x7f39cc06c7c0 2024-03-14T13:08:26.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 msgr2=0x7f39cc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 0x7f39cc06ec70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f39d4004f90 tx=0x7f39d4009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 msgr2=0x7f39e410ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f39d800b790 tx=0x7f39d8009220 comp rx=0 tx=0).stop 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 shutdown_connections 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39cc06c7c0 0x7f39cc06ec70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39e4101d20 0x7f39e410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 --2- 172.21.15.111:0/3958335733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39e41026c0 0x7f39e410ef40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 >> 172.21.15.111:0/3958335733 conn(0x7f39e40f7870 msgr2=0x7f39e40f88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 shutdown_connections 2024-03-14T13:08:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.723+0000 7f39e97e5700 1 -- 172.21.15.111:0/3958335733 wait complete. 2024-03-14T13:08:26.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:26 smithi137 ceph-mon[108145]: osd.3 marked itself down and dead 2024-03-14T13:08:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- 172.21.15.111:0/2867095380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 msgr2=0x7fc4600fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 --2- 172.21.15.111:0/2867095380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc4600fbd80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc454004940 tx=0x7fc454009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- 172.21.15.111:0/2867095380 shutdown_connections 2024-03-14T13:08:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 --2- 172.21.15.111:0/2867095380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc4600fbd80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 --2- 172.21.15.111:0/2867095380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 0x7fc460102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- 172.21.15.111:0/2867095380 >> 172.21.15.111:0/2867095380 conn(0x7fc460072bc0 msgr2=0x7fc460072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- 172.21.15.111:0/2867095380 shutdown_connections 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- 172.21.15.111:0/2867095380 wait complete. 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 Processor -- start 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.824+0000 7fc466f64700 1 -- start start 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc466f64700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 0x7fc460113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc466f64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc466f64700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc460114670 con 0x7fc460102710 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc466f64700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46010da60 con 0x7fc460101d70 2024-03-14T13:08:26.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59504/0 (socket says 172.21.15.111:59504) 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 -- 172.21.15.111:0/2843158072 learned_addr learned my addr 172.21.15.111:0/2843158072 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc464d00700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 0x7fc460113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 -- 172.21.15.111:0/2843158072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 msgr2=0x7fc460113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 0x7fc460113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 -- 172.21.15.111:0/2843158072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc454005040 con 0x7fc460102710 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45ffff700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc454009380 tx=0x7fc454009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.825+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45400e040 con 0x7fc460102710 2024-03-14T13:08:26.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.826+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc454013420 con 0x7fc460102710 2024-03-14T13:08:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.826+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46010dce0 con 0x7fc460102710 2024-03-14T13:08:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.826+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc45401c630 con 0x7fc460102710 2024-03-14T13:08:26.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.826+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46010e2a0 con 0x7fc460102710 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.826+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4600fd4c0 con 0x7fc460102710 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.827+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc4540139b0 con 0x7fc460102710 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.827+0000 7fc45d7fa700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 0x7fc44806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.827+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc45408cb40 con 0x7fc460102710 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.827+0000 7fc464d00700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 0x7fc44806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:26.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.828+0000 7fc464d00700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 0x7fc44806ec70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc458001450 tx=0x7fc45800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:26.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:26.832+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc45405a8f0 con 0x7fc460102710 2024-03-14T13:08:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:26 smithi111 ceph-mon[145314]: osd.3 marked itself down and dead 2024-03-14T13:08:27.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.032+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc460108d30 con 0x7fc44806c7c0 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.041+0000 7fc45d7fa700 1 -- 172.21.15.111:0/2843158072 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fc460108d30 con 0x7fc44806c7c0 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (4m) 25s ago 16m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (2m) 25s ago 16m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (2m) 2m ago 14m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (4m) 25s ago 15m 89.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (10m) 25s ago 10m 64.2M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (10m) 25s ago 10m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:08:27.041 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (10m) 2m ago 9m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (10m) 2m ago 10m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (7m) 25s ago 17m 501M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (5m) 2m ago 14m 449M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (3m) 25s ago 17m 51.8M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (2m) 2m ago 14m 37.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (5m) 25s ago 16m 19.0M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (5m) 2m ago 14m 18.1M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (104s) 25s ago 14m 84.2M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (67s) 25s ago 13m 91.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (31s) 25s ago 13m 15.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (12m) 2m ago 12m 267M 2592M 17.2.7-646-gb92acd1f a07fb176315d 150a88d208c1 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (12m) 2m ago 12m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:08:27.042 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (11m) 2m ago 11m 219M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:08:27.043 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (5m) 25s ago 15m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:08:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 msgr2=0x7fc44806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 0x7fc44806ec70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc458001450 tx=0x7fc45800b040 comp rx=0 tx=0).stop 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 msgr2=0x7fc460114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc454009380 tx=0x7fc454009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 shutdown_connections 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc44806c7c0 0x7fc44806ec70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc460101d70 0x7fc460113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 --2- 172.21.15.111:0/2843158072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc460102710 0x7fc460114020 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.043+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 >> 172.21.15.111:0/2843158072 conn(0x7fc460072bc0 msgr2=0x7fc460065a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.044+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 shutdown_connections 2024-03-14T13:08:27.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.044+0000 7fc466f64700 1 -- 172.21.15.111:0/2843158072 wait complete. 2024-03-14T13:08:27.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 -- 172.21.15.111:0/3260008435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b4102e00 msgr2=0x7f57b41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 --2- 172.21.15.111:0/3260008435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b4102e00 0x7f57b41031d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f57a4004940 tx=0x7f57a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 -- 172.21.15.111:0/3260008435 shutdown_connections 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 --2- 172.21.15.111:0/3260008435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b41037a0 0x7f57b40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 --2- 172.21.15.111:0/3260008435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b4102e00 0x7f57b41031d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 -- 172.21.15.111:0/3260008435 >> 172.21.15.111:0/3260008435 conn(0x7f57b40f89b0 msgr2=0x7f57b40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 -- 172.21.15.111:0/3260008435 shutdown_connections 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.142+0000 7f57bbe60700 1 -- 172.21.15.111:0/3260008435 wait complete. 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 Processor -- start 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 -- start start 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b4102e00 0x7f57b4113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4114650 con 0x7f57b41037a0 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57bbe60700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b410da40 con 0x7f57b4102e00 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57b93fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57b93fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59532/0 (socket says 172.21.15.111:59532) 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.143+0000 7f57b93fb700 1 -- 172.21.15.111:0/1234894360 learned_addr learned my addr 172.21.15.111:0/1234894360 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57b93fb700 1 -- 172.21.15.111:0/1234894360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b4102e00 msgr2=0x7f57b4113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57b93fb700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b4102e00 0x7f57b4113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57b93fb700 1 -- 172.21.15.111:0/1234894360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4005040 con 0x7f57b41037a0 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57b93fb700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c800 tx=0x7f57b0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b000a760 con 0x7f57b41037a0 2024-03-14T13:08:27.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57b000ad70 con 0x7f57b41037a0 2024-03-14T13:08:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b410dd20 con 0x7f57b41037a0 2024-03-14T13:08:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57b0012e80 con 0x7f57b41037a0 2024-03-14T13:08:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.144+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b410e2e0 con 0x7f57b41037a0 2024-03-14T13:08:27.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.145+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b40fe550 con 0x7f57b41037a0 2024-03-14T13:08:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.145+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f57b000a8c0 con 0x7f57b41037a0 2024-03-14T13:08:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.146+0000 7f57aa7fc700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 0x7f57a006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.146+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f57b008b9d0 con 0x7f57b41037a0 2024-03-14T13:08:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.146+0000 7f57b9bfc700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 0x7f57a006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.146+0000 7f57b9bfc700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 0x7f57a006ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57a4004f90 tx=0x7f57a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.150+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f57b0059820 con 0x7f57b41037a0 2024-03-14T13:08:27.414 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:27 smithi137 bash[120593]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3 2024-03-14T13:08:27.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.430+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f57b4109dc0 con 0x7f57b41037a0 2024-03-14T13:08:27.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.433+0000 7f57aa7fc700 1 -- 172.21.15.111:0/1234894360 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7f57b00593b0 con 0x7f57b41037a0 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:27.436 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 2, 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 3 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 6, 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 7 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:08:27.437 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:08:27.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.437+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 msgr2=0x7f57a006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.437+0000 7f57bbe60700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 0x7f57a006ec70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57a4004f90 tx=0x7f57a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.437+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 msgr2=0x7f57b4114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.437+0000 7f57bbe60700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c800 tx=0x7f57b0004470 comp rx=0 tx=0).stop 2024-03-14T13:08:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.445+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 shutdown_connections 2024-03-14T13:08:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.445+0000 7f57bbe60700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f57a006c7c0 0x7f57a006ec70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.445+0000 7f57bbe60700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57b4102e00 0x7f57b4113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.445+0000 7f57bbe60700 1 --2- 172.21.15.111:0/1234894360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57b41037a0 0x7f57b4114000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.445+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 >> 172.21.15.111:0/1234894360 conn(0x7f57b40f89b0 msgr2=0x7f57b40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.450+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 shutdown_connections 2024-03-14T13:08:27.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.450+0000 7f57bbe60700 1 -- 172.21.15.111:0/1234894360 wait complete. 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- 172.21.15.111:0/2122513512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 msgr2=0x7fc97c0fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 --2- 172.21.15.111:0/2122513512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c0fbfa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc96c004940 tx=0x7fc96c009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- 172.21.15.111:0/2122513512 shutdown_connections 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 --2- 172.21.15.111:0/2122513512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 --2- 172.21.15.111:0/2122513512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c0fbfa0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- 172.21.15.111:0/2122513512 >> 172.21.15.111:0/2122513512 conn(0x7fc97c0f7880 msgr2=0x7fc97c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- 172.21.15.111:0/2122513512 shutdown_connections 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- 172.21.15.111:0/2122513512 wait complete. 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 Processor -- start 2024-03-14T13:08:27.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.551+0000 7fc983a7b700 1 -- start start 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c1163a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1169f0 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c10fdc0 con 0x7fc97c0fc4e0 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59550/0 (socket says 172.21.15.111:59550) 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 -- 172.21.15.111:0/3932591379 learned_addr learned my addr 172.21.15.111:0/3932591379 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981016700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c1163a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 -- 172.21.15.111:0/3932591379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 msgr2=0x7fc97c1163a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c1163a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 -- 172.21.15.111:0/3932591379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c005040 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981016700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c1163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc981817700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc96c000c00 tx=0x7fc96c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc96c021070 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c110040 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc96c013e00 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc96c00e070 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.552+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c110600 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.553+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c106970 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.554+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc96c01c690 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.554+0000 7fc9727fc700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 0x7fc96806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.554+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc96c08cb80 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.554+0000 7fc981016700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 0x7fc96806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.555+0000 7fc981016700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 0x7fc96806ec70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc978004600 tx=0x7fc97800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.559+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc96c05a930 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:27 smithi137 ceph-mon[108145]: pgmap v109: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:08:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:27 smithi137 ceph-mon[108145]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:27.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:27 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:08:27.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:27 smithi137 ceph-mon[108145]: osdmap e66: 6 total, 5 up, 6 in 2024-03-14T13:08:27.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.790+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc97c10bee0 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.791+0000 7fc9727fc700 1 -- 172.21.15.111:0/3932591379 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7fc96c05a4c0 con 0x7fc97c0fbbd0 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:08:27.791 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:08:27.792 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:08:27.793 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 msgr2=0x7fc96806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 0x7fc96806ec70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc978004600 tx=0x7fc97800b040 comp rx=0 tx=0).stop 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 msgr2=0x7fc97c115e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc96c000c00 tx=0x7fc96c009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 shutdown_connections 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc96806c7c0 0x7fc96806ec70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc97c0fbbd0 0x7fc97c115e60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 --2- 172.21.15.111:0/3932591379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc97c0fc4e0 0x7fc97c1163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 >> 172.21.15.111:0/3932591379 conn(0x7fc97c0f7880 msgr2=0x7fc97c0f7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 shutdown_connections 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.793+0000 7fc983a7b700 1 -- 172.21.15.111:0/3932591379 wait complete. 2024-03-14T13:08:27.794 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:08:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:27 smithi111 ceph-mon[145314]: pgmap v109: 65 pgs: 65 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:08:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:27 smithi111 ceph-mon[145314]: from='client.34282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:27 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:08:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:27 smithi111 ceph-mon[145314]: osdmap e66: 6 total, 5 up, 6 in 2024-03-14T13:08:27.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- 172.21.15.111:0/1615300547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60101d70 msgr2=0x7f6b60102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/1615300547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60101d70 0x7f6b60102140 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6b54004940 tx=0x7f6b54009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- 172.21.15.111:0/1615300547 shutdown_connections 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/1615300547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60102710 0x7f6b600fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/1615300547 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60101d70 0x7f6b60102140 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- 172.21.15.111:0/1615300547 >> 172.21.15.111:0/1615300547 conn(0x7f6b60072bc0 msgr2=0x7f6b60072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- 172.21.15.111:0/1615300547 shutdown_connections 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- 172.21.15.111:0/1615300547 wait complete. 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 Processor -- start 2024-03-14T13:08:27.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.894+0000 7f6b66ec0700 1 -- start start 2024-03-14T13:08:27.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60101d70 0x7f6b60102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601036a0 con 0x7f6b60102710 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b601037e0 con 0x7f6b60101d70 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59566/0 (socket says 172.21.15.111:59566) 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 -- 172.21.15.111:0/3129529035 learned_addr learned my addr 172.21.15.111:0/3129529035 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b64c5c700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60101d70 0x7f6b60102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 -- 172.21.15.111:0/3129529035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60101d70 msgr2=0x7f6b60102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60101d70 0x7f6b60102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 -- 172.21.15.111:0/3129529035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b54005040 con 0x7f6b60102710 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5ffff700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00c890 tx=0x7f6b4c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b4c00a700 con 0x7f6b60102710 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b60119d60 con 0x7f6b60102710 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b4c00ad10 con 0x7f6b60102710 2024-03-14T13:08:27.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b4c012e70 con 0x7f6b60102710 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.895+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6011a1e0 con 0x7f6b60102710 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.896+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b600fd4c0 con 0x7f6b60102710 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.897+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6b4c00a860 con 0x7f6b60102710 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.897+0000 7f6b5d7fa700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 0x7f6b5006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.897+0000 7f6b64c5c700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 0x7f6b5006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.897+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f6b4c08ba40 con 0x7f6b60102710 2024-03-14T13:08:27.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.898+0000 7f6b64c5c700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 0x7f6b5006ec70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b54004f90 tx=0x7f6b54009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:27.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:27.902+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6b4c055d60 con 0x7f6b60102710 2024-03-14T13:08:28.111 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.111+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b6004bed0 con 0x7f6b5006c7c0 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.113+0000 7f6b5d7fa700 1 -- 172.21.15.111:0/3129529035 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f6b6004bed0 con 0x7f6b5006c7c0 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:08:28.112 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "9/21 daemons upgraded", 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:08:28.113 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:08:28.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 msgr2=0x7f6b5006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.114 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 0x7f6b5006ec70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6b54004f90 tx=0x7f6b54009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 msgr2=0x7f6b60103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00c890 tx=0x7f6b4c0044c0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 shutdown_connections 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6b5006c7c0 0x7f6b5006ec70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6b60101d70 0x7f6b60102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 --2- 172.21.15.111:0/3129529035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b60102710 0x7f6b60103110 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 >> 172.21.15.111:0/3129529035 conn(0x7f6b60072bc0 msgr2=0x7f6b600659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 shutdown_connections 2024-03-14T13:08:28.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.115+0000 7f6b66ec0700 1 -- 172.21.15.111:0/3129529035 wait complete. 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- 172.21.15.111:0/1114463548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 msgr2=0x7f08341020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 --2- 172.21.15.111:0/1114463548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f08341020e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0824004940 tx=0x7f0824009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- 172.21.15.111:0/1114463548 shutdown_connections 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 --2- 172.21.15.111:0/1114463548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08340fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 --2- 172.21.15.111:0/1114463548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f08341020e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- 172.21.15.111:0/1114463548 >> 172.21.15.111:0/1114463548 conn(0x7f08340f7880 msgr2=0x7f08340f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- 172.21.15.111:0/1114463548 shutdown_connections 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- 172.21.15.111:0/1114463548 wait complete. 2024-03-14T13:08:28.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 Processor -- start 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- start start 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08341131d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0834113820 con 0x7f0834101d10 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f0838817700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083410cbf0 con 0x7f08341026b0 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.215+0000 7f083259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59584/0 (socket says 172.21.15.111:59584) 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 -- 172.21.15.111:0/3088705752 learned_addr learned my addr 172.21.15.111:0/3088705752 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0831d9b700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08341131d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 -- 172.21.15.111:0/3088705752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 msgr2=0x7f08341131d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08341131d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 -- 172.21.15.111:0/3088705752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824005040 con 0x7f0834101d10 2024-03-14T13:08:28.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0831d9b700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08341131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f083259c700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0824000c00 tx=0x7f0824009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0824021070 con 0x7f0834101d10 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0824013de0 con 0x7f0834101d10 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083410ce70 con 0x7f0834101d10 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f082400e070 con 0x7f0834101d10 2024-03-14T13:08:28.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.216+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083410d3d0 con 0x7f0834101d10 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.217+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08340fd460 con 0x7f0834101d10 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.218+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f082401c600 con 0x7f0834101d10 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.218+0000 7f0822ffd700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 0x7f081c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.218+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f082408c9c0 con 0x7f0834101d10 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.218+0000 7f0831d9b700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 0x7f081c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:28.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.218+0000 7f0831d9b700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 0x7f081c06ec70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0828004600 tx=0x7f082800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:28.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.222+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f082405a810 con 0x7f0834101d10 2024-03-14T13:08:28.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.498+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f083405f5f0 con 0x7f0834101d10 2024-03-14T13:08:28.498 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.498+0000 7f0822ffd700 1 -- 172.21.15.111:0/3088705752 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+100 (secure 0 0 0) 0x7f082405a3a0 con 0x7f0834101d10 2024-03-14T13:08:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN 1 osds down 2024-03-14T13:08:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-14T13:08:28.498 INFO:teuthology.orchestra.run.smithi111.stdout: osd.3 (root=default,host=smithi137) is down 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 msgr2=0x7f081c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 0x7f081c06ec70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0828004600 tx=0x7f082800b040 comp rx=0 tx=0).stop 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 msgr2=0x7f0834112c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0824000c00 tx=0x7f0824009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 shutdown_connections 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f081c06c7c0 0x7f081c06ec70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0834101d10 0x7f0834112c90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 --2- 172.21.15.111:0/3088705752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08341026b0 0x7f08341131d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 >> 172.21.15.111:0/3088705752 conn(0x7f08340f7880 msgr2=0x7f08340f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 shutdown_connections 2024-03-14T13:08:28.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:28.500+0000 7f0838817700 1 -- 172.21.15.111:0/3088705752 wait complete. 2024-03-14T13:08:28.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:28 smithi137 ceph-mon[108145]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:28.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:28 smithi137 ceph-mon[108145]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:28.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:28 smithi137 ceph-mon[108145]: osdmap e67: 6 total, 5 up, 6 in 2024-03-14T13:08:28.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:28 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1234894360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:28.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:28 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3932591379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:08:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:28 smithi111 ceph-mon[145314]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:28 smithi111 ceph-mon[145314]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:28 smithi111 ceph-mon[145314]: osdmap e67: 6 total, 5 up, 6 in 2024-03-14T13:08:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:28 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1234894360' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:28 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3932591379' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:08:29.256 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:28 smithi137 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.3.service: Succeeded. 2024-03-14T13:08:29.256 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:28 smithi137 systemd[1]: Stopped Ceph osd.3 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:08:29.256 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:29 smithi137 systemd[1]: Starting Ceph osd.3 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:08:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:29 smithi137 ceph-mon[108145]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:29.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:29 smithi137 ceph-mon[108145]: pgmap v112: 65 pgs: 16 stale+active+clean, 49 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-03-14T13:08:29.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:29 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3088705752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:08:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:29 smithi111 ceph-mon[145314]: from='client.34302 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:08:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:29 smithi111 ceph-mon[145314]: pgmap v112: 65 pgs: 16 stale+active+clean, 49 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 3 op/s 2024-03-14T13:08:29.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:29 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3088705752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:08:30.396 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-14T13:08:30.396 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-14T13:08:30.396 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-14T13:08:30.396 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-14T13:08:30.396 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-03-14T13:08:30.397 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-14T13:08:30.397 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:30 smithi137 bash[120869]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-14T13:08:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:30 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 45/321 objects degraded (14.019%), 12 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:30 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 45/321 objects degraded (14.019%), 12 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:31.480 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-osd[121166]: -- 172.21.15.137:0/2000930274 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a53a8151e0 con 0x55a53a7e5000 2024-03-14T13:08:31.480 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:31 smithi137 bash[121128]: 9df61411961eba36bd5876d07afe8e5de4de3923c9aa4e3aabe70bc181d5f140 2024-03-14T13:08:31.480 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:31 smithi137 systemd[1]: Started Ceph osd.3 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:08:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-mon[108145]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 3 stale+active+clean, 13 active+undersized, 37 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 45/321 objects degraded (14.019%) 2024-03-14T13:08:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:31 smithi111 ceph-mon[145314]: pgmap v113: 65 pgs: 12 active+undersized+degraded, 3 stale+active+clean, 13 active+undersized, 37 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 45/321 objects degraded (14.019%) 2024-03-14T13:08:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:31 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:31 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:31.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:31 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:32.005 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:31 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:08:31.754+0000 7f4b9f753640 -1 Falling back to public interface 2024-03-14T13:08:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:33 smithi137 ceph-mon[108145]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:33 smithi111 ceph-mon[145314]: pgmap v114: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:35.366 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.366 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.366 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:35 smithi137 ceph-mon[108145]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:35.366 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.366 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:35 smithi111 ceph-mon[145314]: pgmap v115: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:35.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:35.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:36.755 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:36 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:08:36.311+0000 7f4b9f753640 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-14T13:08:37.256 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:36 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:08:36.816+0000 7f4b9f753640 -1 osd.3 65 log_to_monitors true 2024-03-14T13:08:37.600 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:08:37 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:08:37.327+0000 7f4b8e3a7700 -1 osd.3 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:08:37.600 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:37 smithi137 ceph-mon[108145]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:37.600 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:37 smithi137 ceph-mon[108145]: from='osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T13:08:37.600 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:37 smithi137 ceph-mon[108145]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T13:08:37.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:37 smithi111 ceph-mon[145314]: pgmap v116: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:37.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:37 smithi111 ceph-mon[145314]: from='osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T13:08:37.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:37 smithi111 ceph-mon[145314]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-14T13:08:38.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: osdmap e68: 6 total, 5 up, 6 in 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:08:38.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:38 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: osdmap e68: 6 total, 5 up, 6 in 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:08:38.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:38 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:08:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476] boot 2024-03-14T13:08:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: osdmap e69: 6 total, 6 up, 6 in 2024-03-14T13:08:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:39.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:39 smithi111 ceph-mon[145314]: osdmap e70: 6 total, 6 up, 6 in 2024-03-14T13:08:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: pgmap v118: 65 pgs: 19 active+undersized+degraded, 19 active+undersized, 27 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 66/321 objects degraded (20.561%) 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: osd.3 [v2:172.21.15.137:6800/137240476,v1:172.21.15.137:6801/137240476] boot 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: osdmap e69: 6 total, 6 up, 6 in 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-03-14T13:08:39.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:39.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:39.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:39 smithi137 ceph-mon[108145]: osdmap e70: 6 total, 6 up, 6 in 2024-03-14T13:08:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:40 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 66/321 objects degraded (20.561%), 19 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:40.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:40 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 66/321 objects degraded (20.561%), 19 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:41 smithi137 ceph-mon[108145]: pgmap v121: 65 pgs: 15 active+undersized+degraded, 11 active+undersized, 39 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s; 54/321 objects degraded (16.822%) 2024-03-14T13:08:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:41 smithi111 ceph-mon[145314]: pgmap v121: 65 pgs: 15 active+undersized+degraded, 11 active+undersized, 39 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s; 54/321 objects degraded (16.822%) 2024-03-14T13:08:43.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:43 smithi137 ceph-mon[108145]: pgmap v122: 65 pgs: 10 active+undersized+degraded, 6 active+undersized, 49 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 3 op/s; 40/321 objects degraded (12.461%) 2024-03-14T13:08:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:43 smithi111 ceph-mon[145314]: pgmap v122: 65 pgs: 10 active+undersized+degraded, 6 active+undersized, 49 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 682 B/s rd, 3 op/s; 40/321 objects degraded (12.461%) 2024-03-14T13:08:44.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:44 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 40/321 objects degraded (12.461%), 10 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:44 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 40/321 objects degraded (12.461%), 10 pgs degraded (PG_DEGRADED) 2024-03-14T13:08:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:45 smithi137 ceph-mon[108145]: pgmap v123: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-14T13:08:45.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:45 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/321 objects degraded (12.461%), 10 pgs degraded) 2024-03-14T13:08:45.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:45 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:08:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:45 smithi111 ceph-mon[145314]: pgmap v123: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s 2024-03-14T13:08:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:45 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/321 objects degraded (12.461%), 10 pgs degraded) 2024-03-14T13:08:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:45 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:08:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:47 smithi137 ceph-mon[108145]: pgmap v124: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-14T13:08:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:47 smithi111 ceph-mon[145314]: pgmap v124: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-14T13:08:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:49 smithi137 ceph-mon[108145]: pgmap v125: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:08:49.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:49 smithi111 ceph-mon[145314]: pgmap v125: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:08:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:51 smithi137 ceph-mon[108145]: pgmap v126: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-14T13:08:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:51 smithi111 ceph-mon[145314]: pgmap v126: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-14T13:08:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:53 smithi137 ceph-mon[108145]: pgmap v127: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:08:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:53 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:53 smithi111 ceph-mon[145314]: pgmap v127: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s 2024-03-14T13:08:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:53 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:54 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:54 smithi137 ceph-mon[108145]: Upgrade: osd.4 is safe to restart 2024-03-14T13:08:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:54.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:54 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-14T13:08:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:54 smithi111 ceph-mon[145314]: Upgrade: osd.4 is safe to restart 2024-03-14T13:08:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:54.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:08:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:55 smithi137 ceph-mon[108145]: pgmap v128: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:08:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T13:08:55.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:55 smithi111 ceph-mon[145314]: pgmap v128: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:08:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:08:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-14T13:08:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:08:56.658 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:56 smithi137 ceph-mon[108145]: Upgrade: Updating osd.4 2024-03-14T13:08:56.659 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:56 smithi137 ceph-mon[108145]: Deploying daemon osd.4 on smithi137 2024-03-14T13:08:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:56 smithi111 ceph-mon[145314]: Upgrade: Updating osd.4 2024-03-14T13:08:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:56 smithi111 ceph-mon[145314]: Deploying daemon osd.4 on smithi137 2024-03-14T13:08:57.692 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:57 smithi137 ceph-mon[108145]: pgmap v129: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:08:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:57 smithi111 ceph-mon[145314]: pgmap v129: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:08:58.256 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:08:57 smithi137 systemd[1]: Stopping Ceph osd.4 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:08:58.256 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:08:58 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[64676]: 2024-03-14T13:08:58.210+0000 7f124e3a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:08:58.256 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:08:58 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[64676]: 2024-03-14T13:08:58.210+0000 7f124e3a5700 -1 osd.4 70 *** Got signal Terminated *** 2024-03-14T13:08:58.256 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:08:58 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[64676]: 2024-03-14T13:08:58.210+0000 7f124e3a5700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:08:58.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 -- 172.21.15.111:0/2622156667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00fb4a0 msgr2=0x7f2db00fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 --2- 172.21.15.111:0/2622156667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fb910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d98004940 tx=0x7f2d98009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 -- 172.21.15.111:0/2622156667 shutdown_connections 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 --2- 172.21.15.111:0/2622156667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fb910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 --2- 172.21.15.111:0/2622156667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db0102e70 0x7f2db0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 -- 172.21.15.111:0/2622156667 >> 172.21.15.111:0/2622156667 conn(0x7f2db00714c0 msgr2=0x7f2db00718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 -- 172.21.15.111:0/2622156667 shutdown_connections 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.608+0000 7f2db6237700 1 -- 172.21.15.111:0/2622156667 wait complete. 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.609+0000 7f2db6237700 1 Processor -- start 2024-03-14T13:08:58.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.609+0000 7f2db6237700 1 -- start start 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.609+0000 7f2db6237700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.609+0000 7f2db6237700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.609+0000 7f2db6237700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00fd150 con 0x7f2db0102e70 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2db6237700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00fd290 con 0x7f2db00fb4a0 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35392/0 (socket says 172.21.15.111:35392) 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 -- 172.21.15.111:0/4134232868 learned_addr learned my addr 172.21.15.111:0/4134232868 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daf7fe700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 -- 172.21.15.111:0/4134232868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db00fb4a0 msgr2=0x7f2db00fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 -- 172.21.15.111:0/4134232868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98005040 con 0x7f2db0102e70 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2daeffd700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009180 tx=0x7f2d98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:58.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d9800e040 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db00fd4e0 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d98004290 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d98013e60 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.610+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db011a160 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.611+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0100170 con 0x7f2db0102e70 2024-03-14T13:08:58.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.612+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2d980136b0 con 0x7f2db0102e70 2024-03-14T13:08:58.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.612+0000 7f2da7fff700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 0x7f2d9c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.612+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f2d9808cc30 con 0x7f2db0102e70 2024-03-14T13:08:58.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.612+0000 7f2daf7fe700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 0x7f2d9c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.613+0000 7f2daf7fe700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 0x7f2d9c06ec00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2da0007b20 tx=0x7f2da0009e20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:58.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.617+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2d98056f50 con 0x7f2db0102e70 2024-03-14T13:08:58.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:58 smithi137 ceph-mon[108145]: osd.4 marked itself down and dead 2024-03-14T13:08:58.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.829+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2db0109e20 con 0x7f2d9c06c750 2024-03-14T13:08:58.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.831+0000 7f2da7fff700 1 -- 172.21.15.111:0/4134232868 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f2db0109e20 con 0x7f2d9c06c750 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 msgr2=0x7f2d9c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 0x7f2d9c06ec00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2da0007b20 tx=0x7f2da0009e20 comp rx=0 tx=0).stop 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 msgr2=0x7f2db00fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009180 tx=0x7f2d98009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 shutdown_connections 2024-03-14T13:08:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2d9c06c750 0x7f2d9c06ec00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2db00fb4a0 0x7f2db00fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 --2- 172.21.15.111:0/4134232868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db0102e70 0x7f2db00fcbc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 >> 172.21.15.111:0/4134232868 conn(0x7f2db00714c0 msgr2=0x7f2db00fa7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 shutdown_connections 2024-03-14T13:08:58.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.833+0000 7f2db6237700 1 -- 172.21.15.111:0/4134232868 wait complete. 2024-03-14T13:08:58.842 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:08:58.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:58 smithi111 ceph-mon[145314]: osd.4 marked itself down and dead 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 -- 172.21.15.111:0/104067642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 msgr2=0x7fc5dc109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/104067642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc109180 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8004970 tx=0x7fc5d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 -- 172.21.15.111:0/104067642 shutdown_connections 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/104067642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc109180 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/104067642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 0x7fc5dc0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 -- 172.21.15.111:0/104067642 >> 172.21.15.111:0/104067642 conn(0x7fc5dc0f8980 msgr2=0x7fc5dc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 -- 172.21.15.111:0/104067642 shutdown_connections 2024-03-14T13:08:58.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.934+0000 7fc5e38ba700 1 -- 172.21.15.111:0/104067642 wait complete. 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 Processor -- start 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 -- start start 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 0x7fc5dc115c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc116810 con 0x7fc5dc070c10 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e38ba700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc10fbe0 con 0x7fc5dc070300 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e0e55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e0e55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35424/0 (socket says 172.21.15.111:35424) 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e0e55700 1 -- 172.21.15.111:0/3403789212 learned_addr learned my addr 172.21.15.111:0/3403789212 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.935+0000 7fc5e1656700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 0x7fc5dc115c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e0e55700 1 -- 172.21.15.111:0/3403789212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 msgr2=0x7fc5dc115c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e0e55700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 0x7fc5dc115c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e0e55700 1 -- 172.21.15.111:0/3403789212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8005040 con 0x7fc5dc070c10 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e0e55700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009380 tx=0x7fc5d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5d800e040 con 0x7fc5dc070c10 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5d8013420 con 0x7fc5dc070c10 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc10fe60 con 0x7fc5dc070c10 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc5d801c630 con 0x7fc5dc070c10 2024-03-14T13:08:58.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.936+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc1103a0 con 0x7fc5dc070c10 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.937+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc071a40 con 0x7fc5dc070c10 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.937+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc5d801c8a0 con 0x7fc5dc070c10 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.938+0000 7fc5d1ffb700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 0x7fc5c806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.938+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc5d808ca90 con 0x7fc5dc070c10 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.938+0000 7fc5e1656700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 0x7fc5c806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:58.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.938+0000 7fc5e1656700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 0x7fc5c806ec70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009fd0 tx=0x7fc5cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:58.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:58.942+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc5d805a8e0 con 0x7fc5dc070c10 2024-03-14T13:08:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.152+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5dc10bfd0 con 0x7fc5c806c7c0 2024-03-14T13:08:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.153+0000 7fc5d1ffb700 1 -- 172.21.15.111:0/3403789212 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc5dc10bfd0 con 0x7fc5c806c7c0 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 msgr2=0x7fc5c806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 0x7fc5c806ec70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009fd0 tx=0x7fc5cc00b040 comp rx=0 tx=0).stop 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 msgr2=0x7fc5dc1161c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009380 tx=0x7fc5d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 shutdown_connections 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc5c806c7c0 0x7fc5c806ec70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5dc070300 0x7fc5dc115c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 --2- 172.21.15.111:0/3403789212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5dc070c10 0x7fc5dc1161c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.155+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 >> 172.21.15.111:0/3403789212 conn(0x7fc5dc0f8980 msgr2=0x7fc5dc0f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.156+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 shutdown_connections 2024-03-14T13:08:59.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.156+0000 7fc5e38ba700 1 -- 172.21.15.111:0/3403789212 wait complete. 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2677199194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4101d70 msgr2=0x7f0bd4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2677199194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4101d70 0x7f0bd4102140 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004940 tx=0x7f0bc8009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2677199194 shutdown_connections 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2677199194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4102710 0x7f0bd40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2677199194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4101d70 0x7f0bd4102140 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2677199194 >> 172.21.15.111:0/2677199194 conn(0x7f0bd4072bc0 msgr2=0x7f0bd4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2677199194 shutdown_connections 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.255+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2677199194 wait complete. 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 Processor -- start 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 -- start start 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4101d70 0x7f0bd4115be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd4116800 con 0x7f0bd4102710 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bdbe21700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd410fbf0 con 0x7f0bd4101d70 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35430/0 (socket says 172.21.15.111:35430) 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 -- 172.21.15.111:0/2782789422 learned_addr learned my addr 172.21.15.111:0/2782789422 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd9bbd700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4101d70 0x7f0bd4115be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 -- 172.21.15.111:0/2782789422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4101d70 msgr2=0x7f0bd4115be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4101d70 0x7f0bd4115be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.256+0000 7f0bd93bc700 1 -- 172.21.15.111:0/2782789422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8005040 con 0x7f0bd4102710 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bd93bc700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000b790 tx=0x7f0bd0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd000f6f0 con 0x7f0bd4102710 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd000fd00 con 0x7f0bd4102710 2024-03-14T13:08:59.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd410fed0 con 0x7f0bd4102710 2024-03-14T13:08:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0bd00103a0 con 0x7f0bd4102710 2024-03-14T13:08:59.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.257+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd4110490 con 0x7f0bd4102710 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.258+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd40fd4c0 con 0x7f0bd4102710 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.258+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0bd000f850 con 0x7f0bd4102710 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.259+0000 7f0bc67fc700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 0x7f0bc006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.259+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0bd008b710 con 0x7f0bd4102710 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.259+0000 7f0bd9bbd700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 0x7f0bc006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.259+0000 7f0bd9bbd700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 0x7f0bc006ec70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004f90 tx=0x7f0bc8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.263+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0bd0055ad0 con 0x7f0bd4102710 2024-03-14T13:08:59.452 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:59 smithi137 ceph-mon[108145]: pgmap v130: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:08:59.452 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:59 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:08:59.452 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:08:59 smithi137 ceph-mon[108145]: osdmap e71: 6 total, 5 up, 6 in 2024-03-14T13:08:59.453 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:08:59 smithi137 bash[127458]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4 2024-03-14T13:08:59.492 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.491+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bd4108da0 con 0x7f0bc006c7c0 2024-03-14T13:08:59.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.501+0000 7f0bc67fc700 1 -- 172.21.15.111:0/2782789422 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f0bd4108da0 con 0x7f0bc006c7c0 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (5m) 57s ago 16m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (2m) 57s ago 17m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (2m) 25s ago 15m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (4m) 57s ago 16m 89.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (10m) 57s ago 10m 64.2M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (10m) 57s ago 10m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (10m) 25s ago 10m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (10m) 25s ago 10m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (7m) 57s ago 18m 501M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (6m) 25s ago 15m 451M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:08:59.502 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (3m) 57s ago 18m 51.8M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (3m) 25s ago 15m 38.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 57s ago 16m 19.0M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 25s ago 15m 18.2M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (2m) 57s ago 14m 84.2M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (100s) 57s ago 14m 91.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (63s) 57s ago 13m 15.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (28s) 25s ago 13m 13.9M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (12m) 25s ago 12m 232M 2592M 17.2.7-646-gb92acd1f a07fb176315d 2c24493e8a96 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (12m) 25s ago 12m 220M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:08:59.503 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (5m) 57s ago 15m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 msgr2=0x7f0bc006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 0x7f0bc006ec70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8004f90 tx=0x7f0bc8009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 msgr2=0x7f0bd4116120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000b790 tx=0x7f0bd0009220 comp rx=0 tx=0).stop 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 shutdown_connections 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0bc006c7c0 0x7f0bc006ec70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0bd4101d70 0x7f0bd4115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 --2- 172.21.15.111:0/2782789422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0bd4102710 0x7f0bd4116120 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.503+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 >> 172.21.15.111:0/2782789422 conn(0x7f0bd4072bc0 msgr2=0x7f0bd4065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.504+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 shutdown_connections 2024-03-14T13:08:59.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.504+0000 7f0bdbe21700 1 -- 172.21.15.111:0/2782789422 wait complete. 2024-03-14T13:08:59.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 -- 172.21.15.111:0/820799055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fde90 msgr2=0x7fb4880fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 --2- 172.21.15.111:0/820799055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fde90 0x7fb4880fe260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb47c004940 tx=0x7fb47c009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 -- 172.21.15.111:0/820799055 shutdown_connections 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 --2- 172.21.15.111:0/820799055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fe830 0x7fb488106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 --2- 172.21.15.111:0/820799055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fde90 0x7fb4880fe260 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 -- 172.21.15.111:0/820799055 >> 172.21.15.111:0/820799055 conn(0x7fb488072bc0 msgr2=0x7fb488072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 -- 172.21.15.111:0/820799055 shutdown_connections 2024-03-14T13:08:59.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.602+0000 7fb48d412700 1 -- 172.21.15.111:0/820799055 wait complete. 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 Processor -- start 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 -- start start 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fde90 0x7fb488113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488114600 con 0x7fb4880fe830 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb48d412700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48810d9f0 con 0x7fb4880fde90 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35450/0 (socket says 172.21.15.111:35450) 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 -- 172.21.15.111:0/2875551381 learned_addr learned my addr 172.21.15.111:0/2875551381 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:59.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb486ffd700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fde90 0x7fb488113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 -- 172.21.15.111:0/2875551381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fde90 msgr2=0x7fb488113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fde90 0x7fb488113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 -- 172.21.15.111:0/2875551381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c005040 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.603+0000 7fb4867fc700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb478002a80 tx=0x7fb4780046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.604+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb478007cc0 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.604+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb47800f040 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.604+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48810dcd0 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.604+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb47800da60 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.604+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48810e210 con 0x7fb4880fe830 2024-03-14T13:08:59.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.605+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488065e30 con 0x7fb4880fe830 2024-03-14T13:08:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.605+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb47800dbc0 con 0x7fb4880fe830 2024-03-14T13:08:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.605+0000 7fb46f7fe700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 0x7fb47006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.606+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fb47808b640 con 0x7fb4880fe830 2024-03-14T13:08:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.606+0000 7fb486ffd700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 0x7fb47006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.606+0000 7fb486ffd700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 0x7fb47006ec70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00b7e0 tx=0x7fb47c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.610+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb478059370 con 0x7fb4880fe830 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.883+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb488109dc0 con 0x7fb4880fe830 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.884+0000 7fb46f7fe700 1 -- 172.21.15.111:0/2875551381 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7fb478058f00 con 0x7fb4880fe830 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:08:59.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 1, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 5, 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 8 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:08:59.885 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:08:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 msgr2=0x7fb47006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 0x7fb47006ec70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00b7e0 tx=0x7fb47c009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 msgr2=0x7fb488113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb478002a80 tx=0x7fb4780046b0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 shutdown_connections 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb47006c7c0 0x7fb47006ec70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb4880fde90 0x7fb488113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 --2- 172.21.15.111:0/2875551381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4880fe830 0x7fb488113fb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 >> 172.21.15.111:0/2875551381 conn(0x7fb488072bc0 msgr2=0x7fb4881037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 shutdown_connections 2024-03-14T13:08:59.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.886+0000 7fb48d412700 1 -- 172.21.15.111:0/2875551381 wait complete. 2024-03-14T13:08:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:59 smithi111 ceph-mon[145314]: pgmap v130: 65 pgs: 65 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:08:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:59 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:08:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:08:59 smithi111 ceph-mon[145314]: osdmap e71: 6 total, 5 up, 6 in 2024-03-14T13:08:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- 172.21.15.111:0/2802375045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c102710 msgr2=0x7f3e0c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 --2- 172.21.15.111:0/2802375045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c102710 0x7f3e0c0fbd80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3df8004940 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-03-14T13:08:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- 172.21.15.111:0/2802375045 shutdown_connections 2024-03-14T13:08:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 --2- 172.21.15.111:0/2802375045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c102710 0x7f3e0c0fbd80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 --2- 172.21.15.111:0/2802375045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c101d70 0x7f3e0c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- 172.21.15.111:0/2802375045 >> 172.21.15.111:0/2802375045 conn(0x7f3e0c072bc0 msgr2=0x7f3e0c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- 172.21.15.111:0/2802375045 shutdown_connections 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- 172.21.15.111:0/2802375045 wait complete. 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 Processor -- start 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.986+0000 7f3e13196700 1 -- start start 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e13196700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e13196700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c102710 0x7f3e0c10f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e13196700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c10f640 con 0x7f3e0c101d70 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e13196700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c10f780 con 0x7f3e0c102710 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35460/0 (socket says 172.21.15.111:35460) 2024-03-14T13:08:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 -- 172.21.15.111:0/2551388269 learned_addr learned my addr 172.21.15.111:0/2551388269 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e0bfff700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c102710 0x7f3e0c10f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 -- 172.21.15.111:0/2551388269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c102710 msgr2=0x7f3e0c10f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c102710 0x7f3e0c10f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 -- 172.21.15.111:0/2551388269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8005040 con 0x7f3e0c101d70 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e10f32700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000eb70 tx=0x7f3e0000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.987+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e00007cc0 con 0x7f3e0c101d70 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.988+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e0000f040 con 0x7f3e0c101d70 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.988+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e0000c9f0 con 0x7f3e0c101d70 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.988+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c117a70 con 0x7f3e0c101d70 2024-03-14T13:08:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.988+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c117f80 con 0x7f3e0c101d70 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.989+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c104240 con 0x7f3e0c101d70 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.989+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3e0000cb50 con 0x7f3e0c101d70 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.989+0000 7f3e097fa700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 0x7f3dfc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.990+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f3e0008b5c0 con 0x7f3e0c101d70 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.990+0000 7f3e0bfff700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 0x7f3dfc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:08:59.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.990+0000 7f3e0bfff700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 0x7f3dfc06ec80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c066020 tx=0x7f3df8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:08:59.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:08:59.994+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3e00055860 con 0x7f3e0c101d70 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.230+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3e0c108d60 con 0x7f3e0c101d70 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.230+0000 7f3e097fa700 1 -- 172.21.15.111:0/2551388269 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f3e00058e80 con 0x7f3e0c101d70 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:09:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:09:00.232 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:09:00.233 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:00.234 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:00.234 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:00.234 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 msgr2=0x7f3dfc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 0x7f3dfc06ec80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c066020 tx=0x7f3df8009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 msgr2=0x7f3e0c10ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000eb70 tx=0x7f3e0000b040 comp rx=0 tx=0).stop 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 shutdown_connections 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3dfc06c7d0 0x7f3dfc06ec80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e0c101d70 0x7f3e0c10ebc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 --2- 172.21.15.111:0/2551388269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3e0c102710 0x7f3e0c10f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.232+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 >> 172.21.15.111:0/2551388269 conn(0x7f3e0c072bc0 msgr2=0x7f3e0c0fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.233+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 shutdown_connections 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.233+0000 7f3e13196700 1 -- 172.21.15.111:0/2551388269 wait complete. 2024-03-14T13:09:00.235 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 -- 172.21.15.111:0/2561380411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690101d70 msgr2=0x7fc690102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 --2- 172.21.15.111:0/2561380411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690101d70 0x7fc690102140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc684004940 tx=0x7fc684009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 -- 172.21.15.111:0/2561380411 shutdown_connections 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 --2- 172.21.15.111:0/2561380411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690102710 0x7fc6900fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 --2- 172.21.15.111:0/2561380411 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690101d70 0x7fc690102140 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 -- 172.21.15.111:0/2561380411 >> 172.21.15.111:0/2561380411 conn(0x7fc690072bc0 msgr2=0x7fc690072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 -- 172.21.15.111:0/2561380411 shutdown_connections 2024-03-14T13:09:00.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.334+0000 7fc6975bf700 1 -- 172.21.15.111:0/2561380411 wait complete. 2024-03-14T13:09:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 Processor -- start 2024-03-14T13:09:00.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 -- start start 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690101d70 0x7fc6901129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc690113530 con 0x7fc690102710 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc6975bf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69010c920 con 0x7fc690101d70 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc694b5a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc694b5a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33080/0 (socket says 172.21.15.111:33080) 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc694b5a700 1 -- 172.21.15.111:0/3067603891 learned_addr learned my addr 172.21.15.111:0/3067603891 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.335+0000 7fc69535b700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690101d70 0x7fc6901129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc694b5a700 1 -- 172.21.15.111:0/3067603891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690101d70 msgr2=0x7fc6901129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc694b5a700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690101d70 0x7fc6901129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc694b5a700 1 -- 172.21.15.111:0/3067603891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc684005040 con 0x7fc690102710 2024-03-14T13:09:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc694b5a700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00fce0 tx=0x7fc68c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68c007ea0 con 0x7fc690102710 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69010cba0 con 0x7fc690102710 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc68c00dcd0 con 0x7fc690102710 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc68c012460 con 0x7fc690102710 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.336+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69010d160 con 0x7fc690102710 2024-03-14T13:09:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.337+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6900fd4c0 con 0x7fc690102710 2024-03-14T13:09:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.338+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc68c00d820 con 0x7fc690102710 2024-03-14T13:09:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.338+0000 7fc681ffb700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 0x7fc67c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.338+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fc68c08e140 con 0x7fc690102710 2024-03-14T13:09:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.338+0000 7fc69535b700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 0x7fc67c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.338+0000 7fc69535b700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 0x7fc67c06ec70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc684004f90 tx=0x7fc684009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:00.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.342+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc68c05be70 con 0x7fc690102710 2024-03-14T13:09:00.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.551+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc690108da0 con 0x7fc67c06c7c0 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.552+0000 7fc681ffb700 1 -- 172.21.15.111:0/3067603891 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fc690108da0 con 0x7fc67c06c7c0 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:09:00.553 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "10/21 daemons upgraded", 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:09:00.554 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:09:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 msgr2=0x7fc67c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 0x7fc67c06ec70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc684004f90 tx=0x7fc684009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 msgr2=0x7fc690112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc68c00fce0 tx=0x7fc68c004600 comp rx=0 tx=0).stop 2024-03-14T13:09:00.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 shutdown_connections 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc67c06c7c0 0x7fc67c06ec70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc690101d70 0x7fc6901129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 --2- 172.21.15.111:0/3067603891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc690102710 0x7fc690112ee0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 >> 172.21.15.111:0/3067603891 conn(0x7fc690072bc0 msgr2=0x7fc6900fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 shutdown_connections 2024-03-14T13:09:00.556 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.555+0000 7fc6975bf700 1 -- 172.21.15.111:0/3067603891 wait complete. 2024-03-14T13:09:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- 172.21.15.111:0/1981687481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430102710 msgr2=0x7fa4300fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 --2- 172.21.15.111:0/1981687481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430102710 0x7fa4300fbd80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa424004940 tx=0x7fa424009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- 172.21.15.111:0/1981687481 shutdown_connections 2024-03-14T13:09:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 --2- 172.21.15.111:0/1981687481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430102710 0x7fa4300fbd80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 --2- 172.21.15.111:0/1981687481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430101d70 0x7fa430102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- 172.21.15.111:0/1981687481 >> 172.21.15.111:0/1981687481 conn(0x7fa430072bc0 msgr2=0x7fa430072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- 172.21.15.111:0/1981687481 shutdown_connections 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- 172.21.15.111:0/1981687481 wait complete. 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 Processor -- start 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.655+0000 7fa434fc0700 1 -- start start 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430102710 0x7fa430112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa430113550 con 0x7fa430101d70 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43010c940 con 0x7fa430102710 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33098/0 (socket says 172.21.15.111:33098) 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 -- 172.21.15.111:0/2752282765 learned_addr learned my addr 172.21.15.111:0/2752282765 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42dd9b700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430102710 0x7fa430112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 -- 172.21.15.111:0/2752282765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430102710 msgr2=0x7fa430112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430102710 0x7fa430112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 -- 172.21.15.111:0/2752282765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa424005040 con 0x7fa430101d70 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa42e59c700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa42000fce0 tx=0x7fa4200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa420007ea0 con 0x7fa430101d70 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa42000dd80 con 0x7fa430101d70 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43010cbc0 con 0x7fa430101d70 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa420012460 con 0x7fa430101d70 2024-03-14T13:09:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.656+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43010d150 con 0x7fa430101d70 2024-03-14T13:09:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.657+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4300fd4c0 con 0x7fa430101d70 2024-03-14T13:09:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.658+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa420009360 con 0x7fa430101d70 2024-03-14T13:09:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.658+0000 7fa41effd700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 0x7fa41806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.658+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fa42008e0d0 con 0x7fa430101d70 2024-03-14T13:09:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.658+0000 7fa42dd9b700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 0x7fa41806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.659+0000 7fa42dd9b700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 0x7fa41806ec70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa424004890 tx=0x7fa424009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:00.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.663+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa42005be00 con 0x7fa430101d70 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: osdmap e72: 6 total, 5 up, 6 in 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2875551381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:00.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:00 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2551388269' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: osdmap e72: 6 total, 5 up, 6 in 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2875551381' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:00 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2551388269' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.934+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa43004bed0 con 0x7fa430101d70 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.934+0000 7fa41effd700 1 -- 172.21.15.111:0/2752282765 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+605 (secure 0 0 0) 0x7fa42005b990 con 0x7fa430101d70 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 18/321 objects degraded (5.607%), 6 pgs degraded 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: osd.4 (root=default,host=smithi137) is down 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 18/321 objects degraded (5.607%), 6 pgs degraded 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.f is active+undersized+degraded, acting [0,5] 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.13 is active+undersized+degraded, acting [0,2] 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.18 is active+undersized+degraded, acting [5,3] 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.19 is active+undersized+degraded, acting [0,2] 2024-03-14T13:09:00.935 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-03-14T13:09:00.936 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.1f is active+undersized+degraded, acting [0,3] 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 msgr2=0x7fa41806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 0x7fa41806ec70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa424004890 tx=0x7fa424009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 msgr2=0x7fa4301129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa42000fce0 tx=0x7fa4200046b0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 shutdown_connections 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa41806c7c0 0x7fa41806ec70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa430101d70 0x7fa4301129c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 --2- 172.21.15.111:0/2752282765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa430102710 0x7fa430112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:00.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 >> 172.21.15.111:0/2752282765 conn(0x7fa430072bc0 msgr2=0x7fa4300fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:00.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.936+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 shutdown_connections 2024-03-14T13:09:00.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:00.937+0000 7fa434fc0700 1 -- 172.21.15.111:0/2752282765 wait complete. 2024-03-14T13:09:01.454 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:01 smithi137 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.4.service: Succeeded. 2024-03-14T13:09:01.454 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:01 smithi137 systemd[1]: Stopped Ceph osd.4 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:09:01.454 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:01 smithi137 systemd[1]: Starting Ceph osd.4 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:09:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:01 smithi137 ceph-mon[108145]: pgmap v133: 65 pgs: 6 active+undersized+degraded, 3 stale+active+clean, 7 active+undersized, 49 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 18/321 objects degraded (5.607%) 2024-03-14T13:09:01.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:01 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 18/321 objects degraded (5.607%), 6 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:01.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:01 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2752282765' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:09:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:01 smithi111 ceph-mon[145314]: pgmap v133: 65 pgs: 6 active+undersized+degraded, 3 stale+active+clean, 7 active+undersized, 49 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 18/321 objects degraded (5.607%) 2024-03-14T13:09:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:01 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 18/321 objects degraded (5.607%), 6 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:01 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2752282765' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:09:02.456 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-14T13:09:02.457 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:02 smithi137 bash[127732]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-14T13:09:02.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:02 smithi137 ceph-mon[108145]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:02 smithi111 ceph-mon[145314]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:03.464 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-osd[128053]: -- 172.21.15.137:0/175003299 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56152bff91e0 con 0x56152bfc9000 2024-03-14T13:09:03.464 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:03 smithi137 bash[128015]: 5382a85f980927aecfbdfbabe448583c2fd1db0be2276733ba170ba31371d566 2024-03-14T13:09:03.464 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:03 smithi137 systemd[1]: Started Ceph osd.4 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:09:03.755 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:09:03.684+0000 7fc92a06f640 -1 Falling back to public interface 2024-03-14T13:09:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-mon[108145]: pgmap v134: 65 pgs: 10 active+undersized+degraded, 1 stale+active+clean, 13 active+undersized, 41 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 28/321 objects degraded (8.723%) 2024-03-14T13:09:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:03.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:03 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:03 smithi111 ceph-mon[145314]: pgmap v134: 65 pgs: 10 active+undersized+degraded, 1 stale+active+clean, 13 active+undersized, 41 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 2 op/s; 28/321 objects degraded (8.723%) 2024-03-14T13:09:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:03 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:05.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:05 smithi137 ceph-mon[108145]: pgmap v135: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:05 smithi111 ceph-mon[145314]: pgmap v135: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:07.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:07 smithi137 ceph-mon[108145]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:07.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:07 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:07 smithi111 ceph-mon[145314]: pgmap v136: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 0 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:07.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:07.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:07 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:08.341 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:08 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:09:08.175+0000 7fc92a06f640 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-14T13:09:09.006 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:08 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:09:08.659+0000 7fc92a06f640 -1 osd.4 70 log_to_monitors true 2024-03-14T13:09:09.636 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:09:09.341+0000 7fc918cc3700 -1 osd.4 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:09:09.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-mon[108145]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:09.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-mon[108145]: from='osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T13:09:09.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-mon[108145]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T13:09:09.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:09.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:09 smithi111 ceph-mon[145314]: pgmap v137: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.5 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 1 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:09 smithi111 ceph-mon[145314]: from='osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T13:09:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:09 smithi111 ceph-mon[145314]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-14T13:09:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:10 smithi111 ceph-mon[145314]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T13:09:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:10 smithi111 ceph-mon[145314]: osdmap e73: 6 total, 5 up, 6 in 2024-03-14T13:09:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:10 smithi111 ceph-mon[145314]: from='osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:10 smithi111 ceph-mon[145314]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:10.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:10 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 52/321 objects degraded (16.199%), 16 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:10.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:10 smithi137 ceph-mon[108145]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-14T13:09:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:10 smithi137 ceph-mon[108145]: osdmap e73: 6 total, 5 up, 6 in 2024-03-14T13:09:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:10 smithi137 ceph-mon[108145]: from='osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:10 smithi137 ceph-mon[108145]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:10 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 52/321 objects degraded (16.199%), 16 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:11.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674] boot 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: osdmap e74: 6 total, 6 up, 6 in 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:11.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:11 smithi111 ceph-mon[145314]: osdmap e75: 6 total, 6 up, 6 in 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: pgmap v139: 65 pgs: 16 active+undersized+degraded, 18 active+undersized, 31 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 52/321 objects degraded (16.199%) 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: osd.4 [v2:172.21.15.137:6808/2798642674,v1:172.21.15.137:6809/2798642674] boot 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: osdmap e74: 6 total, 6 up, 6 in 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:11.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:11.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:11 smithi137 ceph-mon[108145]: osdmap e75: 6 total, 6 up, 6 in 2024-03-14T13:09:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:12 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:12.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:12 smithi111 ceph-mon[145314]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-14T13:09:12.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:12 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:12.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:12 smithi137 ceph-mon[108145]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-14T13:09:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:13 smithi111 ceph-mon[145314]: pgmap v142: 65 pgs: 7 peering, 12 active+undersized+degraded, 14 active+undersized, 32 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 42/321 objects degraded (13.084%) 2024-03-14T13:09:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:13 smithi137 ceph-mon[108145]: pgmap v142: 65 pgs: 7 peering, 12 active+undersized+degraded, 14 active+undersized, 32 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 4 op/s; 42/321 objects degraded (13.084%) 2024-03-14T13:09:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:15 smithi111 ceph-mon[145314]: pgmap v143: 65 pgs: 7 peering, 7 active+undersized+degraded, 9 active+undersized, 42 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/321 objects degraded (6.231%) 2024-03-14T13:09:15.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:15 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 20/321 objects degraded (6.231%), 7 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:15 smithi137 ceph-mon[108145]: pgmap v143: 65 pgs: 7 peering, 7 active+undersized+degraded, 9 active+undersized, 42 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2 op/s; 20/321 objects degraded (6.231%) 2024-03-14T13:09:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:15 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 20/321 objects degraded (6.231%), 7 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:16.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:16 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/321 objects degraded (6.231%), 7 pgs degraded) 2024-03-14T13:09:16.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:16 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:09:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:16 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 20/321 objects degraded (6.231%), 7 pgs degraded) 2024-03-14T13:09:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:16 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:09:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:17 smithi137 ceph-mon[108145]: pgmap v144: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:09:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:17 smithi111 ceph-mon[145314]: pgmap v144: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:09:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:19 smithi137 ceph-mon[108145]: pgmap v145: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-14T13:09:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:19 smithi111 ceph-mon[145314]: pgmap v145: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-14T13:09:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:21 smithi137 ceph-mon[108145]: pgmap v146: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-14T13:09:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:21 smithi111 ceph-mon[145314]: pgmap v146: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-14T13:09:23.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:23 smithi137 ceph-mon[108145]: pgmap v147: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-14T13:09:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:23 smithi111 ceph-mon[145314]: pgmap v147: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 4 op/s 2024-03-14T13:09:25.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:25 smithi111 ceph-mon[145314]: pgmap v148: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:09:25.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:25.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:25.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:25 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:25 smithi137 ceph-mon[108145]: pgmap v148: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s rd, 3 op/s 2024-03-14T13:09:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:25.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:25 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:26.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:26 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:26.658 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:26 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:27.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:27.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: Upgrade: osd.5 is safe to restart 2024-03-14T13:09:27.554 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: pgmap v149: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:09:27.554 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:27.554 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T13:09:27.554 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:27 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:27.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-14T13:09:27.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: Upgrade: osd.5 is safe to restart 2024-03-14T13:09:27.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: pgmap v149: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:09:27.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:27.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-14T13:09:27.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:27 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:28.462 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:28 smithi137 ceph-mon[108145]: Upgrade: Updating osd.5 2024-03-14T13:09:28.462 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:28 smithi137 ceph-mon[108145]: Deploying daemon osd.5 on smithi137 2024-03-14T13:09:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:28 smithi111 ceph-mon[145314]: Upgrade: Updating osd.5 2024-03-14T13:09:28.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:28 smithi111 ceph-mon[145314]: Deploying daemon osd.5 on smithi137 2024-03-14T13:09:29.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:29 smithi137 ceph-mon[108145]: pgmap v150: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:09:29.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:29 smithi111 ceph-mon[145314]: pgmap v150: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:09:30.424 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:30 smithi137 systemd[1]: Stopping Ceph osd.5 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:09:30.424 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:30 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[72682]: 2024-03-14T13:09:30.358+0000 7f4751325700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:09:30.424 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:30 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[72682]: 2024-03-14T13:09:30.358+0000 7f4751325700 -1 osd.5 75 *** Got signal Terminated *** 2024-03-14T13:09:30.424 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:30 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[72682]: 2024-03-14T13:09:30.358+0000 7f4751325700 -1 osd.5 75 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:09:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:30 smithi137 ceph-mon[108145]: osd.5 marked itself down and dead 2024-03-14T13:09:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:30 smithi111 ceph-mon[145314]: osd.5 marked itself down and dead 2024-03-14T13:09:31.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- 172.21.15.111:0/3070274134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 msgr2=0x7fabd81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 --2- 172.21.15.111:0/3070274134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd81020f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fabcc004940 tx=0x7fabcc009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- 172.21.15.111:0/3070274134 shutdown_connections 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 --2- 172.21.15.111:0/3070274134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 0x7fabd80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 --2- 172.21.15.111:0/3070274134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd81020f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- 172.21.15.111:0/3070274134 >> 172.21.15.111:0/3070274134 conn(0x7fabd80f78d0 msgr2=0x7fabd80f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- 172.21.15.111:0/3070274134 shutdown_connections 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- 172.21.15.111:0/3070274134 wait complete. 2024-03-14T13:09:31.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 Processor -- start 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.052+0000 7fabde73b700 1 -- start start 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabde73b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabde73b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 0x7fabd810cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabde73b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd810d2a0 con 0x7fabd8101d20 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabde73b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd810d410 con 0x7fabd81026c0 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35012/0 (socket says 172.21.15.111:35012) 2024-03-14T13:09:31.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 -- 172.21.15.111:0/3633742906 learned_addr learned my addr 172.21.15.111:0/3633742906 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd77fe700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 0x7fabd810cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 -- 172.21.15.111:0/3633742906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 msgr2=0x7fabd810cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 0x7fabd810cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 -- 172.21.15.111:0/3633742906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc005040 con 0x7fabd8101d20 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd7fff700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00b3b0 tx=0x7fabcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabcc00e040 con 0x7fabd8101d20 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.053+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd810d690 con 0x7fabd8101d20 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.054+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabcc013420 con 0x7fabd8101d20 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.054+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fabcc01c610 con 0x7fabd8101d20 2024-03-14T13:09:31.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.054+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd811a260 con 0x7fabd8101d20 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.055+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd80fd470 con 0x7fabd8101d20 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.055+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fabcc013a00 con 0x7fabd8101d20 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.056+0000 7fabd4ff9700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 0x7fabc006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.056+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7fabcc08c940 con 0x7fabd8101d20 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.056+0000 7fabd77fe700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 0x7fabc006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.056+0000 7fabd77fe700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 0x7fabc006ec70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabd80668f0 tx=0x7fabc800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.060+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fabcc05a6f0 con 0x7fabd8101d20 2024-03-14T13:09:31.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.273+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fabd810d690 con 0x7fabc006c7c0 2024-03-14T13:09:31.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.274+0000 7fabd4ff9700 1 -- 172.21.15.111:0/3633742906 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fabd810d690 con 0x7fabc006c7c0 2024-03-14T13:09:31.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 msgr2=0x7fabc006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 0x7fabc006ec70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabd80668f0 tx=0x7fabc800b040 comp rx=0 tx=0).stop 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 msgr2=0x7fabd810c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00b3b0 tx=0x7fabcc009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 shutdown_connections 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fabc006c7c0 0x7fabc006ec70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd8101d20 0x7fabd810c820 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 --2- 172.21.15.111:0/3633742906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabd81026c0 0x7fabd810cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 >> 172.21.15.111:0/3633742906 conn(0x7fabd80f78d0 msgr2=0x7fabd80f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 shutdown_connections 2024-03-14T13:09:31.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.276+0000 7fabde73b700 1 -- 172.21.15.111:0/3633742906 wait complete. 2024-03-14T13:09:31.286 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- 172.21.15.111:0/2560742751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 msgr2=0x7f1ed0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/2560742751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0102140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4004940 tx=0x7f1ec4009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- 172.21.15.111:0/2560742751 shutdown_connections 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/2560742751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 0x7f1ed00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/2560742751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0102140 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- 172.21.15.111:0/2560742751 >> 172.21.15.111:0/2560742751 conn(0x7f1ed0072bc0 msgr2=0x7f1ed0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- 172.21.15.111:0/2560742751 shutdown_connections 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- 172.21.15.111:0/2560742751 wait complete. 2024-03-14T13:09:31.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 Processor -- start 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.380+0000 7f1ed6f44700 1 -- start start 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 0x7f1ed0103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0103df0 con 0x7f1ed0101d70 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed0103f30 con 0x7f1ed0102710 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35030/0 (socket says 172.21.15.111:35030) 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 -- 172.21.15.111:0/1920535029 learned_addr learned my addr 172.21.15.111:0/1920535029 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ecffff700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 0x7f1ed0103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 -- 172.21.15.111:0/1920535029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 msgr2=0x7f1ed0103860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 0x7f1ed0103860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 -- 172.21.15.111:0/1920535029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4005040 con 0x7f1ed0101d70 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed4ce0700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3b0 tx=0x7f1ec4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec400e040 con 0x7f1ed0101d70 2024-03-14T13:09:31.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ec4013420 con 0x7f1ed0101d70 2024-03-14T13:09:31.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed0104180 con 0x7f1ed0101d70 2024-03-14T13:09:31.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec401c610 con 0x7f1ed0101d70 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.381+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed011a1d0 con 0x7f1ed0101d70 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.382+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed00fd4c0 con 0x7f1ed0101d70 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.383+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1ec4013a00 con 0x7f1ed0101d70 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.383+0000 7f1ecd7fa700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 0x7f1ec006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.383+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f1ec408cb70 con 0x7f1ed0101d70 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.383+0000 7f1ecffff700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 0x7f1ec006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.384+0000 7f1ecffff700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 0x7f1ec006ec70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0104c60 tx=0x7f1ebc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.387 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.388+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1ec4056e90 con 0x7f1ed0101d70 2024-03-14T13:09:31.423 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:31 smithi137 bash[133593]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5 2024-03-14T13:09:31.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.613+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ed0104a10 con 0x7f1ec006c7c0 2024-03-14T13:09:31.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.614+0000 7f1ecd7fa700 1 -- 172.21.15.111:0/1920535029 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1ed0104a10 con 0x7f1ec006c7c0 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 msgr2=0x7f1ec006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 0x7f1ec006ec70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0104c60 tx=0x7f1ebc00b040 comp rx=0 tx=0).stop 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 msgr2=0x7f1ed0103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400b3b0 tx=0x7f1ec4009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 shutdown_connections 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ec006c7c0 0x7f1ec006ec70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1ed0101d70 0x7f1ed0103320 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 --2- 172.21.15.111:0/1920535029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ed0102710 0x7f1ed0103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.616+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 >> 172.21.15.111:0/1920535029 conn(0x7f1ed0072bc0 msgr2=0x7f1ed0105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.617+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 shutdown_connections 2024-03-14T13:09:31.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.617+0000 7f1ed6f44700 1 -- 172.21.15.111:0/1920535029 wait complete. 2024-03-14T13:09:31.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 -- 172.21.15.111:0/2246700159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 msgr2=0x7f31a0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2246700159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0106f00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3194004940 tx=0x7f3194009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 -- 172.21.15.111:0/2246700159 shutdown_connections 2024-03-14T13:09:31.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2246700159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0106f00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2246700159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 0x7f31a00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.716+0000 7f31a51d6700 1 -- 172.21.15.111:0/2246700159 >> 172.21.15.111:0/2246700159 conn(0x7f31a0072bc0 msgr2=0x7f31a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 -- 172.21.15.111:0/2246700159 shutdown_connections 2024-03-14T13:09:31.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 -- 172.21.15.111:0/2246700159 wait complete. 2024-03-14T13:09:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 Processor -- start 2024-03-14T13:09:31.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 -- start start 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 0x7f31a0113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a0114630 con 0x7f31a00fe830 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.717+0000 7f31a51d6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a010da20 con 0x7f31a00fde90 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35046/0 (socket says 172.21.15.111:35046) 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 -- 172.21.15.111:0/2684379073 learned_addr learned my addr 172.21.15.111:0/2684379073 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319ed9d700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 0x7f31a0113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 -- 172.21.15.111:0/2684379073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 msgr2=0x7f31a0113aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 0x7f31a0113aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 -- 172.21.15.111:0/2684379073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194005040 con 0x7f31a00fe830 2024-03-14T13:09:31.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f319e59c700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3194000c00 tx=0x7f3194009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3194021070 con 0x7f31a00fe830 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a010dca0 con 0x7f31a00fe830 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3194013e00 con 0x7f31a00fe830 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f319400e070 con 0x7f31a00fe830 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.718+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a010e1e0 con 0x7f31a00fe830 2024-03-14T13:09:31.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.719+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a0101980 con 0x7f31a00fe830 2024-03-14T13:09:31.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.720+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3194013950 con 0x7f31a00fe830 2024-03-14T13:09:31.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.720+0000 7f31877fe700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 0x7f318806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:31.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.720+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f319408ca60 con 0x7f31a00fe830 2024-03-14T13:09:31.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.720+0000 7f319ed9d700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 0x7f318806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:31.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.720+0000 7f319ed9d700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 0x7f318806ec70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3190009b80 tx=0x7f319000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:31.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.724+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f319405a810 con 0x7f31a00fe830 2024-03-14T13:09:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:31 smithi137 ceph-mon[108145]: pgmap v151: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:09:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:31 smithi137 ceph-mon[108145]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:09:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:31 smithi137 ceph-mon[108145]: osdmap e76: 6 total, 5 up, 6 in 2024-03-14T13:09:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:31 smithi111 ceph-mon[145314]: pgmap v151: 65 pgs: 65 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:09:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:31 smithi111 ceph-mon[145314]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-14T13:09:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:31 smithi111 ceph-mon[145314]: osdmap e76: 6 total, 5 up, 6 in 2024-03-14T13:09:31.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.926+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a010f0f0 con 0x7f318806c7c0 2024-03-14T13:09:31.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.936+0000 7f31877fe700 1 -- 172.21.15.111:0/2684379073 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f31a010f0f0 con 0x7f318806c7c0 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (5m) 90s ago 17m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (3m) 90s ago 17m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (3m) 25s ago 16m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (5m) 90s ago 16m 89.6M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (11m) 90s ago 11m 64.2M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (11m) 90s ago 11m 120M - 17.2.7-646-gb92acd1f a07fb176315d ac87bb3988c1 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (11m) 25s ago 11m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (11m) 25s ago 11m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:09:31.936 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (8m) 90s ago 18m 501M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (6m) 25s ago 15m 451M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (4m) 90s ago 18m 51.8M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (3m) 25s ago 15m 40.1M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 90s ago 17m 19.0M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 25s ago 15m 18.1M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (2m) 90s ago 15m 84.2M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (2m) 90s ago 14m 91.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (96s) 90s ago 14m 15.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (61s) 25s ago 13m 99.0M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (29s) 25s ago 13m 13.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (12m) 25s ago 12m 221M 2592M 17.2.7-646-gb92acd1f a07fb176315d 5c2759124485 2024-03-14T13:09:31.937 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (6m) 90s ago 16m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 msgr2=0x7f318806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 0x7f318806ec70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3190009b80 tx=0x7f319000b040 comp rx=0 tx=0).stop 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 msgr2=0x7f31a0113fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3194000c00 tx=0x7f3194009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 shutdown_connections 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f318806c7c0 0x7f318806ec70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31a00fde90 0x7f31a0113aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 --2- 172.21.15.111:0/2684379073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31a00fe830 0x7f31a0113fe0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:31.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 >> 172.21.15.111:0/2684379073 conn(0x7f31a0072bc0 msgr2=0x7f31a00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:31.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 shutdown_connections 2024-03-14T13:09:31.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:31.938+0000 7f31a51d6700 1 -- 172.21.15.111:0/2684379073 wait complete. 2024-03-14T13:09:32.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 -- 172.21.15.111:0/449337838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec102e20 msgr2=0x7f46ec1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 --2- 172.21.15.111:0/449337838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec102e20 0x7f46ec1031f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46dc004940 tx=0x7f46dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 -- 172.21.15.111:0/449337838 shutdown_connections 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 --2- 172.21.15.111:0/449337838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec1037c0 0x7f46ec0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 --2- 172.21.15.111:0/449337838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec102e20 0x7f46ec1031f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 -- 172.21.15.111:0/449337838 >> 172.21.15.111:0/449337838 conn(0x7f46ec0f89d0 msgr2=0x7f46ec0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 -- 172.21.15.111:0/449337838 shutdown_connections 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.037+0000 7f46f24ea700 1 -- 172.21.15.111:0/449337838 wait complete. 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 Processor -- start 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 -- start start 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec102e20 0x7f46ec113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec114670 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46f24ea700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec10da60 con 0x7f46ec102e20 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46eb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35064/0 (socket says 172.21.15.111:35064) 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46eb7fe700 1 -- 172.21.15.111:0/2219511243 learned_addr learned my addr 172.21.15.111:0/2219511243 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.038+0000 7f46ebfff700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec102e20 0x7f46ec113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46eb7fe700 1 -- 172.21.15.111:0/2219511243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec102e20 msgr2=0x7f46ec113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46eb7fe700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec102e20 0x7f46ec113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46eb7fe700 1 -- 172.21.15.111:0/2219511243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc005040 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46eb7fe700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f46e0002a50 tx=0x7f46e0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46e0007cc0 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec10dd40 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46e000f040 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46e000d920 con 0x7f46ec1037c0 2024-03-14T13:09:32.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.039+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec10e300 con 0x7f46ec1037c0 2024-03-14T13:09:32.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.040+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec0fe570 con 0x7f46ec1037c0 2024-03-14T13:09:32.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.040+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f46e0009680 con 0x7f46ec1037c0 2024-03-14T13:09:32.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.041+0000 7f46e8ff9700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 0x7f46d406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.041+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f46e008b770 con 0x7f46ec1037c0 2024-03-14T13:09:32.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.041+0000 7f46ebfff700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 0x7f46d406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.041+0000 7f46ebfff700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 0x7f46d406ec70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46dc004f90 tx=0x7f46dc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.045+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f46e00594a0 con 0x7f46ec1037c0 2024-03-14T13:09:32.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.321+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f46ec109de0 con 0x7f46ec1037c0 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.321+0000 7f46e8ff9700 1 -- 172.21.15.111:0/2219511243 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f46e0059030 con 0x7f46ec1037c0 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:09:32.321 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 5 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 4, 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 9 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:09:32.322 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 msgr2=0x7f46d406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 0x7f46d406ec70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f46dc004f90 tx=0x7f46dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 msgr2=0x7f46ec114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f46e0002a50 tx=0x7f46e0004600 comp rx=0 tx=0).stop 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 shutdown_connections 2024-03-14T13:09:32.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f46d406c7c0 0x7f46d406ec70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ec102e20 0x7f46ec113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 --2- 172.21.15.111:0/2219511243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46ec1037c0 0x7f46ec114020 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 >> 172.21.15.111:0/2219511243 conn(0x7f46ec0f89d0 msgr2=0x7f46ec0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 shutdown_connections 2024-03-14T13:09:32.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.323+0000 7f46f24ea700 1 -- 172.21.15.111:0/2219511243 wait complete. 2024-03-14T13:09:32.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 -- 172.21.15.111:0/2700720518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0101d70 msgr2=0x7f62d0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 --2- 172.21.15.111:0/2700720518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0101d70 0x7f62d0102140 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f62c0004940 tx=0x7f62c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 -- 172.21.15.111:0/2700720518 shutdown_connections 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 --2- 172.21.15.111:0/2700720518 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0102710 0x7f62d00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 --2- 172.21.15.111:0/2700720518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0101d70 0x7f62d0102140 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 -- 172.21.15.111:0/2700720518 >> 172.21.15.111:0/2700720518 conn(0x7f62d0072bc0 msgr2=0x7f62d0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 -- 172.21.15.111:0/2700720518 shutdown_connections 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.423+0000 7f62d6641700 1 -- 172.21.15.111:0/2700720518 wait complete. 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 Processor -- start 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 -- start start 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0101d70 0x7f62d0112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01134b0 con 0x7f62d0102710 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62d6641700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d010c8a0 con 0x7f62d0101d70 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35082/0 (socket says 172.21.15.111:35082) 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 -- 172.21.15.111:0/1874516536 learned_addr learned my addr 172.21.15.111:0/1874516536 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cffff700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0101d70 0x7f62d0112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 -- 172.21.15.111:0/1874516536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0101d70 msgr2=0x7f62d0112920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0101d70 0x7f62d0112920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.424+0000 7f62cf7fe700 1 -- 172.21.15.111:0/1874516536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0005040 con 0x7f62d0102710 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62cf7fe700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62c400fce0 tx=0x7f62c40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c4007ea0 con 0x7f62d0102710 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d010cb20 con 0x7f62d0102710 2024-03-14T13:09:32.426 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62c400dd80 con 0x7f62d0102710 2024-03-14T13:09:32.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62c40126a0 con 0x7f62d0102710 2024-03-14T13:09:32.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.425+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d010d0e0 con 0x7f62d0102710 2024-03-14T13:09:32.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.426+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d00fd4c0 con 0x7f62d0102710 2024-03-14T13:09:32.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.426+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f62c400d8d0 con 0x7f62d0102710 2024-03-14T13:09:32.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.427+0000 7f62ccff9700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 0x7f62b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.427+0000 7f62cffff700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 0x7f62b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.427+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f62c408def0 con 0x7f62d0102710 2024-03-14T13:09:32.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.427+0000 7f62cffff700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 0x7f62b806ec70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62c0004f90 tx=0x7f62c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.431+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f62c405bc20 con 0x7f62d0102710 2024-03-14T13:09:32.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.666+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f62d004bed0 con 0x7f62d0102710 2024-03-14T13:09:32.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.667+0000 7f62ccff9700 1 -- 172.21.15.111:0/1874516536 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+1875 (secure 0 0 0) 0x7f62c405b7b0 con 0x7f62d0102710 2024-03-14T13:09:32.666 INFO:teuthology.orchestra.run.smithi111.stdout:e22 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 22 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:05:42.299211+0000 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:09:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 0 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=14482,1=24261} 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:09:32.668 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:14482} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.111:6826/1337358162,v1:172.21.15.111:6827/1337358162] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{0:14494} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.111:6828/1527723924,v1:172.21.15.111:6829/1527723924] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{1:24261} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.137:6824/3856946079,v1:172.21.15.137:6825/3856946079] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:24273} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.137:6826/3644044473,v1:172.21.15.137:6827/3644044473] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 msgr2=0x7f62b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 0x7f62b806ec70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62c0004f90 tx=0x7f62c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 msgr2=0x7f62d0112e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62c400fce0 tx=0x7f62c40046b0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 shutdown_connections 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f62b806c7c0 0x7f62b806ec70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d0101d70 0x7f62d0112920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 --2- 172.21.15.111:0/1874516536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62d0102710 0x7f62d0112e60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 >> 172.21.15.111:0/1874516536 conn(0x7f62d0072bc0 msgr2=0x7f62d00faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 shutdown_connections 2024-03-14T13:09:32.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.669+0000 7f62d6641700 1 -- 172.21.15.111:0/1874516536 wait complete. 2024-03-14T13:09:32.671 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:09:32.726 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:32 smithi137 ceph-mon[108145]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:32.726 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:32 smithi137 ceph-mon[108145]: osdmap e77: 6 total, 5 up, 6 in 2024-03-14T13:09:32.726 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:32 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2219511243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 -- 172.21.15.111:0/726201787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c102f50 msgr2=0x7f514c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 --2- 172.21.15.111:0/726201787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c102f50 0x7f514c103320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5140004940 tx=0x7f5140009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 -- 172.21.15.111:0/726201787 shutdown_connections 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 --2- 172.21.15.111:0/726201787 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c0fb4d0 0x7f514c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 --2- 172.21.15.111:0/726201787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c102f50 0x7f514c103320 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 -- 172.21.15.111:0/726201787 >> 172.21.15.111:0/726201787 conn(0x7f514c0714c0 msgr2=0x7f514c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 -- 172.21.15.111:0/726201787 shutdown_connections 2024-03-14T13:09:32.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.770+0000 7f5151a50700 1 -- 172.21.15.111:0/726201787 wait complete. 2024-03-14T13:09:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 Processor -- start 2024-03-14T13:09:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 -- start start 2024-03-14T13:09:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c102f50 0x7f514c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c114470 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f5151a50700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c10da70 con 0x7f514c102f50 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f514affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f514affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35092/0 (socket says 172.21.15.111:35092) 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.771+0000 7f514affd700 1 -- 172.21.15.111:0/1430802607 learned_addr learned my addr 172.21.15.111:0/1430802607 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f514a7fc700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c102f50 0x7f514c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f514affd700 1 -- 172.21.15.111:0/1430802607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c102f50 msgr2=0x7f514c113e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f514affd700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c102f50 0x7f514c113e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f514affd700 1 -- 172.21.15.111:0/1430802607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5140005040 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f514affd700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5140009380 tx=0x7f5140009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514000e040 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c10dcf0 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5140013420 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f514001c630 con 0x7f514c0fb4d0 2024-03-14T13:09:32.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.772+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c10e230 con 0x7f514c0fb4d0 2024-03-14T13:09:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.773+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c100200 con 0x7f514c0fb4d0 2024-03-14T13:09:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.773+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f514000bb50 con 0x7f514c0fb4d0 2024-03-14T13:09:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.774+0000 7f51337fe700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 0x7f513406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.774+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f514008cbe0 con 0x7f514c0fb4d0 2024-03-14T13:09:32.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.774+0000 7f514a7fc700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 0x7f513406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:32.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.774+0000 7f514a7fc700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 0x7f513406ec70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f514c10ec30 tx=0x7f513c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:32.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.778+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f514005a990 con 0x7f514c0fb4d0 2024-03-14T13:09:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:32 smithi111 ceph-mon[145314]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:32 smithi111 ceph-mon[145314]: osdmap e77: 6 total, 5 up, 6 in 2024-03-14T13:09:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:32 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2219511243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:32.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.989+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f514c10ef30 con 0x7f513406c7c0 2024-03-14T13:09:32.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.991+0000 7f51337fe700 1 -- 172.21.15.111:0/1430802607 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f514c10ef30 con 0x7f513406c7c0 2024-03-14T13:09:32.990 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:09:32.990 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:09:32.990 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:09:32.990 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "11/21 daemons upgraded", 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading osd daemons", 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:09:32.991 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:09:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 msgr2=0x7f513406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 0x7f513406ec70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f514c10ec30 tx=0x7f513c009ee0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 msgr2=0x7f514c1138e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:32.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5140009380 tx=0x7f5140009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 shutdown_connections 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f513406c7c0 0x7f513406ec70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f514c0fb4d0 0x7f514c1138e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 --2- 172.21.15.111:0/1430802607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f514c102f50 0x7f514c113e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 >> 172.21.15.111:0/1430802607 conn(0x7f514c0714c0 msgr2=0x7f514c0fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 shutdown_connections 2024-03-14T13:09:32.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:32.993+0000 7f5151a50700 1 -- 172.21.15.111:0/1430802607 wait complete. 2024-03-14T13:09:33.005 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:32 smithi137 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.5.service: Succeeded. 2024-03-14T13:09:33.005 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:32 smithi137 systemd[1]: Stopped Ceph osd.5 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/479031793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 msgr2=0x7f0e981031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/479031793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e981031d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c004940 tx=0x7f0e8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/479031793 shutdown_connections 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/479031793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 0x7f0e980fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/479031793 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e981031d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/479031793 >> 172.21.15.111:0/479031793 conn(0x7f0e980f89b0 msgr2=0x7f0e980fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.095+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/479031793 shutdown_connections 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/479031793 wait complete. 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 Processor -- start 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 -- start start 2024-03-14T13:09:33.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 0x7f0e98070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98070e20 con 0x7f0e98102e00 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9f0c4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98070f60 con 0x7f0e981037a0 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.096+0000 7f0e9ce60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35120/0 (socket says 172.21.15.111:35120) 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 -- 172.21.15.111:0/4251525682 learned_addr learned my addr 172.21.15.111:0/4251525682 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e97fff700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 0x7f0e98070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 -- 172.21.15.111:0/4251525682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 msgr2=0x7f0e98070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 0x7f0e98070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 -- 172.21.15.111:0/4251525682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c005040 con 0x7f0e98102e00 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9ce60700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009360 tx=0x7f0e8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c00e040 con 0x7f0e98102e00 2024-03-14T13:09:33.098 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e98119da0 con 0x7f0e98102e00 2024-03-14T13:09:33.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e8c013420 con 0x7f0e98102e00 2024-03-14T13:09:33.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e8c01c5d0 con 0x7f0e98102e00 2024-03-14T13:09:33.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.097+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9811a280 con 0x7f0e98102e00 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.098+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e980fe550 con 0x7f0e98102e00 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.099+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0e8c013590 con 0x7f0e98102e00 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.099+0000 7f0e957fa700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 0x7f0e8806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.099+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6324+0+0 (secure 0 0 0) 0x7f0e8c08cb70 con 0x7f0e98102e00 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.099+0000 7f0e97fff700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 0x7f0e8806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:09:33.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.099+0000 7f0e97fff700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 0x7f0e8806ec80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e98071c00 tx=0x7f0e840091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:09:33.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.103+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0e8c056e10 con 0x7f0e98102e00 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.375+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f0e9804bf00 con 0x7f0e98102e00 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.375+0000 7f0e957fa700 1 -- 172.21.15.111:0/4251525682 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+515 (secure 0 0 0) 0x7f0e8c05a430 con 0x7f0e98102e00 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_WARN 1 osds down; Reduced data availability: 4 pgs peering 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] OSD_DOWN: 1 osds down 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout: osd.5 (root=default,host=smithi137) is down 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 4 pgs peering 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.2 is stuck peering for 6m, current state peering, last acting [1,0] 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.3 is stuck peering for 7m, current state peering, last acting [2,1] 2024-03-14T13:09:33.375 INFO:teuthology.orchestra.run.smithi111.stdout: pg 2.17 is stuck peering for 2m, current state peering, last acting [2,1] 2024-03-14T13:09:33.376 INFO:teuthology.orchestra.run.smithi111.stdout: pg 3.5 is stuck peering for 5m, current state peering, last acting [2,3] 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 msgr2=0x7f0e8806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 0x7f0e8806ec80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0e98071c00 tx=0x7f0e840091a0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 msgr2=0x7f0e980702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009360 tx=0x7f0e8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 shutdown_connections 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0e8806c7d0 0x7f0e8806ec80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e98102e00 0x7f0e980702c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 --2- 172.21.15.111:0/4251525682 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e981037a0 0x7f0e98070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 >> 172.21.15.111:0/4251525682 conn(0x7f0e980f89b0 msgr2=0x7f0e980f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 shutdown_connections 2024-03-14T13:09:33.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:09:33.377+0000 7f0e9f0c4700 1 -- 172.21.15.111:0/4251525682 wait complete. 2024-03-14T13:09:33.393 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 systemd[1]: Starting Ceph osd.5 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:09:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: pgmap v154: 65 pgs: 8 peering, 9 stale+active+clean, 48 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-14T13:09:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1874516536' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:09:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:33 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4251525682' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:09:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: from='client.34348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: pgmap v154: 65 pgs: 8 peering, 9 stale+active+clean, 48 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-03-14T13:09:33.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1874516536' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:09:33.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:33 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4251525682' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-03-14T13:09:34.232 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-14T13:09:34.233 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:33 smithi137 bash[133864]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-14T13:09:34.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:34 smithi137 ceph-mon[108145]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:34 smithi111 ceph-mon[145314]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:09:35.169 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-osd[134164]: -- 172.21.15.137:0/2164335138 <== mon.1 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568fb4b11e0 con 0x5568fb481000 2024-03-14T13:09:35.442 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:35 smithi137 bash[134126]: 98cbc02db785ae56bf085978b7597543a50b4a6cffee6900acff7fc80c0e4963 2024-03-14T13:09:35.443 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:35 smithi137 systemd[1]: Started Ceph osd.5 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:09:35.755 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:09:35.616+0000 7f064a499640 -1 Falling back to public interface 2024-03-14T13:09:35.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-mon[108145]: pgmap v155: 65 pgs: 6 active+undersized+degraded, 8 peering, 6 stale+active+clean, 4 active+undersized, 41 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/321 objects degraded (9.346%) 2024-03-14T13:09:35.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-mon[108145]: Health check failed: Degraded data redundancy: 30/321 objects degraded (9.346%), 6 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:35.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:35.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:35.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:35 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:35 smithi111 ceph-mon[145314]: pgmap v155: 65 pgs: 6 active+undersized+degraded, 8 peering, 6 stale+active+clean, 4 active+undersized, 41 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 30/321 objects degraded (9.346%) 2024-03-14T13:09:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:35 smithi111 ceph-mon[145314]: Health check failed: Degraded data redundancy: 30/321 objects degraded (9.346%), 6 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:35.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:35.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:35.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:35 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:37.738 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:37 smithi137 ceph-mon[108145]: pgmap v156: 65 pgs: 11 active+undersized+degraded, 5 peering, 12 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 44/321 objects degraded (13.707%) 2024-03-14T13:09:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:37 smithi111 ceph-mon[145314]: pgmap v156: 65 pgs: 11 active+undersized+degraded, 5 peering, 12 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 44/321 objects degraded (13.707%) 2024-03-14T13:09:39.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:39 smithi137 ceph-mon[108145]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-14T13:09:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:39 smithi111 ceph-mon[145314]: pgmap v157: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-14T13:09:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:39.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:40.359 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:40 smithi137 ceph-mon[108145]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-14T13:09:40.359 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:40 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:40.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:40 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:40.360 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:40 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:09:40.062+0000 7f064a499640 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-14T13:09:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:40 smithi111 ceph-mon[145314]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-14T13:09:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:40 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:40.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:40 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:40.690 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:40 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:09:40.468+0000 7f064a499640 -1 osd.5 75 log_to_monitors true 2024-03-14T13:09:41.505 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:09:41 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:09:41.208+0000 7f06398ee700 -1 osd.5 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-14T13:09:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:41 smithi137 ceph-mon[108145]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-14T13:09:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:41 smithi137 ceph-mon[108145]: from='osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T13:09:41.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:41 smithi137 ceph-mon[108145]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T13:09:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:41 smithi111 ceph-mon[145314]: pgmap v158: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 51/321 objects degraded (15.888%) 2024-03-14T13:09:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:41 smithi111 ceph-mon[145314]: from='osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T13:09:41.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:41 smithi111 ceph-mon[145314]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-14T13:09:42.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:42 smithi137 ceph-mon[108145]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T13:09:42.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:42 smithi137 ceph-mon[108145]: osdmap e78: 6 total, 5 up, 6 in 2024-03-14T13:09:42.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:42 smithi137 ceph-mon[108145]: from='osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:42.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:42 smithi137 ceph-mon[108145]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:42 smithi111 ceph-mon[145314]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-14T13:09:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:42 smithi111 ceph-mon[145314]: osdmap e78: 6 total, 5 up, 6 in 2024-03-14T13:09:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:42 smithi111 ceph-mon[145314]: from='osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:42.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:42 smithi111 ceph-mon[145314]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-03-14T13:09:43.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345] boot 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: osdmap e79: 6 total, 6 up, 6 in 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 51/321 objects degraded (15.888%) 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-14T13:09:43.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-14T13:09:43.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-14T13:09:43.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-14T13:09:43.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:43 smithi137 ceph-mon[108145]: osdmap e80: 6 total, 6 up, 6 in 2024-03-14T13:09:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: osd.5 [v2:172.21.15.137:6816/4240986345,v1:172.21.15.137:6817/4240986345] boot 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: osdmap e79: 6 total, 6 up, 6 in 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: pgmap v161: 65 pgs: 14 active+undersized+degraded, 14 active+undersized, 37 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 0 B/s rd, 3 op/s; 51/321 objects degraded (15.888%) 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-14T13:09:43.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-14T13:09:43.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-14T13:09:43.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd require-osd-release", "release": "reef"}]: dispatch 2024-03-14T13:09:43.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "osd require-osd-release", "release": "reef"}]': finished 2024-03-14T13:09:43.643 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:43 smithi111 ceph-mon[145314]: osdmap e80: 6 total, 6 up, 6 in 2024-03-14T13:09:44.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all osd 2024-03-14T13:09:44.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: Upgrade: Setting require_osd_release to 18 reef 2024-03-14T13:09:44.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-14T13:09:44.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:44.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-14T13:09:44.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:44 smithi137 ceph-mon[108145]: osdmap e81: 6 total, 6 up, 6 in 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all osd 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: Upgrade: Setting require_osd_release to 18 reef 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: Upgrade: Disabling standby-replay for filesystem cephfs 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]: dispatch 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-mon[145314]: osdmap e81: 6 total, 6 up, 6 in 2024-03-14T13:09:44.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:44 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:44.382+0000 7eff5268a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:45.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: pgmap v164: 65 pgs: 7 peering, 9 active+undersized+degraded, 12 active+undersized, 37 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 25/321 objects degraded (7.788%) 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: osdmap e82: 6 total, 6 up, 6 in 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.137:6826/513022872,v1:172.21.15.137:6827/513022872] up:boot 2024-03-14T13:09:45.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/1531215732,v1:172.21.15.111:6829/1531215732] up:boot 2024-03-14T13:09:45.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-14T13:09:45.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-14T13:09:45.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:09:45.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:09:45.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:45 smithi137 ceph-mon[108145]: Health check update: Degraded data redundancy: 51/321 objects degraded (15.888%), 14 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "0"}]': finished 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi137.ryucox=up:active} 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:45.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: pgmap v164: 65 pgs: 7 peering, 9 active+undersized+degraded, 12 active+undersized, 37 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 4 op/s; 25/321 objects degraded (7.788%) 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Upgrade: failing fs cephfs for rapid multi-rank mds upgrade 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-14T13:09:45.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: osdmap e82: 6 total, 6 up, 6 in 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.137:6826/513022872,v1:172.21.15.137:6827/513022872] up:boot 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/1531215732,v1:172.21.15.111:6829/1531215732] up:boot 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 2 up:standby, 2 failed 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:09:45.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:45 smithi111 ceph-mon[145314]: Health check update: Degraded data redundancy: 51/321 objects degraded (15.888%), 14 pgs degraded (PG_DEGRADED) 2024-03-14T13:09:46.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.137:6824/1043472821,v1:172.21.15.137:6825/1043472821] up:boot 2024-03-14T13:09:46.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/3595552801,v1:172.21.15.111:6827/3595552801] up:boot 2024-03-14T13:09:46.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:09:46.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:46.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:09:46.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:46.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:09:46.692 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.137:6824/1043472821,v1:172.21.15.137:6825/1043472821] up:boot 2024-03-14T13:09:46.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/3595552801,v1:172.21.15.111:6827/3595552801] up:boot 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.zmslha", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:09:46.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:09:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:47 smithi111 ceph-mon[145314]: Upgrade: Updating mds.cephfs.smithi111.zmslha 2024-03-14T13:09:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:47 smithi111 ceph-mon[145314]: Deploying daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T13:09:47.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:47 smithi111 ceph-mon[145314]: pgmap v166: 65 pgs: 7 peering, 1 active+undersized+degraded, 1 active+undersized, 56 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2/321 objects degraded (0.623%) 2024-03-14T13:09:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:47 smithi137 ceph-mon[108145]: Upgrade: Updating mds.cephfs.smithi111.zmslha 2024-03-14T13:09:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:47 smithi137 ceph-mon[108145]: Deploying daemon mds.cephfs.smithi111.zmslha on smithi111 2024-03-14T13:09:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:47 smithi137 ceph-mon[108145]: pgmap v166: 65 pgs: 7 peering, 1 active+undersized+degraded, 1 active+undersized, 56 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2/321 objects degraded (0.623%) 2024-03-14T13:09:49.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:49 smithi111 ceph-mon[145314]: osdmap e83: 6 total, 6 up, 6 in 2024-03-14T13:09:49.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:49 smithi111 ceph-mon[145314]: pgmap v168: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:49.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:49 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:09:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:49 smithi137 ceph-mon[108145]: osdmap e83: 6 total, 6 up, 6 in 2024-03-14T13:09:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:49 smithi137 ceph-mon[108145]: pgmap v168: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:49 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:09:50.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:50 smithi111 ceph-mon[145314]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/321 objects degraded (0.623%), 1 pg degraded) 2024-03-14T13:09:50.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:50 smithi137 ceph-mon[108145]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/321 objects degraded (0.623%), 1 pg degraded) 2024-03-14T13:09:51.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:51 smithi137 ceph-mon[108145]: pgmap v169: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:51.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:51 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:51 smithi111 ceph-mon[145314]: pgmap v169: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:51.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:51 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:53.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:53 smithi111 ceph-mon[145314]: pgmap v170: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:53 smithi137 ceph-mon[108145]: pgmap v170: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:55.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: pgmap v171: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:boot 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:55.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:09:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: pgmap v171: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:boot 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:09:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:57.604 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:57 smithi111 ceph-mon[145314]: pgmap v172: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-14T13:09:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:57 smithi137 ceph-mon[108145]: pgmap v172: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 102 B/s rd, 0 op/s 2024-03-14T13:09:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:59 smithi111 ceph-mon[145314]: pgmap v173: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:59 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:59.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:09:59 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:59 smithi137 ceph-mon[108145]: pgmap v173: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:09:59.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:59 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:09:59.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:09:59 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:00.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:59.998+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:10:00.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:59.998+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:10:00.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:59.998+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : fs cephfs is degraded 2024-03-14T13:10:00.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:59.998+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:10:00.334 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:09:59.998+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: fs cephfs is degraded 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:10:01.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:00 smithi111 ceph-mon[145314]: pgmap v174: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:01.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: Health detail: HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: fs cephfs is degraded 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:10:01.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:00 smithi137 ceph-mon[108145]: pgmap v174: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- 172.21.15.111:0/2450582045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4101d70 msgr2=0x7faac4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 --2- 172.21.15.111:0/2450582045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4101d70 0x7faac4102140 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faab8004940 tx=0x7faab8009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- 172.21.15.111:0/2450582045 shutdown_connections 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 --2- 172.21.15.111:0/2450582045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4102710 0x7faac40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 --2- 172.21.15.111:0/2450582045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4101d70 0x7faac4102140 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- 172.21.15.111:0/2450582045 >> 172.21.15.111:0/2450582045 conn(0x7faac4072bc0 msgr2=0x7faac4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- 172.21.15.111:0/2450582045 shutdown_connections 2024-03-14T13:10:03.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- 172.21.15.111:0/2450582045 wait complete. 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 Processor -- start 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.481+0000 7faacb011700 1 -- start start 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faacb011700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 0x7faac41129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faacb011700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faacb011700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac41135d0 con 0x7faac4102710 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faacb011700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac410c930 con 0x7faac4101d70 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46010/0 (socket says 172.21.15.111:46010) 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 -- 172.21.15.111:0/309144199 learned_addr learned my addr 172.21.15.111:0/309144199 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac8dad700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 0x7faac41129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 -- 172.21.15.111:0/309144199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 msgr2=0x7faac41129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 0x7faac41129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 -- 172.21.15.111:0/309144199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8005040 con 0x7faac4102710 2024-03-14T13:10:03.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac8dad700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 0x7faac41129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.482+0000 7faac3fff700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faab000daf0 tx=0x7faab000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.483+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab000ed70 con 0x7faac4102710 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.483+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faab0017920 con 0x7faac4102710 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.483+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac410cbb0 con 0x7faac4102710 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.483+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faab0016970 con 0x7faac4102710 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.483+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac410d170 con 0x7faac4102710 2024-03-14T13:10:03.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.484+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac40fd4c0 con 0x7faac4102710 2024-03-14T13:10:03.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.484+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faab0014070 con 0x7faac4102710 2024-03-14T13:10:03.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.484+0000 7faac17fa700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 0x7faab406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.485+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faab008dc80 con 0x7faac4102710 2024-03-14T13:10:03.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.485+0000 7faac8dad700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 0x7faab406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.485+0000 7faac8dad700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 0x7faab406ec70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faab8004560 tx=0x7faab8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:03.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.489+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faab005b990 con 0x7faac4102710 2024-03-14T13:10:03.621 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:03 smithi111 ceph-mon[145314]: pgmap v175: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:03.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.724+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faac4108e10 con 0x7faab406c7c0 2024-03-14T13:10:03.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.726+0000 7faac17fa700 1 -- 172.21.15.111:0/309144199 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7faac4108e10 con 0x7faab406c7c0 2024-03-14T13:10:03.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 msgr2=0x7faab406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 0x7faab406ec70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faab8004560 tx=0x7faab8009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:03.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 msgr2=0x7faac4112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faab000daf0 tx=0x7faab000b4c0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 shutdown_connections 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faab406c7c0 0x7faab406ec70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faac4101d70 0x7faac41129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 --2- 172.21.15.111:0/309144199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faac4102710 0x7faac4112ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 >> 172.21.15.111:0/309144199 conn(0x7faac4072bc0 msgr2=0x7faac40faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 shutdown_connections 2024-03-14T13:10:03.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.728+0000 7faacb011700 1 -- 172.21.15.111:0/309144199 wait complete. 2024-03-14T13:10:03.740 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:10:03.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:03 smithi137 ceph-mon[108145]: pgmap v175: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:03.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- 172.21.15.111:0/2125137189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa080fb4a0 msgr2=0x7ffa080fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 --2- 172.21.15.111:0/2125137189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa080fb4a0 0x7ffa080fb910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4004940 tx=0x7ff9f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:03.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- 172.21.15.111:0/2125137189 shutdown_connections 2024-03-14T13:10:03.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 --2- 172.21.15.111:0/2125137189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa080fb4a0 0x7ffa080fb910 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 --2- 172.21.15.111:0/2125137189 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa08102e70 0x7ffa08103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- 172.21.15.111:0/2125137189 >> 172.21.15.111:0/2125137189 conn(0x7ffa080714c0 msgr2=0x7ffa080718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:03.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- 172.21.15.111:0/2125137189 shutdown_connections 2024-03-14T13:10:03.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- 172.21.15.111:0/2125137189 wait complete. 2024-03-14T13:10:03.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 Processor -- start 2024-03-14T13:10:03.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.832+0000 7ffa0f238700 1 -- start start 2024-03-14T13:10:03.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0f238700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa080fb4a0 0x7ffa08115d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0f238700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0f238700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa08116940 con 0x7ffa08102e70 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0f238700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0810fc80 con 0x7ffa080fb4a0 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46032/0 (socket says 172.21.15.111:46032) 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 -- 172.21.15.111:0/3537736431 learned_addr learned my addr 172.21.15.111:0/3537736431 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0cfd4700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa080fb4a0 0x7ffa08115d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 -- 172.21.15.111:0/3537736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa080fb4a0 msgr2=0x7ffa08115d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa080fb4a0 0x7ffa08115d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 -- 172.21.15.111:0/3537736431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f4005040 con 0x7ffa08102e70 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa07fff700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009360 tx=0x7ff9f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9f400e040 con 0x7ffa08102e70 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.833+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa0810fea0 con 0x7ffa08102e70 2024-03-14T13:10:03.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.834+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9f4013420 con 0x7ffa08102e70 2024-03-14T13:10:03.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.834+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9f401c5d0 con 0x7ffa08102e70 2024-03-14T13:10:03.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.834+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa081103e0 con 0x7ffa08102e70 2024-03-14T13:10:03.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.835+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa081069e0 con 0x7ffa08102e70 2024-03-14T13:10:03.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.835+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff9f4013590 con 0x7ffa08102e70 2024-03-14T13:10:03.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.835+0000 7ffa057fa700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 0x7ff9f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:03.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.835+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff9f408cc10 con 0x7ffa08102e70 2024-03-14T13:10:03.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.835+0000 7ffa0cfd4700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 0x7ff9f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:03.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.836+0000 7ffa0cfd4700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 0x7ff9f806ec80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009bf0 tx=0x7ff9fc0091a0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:03.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:03.840+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff9f405a8a0 con 0x7ffa08102e70 2024-03-14T13:10:04.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.062+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa08109ea0 con 0x7ff9f806c7d0 2024-03-14T13:10:04.064 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.064+0000 7ffa057fa700 1 -- 172.21.15.111:0/3537736431 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7ffa08109ea0 con 0x7ff9f806c7d0 2024-03-14T13:10:04.066 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 msgr2=0x7ff9f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 0x7ff9f806ec80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009bf0 tx=0x7ff9fc0091a0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 msgr2=0x7ffa08116260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9f4009360 tx=0x7ff9f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 shutdown_connections 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff9f806c7d0 0x7ff9f806ec80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa080fb4a0 0x7ffa08115d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 --2- 172.21.15.111:0/3537736431 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa08102e70 0x7ffa08116260 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 >> 172.21.15.111:0/3537736431 conn(0x7ffa080714c0 msgr2=0x7ffa080fa540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 shutdown_connections 2024-03-14T13:10:04.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.066+0000 7ffa0f238700 1 -- 172.21.15.111:0/3537736431 wait complete. 2024-03-14T13:10:04.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.172+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/517162198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38101d70 msgr2=0x7f1b38102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.172+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/517162198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38101d70 0x7f1b38102140 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b28004940 tx=0x7f1b28009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.172+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/517162198 shutdown_connections 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.172+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/517162198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38102710 0x7f1b380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/517162198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38101d70 0x7f1b38102140 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/517162198 >> 172.21.15.111:0/517162198 conn(0x7f1b38072bc0 msgr2=0x7f1b38072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/517162198 shutdown_connections 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/517162198 wait complete. 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 Processor -- start 2024-03-14T13:10:04.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- start start 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38101d70 0x7f1b38102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38103730 con 0x7f1b38102710 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.173+0000 7f1b3e7d4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b38103870 con 0x7f1b38101d70 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46048/0 (socket says 172.21.15.111:46048) 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 -- 172.21.15.111:0/3058043306 learned_addr learned my addr 172.21.15.111:0/3058043306 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b37fff700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38101d70 0x7f1b38102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 -- 172.21.15.111:0/3058043306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38101d70 msgr2=0x7f1b38102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38101d70 0x7f1b38102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 -- 172.21.15.111:0/3058043306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b28005040 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b377fe700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b790 tx=0x7f1b2c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b2c00f6f0 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b2c00fd00 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b2c0103a0 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b38119d70 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.174+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3811a280 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.175+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b380fd4c0 con 0x7f1b38102710 2024-03-14T13:10:04.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.176+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1b2c00f850 con 0x7f1b38102710 2024-03-14T13:10:04.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.176+0000 7f1b34ff9700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 0x7f1b2006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.176+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1b2c08b920 con 0x7f1b38102710 2024-03-14T13:10:04.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.176+0000 7f1b37fff700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 0x7f1b2006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.177+0000 7f1b37fff700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 0x7f1b2006ec70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1b28004f90 tx=0x7f1b28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.180+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1b2c055b20 con 0x7f1b38102710 2024-03-14T13:10:04.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.393+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b3804bed0 con 0x7f1b2006c7c0 2024-03-14T13:10:04.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.405+0000 7f1b34ff9700 1 -- 172.21.15.111:0/3058043306 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f1b3804bed0 con 0x7f1b2006c7c0 2024-03-14T13:10:04.406 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:10:04.406 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 5s ago 17m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:10:04.406 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (3m) 5s ago 18m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:10:04.406 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (3m) 26s ago 16m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (5m) 5s ago 17m 90.2M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (11m) 5s ago 11m 10.5M - 17.2.7-646-gb92acd1f a07fb176315d 564f07239559 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (11s) 5s ago 11m 12.5M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (11m) 26s ago 11m 67.5M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (11m) 26s ago 11m 115M - 17.2.7-646-gb92acd1f a07fb176315d 2d95f98eec9d 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (8m) 5s ago 19m 504M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (7m) 26s ago 16m 451M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (4m) 5s ago 19m 56.7M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (4m) 26s ago 16m 39.6M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 5s ago 18m 19.2M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 26s ago 16m 18.4M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (3m) 5s ago 15m 88.5M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (2m) 5s ago 15m 95.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (2m) 5s ago 14m 84.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:10:04.407 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (93s) 26s ago 14m 99.9M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:10:04.408 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (61s) 26s ago 13m 94.0M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:10:04.408 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (29s) 26s ago 13m 10.9M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:10:04.408 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (6m) 5s ago 16m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 msgr2=0x7f1b2006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 0x7f1b2006ec70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1b28004f90 tx=0x7f1b28009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 msgr2=0x7f1b38103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b790 tx=0x7f1b2c009220 comp rx=0 tx=0).stop 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 shutdown_connections 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1b2006c7c0 0x7f1b2006ec70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b38101d70 0x7f1b38102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 --2- 172.21.15.111:0/3058043306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b38102710 0x7f1b38103110 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 >> 172.21.15.111:0/3058043306 conn(0x7f1b38072bc0 msgr2=0x7f1b38065970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 shutdown_connections 2024-03-14T13:10:04.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.408+0000 7f1b3e7d4700 1 -- 172.21.15.111:0/3058043306 wait complete. 2024-03-14T13:10:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- 172.21.15.111:0/701389516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50101d70 msgr2=0x7f5e50102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/701389516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50101d70 0x7f5e50102140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5e40004940 tx=0x7f5e40009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- 172.21.15.111:0/701389516 shutdown_connections 2024-03-14T13:10:04.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/701389516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50102710 0x7f5e500fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/701389516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50101d70 0x7f5e50102140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- 172.21.15.111:0/701389516 >> 172.21.15.111:0/701389516 conn(0x7f5e50072bc0 msgr2=0x7f5e50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- 172.21.15.111:0/701389516 shutdown_connections 2024-03-14T13:10:04.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- 172.21.15.111:0/701389516 wait complete. 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 Processor -- start 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.513+0000 7f5e54e4b700 1 -- start start 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e54e4b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50101d70 0x7f5e501129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e54e4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e54e4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e50113530 con 0x7f5e50102710 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e54e4b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5010c920 con 0x7f5e50101d70 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46076/0 (socket says 172.21.15.111:46076) 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 -- 172.21.15.111:0/2049950208 learned_addr learned my addr 172.21.15.111:0/2049950208 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4e59c700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50101d70 0x7f5e501129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 -- 172.21.15.111:0/2049950208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50101d70 msgr2=0x7f5e501129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50101d70 0x7f5e501129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 -- 172.21.15.111:0/2049950208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e40005040 con 0x7f5e50102710 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.514+0000 7f5e4dd9b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c800 tx=0x7f5e44004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.515+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e440106f0 con 0x7f5e50102710 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.515+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e44010d00 con 0x7f5e50102710 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.515+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5010cc00 con 0x7f5e50102710 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.515+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e440113a0 con 0x7f5e50102710 2024-03-14T13:10:04.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.515+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5010d1c0 con 0x7f5e50102710 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.516+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e500ff080 con 0x7f5e50102710 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.516+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5e44012400 con 0x7f5e50102710 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.516+0000 7f5e3effd700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 0x7f5e3806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.517+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e4408b900 con 0x7f5e50102710 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.517+0000 7f5e4e59c700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 0x7f5e3806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.517+0000 7f5e4e59c700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 0x7f5e3806ec70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e40004f90 tx=0x7f5e40009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.521 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.521+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e440596d0 con 0x7f5e50102710 2024-03-14T13:10:04.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.809+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e5010dff0 con 0x7f5e50102710 2024-03-14T13:10:04.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.809+0000 7f5e3effd700 1 -- 172.21.15.111:0/2049950208 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f5e44059260 con 0x7f5e50102710 2024-03-14T13:10:04.810 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 3, 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 1 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:10:04.811 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 3, 2024-03-14T13:10:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 11 2024-03-14T13:10:04.812 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:10:04.812 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:10:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 msgr2=0x7f5e3806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 0x7f5e3806ec70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e40004f90 tx=0x7f5e40009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 msgr2=0x7f5e50112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c800 tx=0x7f5e44004470 comp rx=0 tx=0).stop 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 shutdown_connections 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3806c7c0 0x7f5e3806ec70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e50101d70 0x7f5e501129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 --2- 172.21.15.111:0/2049950208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e50102710 0x7f5e50112ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 >> 172.21.15.111:0/2049950208 conn(0x7f5e50072bc0 msgr2=0x7f5e500fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 shutdown_connections 2024-03-14T13:10:04.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.812+0000 7f5e54e4b700 1 -- 172.21.15.111:0/2049950208 wait complete. 2024-03-14T13:10:04.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.913+0000 7f8fe1453700 1 -- 172.21.15.111:0/165340576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 msgr2=0x7f8fdc0fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.913+0000 7f8fe1453700 1 --2- 172.21.15.111:0/165340576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc0fcdf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0004940 tx=0x7f8fd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:04.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- 172.21.15.111:0/165340576 shutdown_connections 2024-03-14T13:10:04.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 --2- 172.21.15.111:0/165340576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc0fcdf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 --2- 172.21.15.111:0/165340576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- 172.21.15.111:0/165340576 >> 172.21.15.111:0/165340576 conn(0x7f8fdc0f8950 msgr2=0x7f8fdc0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:04.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- 172.21.15.111:0/165340576 shutdown_connections 2024-03-14T13:10:04.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- 172.21.15.111:0/165340576 wait complete. 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 Processor -- start 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- start start 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc114440 con 0x7f8fdc103780 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.914+0000 7f8fe1453700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc10da40 con 0x7f8fdc102de0 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46092/0 (socket says 172.21.15.111:46092) 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 -- 172.21.15.111:0/3483042109 learned_addr learned my addr 172.21.15.111:0/3483042109 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:04.917 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fdaffd700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 -- 172.21.15.111:0/3483042109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 msgr2=0x7f8fdc1138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 -- 172.21.15.111:0/3483042109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0005040 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fdaffd700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fda7fc700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009380 tx=0x7f8fd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd000e040 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fdc10dcc0 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8fd0013420 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd001c630 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.915+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fdc10e280 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.916+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fdc0fe530 con 0x7f8fdc103780 2024-03-14T13:10:04.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.917+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8fd00139e0 con 0x7f8fdc103780 2024-03-14T13:10:04.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.917+0000 7f8fc37fe700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 0x7f8fc406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:04.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.917+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8fd008cc60 con 0x7f8fdc103780 2024-03-14T13:10:04.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.917+0000 7f8fdaffd700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 0x7f8fc406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:04.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.917+0000 7f8fdaffd700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 0x7f8fc406ec70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009fd0 tx=0x7f8fcc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:04.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:04.922+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8fd005a970 con 0x7f8fdc103780 2024-03-14T13:10:05.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: pgmap v176: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:05.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:04 smithi137 ceph-mon[108145]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.123 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.123 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.123 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='client.34396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: pgmap v176: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:05.124 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:04 smithi111 ceph-mon[145314]: from='client.34400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.164+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8fdc109d80 con 0x7f8fdc103780 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.164+0000 7f8fc37fe700 1 -- 172.21.15.111:0/3483042109 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f8fd005a500 con 0x7f8fdc103780 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:e27 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:10:05.165 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:10:05.166 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:05.167 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34376} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/1531215732,v1:172.21.15.111:6829/1531215732] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:05.168 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:05.168 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/513022872,v1:172.21.15.137:6827/513022872] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:05.168 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44287} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/1043472821,v1:172.21.15.137:6825/1043472821] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 msgr2=0x7f8fc406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 0x7f8fc406ec70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009fd0 tx=0x7f8fcc00b040 comp rx=0 tx=0).stop 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 msgr2=0x7f8fdc113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009380 tx=0x7f8fd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 shutdown_connections 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8fc406c7c0 0x7f8fc406ec70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fdc102de0 0x7f8fdc1138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 --2- 172.21.15.111:0/3483042109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fdc103780 0x7f8fdc113df0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 >> 172.21.15.111:0/3483042109 conn(0x7f8fdc0f8950 msgr2=0x7f8fdc0f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 shutdown_connections 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.166+0000 7f8fe1453700 1 -- 172.21.15.111:0/3483042109 wait complete. 2024-03-14T13:10:05.169 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 27 2024-03-14T13:10:05.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/2670496443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 msgr2=0x7fb9e40fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/2670496443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e40fc650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8004940 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/2670496443 shutdown_connections 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/2670496443 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e40fc650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/2670496443 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.267+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/2670496443 >> 172.21.15.111:0/2670496443 conn(0x7fb9e4071530 msgr2=0x7fb9e4071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.268+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/2670496443 shutdown_connections 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.268+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/2670496443 wait complete. 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.268+0000 7fb9e8ce0700 1 Processor -- start 2024-03-14T13:10:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.268+0000 7fb9e8ce0700 1 -- start start 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.268+0000 7fb9e8ce0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e41127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e8ce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e8ce0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4113350 con 0x7fb9e4102800 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e8ce0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e410c950 con 0x7fb9e4101e60 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46118/0 (socket says 172.21.15.111:46118) 2024-03-14T13:10:05.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 -- 172.21.15.111:0/1842181745 learned_addr learned my addr 172.21.15.111:0/1842181745 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 -- 172.21.15.111:0/1842181745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 msgr2=0x7fb9e41127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e259c700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e41127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e41127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 -- 172.21.15.111:0/1842181745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e259c700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e41127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e1d9b700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80093b0 tx=0x7fb9d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d800e040 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e410cbd0 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9d8013420 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9d801c630 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.269+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e410d190 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.270+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e40ff1c0 con 0x7fb9e4102800 2024-03-14T13:10:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.271+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb9d80139e0 con 0x7fb9e4102800 2024-03-14T13:10:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.271+0000 7fb9d2ffd700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 0x7fb9cc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.271+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9d808ccc0 con 0x7fb9e4102800 2024-03-14T13:10:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.271+0000 7fb9e259c700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 0x7fb9cc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.272+0000 7fb9e259c700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 0x7fb9cc06ec70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4007fd0 tx=0x7fb9d4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:05.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.276+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb9d805a9d0 con 0x7fb9e4102800 2024-03-14T13:10:05.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.489+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9e410de10 con 0x7fb9cc06c7c0 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.491+0000 7fb9d2ffd700 1 -- 172.21.15.111:0/1842181745 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb9e410de10 con 0x7fb9cc06c7c0 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:10:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "13/21 daemons upgraded", 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:10:05.492 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:10:05.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 msgr2=0x7fb9cc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 0x7fb9cc06ec70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4007fd0 tx=0x7fb9d4009040 comp rx=0 tx=0).stop 2024-03-14T13:10:05.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 msgr2=0x7fb9e4112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80093b0 tx=0x7fb9d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 shutdown_connections 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb9cc06c7c0 0x7fb9cc06ec70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9e4101e60 0x7fb9e41127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 --2- 172.21.15.111:0/1842181745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e4102800 0x7fb9e4112d00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 >> 172.21.15.111:0/1842181745 conn(0x7fb9e4071530 msgr2=0x7fb9e40fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 shutdown_connections 2024-03-14T13:10:05.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.493+0000 7fb9e8ce0700 1 -- 172.21.15.111:0/1842181745 wait complete. 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/3180502604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fde80 msgr2=0x7f6ed00fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/3180502604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fde80 0x7f6ed00fe250 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0004940 tx=0x7f6ec0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/3180502604 shutdown_connections 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/3180502604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fe820 0x7f6ed0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/3180502604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fde80 0x7f6ed00fe250 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/3180502604 >> 172.21.15.111:0/3180502604 conn(0x7f6ed00f78f0 msgr2=0x7f6ed00f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/3180502604 shutdown_connections 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/3180502604 wait complete. 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.602+0000 7f6ed4dc4700 1 Processor -- start 2024-03-14T13:10:05.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ed4dc4700 1 -- start start 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ed4dc4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fde80 0x7f6ed0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ed4dc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ed4dc4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed0114490 con 0x7f6ed00fe820 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ed4dc4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed010da00 con 0x7f6ed00fde80 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46136/0 (socket says 172.21.15.111:46136) 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 -- 172.21.15.111:0/2828211745 learned_addr learned my addr 172.21.15.111:0/2828211745 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ece59c700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fde80 0x7f6ed0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 -- 172.21.15.111:0/2828211745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fde80 msgr2=0x7f6ed0113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fde80 0x7f6ed0113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 -- 172.21.15.111:0/2828211745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0005040 con 0x7f6ed00fe820 2024-03-14T13:10:05.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.603+0000 7f6ecdd9b700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b790 tx=0x7f6ec4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.604+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec400f770 con 0x7f6ed00fe820 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.604+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed010dce0 con 0x7f6ed00fe820 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.604+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec400fd80 con 0x7f6ed00fe820 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.604+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ec400d310 con 0x7f6ed00fe820 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.604+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed010e220 con 0x7f6ed00fe820 2024-03-14T13:10:05.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.605+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed0066af0 con 0x7f6ed00fe820 2024-03-14T13:10:05.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.605+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6ec400d470 con 0x7f6ed00fe820 2024-03-14T13:10:05.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.605+0000 7f6ebeffd700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 0x7f6eb806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:05.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.606+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ec408ba70 con 0x7f6ed00fe820 2024-03-14T13:10:05.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.606+0000 7f6ece59c700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 0x7f6eb806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:05.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.606+0000 7f6ece59c700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 0x7f6eb806ec70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0004f90 tx=0x7f6ec0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:05.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.611+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ec40597f0 con 0x7f6ed00fe820 2024-03-14T13:10:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2049950208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3483042109' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:10:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:05.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:05.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:05 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:05.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.901+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6ed004c710 con 0x7f6ed00fe820 2024-03-14T13:10:05.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.901+0000 7f6ebeffd700 1 -- 172.21.15.111:0/2828211745 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6ec4059380 con 0x7f6ed00fe820 2024-03-14T13:10:05.902 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:10:05.902 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:10:05.902 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:10:05.902 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:10:05.902 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 msgr2=0x7f6eb806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 0x7f6eb806ec70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0004f90 tx=0x7f6ec0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 msgr2=0x7f6ed0113db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b790 tx=0x7f6ec4009250 comp rx=0 tx=0).stop 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 shutdown_connections 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6eb806c7c0 0x7f6eb806ec70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed00fde80 0x7f6ed0113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 --2- 172.21.15.111:0/2828211745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed00fe820 0x7f6ed0113db0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 >> 172.21.15.111:0/2828211745 conn(0x7f6ed00f78f0 msgr2=0x7f6ed01036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 shutdown_connections 2024-03-14T13:10:05.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:05.903+0000 7f6ed4dc4700 1 -- 172.21.15.111:0/2828211745 wait complete. 2024-03-14T13:10:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2049950208' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:06.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3483042109' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:10:06.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:06.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:06.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi111.yzwmcx", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:06.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:05 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:06 smithi137 ceph-mon[108145]: Upgrade: Updating mds.cephfs.smithi111.yzwmcx 2024-03-14T13:10:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:06 smithi137 ceph-mon[108145]: Deploying daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T13:10:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:06 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2828211745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:10:07.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:06 smithi137 ceph-mon[108145]: pgmap v177: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:07.095 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:06 smithi111 ceph-mon[145314]: Upgrade: Updating mds.cephfs.smithi111.yzwmcx 2024-03-14T13:10:07.095 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:06 smithi111 ceph-mon[145314]: Deploying daemon mds.cephfs.smithi111.yzwmcx on smithi111 2024-03-14T13:10:07.095 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:06 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2828211745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:10:07.096 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:06 smithi111 ceph-mon[145314]: pgmap v177: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:08 smithi111 ceph-mon[145314]: osdmap e84: 6 total, 6 up, 6 in 2024-03-14T13:10:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:08 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:08 smithi111 ceph-mon[145314]: pgmap v179: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:09.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:08 smithi137 ceph-mon[108145]: osdmap e84: 6 total, 6 up, 6 in 2024-03-14T13:10:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:08 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:09.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:08 smithi137 ceph-mon[108145]: pgmap v179: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:10.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:10:10.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:10.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:10:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:10 smithi137 ceph-mon[108145]: pgmap v180: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:11.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:10 smithi111 ceph-mon[145314]: pgmap v180: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:13.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:13 smithi111 ceph-mon[145314]: pgmap v181: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:13 smithi137 ceph-mon[108145]: pgmap v181: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:boot 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:14.642 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:14 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:boot 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:14.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:14 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:15 smithi137 ceph-mon[108145]: pgmap v182: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:15 smithi111 ceph-mon[145314]: pgmap v182: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:17 smithi137 ceph-mon[108145]: pgmap v183: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:17 smithi111 ceph-mon[145314]: pgmap v183: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:19.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:18 smithi137 ceph-mon[108145]: pgmap v184: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:19.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:18 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:18 smithi111 ceph-mon[145314]: pgmap v184: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:19.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:19.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:18 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:21.638 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:21 smithi111 ceph-mon[145314]: pgmap v185: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:21 smithi137 ceph-mon[108145]: pgmap v185: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.365 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:23 smithi137 ceph-mon[108145]: pgmap v186: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:23.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:23 smithi111 ceph-mon[145314]: pgmap v186: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: Upgrade: Updating mds.cephfs.smithi137.ryucox 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: Deploying daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:24 smithi137 ceph-mon[108145]: pgmap v187: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: Upgrade: Updating mds.cephfs.smithi137.ryucox 2024-03-14T13:10:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.ryucox", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:25.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: Deploying daemon mds.cephfs.smithi137.ryucox on smithi137 2024-03-14T13:10:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:25.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:24 smithi111 ceph-mon[145314]: pgmap v187: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:26.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:26 smithi137 ceph-mon[108145]: osdmap e85: 6 total, 6 up, 6 in 2024-03-14T13:10:26.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:26 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:26 smithi111 ceph-mon[145314]: osdmap e85: 6 total, 6 up, 6 in 2024-03-14T13:10:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:26 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:27 smithi137 ceph-mon[108145]: pgmap v189: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:27 smithi111 ceph-mon[145314]: pgmap v189: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:29.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:29 smithi137 ceph-mon[108145]: pgmap v190: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:29 smithi111 ceph-mon[145314]: pgmap v190: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: pgmap v191: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] up:boot 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:31.256 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:30 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:10:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:31.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: pgmap v191: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] up:boot 2024-03-14T13:10:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:31.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:30 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:10:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:33 smithi111 ceph-mon[145314]: pgmap v192: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:33.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:33 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:33 smithi137 ceph-mon[108145]: pgmap v192: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:33.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:33 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:34.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:34 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:34.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:34 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:34 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:35.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:35 smithi137 ceph-mon[108145]: pgmap v193: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:35 smithi111 ceph-mon[145314]: pgmap v193: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:36.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 -- 172.21.15.111:0/3843004507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4101d70 msgr2=0x7f9bc4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3843004507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4101d70 0x7f9bc4102140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4004940 tx=0x7f9bb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 -- 172.21.15.111:0/3843004507 shutdown_connections 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3843004507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4102710 0x7f9bc40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3843004507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4101d70 0x7f9bc4102140 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 -- 172.21.15.111:0/3843004507 >> 172.21.15.111:0/3843004507 conn(0x7f9bc4072bc0 msgr2=0x7f9bc4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 -- 172.21.15.111:0/3843004507 shutdown_connections 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.018+0000 7f9bca64d700 1 -- 172.21.15.111:0/3843004507 wait complete. 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 Processor -- start 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 -- start start 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4101d70 0x7f9bc41129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc4113530 con 0x7f9bc4102710 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bca64d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc410c920 con 0x7f9bc4101d70 2024-03-14T13:10:36.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bc37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49116/0 (socket says 172.21.15.111:49116) 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.019+0000 7f9bc37fe700 1 -- 172.21.15.111:0/3000123315 learned_addr learned my addr 172.21.15.111:0/3000123315 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc3fff700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4101d70 0x7f9bc41129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc37fe700 1 -- 172.21.15.111:0/3000123315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4101d70 msgr2=0x7f9bc41129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc37fe700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4101d70 0x7f9bc41129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc37fe700 1 -- 172.21.15.111:0/3000123315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb4005040 con 0x7f9bc4102710 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc37fe700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9bac002a50 tx=0x7f9bac004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bac007cc0 con 0x7f9bc4102710 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc410cc00 con 0x7f9bc4102710 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bac00f040 con 0x7f9bc4102710 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bac00d990 con 0x7f9bc4102710 2024-03-14T13:10:36.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.020+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc410d1c0 con 0x7f9bc4102710 2024-03-14T13:10:36.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.021+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc40ff080 con 0x7f9bc4102710 2024-03-14T13:10:36.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.022+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9bac009710 con 0x7f9bc4102710 2024-03-14T13:10:36.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.022+0000 7f9bc0ff9700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 0x7f9bb006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.022+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9bac08b8b0 con 0x7f9bc4102710 2024-03-14T13:10:36.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.022+0000 7f9bc3fff700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 0x7f9bb006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.023+0000 7f9bc3fff700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 0x7f9bb006ec80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400b3b0 tx=0x7f9bb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.027+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9bac059540 con 0x7f9bc4102710 2024-03-14T13:10:36.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.236+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bc4112670 con 0x7f9bb006c7d0 2024-03-14T13:10:36.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.238+0000 7f9bc0ff9700 1 -- 172.21.15.111:0/3000123315 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9bc4112670 con 0x7f9bb006c7d0 2024-03-14T13:10:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 msgr2=0x7f9bb006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 0x7f9bb006ec80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400b3b0 tx=0x7f9bb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 msgr2=0x7f9bc4112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9bac002a50 tx=0x7f9bac004600 comp rx=0 tx=0).stop 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 shutdown_connections 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9bb006c7d0 0x7f9bb006ec80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc4101d70 0x7f9bc41129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 --2- 172.21.15.111:0/3000123315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9bc4102710 0x7f9bc4112ee0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 >> 172.21.15.111:0/3000123315 conn(0x7f9bc4072bc0 msgr2=0x7f9bc40fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 shutdown_connections 2024-03-14T13:10:36.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.240+0000 7f9bca64d700 1 -- 172.21.15.111:0/3000123315 wait complete. 2024-03-14T13:10:36.250 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 -- 172.21.15.111:0/1386999928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8102690 msgr2=0x7fd9d80fbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 --2- 172.21.15.111:0/1386999928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8102690 0x7fd9d80fbd00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4004940 tx=0x7fd9d4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 -- 172.21.15.111:0/1386999928 shutdown_connections 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 --2- 172.21.15.111:0/1386999928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8102690 0x7fd9d80fbd00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 --2- 172.21.15.111:0/1386999928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8101cf0 0x7fd9d81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 -- 172.21.15.111:0/1386999928 >> 172.21.15.111:0/1386999928 conn(0x7fd9d80f7840 msgr2=0x7fd9d80f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 -- 172.21.15.111:0/1386999928 shutdown_connections 2024-03-14T13:10:36.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 -- 172.21.15.111:0/1386999928 wait complete. 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.342+0000 7fd9df73f700 1 Processor -- start 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 -- start start 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8102690 0x7fd9d810ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810f480 con 0x7fd9d8101cf0 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d810f5c0 con 0x7fd9d8102690 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49138/0 (socket says 172.21.15.111:49138) 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 -- 172.21.15.111:0/3444059632 learned_addr learned my addr 172.21.15.111:0/3444059632 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dccda700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8102690 0x7fd9d810ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 -- 172.21.15.111:0/3444059632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8102690 msgr2=0x7fd9d810ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8102690 0x7fd9d810ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 -- 172.21.15.111:0/3444059632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4005040 con 0x7fd9d8101cf0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9dd4db700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800eb40 tx=0x7fd9c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9c8007cc0 con 0x7fd9d8101cf0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9c800f040 con 0x7fd9d8101cf0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d810f840 con 0x7fd9d8101cf0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.343+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9c800c990 con 0x7fd9d8101cf0 2024-03-14T13:10:36.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.344+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d8117d70 con 0x7fd9d8101cf0 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.344+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d80fd440 con 0x7fd9d8101cf0 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.345+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd9c800caf0 con 0x7fd9d8101cf0 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.345+0000 7fd9cdffb700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 0x7fd9c406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.345+0000 7fd9dccda700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 0x7fd9c406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.345+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd9c808b5a0 con 0x7fd9d8101cf0 2024-03-14T13:10:36.346 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.346+0000 7fd9dccda700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 0x7fd9c406ec70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400b0d0 tx=0x7fd9d4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.350+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd9c80558d0 con 0x7fd9d8101cf0 2024-03-14T13:10:36.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.559+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9d8066900 con 0x7fd9c406c7c0 2024-03-14T13:10:36.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.560+0000 7fd9cdffb700 1 -- 172.21.15.111:0/3444059632 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fd9d8066900 con 0x7fd9c406c7c0 2024-03-14T13:10:36.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 msgr2=0x7fd9c406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 0x7fd9c406ec70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400b0d0 tx=0x7fd9d4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 msgr2=0x7fd9d810ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd9c800eb40 tx=0x7fd9c800b040 comp rx=0 tx=0).stop 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 shutdown_connections 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd9c406c7c0 0x7fd9c406ec70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9d8101cf0 0x7fd9d810ea00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 --2- 172.21.15.111:0/3444059632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9d8102690 0x7fd9d810ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 >> 172.21.15.111:0/3444059632 conn(0x7fd9d80f7840 msgr2=0x7fd9d80f88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 shutdown_connections 2024-03-14T13:10:36.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.562+0000 7fd9df73f700 1 -- 172.21.15.111:0/3444059632 wait complete. 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- 172.21.15.111:0/2698436368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 msgr2=0x7f40a40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 --2- 172.21.15.111:0/2698436368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a40fbd80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4098004940 tx=0x7f4098009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- 172.21.15.111:0/2698436368 shutdown_connections 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 --2- 172.21.15.111:0/2698436368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a40fbd80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 --2- 172.21.15.111:0/2698436368 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- 172.21.15.111:0/2698436368 >> 172.21.15.111:0/2698436368 conn(0x7f40a4072bc0 msgr2=0x7f40a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- 172.21.15.111:0/2698436368 shutdown_connections 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- 172.21.15.111:0/2698436368 wait complete. 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 Processor -- start 2024-03-14T13:10:36.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.663+0000 7f40a90a6700 1 -- start start 2024-03-14T13:10:36.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a90a6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4113aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a90a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a4113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a90a6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a4114630 con 0x7f40a4102710 2024-03-14T13:10:36.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a90a6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a410da20 con 0x7f40a4101d70 2024-03-14T13:10:36.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a4113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a2d9d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4113aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a4113fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49154/0 (socket says 172.21.15.111:49154) 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a259c700 1 -- 172.21.15.111:0/1102917362 learned_addr learned my addr 172.21.15.111:0/1102917362 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a2d9d700 1 -- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 msgr2=0x7f40a4113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a2d9d700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a4113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a2d9d700 1 -- 172.21.15.111:0/1102917362 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4098005040 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a2d9d700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4113aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4094009fd0 tx=0x7f409400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4094004e70 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f409400f040 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40a410dca0 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.664+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f409400c930 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.665+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a410e260 con 0x7f40a4101d70 2024-03-14T13:10:36.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.665+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a40fd4c0 con 0x7f40a4101d70 2024-03-14T13:10:36.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.666+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4094007cc0 con 0x7f40a4101d70 2024-03-14T13:10:36.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.666+0000 7f408b7fe700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 0x7f408c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.666+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f409408b6f0 con 0x7f40a4101d70 2024-03-14T13:10:36.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.666+0000 7f40a259c700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 0x7f408c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.667+0000 7f40a259c700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 0x7f408c06ec70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40a410ebe0 tx=0x7f4098009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.671+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4094059300 con 0x7f40a4101d70 2024-03-14T13:10:36.872 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.872+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f40a4108d30 con 0x7f408c06c7c0 2024-03-14T13:10:36.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.882+0000 7f408b7fe700 1 -- 172.21.15.111:0/1102917362 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f40a4108d30 con 0x7f408c06c7c0 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 19s ago 18m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (4m) 19s ago 18m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (4m) 4s ago 17m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (6m) 19s ago 17m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (24s) 19s ago 12m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (43s) 19s ago 12m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (12m) 4s ago 12m 10.6M - 17.2.7-646-gb92acd1f a07fb176315d dfb0e9bec1b5 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (7s) 4s ago 12m 10.2M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:10:36.882 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (9m) 19s ago 19m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (8m) 4s ago 16m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (5m) 19s ago 19m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (4m) 4s ago 16m 44.3M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 19s ago 18m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 4s ago 17m 18.5M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (3m) 19s ago 16m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (3m) 19s ago 15m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (2m) 19s ago 15m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (2m) 4s ago 14m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (94s) 4s ago 14m 94.1M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (62s) 4s ago 13m 86.5M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:10:36.883 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (7m) 19s ago 17m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:10:36.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 msgr2=0x7f408c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 0x7f408c06ec70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40a410ebe0 tx=0x7f4098009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 msgr2=0x7f40a4113aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4113aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4094009fd0 tx=0x7f409400b040 comp rx=0 tx=0).stop 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 shutdown_connections 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f408c06c7c0 0x7f408c06ec70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40a4101d70 0x7f40a4113aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 --2- 172.21.15.111:0/1102917362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40a4102710 0x7f40a4113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 >> 172.21.15.111:0/1102917362 conn(0x7f40a4072bc0 msgr2=0x7f40a4065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 shutdown_connections 2024-03-14T13:10:36.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.884+0000 7f40a90a6700 1 -- 172.21.15.111:0/1102917362 wait complete. 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- 172.21.15.111:0/658998140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 msgr2=0x7f99e8103010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/658998140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8103010 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99dc004940 tx=0x7f99dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- 172.21.15.111:0/658998140 shutdown_connections 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/658998140 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 0x7f99e80fcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/658998140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8103010 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- 172.21.15.111:0/658998140 >> 172.21.15.111:0/658998140 conn(0x7f99e80f8830 msgr2=0x7f99e80fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:36.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- 172.21.15.111:0/658998140 shutdown_connections 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- 172.21.15.111:0/658998140 wait complete. 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 Processor -- start 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.984+0000 7f99ecd9f700 1 -- start start 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99ecd9f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99ecd9f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 0x7f99e8113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99ecd9f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e81144e0 con 0x7f99e8102c40 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99ecd9f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e810d8d0 con 0x7f99e81035e0 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49182/0 (socket says 172.21.15.111:49182) 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 -- 172.21.15.111:0/311433005 learned_addr learned my addr 172.21.15.111:0/311433005 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e6ffd700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 0x7f99e8113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 -- 172.21.15.111:0/311433005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 msgr2=0x7f99e8113e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 0x7f99e8113e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:36.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 -- 172.21.15.111:0/311433005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc005040 con 0x7f99e8102c40 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99e77fe700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009180 tx=0x7f99dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99dc00e040 con 0x7f99e8102c40 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99dc004290 con 0x7f99e8102c40 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e810db50 con 0x7f99e8102c40 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.985+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99dc013e60 con 0x7f99e8102c40 2024-03-14T13:10:36.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.986+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e810e110 con 0x7f99e8102c40 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.987+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e80fe390 con 0x7f99e8102c40 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.987+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f99dc0136b0 con 0x7f99e8102c40 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.987+0000 7f99cffff700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 0x7f99d006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.987+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f99dc08ccd0 con 0x7f99e8102c40 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.987+0000 7f99e6ffd700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 0x7f99d006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:36.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.988+0000 7f99e6ffd700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 0x7f99d006ec00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99e810ea90 tx=0x7f99d800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:36.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:36.992+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f99dc05aa90 con 0x7f99e8102c40 2024-03-14T13:10:37.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.264+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f99e80fe390 con 0x7f99e8102c40 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.264+0000 7f99cffff700 1 -- 172.21.15.111:0/311433005 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+842 (secure 0 0 0) 0x7f99dc05a620 con 0x7f99e8102c40 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:10:37.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 1, 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 3 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.7-646-gb92acd1f (b92acd1f9b519af8166ff5b5bae92bf72f1fd797) quincy (stable)": 1, 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 13 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:10:37.265 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 msgr2=0x7f99d006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 0x7f99d006ec00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f99e810ea90 tx=0x7f99d800b040 comp rx=0 tx=0).stop 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 msgr2=0x7f99e8113950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f99dc009180 tx=0x7f99dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 shutdown_connections 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f99d006c750 0x7f99d006ec00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99e8102c40 0x7f99e8113950 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 --2- 172.21.15.111:0/311433005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e81035e0 0x7f99e8113e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.266 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.266+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 >> 172.21.15.111:0/311433005 conn(0x7f99e80f8830 msgr2=0x7f99e80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.267+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 shutdown_connections 2024-03-14T13:10:37.267 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.267+0000 7f99ecd9f700 1 -- 172.21.15.111:0/311433005 wait complete. 2024-03-14T13:10:37.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- 172.21.15.111:0/3051533198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 msgr2=0x7f6cf40fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 --2- 172.21.15.111:0/3051533198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40fd5a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4004940 tx=0x7f6ce4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- 172.21.15.111:0/3051533198 shutdown_connections 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 --2- 172.21.15.111:0/3051533198 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf40fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 --2- 172.21.15.111:0/3051533198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40fd5a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- 172.21.15.111:0/3051533198 >> 172.21.15.111:0/3051533198 conn(0x7f6cf40f89b0 msgr2=0x7f6cf40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- 172.21.15.111:0/3051533198 shutdown_connections 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- 172.21.15.111:0/3051533198 wait complete. 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 Processor -- start 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.372+0000 7f6cfc579700 1 -- start start 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf4072210 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf4070d40 con 0x7f6cf40fe9d0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49206/0 (socket says 172.21.15.111:49206) 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 -- 172.21.15.111:0/1615808541 learned_addr learned my addr 172.21.15.111:0/1615808541 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cf9b14700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 -- 172.21.15.111:0/1615808541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 msgr2=0x7f6cf4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 -- 172.21.15.111:0/1615808541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4005040 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cf9b14700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfa315700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400b3b0 tx=0x7f6ce4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ce400e040 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ce4013420 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf4070fc0 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ce401c630 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.373+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf4119fc0 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.374+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf40fe570 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.375+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6ce4013970 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.375+0000 7f6ceaffd700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 0x7f6ce006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.375+0000 7f6cf9b14700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 0x7f6ce006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.375+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ce408ccc0 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.376+0000 7f6cf9b14700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 0x7f6ce006ec70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009fd0 tx=0x7f6cf0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:37.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.380+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ce4056f40 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.619+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6cf411a1b0 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.620+0000 7f6ceaffd700 1 -- 172.21.15.111:0/1615808541 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 31 v31) v1 ==== 76+0+1843 (secure 0 0 0) 0x7f6ce405a560 con 0x7f6cf40fd1d0 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:e31 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:10:37.620 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:10:37.621 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:37.622 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44279} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/513022872,v1:172.21.15.137:6827/513022872] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:37.623 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:10:37.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 msgr2=0x7f6ce006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 0x7f6ce006ec70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009fd0 tx=0x7f6cf0009ee0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 msgr2=0x7f6cf40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400b3b0 tx=0x7f6ce4009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 shutdown_connections 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ce006c7c0 0x7f6ce006ec70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cf40fd1d0 0x7f6cf40702c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 --2- 172.21.15.111:0/1615808541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6cf40fe9d0 0x7f6cf4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 >> 172.21.15.111:0/1615808541 conn(0x7f6cf40f89b0 msgr2=0x7f6cf40fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 shutdown_connections 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.622+0000 7f6cfc579700 1 -- 172.21.15.111:0/1615808541 wait complete. 2024-03-14T13:10:37.624 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 31 2024-03-14T13:10:37.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/914668479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d00fb540 msgr2=0x7fb0d00fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/914668479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d00fb540 0x7fb0d00fb990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0004940 tx=0x7fb0c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/914668479 shutdown_connections 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/914668479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d00fb540 0x7fb0d00fb990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/914668479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d0102fd0 0x7fb0d01033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/914668479 >> 172.21.15.111:0/914668479 conn(0x7fb0d0071530 msgr2=0x7fb0d0071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.723+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/914668479 shutdown_connections 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/914668479 wait complete. 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 Processor -- start 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 -- start start 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d00fb540 0x7fb0d01138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d0114460 con 0x7fb0d0102fd0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d7f7f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d010da60 con 0x7fb0d00fb540 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d551a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d551a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49232/0 (socket says 172.21.15.111:49232) 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d551a700 1 -- 172.21.15.111:0/2376231482 learned_addr learned my addr 172.21.15.111:0/2376231482 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.724+0000 7fb0d5d1b700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d00fb540 0x7fb0d01138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d551a700 1 -- 172.21.15.111:0/2376231482 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d00fb540 msgr2=0x7fb0d01138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d551a700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d00fb540 0x7fb0d01138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d551a700 1 -- 172.21.15.111:0/2376231482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0005040 con 0x7fb0d0102fd0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d551a700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009360 tx=0x7fb0c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0c000e040 con 0x7fb0d0102fd0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0c0013420 con 0x7fb0d0102fd0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d010dce0 con 0x7fb0d0102fd0 2024-03-14T13:10:37.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0c001c5d0 con 0x7fb0d0102fd0 2024-03-14T13:10:37.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.725+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d010e2a0 con 0x7fb0d0102fd0 2024-03-14T13:10:37.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.726+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d01053b0 con 0x7fb0d0102fd0 2024-03-14T13:10:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.726+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb0c0013590 con 0x7fb0d0102fd0 2024-03-14T13:10:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.727+0000 7fb0c67fc700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 0x7fb0bc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.727+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb0c008cc10 con 0x7fb0d0102fd0 2024-03-14T13:10:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.727+0000 7fb0d5d1b700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 0x7fb0bc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:37.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.727+0000 7fb0d5d1b700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 0x7fb0bc06ec70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc004f40 tx=0x7fb0cc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:37.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.731+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb0c005a8a0 con 0x7fb0d0102fd0 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: pgmap v194: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/311433005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.757 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:37 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='client.34426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: pgmap v194: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/311433005' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:37 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:10:37.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.945+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0d01135a0 con 0x7fb0bc06c7c0 2024-03-14T13:10:37.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.946+0000 7fb0c67fc700 1 -- 172.21.15.111:0/2376231482 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fb0d01135a0 con 0x7fb0bc06c7c0 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "15/21 daemons upgraded", 2024-03-14T13:10:37.946 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:10:37.947 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:10:37.947 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 msgr2=0x7fb0bc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 0x7fb0bc06ec70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc004f40 tx=0x7fb0cc00b040 comp rx=0 tx=0).stop 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 msgr2=0x7fb0d0113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009360 tx=0x7fb0c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 shutdown_connections 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb0bc06c7c0 0x7fb0bc06ec70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0d00fb540 0x7fb0d01138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 --2- 172.21.15.111:0/2376231482 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0d0102fd0 0x7fb0d0113e10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 >> 172.21.15.111:0/2376231482 conn(0x7fb0d0071530 msgr2=0x7fb0d00fa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 shutdown_connections 2024-03-14T13:10:37.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:37.948+0000 7fb0d7f7f700 1 -- 172.21.15.111:0/2376231482 wait complete. 2024-03-14T13:10:38.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 -- 172.21.15.111:0/513083041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 msgr2=0x7f92601031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 --2- 172.21.15.111:0/513083041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f92601031f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f925c004940 tx=0x7f925c009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 -- 172.21.15.111:0/513083041 shutdown_connections 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 --2- 172.21.15.111:0/513083041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 0x7f92600fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 --2- 172.21.15.111:0/513083041 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f92601031f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 -- 172.21.15.111:0/513083041 >> 172.21.15.111:0/513083041 conn(0x7f92600f89d0 msgr2=0x7f92600fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 -- 172.21.15.111:0/513083041 shutdown_connections 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.048+0000 7f9268428700 1 -- 172.21.15.111:0/513083041 wait complete. 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 Processor -- start 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 -- start start 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 0x7f9260114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:38.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9260114670 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f9268428700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926010da60 con 0x7f92601037c0 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49262/0 (socket says 172.21.15.111:49262) 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 -- 172.21.15.111:0/3666107950 learned_addr learned my addr 172.21.15.111:0/3666107950 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92659c3700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 0x7f9260114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 -- 172.21.15.111:0/3666107950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 msgr2=0x7f9260114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 0x7f9260114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 -- 172.21.15.111:0/3666107950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c005040 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.049+0000 7f92661c4700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f925c00b3b0 tx=0x7f925c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.050+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f925c00e040 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.050+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f925c013420 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.050+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f925c01c630 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.050+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926010dce0 con 0x7f9260102e20 2024-03-14T13:10:38.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.050+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926010e2a0 con 0x7f9260102e20 2024-03-14T13:10:38.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.051+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92600fe570 con 0x7f9260102e20 2024-03-14T13:10:38.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.051+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f925c013970 con 0x7f9260102e20 2024-03-14T13:10:38.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.051+0000 7f9256ffd700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 0x7f924c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:10:38.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.052+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f925c08cbb0 con 0x7f9260102e20 2024-03-14T13:10:38.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.052+0000 7f92659c3700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 0x7f924c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:10:38.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.052+0000 7f92659c3700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 0x7f924c06ec70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f926010ec20 tx=0x7f925000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:10:38.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.056+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f925c093080 con 0x7f9260102e20 2024-03-14T13:10:38.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.324+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f926004bed0 con 0x7f9260102e20 2024-03-14T13:10:38.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.324+0000 7f9256ffd700 1 -- 172.21.15.111:0/3666107950 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f925c05a980 con 0x7f9260102e20 2024-03-14T13:10:38.324 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:10:38.324 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:10:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:10:38.325 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:10:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:10:38.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.326+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 msgr2=0x7f924c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:38.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.326+0000 7f9268428700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 0x7f924c06ec70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f926010ec20 tx=0x7f925000b040 comp rx=0 tx=0).stop 2024-03-14T13:10:38.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.326+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 msgr2=0x7f9260113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:10:38.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.326+0000 7f9268428700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f925c00b3b0 tx=0x7f925c009e30 comp rx=0 tx=0).stop 2024-03-14T13:10:38.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 shutdown_connections 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f924c06c7c0 0x7f924c06ec70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9260102e20 0x7f9260113ae0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 --2- 172.21.15.111:0/3666107950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92601037c0 0x7f9260114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 >> 172.21.15.111:0/3666107950 conn(0x7f92600f89d0 msgr2=0x7f92600f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 shutdown_connections 2024-03-14T13:10:38.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:10:38.327+0000 7f9268428700 1 -- 172.21.15.111:0/3666107950 wait complete. 2024-03-14T13:10:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:38 smithi137 ceph-mon[108145]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:38 smithi137 ceph-mon[108145]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:38 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1615808541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:10:38.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:38 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3666107950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:10:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:38 smithi111 ceph-mon[145314]: from='client.34430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:38 smithi111 ceph-mon[145314]: from='client.44317 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:38 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1615808541' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:10:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:38 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3666107950' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: pgmap v195: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:39.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:10:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: pgmap v195: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi137.iiqzcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-14T13:10:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:10:39.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:40.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:40 smithi137 ceph-mon[108145]: Upgrade: Updating mds.cephfs.smithi137.iiqzcp 2024-03-14T13:10:40.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:40 smithi137 ceph-mon[108145]: Deploying daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T13:10:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:40 smithi111 ceph-mon[145314]: Upgrade: Updating mds.cephfs.smithi137.iiqzcp 2024-03-14T13:10:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:40 smithi111 ceph-mon[145314]: Deploying daemon mds.cephfs.smithi137.iiqzcp on smithi137 2024-03-14T13:10:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:41 smithi111 ceph-mon[145314]: pgmap v196: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:41 smithi111 ceph-mon[145314]: osdmap e86: 6 total, 6 up, 6 in 2024-03-14T13:10:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:41 smithi137 ceph-mon[108145]: pgmap v196: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:42.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:41 smithi137 ceph-mon[108145]: osdmap e86: 6 total, 6 up, 6 in 2024-03-14T13:10:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:42 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:43.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:42 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 3 up:standby, 2 failed 2024-03-14T13:10:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:43 smithi111 ceph-mon[145314]: pgmap v198: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:43 smithi137 ceph-mon[108145]: pgmap v198: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:45.740 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:45 smithi137 ceph-mon[108145]: pgmap v199: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:45 smithi111 ceph-mon[145314]: pgmap v199: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: pgmap v200: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] up:boot 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:47.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: pgmap v200: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] up:boot 2024-03-14T13:10:47.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:10:47.141 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:10:49.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:49 smithi111 ceph-mon[145314]: pgmap v201: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:49.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:49 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:49 smithi137 ceph-mon[108145]: pgmap v201: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:49.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:49 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:10:51.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:51 smithi137 ceph-mon[108145]: pgmap v202: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:51 smithi111 ceph-mon[145314]: pgmap v202: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:53 smithi137 ceph-mon[108145]: pgmap v203: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:53 smithi111 ceph-mon[145314]: pgmap v203: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:10:55.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:55 smithi137 ceph-mon[108145]: pgmap v204: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:55 smithi111 ceph-mon[145314]: pgmap v204: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:57 smithi137 ceph-mon[108145]: pgmap v205: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:57 smithi111 ceph-mon[145314]: pgmap v205: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:10:59 smithi137 ceph-mon[108145]: pgmap v206: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:10:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:10:59 smithi111 ceph-mon[145314]: pgmap v206: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:01.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:01 smithi137 ceph-mon[108145]: pgmap v207: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:01 smithi111 ceph-mon[145314]: pgmap v207: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:03.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:03 smithi137 ceph-mon[108145]: pgmap v208: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:03 smithi111 ceph-mon[145314]: pgmap v208: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:05 smithi137 ceph-mon[108145]: pgmap v209: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:05 smithi111 ceph-mon[145314]: pgmap v209: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:07.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:07 smithi137 ceph-mon[108145]: pgmap v210: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:07 smithi111 ceph-mon[145314]: pgmap v210: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 -- 172.21.15.111:0/3516763300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fde10 msgr2=0x7fed080fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 --2- 172.21.15.111:0/3516763300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fde10 0x7fed080fe1e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fecf8004940 tx=0x7fecf8009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 -- 172.21.15.111:0/3516763300 shutdown_connections 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 --2- 172.21.15.111:0/3516763300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fe7b0 0x7fed08106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 --2- 172.21.15.111:0/3516763300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fde10 0x7fed080fe1e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 -- 172.21.15.111:0/3516763300 >> 172.21.15.111:0/3516763300 conn(0x7fed080f7880 msgr2=0x7fed080f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 -- 172.21.15.111:0/3516763300 shutdown_connections 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.439+0000 7fed0e449700 1 -- 172.21.15.111:0/3516763300 wait complete. 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 Processor -- start 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 -- start start 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fde10 0x7fed080fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080fc710 con 0x7fed080fe7b0 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.440+0000 7fed0e449700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed080fc850 con 0x7fed080fde10 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46962/0 (socket says 172.21.15.111:46962) 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 -- 172.21.15.111:0/1386912197 learned_addr learned my addr 172.21.15.111:0/1386912197 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:08.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed07fff700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fde10 0x7fed080fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 -- 172.21.15.111:0/1386912197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fde10 msgr2=0x7fed080fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fde10 0x7fed080fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 -- 172.21.15.111:0/1386912197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8005040 con 0x7fed080fe7b0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed077fe700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c830 tx=0x7fecf0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecf00106f0 con 0x7fed080fe7b0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecf0010d00 con 0x7fed080fe7b0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0811a0e0 con 0x7fed080fe7b0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fecf0012e90 con 0x7fed080fe7b0 2024-03-14T13:11:08.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.441+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0811a4e0 con 0x7fed080fe7b0 2024-03-14T13:11:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.442+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed08066770 con 0x7fed080fe7b0 2024-03-14T13:11:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.443+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fecf0010850 con 0x7fed080fe7b0 2024-03-14T13:11:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.443+0000 7fed04ff9700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 0x7fecf406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.443+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fecf008b990 con 0x7fed080fe7b0 2024-03-14T13:11:08.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.443+0000 7fed07fff700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 0x7fecf406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.444+0000 7fed07fff700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 0x7fecf406ec70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fecf8004f90 tx=0x7fecf8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:08.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.448+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fecf0055cd0 con 0x7fed080fe7b0 2024-03-14T13:11:08.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.658+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed0804bed0 con 0x7fecf406c7c0 2024-03-14T13:11:08.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.659+0000 7fed04ff9700 1 -- 172.21.15.111:0/1386912197 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fed0804bed0 con 0x7fecf406c7c0 2024-03-14T13:11:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 msgr2=0x7fecf406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 0x7fecf406ec70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fecf8004f90 tx=0x7fecf8009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:08.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 msgr2=0x7fed080fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c830 tx=0x7fecf0004470 comp rx=0 tx=0).stop 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 shutdown_connections 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fecf406c7c0 0x7fecf406ec70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed080fde10 0x7fed080fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 --2- 172.21.15.111:0/1386912197 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed080fe7b0 0x7fed080fc180 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 >> 172.21.15.111:0/1386912197 conn(0x7fed080f7880 msgr2=0x7fed081030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 shutdown_connections 2024-03-14T13:11:08.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.661+0000 7fed0e449700 1 -- 172.21.15.111:0/1386912197 wait complete. 2024-03-14T13:11:08.673 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 -- 172.21.15.111:0/97811572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce040fb4c0 msgr2=0x7fce040fb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 --2- 172.21.15.111:0/97811572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce040fb4c0 0x7fce040fb930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdec004a50 tx=0x7fcdec00af40 comp rx=0 tx=0).stop 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 -- 172.21.15.111:0/97811572 shutdown_connections 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 --2- 172.21.15.111:0/97811572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce040fb4c0 0x7fce040fb930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 --2- 172.21.15.111:0/97811572 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce04101520 0x7fce041018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 -- 172.21.15.111:0/97811572 >> 172.21.15.111:0/97811572 conn(0x7fce04072310 msgr2=0x7fce04072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 -- 172.21.15.111:0/97811572 shutdown_connections 2024-03-14T13:11:08.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.762+0000 7fce09745700 1 -- 172.21.15.111:0/97811572 wait complete. 2024-03-14T13:11:08.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 Processor -- start 2024-03-14T13:11:08.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 -- start start 2024-03-14T13:11:08.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 0x7fce041115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04112150 con 0x7fce04101520 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce09745700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0410b750 con 0x7fce040fb4c0 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce027fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce027fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46972/0 (socket says 172.21.15.111:46972) 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.763+0000 7fce027fc700 1 -- 172.21.15.111:0/3907152088 learned_addr learned my addr 172.21.15.111:0/3907152088 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce02ffd700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 0x7fce041115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce027fc700 1 -- 172.21.15.111:0/3907152088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 msgr2=0x7fce041115c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce027fc700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 0x7fce041115c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce027fc700 1 -- 172.21.15.111:0/3907152088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec005040 con 0x7fce04101520 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce02ffd700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 0x7fce041115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce027fc700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c870 tx=0x7fcdec00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec00f040 con 0x7fce04101520 2024-03-14T13:11:08.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0410b9d0 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdec0148d0 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdec01d6c0 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.764+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce0410bf90 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.765+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040fcc00 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.765+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcdec027440 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.766+0000 7fcdfb7fe700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 0x7fcdf006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.766+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdec08d830 con 0x7fce04101520 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.766+0000 7fce02ffd700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 0x7fcdf006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:08.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.766+0000 7fce02ffd700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 0x7fcdf006ec70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4004f00 tx=0x7fcdf4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:08.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.770+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcdec05b540 con 0x7fce04101520 2024-03-14T13:11:08.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.983+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce04107bc0 con 0x7fcdf006c7c0 2024-03-14T13:11:08.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.984+0000 7fcdfb7fe700 1 -- 172.21.15.111:0/3907152088 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fce04107bc0 con 0x7fcdf006c7c0 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 msgr2=0x7fcdf006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 0x7fcdf006ec70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4004f00 tx=0x7fcdf4009040 comp rx=0 tx=0).stop 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 msgr2=0x7fce04111b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00c870 tx=0x7fcdec00af40 comp rx=0 tx=0).stop 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 shutdown_connections 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdf006c7c0 0x7fcdf006ec70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce040fb4c0 0x7fce041115c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 --2- 172.21.15.111:0/3907152088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce04101520 0x7fce04111b00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:08.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 >> 172.21.15.111:0/3907152088 conn(0x7fce04072310 msgr2=0x7fce040fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:08.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 shutdown_connections 2024-03-14T13:11:08.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:08.986+0000 7fce09745700 1 -- 172.21.15.111:0/3907152088 wait complete. 2024-03-14T13:11:09.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1154946773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b18101d20 msgr2=0x7f5b181020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1154946773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b18101d20 0x7f5b181020f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5b08004940 tx=0x7f5b08009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:09.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1154946773 shutdown_connections 2024-03-14T13:11:09.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1154946773 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b181026c0 0x7f5b180fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1154946773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b18101d20 0x7f5b181020f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1154946773 >> 172.21.15.111:0/1154946773 conn(0x7f5b180f78d0 msgr2=0x7f5b180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1154946773 shutdown_connections 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.086+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1154946773 wait complete. 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 Processor -- start 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 -- start start 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b18101d20 0x7f5b18113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b181146a0 con 0x7f5b181026c0 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1c88a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1810da90 con 0x7f5b18101d20 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b15d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b15d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46982/0 (socket says 172.21.15.111:46982) 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b15d9b700 1 -- 172.21.15.111:0/1568373899 learned_addr learned my addr 172.21.15.111:0/1568373899 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.087+0000 7f5b1659c700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b18101d20 0x7f5b18113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b15d9b700 1 -- 172.21.15.111:0/1568373899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b18101d20 msgr2=0x7f5b18113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b15d9b700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b18101d20 0x7f5b18113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b15d9b700 1 -- 172.21.15.111:0/1568373899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08005040 con 0x7f5b181026c0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b15d9b700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c800 tx=0x7f5b0c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b0c0106f0 con 0x7f5b181026c0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b0c010d00 con 0x7f5b181026c0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1810dd70 con 0x7f5b181026c0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b0c0113a0 con 0x7f5b181026c0 2024-03-14T13:11:09.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.088+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1810e330 con 0x7f5b181026c0 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.089+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b180fd470 con 0x7f5b181026c0 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.089+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5b0c012400 con 0x7f5b181026c0 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.090+0000 7f5b06ffd700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 0x7f5b0006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.090+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b0c08b900 con 0x7f5b181026c0 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.090+0000 7f5b1659c700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 0x7f5b0006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.090+0000 7f5b1659c700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 0x7f5b0006ec70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b08004f90 tx=0x7f5b08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.094+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5b0c0596d0 con 0x7f5b181026c0 2024-03-14T13:11:09.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.295+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b18108ce0 con 0x7f5b0006c7c0 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.304+0000 7f5b06ffd700 1 -- 172.21.15.111:0/1568373899 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f5b18108ce0 con 0x7f5b0006c7c0 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (7m) 51s ago 18m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (4m) 51s ago 19m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (4m) 21s ago 17m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (7m) 51s ago 18m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (56s) 51s ago 12m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (76s) 51s ago 12m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (24s) 21s ago 12m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (39s) 21s ago 12m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:11:09.305 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (9m) 51s ago 20m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (8m) 21s ago 17m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (5m) 51s ago 20m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (5m) 21s ago 17m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 51s ago 19m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (8m) 21s ago 17m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (4m) 51s ago 16m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (3m) 51s ago 16m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (3m) 51s ago 15m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (2m) 21s ago 15m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (2m) 21s ago 14m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (94s) 21s ago 14m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:11:09.306 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (7m) 51s ago 18m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.306+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 msgr2=0x7f5b0006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.306+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 0x7f5b0006ec70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5b08004f90 tx=0x7f5b08009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.306+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 msgr2=0x7f5b18114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.306+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00c800 tx=0x7f5b0c004470 comp rx=0 tx=0).stop 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 shutdown_connections 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b0006c7c0 0x7f5b0006ec70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b18101d20 0x7f5b18113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 --2- 172.21.15.111:0/1568373899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b181026c0 0x7f5b18114050 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 >> 172.21.15.111:0/1568373899 conn(0x7f5b180f78d0 msgr2=0x7f5b180f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:09.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 shutdown_connections 2024-03-14T13:11:09.308 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.307+0000 7f5b1c88a700 1 -- 172.21.15.111:0/1568373899 wait complete. 2024-03-14T13:11:09.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 -- 172.21.15.111:0/1886888856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 msgr2=0x7f29440fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 --2- 172.21.15.111:0/1886888856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f29440fbd80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2930004940 tx=0x7f2930009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 -- 172.21.15.111:0/1886888856 shutdown_connections 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 --2- 172.21.15.111:0/1886888856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f29440fbd80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 --2- 172.21.15.111:0/1886888856 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 0x7f2944102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.406+0000 7f294b046700 1 -- 172.21.15.111:0/1886888856 >> 172.21.15.111:0/1886888856 conn(0x7f2944072bc0 msgr2=0x7f2944072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.407+0000 7f294b046700 1 -- 172.21.15.111:0/1886888856 shutdown_connections 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.407+0000 7f294b046700 1 -- 172.21.15.111:0/1886888856 wait complete. 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.407+0000 7f294b046700 1 Processor -- start 2024-03-14T13:11:09.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.407+0000 7f294b046700 1 -- start start 2024-03-14T13:11:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f294b046700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 0x7f29441129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f294b046700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f294b046700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2944113550 con 0x7f2944102710 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f294b046700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294410c940 con 0x7f2944101d70 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47000/0 (socket says 172.21.15.111:47000) 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 -- 172.21.15.111:0/1444052003 learned_addr learned my addr 172.21.15.111:0/1444052003 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2948de2700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 0x7f29441129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 -- 172.21.15.111:0/1444052003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 msgr2=0x7f29441129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 0x7f29441129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 -- 172.21.15.111:0/1444052003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2930005040 con 0x7f2944102710 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f2943fff700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2930009390 tx=0x7f2930009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f293000e040 con 0x7f2944102710 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294410cbc0 con 0x7f2944102710 2024-03-14T13:11:09.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2930004290 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.408+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2930013e70 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.409+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294410d180 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.409+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29440ff080 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.410+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2930013690 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.410+0000 7f29417fa700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 0x7f293406ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.410+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f293008cc70 con 0x7f2944102710 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.410+0000 7f2948de2700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 0x7f293406ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.411+0000 7f2948de2700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 0x7f293406ec10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2938004f40 tx=0x7f293800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.415+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f293005aa30 con 0x7f2944102710 2024-03-14T13:11:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.690+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f294410e010 con 0x7f2944102710 2024-03-14T13:11:09.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.691+0000 7f29417fa700 1 -- 172.21.15.111:0/1444052003 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f2930013420 con 0x7f2944102710 2024-03-14T13:11:09.691 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:11:09.691 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:11:09.692 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:11:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 msgr2=0x7f293406ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 0x7f293406ec10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2938004f40 tx=0x7f293800b040 comp rx=0 tx=0).stop 2024-03-14T13:11:09.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 msgr2=0x7f2944112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2930009390 tx=0x7f2930009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 shutdown_connections 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f293406c760 0x7f293406ec10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2944101d70 0x7f29441129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 --2- 172.21.15.111:0/1444052003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2944102710 0x7f2944112f00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 >> 172.21.15.111:0/1444052003 conn(0x7f2944072bc0 msgr2=0x7f29440fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 shutdown_connections 2024-03-14T13:11:09.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.693+0000 7f294b046700 1 -- 172.21.15.111:0/1444052003 wait complete. 2024-03-14T13:11:09.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:09 smithi137 ceph-mon[108145]: pgmap v211: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:09.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:09.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- 172.21.15.111:0/1745335066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 msgr2=0x7fb9a80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 --2- 172.21.15.111:0/1745335066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a80fbd80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4004940 tx=0x7fb9a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- 172.21.15.111:0/1745335066 shutdown_connections 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 --2- 172.21.15.111:0/1745335066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a80fbd80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 --2- 172.21.15.111:0/1745335066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- 172.21.15.111:0/1745335066 >> 172.21.15.111:0/1745335066 conn(0x7fb9a8072bc0 msgr2=0x7fb9a8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- 172.21.15.111:0/1745335066 shutdown_connections 2024-03-14T13:11:09.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- 172.21.15.111:0/1745335066 wait complete. 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 Processor -- start 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- start start 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a81129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8113550 con 0x7fb9a8102710 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.793+0000 7fb9af30c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a810c940 con 0x7fb9a8101d70 2024-03-14T13:11:09.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47030/0 (socket says 172.21.15.111:47030) 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 -- 172.21.15.111:0/2898217413 learned_addr learned my addr 172.21.15.111:0/2898217413 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ad0a8700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a81129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 -- 172.21.15.111:0/2898217413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 msgr2=0x7fb9a81129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a81129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 -- 172.21.15.111:0/2898217413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4005040 con 0x7fb9a8102710 2024-03-14T13:11:09.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ad0a8700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9ac8a7700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400b3b0 tx=0x7fb9a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9a400e040 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a810cbc0 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9a400bc90 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9a4013660 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.794+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a810d120 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.795+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a80fd4c0 con 0x7fb9a8102710 2024-03-14T13:11:09.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.796+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb9a401e420 con 0x7fb9a8102710 2024-03-14T13:11:09.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.796+0000 7fb99dffb700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 0x7fb99406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:09.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.797+0000 7fb9ad0a8700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 0x7fb99406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:09.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.797+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb9a408fa70 con 0x7fb9a8102710 2024-03-14T13:11:09.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.797+0000 7fb9ad0a8700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 0x7fb99406ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb998007fd0 tx=0x7fb998009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:09.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:09.801+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb9a405d680 con 0x7fb9a8102710 2024-03-14T13:11:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:09 smithi111 ceph-mon[145314]: pgmap v211: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.038+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb9a805f5f0 con 0x7fb9a8102710 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.038+0000 7fb99dffb700 1 -- 172.21.15.111:0/2898217413 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fb9a405d210 con 0x7fb9a8102710 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:11:10.039 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:11:10.040 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:11:10.041 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:10.042 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:10.042 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:10.042 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:10.042 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:10.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 msgr2=0x7fb99406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 0x7fb99406ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb998007fd0 tx=0x7fb998009040 comp rx=0 tx=0).stop 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 msgr2=0x7fb9a8112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb9a400b3b0 tx=0x7fb9a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 shutdown_connections 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb99406c7d0 0x7fb99406ec80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb9a8101d70 0x7fb9a81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 --2- 172.21.15.111:0/2898217413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9a8102710 0x7fb9a8112f00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.040+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 >> 172.21.15.111:0/2898217413 conn(0x7fb9a8072bc0 msgr2=0x7fb9a80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.041+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 shutdown_connections 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.041+0000 7fb9af30c700 1 -- 172.21.15.111:0/2898217413 wait complete. 2024-03-14T13:11:10.043 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:11:10.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 -- 172.21.15.111:0/1643061669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e81037b0 msgr2=0x7fb8e80fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 --2- 172.21.15.111:0/1643061669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e81037b0 0x7fb8e80fce20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8004940 tx=0x7fb8d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 -- 172.21.15.111:0/1643061669 shutdown_connections 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 --2- 172.21.15.111:0/1643061669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e81037b0 0x7fb8e80fce20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 --2- 172.21.15.111:0/1643061669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e8102e10 0x7fb8e81031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 -- 172.21.15.111:0/1643061669 >> 172.21.15.111:0/1643061669 conn(0x7fb8e80f89a0 msgr2=0x7fb8e80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 -- 172.21.15.111:0/1643061669 shutdown_connections 2024-03-14T13:11:10.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.141+0000 7fb8ed100700 1 -- 172.21.15.111:0/1643061669 wait complete. 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 Processor -- start 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 -- start start 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e81037b0 0x7fb8e8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81146f0 con 0x7fb8e8102e10 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8ed100700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e810da50 con 0x7fb8e81037b0 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47034/0 (socket says 172.21.15.111:47034) 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 -- 172.21.15.111:0/4216123784 learned_addr learned my addr 172.21.15.111:0/4216123784 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e659c700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e81037b0 0x7fb8e8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 -- 172.21.15.111:0/4216123784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e81037b0 msgr2=0x7fb8e8114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e81037b0 0x7fb8e8114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.142+0000 7fb8e6d9d700 1 -- 172.21.15.111:0/4216123784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d8005040 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8e6d9d700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00b790 tx=0x7fb8dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8dc00f770 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8dc00fd80 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e810dd30 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8dc00d310 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.143+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e810e2f0 con 0x7fb8e8102e10 2024-03-14T13:11:10.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.144+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e80fe560 con 0x7fb8e8102e10 2024-03-14T13:11:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.144+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb8dc010420 con 0x7fb8e8102e10 2024-03-14T13:11:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.145+0000 7fb8cf7fe700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 0x7fb8d0073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.145+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb8dc08b960 con 0x7fb8e8102e10 2024-03-14T13:11:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.145+0000 7fb8e659c700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 0x7fb8d0073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.145+0000 7fb8e659c700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 0x7fb8d0073080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800b3b0 tx=0x7fb8d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:10.149 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.149+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb8dc059730 con 0x7fb8e8102e10 2024-03-14T13:11:10.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.361+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8e8109e90 con 0x7fb8d0070bd0 2024-03-14T13:11:10.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.363+0000 7fb8cf7fe700 1 -- 172.21.15.111:0/4216123784 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb8e8109e90 con 0x7fb8d0070bd0 2024-03-14T13:11:10.363 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:11:10.363 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:11:10.363 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:11:10.364 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:11:10.365 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 msgr2=0x7fb8d0073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 0x7fb8d0073080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800b3b0 tx=0x7fb8d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 msgr2=0x7fb8e8113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00b790 tx=0x7fb8dc009250 comp rx=0 tx=0).stop 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 shutdown_connections 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb8d0070bd0 0x7fb8d0073080 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb8e8102e10 0x7fb8e8113ad0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 --2- 172.21.15.111:0/4216123784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8e81037b0 0x7fb8e8114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 >> 172.21.15.111:0/4216123784 conn(0x7fb8e80f89a0 msgr2=0x7fb8e80f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 shutdown_connections 2024-03-14T13:11:10.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.365+0000 7fb8ed100700 1 -- 172.21.15.111:0/4216123784 wait complete. 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.464+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/2461852612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea8102e70 msgr2=0x7f3ea8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.464+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/2461852612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea8102e70 0x7f3ea8103240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4004940 tx=0x7f3ea4009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/2461852612 shutdown_connections 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/2461852612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea80fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/2461852612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea8102e70 0x7f3ea8103240 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/2461852612 >> 172.21.15.111:0/2461852612 conn(0x7f3ea80714c0 msgr2=0x7f3ea80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:10.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/2461852612 shutdown_connections 2024-03-14T13:11:10.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/2461852612 wait complete. 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 Processor -- start 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 -- start start 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.465+0000 7f3eaf9fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3eaf9fa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea8102e70 0x7f3ea8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3eaf9fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8114600 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3eaf9fa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea810d9f0 con 0x7f3ea8102e70 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51804/0 (socket says 172.21.15.111:51804) 2024-03-14T13:11:10.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 -- 172.21.15.111:0/3860546118 learned_addr learned my addr 172.21.15.111:0/3860546118 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3eacf95700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea8102e70 0x7f3ea8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 -- 172.21.15.111:0/3860546118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea8102e70 msgr2=0x7f3ea8113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea8102e70 0x7f3ea8113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 -- 172.21.15.111:0/3860546118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4005040 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.466+0000 7f3ead796700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400b3b0 tx=0x7f3ea4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.468+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea400e040 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.468+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ea4013420 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.468+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea810dc70 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.468+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ea401c610 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.468+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea810e230 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.469+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea8100110 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.469+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3ea400ba80 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.470+0000 7f3e9dffb700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 0x7f3e9406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:10.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.470+0000 7f3eacf95700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 0x7f3e9406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:10.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.470+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3ea408cb60 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.470+0000 7f3eacf95700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 0x7f3e9406ec70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ea810ebb0 tx=0x7f3e9800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:10.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.476+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3ea405a8b0 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.747+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f3ea810f0c0 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.747+0000 7f3e9dffb700 1 -- 172.21.15.111:0/3860546118 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3ea405a440 con 0x7f3ea80fb4a0 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:11:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 msgr2=0x7f3e9406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 0x7f3e9406ec70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3ea810ebb0 tx=0x7f3e9800b040 comp rx=0 tx=0).stop 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 msgr2=0x7f3ea8113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ea400b3b0 tx=0x7f3ea4009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 shutdown_connections 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3e9406c7c0 0x7f3e9406ec70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ea80fb4a0 0x7f3ea8113a70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 --2- 172.21.15.111:0/3860546118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ea8102e70 0x7f3ea8113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 >> 172.21.15.111:0/3860546118 conn(0x7f3ea80714c0 msgr2=0x7f3ea80fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.749+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 shutdown_connections 2024-03-14T13:11:10.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:10.750+0000 7f3eaf9fa700 1 -- 172.21.15.111:0/3860546118 wait complete. 2024-03-14T13:11:10.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:10 smithi137 ceph-mon[108145]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:10 smithi137 ceph-mon[108145]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:10 smithi137 ceph-mon[108145]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:10 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1444052003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:11:10.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:10 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2898217413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:11:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:10 smithi111 ceph-mon[145314]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:10 smithi111 ceph-mon[145314]: from='client.34460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:10 smithi111 ceph-mon[145314]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:10 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1444052003' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:11:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:10 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2898217413' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:11:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:11 smithi137 ceph-mon[108145]: pgmap v212: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:11 smithi137 ceph-mon[108145]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:11.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:11 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3860546118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:11:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:11 smithi111 ceph-mon[145314]: pgmap v212: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:11 smithi111 ceph-mon[145314]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:11 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3860546118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:11:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:13 smithi137 ceph-mon[108145]: pgmap v213: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:13 smithi111 ceph-mon[145314]: pgmap v213: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:15 smithi137 ceph-mon[108145]: pgmap v214: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:15 smithi111 ceph-mon[145314]: pgmap v214: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:17 smithi137 ceph-mon[108145]: pgmap v215: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:17 smithi111 ceph-mon[145314]: pgmap v215: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:19 smithi137 ceph-mon[108145]: pgmap v216: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:19 smithi111 ceph-mon[145314]: pgmap v216: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:21 smithi137 ceph-mon[108145]: pgmap v217: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:21 smithi111 ceph-mon[145314]: pgmap v217: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:23 smithi137 ceph-mon[108145]: pgmap v218: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:23 smithi111 ceph-mon[145314]: pgmap v218: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:24.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:25.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:25 smithi137 ceph-mon[108145]: pgmap v219: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:25 smithi111 ceph-mon[145314]: pgmap v219: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:27 smithi137 ceph-mon[108145]: pgmap v220: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:27 smithi111 ceph-mon[145314]: pgmap v220: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:29 smithi137 ceph-mon[108145]: pgmap v221: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:29 smithi111 ceph-mon[145314]: pgmap v221: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:31 smithi137 ceph-mon[108145]: pgmap v222: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:31 smithi111 ceph-mon[145314]: pgmap v222: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:33 smithi111 ceph-mon[145314]: pgmap v223: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:33 smithi137 ceph-mon[108145]: pgmap v223: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:35 smithi111 ceph-mon[145314]: pgmap v224: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:35 smithi137 ceph-mon[108145]: pgmap v224: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:37 smithi111 ceph-mon[145314]: pgmap v225: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:37 smithi137 ceph-mon[108145]: pgmap v225: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:39 smithi111 ceph-mon[145314]: pgmap v226: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:39 smithi137 ceph-mon[108145]: pgmap v226: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 -- 172.21.15.111:0/1582111255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 msgr2=0x7f3438102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 --2- 172.21.15.111:0/1582111255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f3438102140 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f342c004940 tx=0x7f342c009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 -- 172.21.15.111:0/1582111255 shutdown_connections 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 --2- 172.21.15.111:0/1582111255 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f34380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 --2- 172.21.15.111:0/1582111255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f3438102140 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 -- 172.21.15.111:0/1582111255 >> 172.21.15.111:0/1582111255 conn(0x7f3438072bc0 msgr2=0x7f3438072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 -- 172.21.15.111:0/1582111255 shutdown_connections 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.863+0000 7f343f6a5700 1 -- 172.21.15.111:0/1582111255 wait complete. 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 Processor -- start 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 -- start start 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:40.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f343810f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343810f660 con 0x7f3438101d70 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343f6a5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343810f7a0 con 0x7f3438102710 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47350/0 (socket says 172.21.15.111:47350) 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 -- 172.21.15.111:0/2461679135 learned_addr learned my addr 172.21.15.111:0/2461679135 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343cc40700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f343810f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 -- 172.21.15.111:0/2461679135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 msgr2=0x7f343810f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f343810f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343d441700 1 -- 172.21.15.111:0/2461679135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342c005040 con 0x7f3438101d70 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.864+0000 7f343cc40700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f343810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f343d441700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f342c009180 tx=0x7f342c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c00e040 con 0x7f3438101d70 2024-03-14T13:11:40.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3438117a90 con 0x7f3438101d70 2024-03-14T13:11:40.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f342c004290 con 0x7f3438101d70 2024-03-14T13:11:40.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f342c01c3f0 con 0x7f3438101d70 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.865+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3438118020 con 0x7f3438101d70 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.866+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34380ff080 con 0x7f3438101d70 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.867+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f342c013790 con 0x7f3438101d70 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.867+0000 7f3429ffb700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 0x7f342406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.867+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f342c08cd30 con 0x7f3438101d70 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.867+0000 7f343cc40700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 0x7f342406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:40.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.867+0000 7f343cc40700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 0x7f342406ec70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3434007fd0 tx=0x7f3434009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:40.871 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:40.872+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f342c056fb0 con 0x7f3438101d70 2024-03-14T13:11:41.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.081+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f343804bed0 con 0x7f342406c7c0 2024-03-14T13:11:41.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.083+0000 7f3429ffb700 1 -- 172.21.15.111:0/2461679135 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f343804bed0 con 0x7f342406c7c0 2024-03-14T13:11:41.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 msgr2=0x7f342406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 0x7f342406ec70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3434007fd0 tx=0x7f3434009040 comp rx=0 tx=0).stop 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 msgr2=0x7f343810ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f342c009180 tx=0x7f342c009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 shutdown_connections 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f342406c7c0 0x7f342406ec70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3438101d70 0x7f343810ebe0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 --2- 172.21.15.111:0/2461679135 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3438102710 0x7f343810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 >> 172.21.15.111:0/2461679135 conn(0x7f3438072bc0 msgr2=0x7f34380fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 shutdown_connections 2024-03-14T13:11:41.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.085+0000 7f343f6a5700 1 -- 172.21.15.111:0/2461679135 wait complete. 2024-03-14T13:11:41.094 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:11:41.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 -- 172.21.15.111:0/3239477360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 msgr2=0x7f67940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 --2- 172.21.15.111:0/3239477360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f67940fbd80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6784004940 tx=0x7f6784009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 -- 172.21.15.111:0/3239477360 shutdown_connections 2024-03-14T13:11:41.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 --2- 172.21.15.111:0/3239477360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f67940fbd80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 --2- 172.21.15.111:0/3239477360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 0x7f6794102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 -- 172.21.15.111:0/3239477360 >> 172.21.15.111:0/3239477360 conn(0x7f6794072bc0 msgr2=0x7f6794072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 -- 172.21.15.111:0/3239477360 shutdown_connections 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 -- 172.21.15.111:0/3239477360 wait complete. 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.186+0000 7f6799d8c700 1 Processor -- start 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6799d8c700 1 -- start start 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6799d8c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 0x7f6794113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6799d8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6799d8c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6794114650 con 0x7f6794102710 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6799d8c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679410da40 con 0x7f6794101d70 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47372/0 (socket says 172.21.15.111:47372) 2024-03-14T13:11:41.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 -- 172.21.15.111:0/2173759857 learned_addr learned my addr 172.21.15.111:0/2173759857 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f67937fe700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 0x7f6794113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 -- 172.21.15.111:0/2173759857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 msgr2=0x7f6794113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 0x7f6794113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 -- 172.21.15.111:0/2173759857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784005040 con 0x7f6794102710 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.187+0000 7f6792ffd700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6784009380 tx=0x7f6784009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.188+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f678400e040 con 0x7f6794102710 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.188+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6784013420 con 0x7f6794102710 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.188+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679410dcc0 con 0x7f6794102710 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.188+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f678401c630 con 0x7f6794102710 2024-03-14T13:11:41.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.188+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679410e280 con 0x7f6794102710 2024-03-14T13:11:41.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.189+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940fd4c0 con 0x7f6794102710 2024-03-14T13:11:41.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.189+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f67840139b0 con 0x7f6794102710 2024-03-14T13:11:41.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.189+0000 7f678bfff700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 0x7f678006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.190+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f678408cbe0 con 0x7f6794102710 2024-03-14T13:11:41.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.190+0000 7f67937fe700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 0x7f678006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.190+0000 7f67937fe700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 0x7f678006ec70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f677c004f40 tx=0x7f677c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.194+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f678405a8f0 con 0x7f6794102710 2024-03-14T13:11:41.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.404+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f67940fd4c0 con 0x7f678006c7c0 2024-03-14T13:11:41.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.406+0000 7f678bfff700 1 -- 172.21.15.111:0/2173759857 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f67940fd4c0 con 0x7f678006c7c0 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 msgr2=0x7f678006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 0x7f678006ec70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f677c004f40 tx=0x7f677c00b040 comp rx=0 tx=0).stop 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 msgr2=0x7f6794114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6784009380 tx=0x7f6784009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 shutdown_connections 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f678006c7c0 0x7f678006ec70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6794101d70 0x7f6794113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 --2- 172.21.15.111:0/2173759857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6794102710 0x7f6794114000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 >> 172.21.15.111:0/2173759857 conn(0x7f6794072bc0 msgr2=0x7f67940659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 shutdown_connections 2024-03-14T13:11:41.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.408+0000 7f6799d8c700 1 -- 172.21.15.111:0/2173759857 wait complete. 2024-03-14T13:11:41.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 -- 172.21.15.111:0/2880601631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 msgr2=0x7f9c9c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/2880601631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c1031f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9c90004940 tx=0x7f9c90009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 -- 172.21.15.111:0/2880601631 shutdown_connections 2024-03-14T13:11:41.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/2880601631 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 0x7f9c9c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/2880601631 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c1031f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 -- 172.21.15.111:0/2880601631 >> 172.21.15.111:0/2880601631 conn(0x7f9c9c0f89d0 msgr2=0x7f9c9c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 -- 172.21.15.111:0/2880601631 shutdown_connections 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 -- 172.21.15.111:0/2880601631 wait complete. 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.508+0000 7f9ca2a67700 1 Processor -- start 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca2a67700 1 -- start start 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca2a67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca2a67700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 0x7f9c9c113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca2a67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c114610 con 0x7f9c9c102e20 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca2a67700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c10da00 con 0x7f9c9c1037c0 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47398/0 (socket says 172.21.15.111:47398) 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 -- 172.21.15.111:0/3044290127 learned_addr learned my addr 172.21.15.111:0/3044290127 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:41.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9c9bfff700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 0x7f9c9c113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 -- 172.21.15.111:0/3044290127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 msgr2=0x7f9c9c113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 0x7f9c9c113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 -- 172.21.15.111:0/3044290127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c90005040 con 0x7f9c9c102e20 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.509+0000 7f9ca0803700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009180 tx=0x7f9c90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9000e040 con 0x7f9c9c102e20 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c90004290 con 0x7f9c9c102e20 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c9001c3f0 con 0x7f9c9c102e20 2024-03-14T13:11:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c10dc80 con 0x7f9c9c102e20 2024-03-14T13:11:41.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c10e240 con 0x7f9c9c102e20 2024-03-14T13:11:41.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.510+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c0fe570 con 0x7f9c9c102e20 2024-03-14T13:11:41.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.511+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9c90013790 con 0x7f9c9c102e20 2024-03-14T13:11:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.511+0000 7f9c997fa700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 0x7f9c8c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.511+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c9008cc30 con 0x7f9c9c102e20 2024-03-14T13:11:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.512+0000 7f9c9bfff700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 0x7f9c8c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.512+0000 7f9c9bfff700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 0x7f9c8c06ec70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c88009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.515 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.516+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c9005aa00 con 0x7f9c9c102e20 2024-03-14T13:11:41.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.716+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9c109e50 con 0x7f9c8c06c7c0 2024-03-14T13:11:41.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.726+0000 7f9c997fa700 1 -- 172.21.15.111:0/3044290127 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9c9c109e50 con 0x7f9c8c06c7c0 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (7m) 83s ago 19m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (5m) 83s ago 19m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (5m) 53s ago 18m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (7m) 83s ago 18m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (88s) 83s ago 13m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (108s) 83s ago 13m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (56s) 53s ago 13m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (72s) 53s ago 13m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (10m) 83s ago 20m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (9m) 53s ago 18m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (6m) 83s ago 21m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (6m) 53s ago 18m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:11:41.726 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (8m) 83s ago 19m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (8m) 53s ago 18m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (4m) 83s ago 17m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (4m) 83s ago 16m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (3m) 83s ago 16m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (3m) 53s ago 15m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (2m) 53s ago 15m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (2m) 53s ago 14m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:11:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (8m) 83s ago 18m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.728+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 msgr2=0x7f9c8c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.728+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 0x7f9c8c06ec70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009fd0 tx=0x7f9c88009f60 comp rx=0 tx=0).stop 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.728+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 msgr2=0x7f9c9c113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.728+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9c90009180 tx=0x7f9c90009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 shutdown_connections 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c8c06c7c0 0x7f9c8c06ec70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c9c102e20 0x7f9c9c113a80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 --2- 172.21.15.111:0/3044290127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c9c1037c0 0x7f9c9c113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 >> 172.21.15.111:0/3044290127 conn(0x7f9c9c0f89d0 msgr2=0x7f9c9c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 shutdown_connections 2024-03-14T13:11:41.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.729+0000 7f9ca2a67700 1 -- 172.21.15.111:0/3044290127 wait complete. 2024-03-14T13:11:41.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.827+0000 7f4cd1587700 1 -- 172.21.15.111:0/3491529396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fde20 msgr2=0x7f4ccc0fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.827+0000 7f4cd1587700 1 --2- 172.21.15.111:0/3491529396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fe1f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0004940 tx=0x7f4cc0009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- 172.21.15.111:0/3491529396 shutdown_connections 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 --2- 172.21.15.111:0/3491529396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 --2- 172.21.15.111:0/3491529396 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fe1f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- 172.21.15.111:0/3491529396 >> 172.21.15.111:0/3491529396 conn(0x7f4ccc0f7870 msgr2=0x7f4ccc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- 172.21.15.111:0/3491529396 shutdown_connections 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- 172.21.15.111:0/3491529396 wait complete. 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 Processor -- start 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- start start 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0fdba0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.828+0000 7f4cd1587700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc0fc6d0 con 0x7f4ccc0fde20 2024-03-14T13:11:41.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47416/0 (socket says 172.21.15.111:47416) 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 -- 172.21.15.111:0/1898225066 learned_addr learned my addr 172.21.15.111:0/1898225066 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4ccaffd700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 -- 172.21.15.111:0/1898225066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fde20 msgr2=0x7f4ccc0fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 -- 172.21.15.111:0/1898225066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0005040 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cca7fc700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc002a50 tx=0x7f4cbc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cbc007cc0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc0fc9b0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cbc00f040 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4cbc009540 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.829+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc119fd0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.830+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc066b70 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.831 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.831+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4cbc0096a0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.831+0000 7f4cb37fe700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 0x7f4cb406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:41.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.831+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4cbc08b890 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.831+0000 7f4ccaffd700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 0x7f4cb406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:41.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.832+0000 7f4ccaffd700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 0x7f4cb406ec70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0004560 tx=0x7f4cc0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:41.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:41.836+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4cbc055a90 con 0x7f4ccc0fe7c0 2024-03-14T13:11:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:41 smithi111 ceph-mon[145314]: pgmap v227: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:41 smithi137 ceph-mon[108145]: pgmap v227: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:42.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.107+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4ccc11a1c0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:42.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.108+0000 7f4cb37fe700 1 -- 172.21.15.111:0/1898225066 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f4cbc0590b0 con 0x7f4ccc0fe7c0 2024-03-14T13:11:42.107 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:11:42.107 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:11:42.108 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:11:42.109 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:11:42.109 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 msgr2=0x7f4cb406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.109 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 0x7f4cb406ec70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0004560 tx=0x7f4cc0009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 msgr2=0x7f4ccc0fc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc002a50 tx=0x7f4cbc0046b0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 shutdown_connections 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4cb406c7c0 0x7f4cb406ec70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ccc0fde20 0x7f4ccc0fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 --2- 172.21.15.111:0/1898225066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ccc0fe7c0 0x7f4ccc0fc190 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 >> 172.21.15.111:0/1898225066 conn(0x7f4ccc0f7870 msgr2=0x7f4ccc1069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 shutdown_connections 2024-03-14T13:11:42.110 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.110+0000 7f4cd1587700 1 -- 172.21.15.111:0/1898225066 wait complete. 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 -- 172.21.15.111:0/2930705353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 msgr2=0x7f0c4c0fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 --2- 172.21.15.111:0/2930705353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c0fcc70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c004940 tx=0x7f0c3c009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 -- 172.21.15.111:0/2930705353 shutdown_connections 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 --2- 172.21.15.111:0/2930705353 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c0fcc70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 --2- 172.21.15.111:0/2930705353 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 0x7f0c4c1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 -- 172.21.15.111:0/2930705353 >> 172.21.15.111:0/2930705353 conn(0x7f0c4c0f6bd0 msgr2=0x7f0c4c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 -- 172.21.15.111:0/2930705353 shutdown_connections 2024-03-14T13:11:42.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.210+0000 7f0c51579700 1 -- 172.21.15.111:0/2930705353 wait complete. 2024-03-14T13:11:42.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 Processor -- start 2024-03-14T13:11:42.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 -- start start 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 0x7f0c4c10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0fc580 con 0x7f0c4c0fc800 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c51579700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c10d310 con 0x7f0c4c101cf0 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47434/0 (socket says 172.21.15.111:47434) 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 -- 172.21.15.111:0/382668187 learned_addr learned my addr 172.21.15.111:0/382668187 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4a7fc700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 0x7f0c4c10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 -- 172.21.15.111:0/382668187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 msgr2=0x7f0c4c10cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.212 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 0x7f0c4c10cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.211+0000 7f0c4affd700 1 -- 172.21.15.111:0/382668187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c005040 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c4affd700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0c40002a50 tx=0x7f0c400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c40007cc0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c10d5f0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c4000f040 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c40009540 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.212+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c11a0c0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.213+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c0fb310 con 0x7f0c4c0fc800 2024-03-14T13:11:42.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.214+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0c400096a0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.214+0000 7f0c337fe700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 0x7f0c3406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.214+0000 7f0c4a7fc700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 0x7f0c3406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.214+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0c4008be90 con 0x7f0c4c0fc800 2024-03-14T13:11:42.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.215+0000 7f0c4a7fc700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 0x7f0c3406ec70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00b3b0 tx=0x7f0c3c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.218+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0c40059ba0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.453+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0c4c108ca0 con 0x7f0c4c0fc800 2024-03-14T13:11:42.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.454+0000 7f0c337fe700 1 -- 172.21.15.111:0/382668187 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f0c40059730 con 0x7f0c4c0fc800 2024-03-14T13:11:42.453 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:11:42.453 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:11:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:11:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:42.456 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 msgr2=0x7f0c3406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 0x7f0c3406ec70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00b3b0 tx=0x7f0c3c009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 msgr2=0x7f0c4c10c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0c40002a50 tx=0x7f0c400046b0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 shutdown_connections 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0c3406c7c0 0x7f0c3406ec70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c4c0fc800 0x7f0c4c10c890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.457 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 --2- 172.21.15.111:0/382668187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c4c101cf0 0x7f0c4c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 >> 172.21.15.111:0/382668187 conn(0x7f0c4c0f6bd0 msgr2=0x7f0c4c0f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 shutdown_connections 2024-03-14T13:11:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.456+0000 7f0c51579700 1 -- 172.21.15.111:0/382668187 wait complete. 2024-03-14T13:11:42.458 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:11:42.557 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 -- 172.21.15.111:0/1714273749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc0fb4a0 msgr2=0x7f9fcc0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/1714273749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fb910 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc004940 tx=0x7f9fbc009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 -- 172.21.15.111:0/1714273749 shutdown_connections 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/1714273749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fb910 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/1714273749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc102e70 0x7f9fcc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 -- 172.21.15.111:0/1714273749 >> 172.21.15.111:0/1714273749 conn(0x7f9fcc0714c0 msgr2=0x7f9fcc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 -- 172.21.15.111:0/1714273749 shutdown_connections 2024-03-14T13:11:42.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.558+0000 7f9fd10cf700 1 -- 172.21.15.111:0/1714273749 wait complete. 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 Processor -- start 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 -- start start 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0fd170 con 0x7f9fcc102e70 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fd10cf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0fd2e0 con 0x7f9fcc0fb4a0 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47448/0 (socket says 172.21.15.111:47448) 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 -- 172.21.15.111:0/767506176 learned_addr learned my addr 172.21.15.111:0/767506176 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:42.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fcad9d700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 -- 172.21.15.111:0/767506176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0fb4a0 msgr2=0x7f9fcc0fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 -- 172.21.15.111:0/767506176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc005040 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.559+0000 7f9fca59c700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009390 tx=0x7f9fbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.560+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fbc00e040 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.560+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fbc013420 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.560+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc119d70 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.560+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fbc01c630 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.560+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc11a180 con 0x7f9fcc102e70 2024-03-14T13:11:42.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.561+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc100110 con 0x7f9fcc102e70 2024-03-14T13:11:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.561+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9fbc0139e0 con 0x7f9fcc102e70 2024-03-14T13:11:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.561+0000 7f9fb37fe700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 0x7f9fb406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.562+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9fbc08ccd0 con 0x7f9fcc102e70 2024-03-14T13:11:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.562+0000 7f9fcad9d700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 0x7f9fb406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.562+0000 7f9fcad9d700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 0x7f9fb406ec80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0004d60 tx=0x7f9fc000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.566+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9fbc056f50 con 0x7f9fcc102e70 2024-03-14T13:11:42.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.774+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fcc109e00 con 0x7f9fb406c7d0 2024-03-14T13:11:42.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.775+0000 7f9fb37fe700 1 -- 172.21.15.111:0/767506176 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9fcc109e00 con 0x7f9fb406c7d0 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:11:42.775 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 msgr2=0x7f9fb406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 0x7f9fb406ec80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0004d60 tx=0x7f9fc000b040 comp rx=0 tx=0).stop 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 msgr2=0x7f9fcc0fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc009390 tx=0x7f9fbc009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 shutdown_connections 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fb406c7d0 0x7f9fb406ec80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0fb4a0 0x7f9fcc0fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 --2- 172.21.15.111:0/767506176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9fcc102e70 0x7f9fcc0fcbc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 >> 172.21.15.111:0/767506176 conn(0x7f9fcc0714c0 msgr2=0x7f9fcc0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 shutdown_connections 2024-03-14T13:11:42.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.777+0000 7f9fd10cf700 1 -- 172.21.15.111:0/767506176 wait complete. 2024-03-14T13:11:42.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 -- 172.21.15.111:0/2462468690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 msgr2=0x7f9894102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2462468690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f9894102140 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9884004940 tx=0x7f9884009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 -- 172.21.15.111:0/2462468690 shutdown_connections 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2462468690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 0x7f98940fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2462468690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f9894102140 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 -- 172.21.15.111:0/2462468690 >> 172.21.15.111:0/2462468690 conn(0x7f9894072bc0 msgr2=0x7f9894072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 -- 172.21.15.111:0/2462468690 shutdown_connections 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.877+0000 7f989c4a5700 1 -- 172.21.15.111:0/2462468690 wait complete. 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 Processor -- start 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 -- start start 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 0x7f989410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989410d250 con 0x7f9894101d70 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989c4a5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989410d3c0 con 0x7f9894102710 2024-03-14T13:11:42.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989a241700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989a241700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47470/0 (socket says 172.21.15.111:47470) 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f989a241700 1 -- 172.21.15.111:0/2988053313 learned_addr learned my addr 172.21.15.111:0/2988053313 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.878+0000 7f9899a40700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 0x7f989410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989a241700 1 -- 172.21.15.111:0/2988053313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 msgr2=0x7f989410cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989a241700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 0x7f989410cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989a241700 1 -- 172.21.15.111:0/2988053313 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9884005040 con 0x7f9894101d70 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989a241700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98840040b0 tx=0x7f9884009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9884021070 con 0x7f9894101d70 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9884013de0 con 0x7f9894101d70 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989410d640 con 0x7f9894101d70 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f988400e070 con 0x7f9894101d70 2024-03-14T13:11:42.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.879+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989411a190 con 0x7f9894101d70 2024-03-14T13:11:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.880+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98940ff080 con 0x7f9894101d70 2024-03-14T13:11:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.880+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9884013930 con 0x7f9894101d70 2024-03-14T13:11:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.881+0000 7f988affd700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 0x7f988006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:11:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.881+0000 7f9899a40700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 0x7f988006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:11:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.881+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f988408c870 con 0x7f9894101d70 2024-03-14T13:11:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.881+0000 7f9899a40700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 0x7f988006ec70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9894066000 tx=0x7f989000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:11:42.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:42.885+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f988405a640 con 0x7f9894101d70 2024-03-14T13:11:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:42 smithi111 ceph-mon[145314]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:42 smithi111 ceph-mon[145314]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:42 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1898225066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:11:42.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:42 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/382668187' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:11:43.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:42 smithi137 ceph-mon[108145]: from='client.34484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:42 smithi137 ceph-mon[108145]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:42 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1898225066' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:11:43.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:42 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/382668187' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:11:43.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.157+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f989410d640 con 0x7f9894101d70 2024-03-14T13:11:43.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.157+0000 7f988affd700 1 -- 172.21.15.111:0/2988053313 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f989410d640 con 0x7f9894101d70 2024-03-14T13:11:43.157 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:11:43.157 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:11:43.157 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:11:43.157 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:11:43.157 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 msgr2=0x7f988006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 0x7f988006ec70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9894066000 tx=0x7f989000b040 comp rx=0 tx=0).stop 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 msgr2=0x7f989410c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98840040b0 tx=0x7f9884009e30 comp rx=0 tx=0).stop 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 shutdown_connections 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f988006c7c0 0x7f988006ec70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9894101d70 0x7f989410c7d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 --2- 172.21.15.111:0/2988053313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9894102710 0x7f989410cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 >> 172.21.15.111:0/2988053313 conn(0x7f9894072bc0 msgr2=0x7f98940faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 shutdown_connections 2024-03-14T13:11:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:11:43.159+0000 7f989c4a5700 1 -- 172.21.15.111:0/2988053313 wait complete. 2024-03-14T13:11:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:43 smithi111 ceph-mon[145314]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:43 smithi111 ceph-mon[145314]: pgmap v228: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:43 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2988053313' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:11:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:43 smithi137 ceph-mon[108145]: from='client.34492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:43 smithi137 ceph-mon[108145]: pgmap v228: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:43 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2988053313' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:11:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:44 smithi111 ceph-mon[145314]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:45.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:44 smithi137 ceph-mon[108145]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:11:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:45 smithi111 ceph-mon[145314]: pgmap v229: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:45 smithi137 ceph-mon[108145]: pgmap v229: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:46.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:46 smithi111 ceph-mon[145314]: pgmap v230: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:47.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:46 smithi137 ceph-mon[108145]: pgmap v230: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:49.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:49 smithi111 ceph-mon[145314]: pgmap v231: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:49 smithi137 ceph-mon[108145]: pgmap v231: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:51 smithi137 ceph-mon[108145]: pgmap v232: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:51 smithi111 ceph-mon[145314]: pgmap v232: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:53 smithi137 ceph-mon[108145]: pgmap v233: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:53 smithi111 ceph-mon[145314]: pgmap v233: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:11:55.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:55 smithi137 ceph-mon[108145]: pgmap v234: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:55 smithi111 ceph-mon[145314]: pgmap v234: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:57 smithi137 ceph-mon[108145]: pgmap v235: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:57 smithi111 ceph-mon[145314]: pgmap v235: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:11:59 smithi137 ceph-mon[108145]: pgmap v236: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:11:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:11:59 smithi111 ceph-mon[145314]: pgmap v236: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:01 smithi137 ceph-mon[108145]: pgmap v237: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:01 smithi111 ceph-mon[145314]: pgmap v237: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:03.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:03 smithi137 ceph-mon[108145]: pgmap v238: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:03 smithi111 ceph-mon[145314]: pgmap v238: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:05 smithi137 ceph-mon[108145]: pgmap v239: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:05 smithi111 ceph-mon[145314]: pgmap v239: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:07.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:07 smithi137 ceph-mon[108145]: pgmap v240: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:07 smithi111 ceph-mon[145314]: pgmap v240: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:09.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:09 smithi137 ceph-mon[108145]: pgmap v241: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:09.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:09 smithi111 ceph-mon[145314]: pgmap v241: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:11.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:11 smithi137 ceph-mon[108145]: pgmap v242: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:11 smithi111 ceph-mon[145314]: pgmap v242: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:13.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 -- 172.21.15.111:0/1316213884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01026b0 msgr2=0x7fb5b00fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 --2- 172.21.15.111:0/1316213884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01026b0 0x7fb5b00fbd20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4004940 tx=0x7fb5a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:13.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 -- 172.21.15.111:0/1316213884 shutdown_connections 2024-03-14T13:12:13.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 --2- 172.21.15.111:0/1316213884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b01026b0 0x7fb5b00fbd20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 --2- 172.21.15.111:0/1316213884 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b0101d10 0x7fb5b01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 -- 172.21.15.111:0/1316213884 >> 172.21.15.111:0/1316213884 conn(0x7fb5b00f7880 msgr2=0x7fb5b00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:13.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 -- 172.21.15.111:0/1316213884 shutdown_connections 2024-03-14T13:12:13.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.272+0000 7fb5b7450700 1 -- 172.21.15.111:0/1316213884 wait complete. 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 Processor -- start 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 -- start start 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 0x7fb5b0113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b0113890 con 0x7fb5b0101d10 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b7450700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b010cc60 con 0x7fb5b01026b0 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50804/0 (socket says 172.21.15.111:50804) 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 -- 172.21.15.111:0/829326939 learned_addr learned my addr 172.21.15.111:0/829326939 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b49eb700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 0x7fb5b0113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 -- 172.21.15.111:0/829326939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 msgr2=0x7fb5b0113240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 0x7fb5b0113240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b51ec700 1 -- 172.21.15.111:0/829326939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4005040 con 0x7fb5b0101d10 2024-03-14T13:12:13.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.273+0000 7fb5b49eb700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 0x7fb5b0113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5b51ec700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c8c0 tx=0x7fb5a00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a000a980 con 0x7fb5b0101d10 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b010cf40 con 0x7fb5b0101d10 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5a000ce60 con 0x7fb5b0101d10 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a00113a0 con 0x7fb5b0101d10 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.274+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b010d500 con 0x7fb5b0101d10 2024-03-14T13:12:13.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.275+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b00fd460 con 0x7fb5b0101d10 2024-03-14T13:12:13.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.275+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb5a0012400 con 0x7fb5b0101d10 2024-03-14T13:12:13.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.276+0000 7fb5adffb700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 0x7fb59806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.276+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb5a008bb90 con 0x7fb5b0101d10 2024-03-14T13:12:13.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.276+0000 7fb5b49eb700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 0x7fb59806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.276+0000 7fb5b49eb700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 0x7fb59806ec70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400b3b0 tx=0x7fb5a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.281+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb5a00598a0 con 0x7fb5b0101d10 2024-03-14T13:12:13.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.495+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5b004bed0 con 0x7fb59806c7c0 2024-03-14T13:12:13.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.497+0000 7fb5adffb700 1 -- 172.21.15.111:0/829326939 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb5b004bed0 con 0x7fb59806c7c0 2024-03-14T13:12:13.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 msgr2=0x7fb59806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 0x7fb59806ec70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400b3b0 tx=0x7fb5a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 msgr2=0x7fb5b0112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c8c0 tx=0x7fb5a00044c0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 shutdown_connections 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb59806c7c0 0x7fb59806ec70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b0101d10 0x7fb5b0112d00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 --2- 172.21.15.111:0/829326939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5b01026b0 0x7fb5b0113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 >> 172.21.15.111:0/829326939 conn(0x7fb5b00f7880 msgr2=0x7fb5b00f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 shutdown_connections 2024-03-14T13:12:13.500 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.499+0000 7fb5b7450700 1 -- 172.21.15.111:0/829326939 wait complete. 2024-03-14T13:12:13.510 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:12:13.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 -- 172.21.15.111:0/375188615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 msgr2=0x7f5c080fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 --2- 172.21.15.111:0/375188615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fbd30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5c04004940 tx=0x7f5c04009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 -- 172.21.15.111:0/375188615 shutdown_connections 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 --2- 172.21.15.111:0/375188615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fbd30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 --2- 172.21.15.111:0/375188615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 0x7f5c081020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 -- 172.21.15.111:0/375188615 >> 172.21.15.111:0/375188615 conn(0x7f5c080f78d0 msgr2=0x7f5c080f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 -- 172.21.15.111:0/375188615 shutdown_connections 2024-03-14T13:12:13.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.600+0000 7f5c10057700 1 -- 172.21.15.111:0/375188615 wait complete. 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 Processor -- start 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 -- start start 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 0x7f5c080fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c080fe470 con 0x7f5c081026c0 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c10057700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c080fe5b0 con 0x7f5c08101d20 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50820/0 (socket says 172.21.15.111:50820) 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 -- 172.21.15.111:0/3361623569 learned_addr learned my addr 172.21.15.111:0/3361623569 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0ddf3700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 0x7f5c080fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 -- 172.21.15.111:0/3361623569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 msgr2=0x7f5c080fd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 0x7f5c080fd9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 -- 172.21.15.111:0/3361623569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c04005040 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5c0d5f2700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5c040093b0 tx=0x7f5c04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.601+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c0400e040 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.602+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c080fe800 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.602+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c04013420 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.602+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c0401c630 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.602+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0811a240 con 0x7f5c081026c0 2024-03-14T13:12:13.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.602+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c080fd470 con 0x7f5c081026c0 2024-03-14T13:12:13.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.603+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5c040139e0 con 0x7f5c081026c0 2024-03-14T13:12:13.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.603+0000 7f5bfe7fc700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 0x7f5bf406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.603+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5c0408cba0 con 0x7f5c081026c0 2024-03-14T13:12:13.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.603+0000 7f5c0ddf3700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 0x7f5bf406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.604+0000 7f5c0ddf3700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 0x7f5bf406ec70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80041d0 tx=0x7f5bf80043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.608+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5c04056ed0 con 0x7f5c081026c0 2024-03-14T13:12:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:13 smithi137 ceph-mon[108145]: pgmap v243: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:13.823 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.823+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c08108d50 con 0x7f5bf406c7c0 2024-03-14T13:12:13.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.824+0000 7f5bfe7fc700 1 -- 172.21.15.111:0/3361623569 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5c08108d50 con 0x7f5bf406c7c0 2024-03-14T13:12:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 msgr2=0x7f5bf406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 0x7f5bf406ec70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5bf80041d0 tx=0x7f5bf80043c0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 msgr2=0x7f5c080fdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5c040093b0 tx=0x7f5c04009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 shutdown_connections 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5bf406c7c0 0x7f5bf406ec70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c08101d20 0x7f5c080fd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 --2- 172.21.15.111:0/3361623569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c081026c0 0x7f5c080fdee0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 >> 172.21.15.111:0/3361623569 conn(0x7f5c080f78d0 msgr2=0x7f5c080f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 shutdown_connections 2024-03-14T13:12:13.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.826+0000 7f5c10057700 1 -- 172.21.15.111:0/3361623569 wait complete. 2024-03-14T13:12:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:13 smithi111 ceph-mon[145314]: pgmap v243: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 -- 172.21.15.111:0/226772746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 msgr2=0x7f89a40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.927 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 --2- 172.21.15.111:0/226772746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a40fbd80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8994004940 tx=0x7f8994009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 -- 172.21.15.111:0/226772746 shutdown_connections 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 --2- 172.21.15.111:0/226772746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a40fbd80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 --2- 172.21.15.111:0/226772746 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 -- 172.21.15.111:0/226772746 >> 172.21.15.111:0/226772746 conn(0x7f89a4072bc0 msgr2=0x7f89a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 -- 172.21.15.111:0/226772746 shutdown_connections 2024-03-14T13:12:13.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.927+0000 7f89a88c1700 1 -- 172.21.15.111:0/226772746 wait complete. 2024-03-14T13:12:13.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 Processor -- start 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 -- start start 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a41129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a4113550 con 0x7f89a4102710 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a88c1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a410c940 con 0x7f89a4101d70 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50832/0 (socket says 172.21.15.111:50832) 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 -- 172.21.15.111:0/537577892 learned_addr learned my addr 172.21.15.111:0/537577892 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a259c700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a41129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 -- 172.21.15.111:0/537577892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 msgr2=0x7f89a41129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a41129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a1d9b700 1 -- 172.21.15.111:0/537577892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994005040 con 0x7f89a4102710 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.928+0000 7f89a259c700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a41129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:12:13.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f89a1d9b700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f899400b3b0 tx=0x7f8994009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f899400e040 con 0x7f89a4102710 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a410cbc0 con 0x7f89a4102710 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f899400bd60 con 0x7f89a4102710 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8994013660 con 0x7f89a4102710 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.929+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a410d120 con 0x7f89a4102710 2024-03-14T13:12:13.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.930+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a40fd4c0 con 0x7f89a4102710 2024-03-14T13:12:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.930+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8994013920 con 0x7f89a4102710 2024-03-14T13:12:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.931+0000 7f899affd700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 0x7f899006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.931+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f899408fa70 con 0x7f89a4102710 2024-03-14T13:12:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.931+0000 7f89a259c700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 0x7f899006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:13.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.931+0000 7f89a259c700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 0x7f899006ec70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f898c009fd0 tx=0x7f898c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:13.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:13.935+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f899405d700 con 0x7f89a4102710 2024-03-14T13:12:14.136 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.136+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f89a4108d10 con 0x7f899006c7c0 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.145+0000 7f899affd700 1 -- 172.21.15.111:0/537577892 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f89a4108d10 con 0x7f899006c7c0 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (8m) 116s ago 19m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (5m) 116s ago 20m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (5m) 86s ago 18m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (8m) 116s ago 19m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (2m) 116s ago 13m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (2m) 116s ago 13m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (88s) 86s ago 13m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:12:14.146 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (104s) 86s ago 13m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (11m) 116s ago 21m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (9m) 86s ago 18m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (7m) 116s ago 21m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (6m) 86s ago 18m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (9m) 116s ago 20m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (9m) 86s ago 18m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (5m) 116s ago 17m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (4m) 116s ago 17m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (4m) 116s ago 16m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (3m) 86s ago 16m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (3m) 86s ago 15m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (2m) 86s ago 15m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:12:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (8m) 116s ago 19m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 msgr2=0x7f899006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 0x7f899006ec70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f898c009fd0 tx=0x7f898c009ee0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 msgr2=0x7f89a4112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f899400b3b0 tx=0x7f8994009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 shutdown_connections 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f899006c7c0 0x7f899006ec70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a4101d70 0x7f89a41129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 --2- 172.21.15.111:0/537577892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a4102710 0x7f89a4112f00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.149 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.147+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 >> 172.21.15.111:0/537577892 conn(0x7f89a4072bc0 msgr2=0x7f89a40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.149 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.148+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 shutdown_connections 2024-03-14T13:12:14.149 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.148+0000 7f89a88c1700 1 -- 172.21.15.111:0/537577892 wait complete. 2024-03-14T13:12:14.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 -- 172.21.15.111:0/75075466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 msgr2=0x7fa3ac1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/75075466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac1033a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa39c004940 tx=0x7fa39c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 -- 172.21.15.111:0/75075466 shutdown_connections 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/75075466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 0x7fa3ac0fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/75075466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac1033a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 -- 172.21.15.111:0/75075466 >> 172.21.15.111:0/75075466 conn(0x7fa3ac071530 msgr2=0x7fa3ac071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 -- 172.21.15.111:0/75075466 shutdown_connections 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.246+0000 7fa3b3edf700 1 -- 172.21.15.111:0/75075466 wait complete. 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 Processor -- start 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 -- start start 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 0x7fa3ac113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac1143e0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b3edf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac10da30 con 0x7fa3ac0fb540 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50854/0 (socket says 172.21.15.111:50854) 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 -- 172.21.15.111:0/183575507 learned_addr learned my addr 172.21.15.111:0/183575507 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b1c7b700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 0x7fa3ac113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 -- 172.21.15.111:0/183575507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 msgr2=0x7fa3ac113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 0x7fa3ac113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.247+0000 7fa3b147a700 1 -- 172.21.15.111:0/183575507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c005040 con 0x7fa3ac102fd0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3b147a700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8002a50 tx=0x7fa3a80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a8007cc0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a800f040 con 0x7fa3ac102fd0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac10dd10 con 0x7fa3ac102fd0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a800d9c0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.250 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.248+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac10e2d0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.249+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac100270 con 0x7fa3ac102fd0 2024-03-14T13:12:14.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.249+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa3a800db20 con 0x7fa3ac102fd0 2024-03-14T13:12:14.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.250+0000 7fa3a27fc700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 0x7fa39806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.250+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa3a808b760 con 0x7fa3ac102fd0 2024-03-14T13:12:14.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.250+0000 7fa3b1c7b700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 0x7fa39806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.250+0000 7fa3b1c7b700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 0x7fa39806ed40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa39c004f90 tx=0x7fa39c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.254+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa3a80593f0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.530+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa3ac10eea0 con 0x7fa3ac102fd0 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.530+0000 7fa3a27fc700 1 -- 172.21.15.111:0/183575507 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fa3a8058f80 con 0x7fa3ac102fd0 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:12:14.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:12:14.532 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:12:14.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.532+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 msgr2=0x7fa39806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.532+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 0x7fa39806ed40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa39c004f90 tx=0x7fa39c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.532+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 msgr2=0x7fa3ac113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8002a50 tx=0x7fa3a80046b0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 shutdown_connections 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa39806c890 0x7fa39806ed40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3ac0fb540 0x7fa3ac113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 --2- 172.21.15.111:0/183575507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3ac102fd0 0x7fa3ac113d90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 >> 172.21.15.111:0/183575507 conn(0x7fa3ac071530 msgr2=0x7fa3ac0fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 shutdown_connections 2024-03-14T13:12:14.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.533+0000 7fa3b3edf700 1 -- 172.21.15.111:0/183575507 wait complete. 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 -- 172.21.15.111:0/576738773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fbc80 msgr2=0x7f40b80fc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 --2- 172.21.15.111:0/576738773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fbc80 0x7f40b80fc050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f40a8004940 tx=0x7f40a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 -- 172.21.15.111:0/576738773 shutdown_connections 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 --2- 172.21.15.111:0/576738773 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fc590 0x7f40b81092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 --2- 172.21.15.111:0/576738773 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fbc80 0x7f40b80fc050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 -- 172.21.15.111:0/576738773 >> 172.21.15.111:0/576738773 conn(0x7f40b8072bc0 msgr2=0x7f40b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 -- 172.21.15.111:0/576738773 shutdown_connections 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.633+0000 7f40bcd03700 1 -- 172.21.15.111:0/576738773 wait complete. 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 Processor -- start 2024-03-14T13:12:14.636 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 -- start start 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fbc80 0x7f40b8115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b81168c0 con 0x7f40b80fc590 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40bcd03700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b810fc90 con 0x7f40b80fbc80 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50860/0 (socket says 172.21.15.111:50860) 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 -- 172.21.15.111:0/3949179288 learned_addr learned my addr 172.21.15.111:0/3949179288 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b659c700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fbc80 0x7f40b8115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 -- 172.21.15.111:0/3949179288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fbc80 msgr2=0x7f40b8115d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fbc80 0x7f40b8115d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.634+0000 7f40b5d9b700 1 -- 172.21.15.111:0/3949179288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a8005040 con 0x7f40b80fc590 2024-03-14T13:12:14.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40b5d9b700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c8c0 tx=0x7f40a00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a000a760 con 0x7f40b80fc590 2024-03-14T13:12:14.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40a000ad70 con 0x7f40b80fc590 2024-03-14T13:12:14.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b810ff70 con 0x7f40b80fc590 2024-03-14T13:12:14.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40a00113a0 con 0x7f40b80fc590 2024-03-14T13:12:14.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.635+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b81104b0 con 0x7f40b80fc590 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.636+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b8065e30 con 0x7f40b80fc590 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.636+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f40a0012400 con 0x7f40b80fc590 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.637+0000 7f40aeffd700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 0x7f40a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.637+0000 7f40b659c700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 0x7f40a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.637+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f40a008bac0 con 0x7f40b80fc590 2024-03-14T13:12:14.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.637+0000 7f40b659c700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 0x7f40a406ec70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40a8004f90 tx=0x7f40a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.641+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f40a00597d0 con 0x7f40b80fc590 2024-03-14T13:12:14.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:14 smithi137 ceph-mon[108145]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:14.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.875+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f40b810c020 con 0x7f40b80fc590 2024-03-14T13:12:14.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.876+0000 7f40aeffd700 1 -- 172.21.15.111:0/3949179288 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f40a0059360 con 0x7f40b80fc590 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:12:14.877 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:12:14.878 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 msgr2=0x7f40a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 0x7f40a406ec70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40a8004f90 tx=0x7f40a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 msgr2=0x7f40b8116270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f40a000c8c0 tx=0x7f40a00044c0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 shutdown_connections 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40a406c7c0 0x7f40a406ec70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40b80fbc80 0x7f40b8115d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 --2- 172.21.15.111:0/3949179288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40b80fc590 0x7f40b8116270 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 >> 172.21.15.111:0/3949179288 conn(0x7f40b8072bc0 msgr2=0x7f40b8105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 shutdown_connections 2024-03-14T13:12:14.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.878+0000 7f40bcd03700 1 -- 172.21.15.111:0/3949179288 wait complete. 2024-03-14T13:12:14.881 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:12:14.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:14 smithi111 ceph-mon[145314]: from='client.34512 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:14.979 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/1759482419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 msgr2=0x7f4f180fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/1759482419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f180fbd30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c004940 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/1759482419 shutdown_connections 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/1759482419 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f180fbd30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/1759482419 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f181020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/1759482419 >> 172.21.15.111:0/1759482419 conn(0x7f4f180f78d0 msgr2=0x7f4f180f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/1759482419 shutdown_connections 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/1759482419 wait complete. 2024-03-14T13:12:14.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.979+0000 7f4f1e7ca700 1 Processor -- start 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 -- start start 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f1810c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1810d310 con 0x7f4f181026c0 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1810d480 con 0x7f4f18101d20 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50878/0 (socket says 172.21.15.111:50878) 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 -- 172.21.15.111:0/3112363893 learned_addr learned my addr 172.21.15.111:0/3112363893 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f17fff700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f1810c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 -- 172.21.15.111:0/3112363893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 msgr2=0x7f4f1810c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f1810c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 -- 172.21.15.111:0/3112363893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c005040 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f17fff700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f1810c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f177fe700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0093b0 tx=0x7f4f0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0c00e040 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.980+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1810d700 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.981+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f0c013420 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.981+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f0c01c630 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.981+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1811a280 con 0x7f4f181026c0 2024-03-14T13:12:14.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.981+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f180fd470 con 0x7f4f181026c0 2024-03-14T13:12:14.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.982+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4f0c0139e0 con 0x7f4f181026c0 2024-03-14T13:12:14.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.982+0000 7f4f14ff9700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 0x7f4f0006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:14.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.982+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4f0c08ca40 con 0x7f4f181026c0 2024-03-14T13:12:14.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.983+0000 7f4f17fff700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 0x7f4f0006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:14.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.983+0000 7f4f17fff700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 0x7f4f0006ec70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f08007fd0 tx=0x7f4f08009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:14.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:14.987+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4f0c05a800 con 0x7f4f181026c0 2024-03-14T13:12:15.198 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.197+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f1810d700 con 0x7f4f0006c7c0 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.198+0000 7f4f14ff9700 1 -- 172.21.15.111:0/3112363893 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4f1810d700 con 0x7f4f0006c7c0 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:12:15.199 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:12:15.200 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:12:15.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.200+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 msgr2=0x7f4f0006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 0x7f4f0006ec70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4f08007fd0 tx=0x7f4f08009040 comp rx=0 tx=0).stop 2024-03-14T13:12:15.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 msgr2=0x7f4f1810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0093b0 tx=0x7f4f0c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:15.201 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 shutdown_connections 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4f0006c7c0 0x7f4f0006ec70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f18101d20 0x7f4f1810c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 --2- 172.21.15.111:0/3112363893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f181026c0 0x7f4f1810cdd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 >> 172.21.15.111:0/3112363893 conn(0x7f4f180f78d0 msgr2=0x7f4f180f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 shutdown_connections 2024-03-14T13:12:15.202 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.201+0000 7f4f1e7ca700 1 -- 172.21.15.111:0/3112363893 wait complete. 2024-03-14T13:12:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.301+0000 7f200d24e700 1 -- 172.21.15.111:0/94273053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 msgr2=0x7f20080fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.301+0000 7f200d24e700 1 --2- 172.21.15.111:0/94273053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fe260 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc004940 tx=0x7f1ffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- 172.21.15.111:0/94273053 shutdown_connections 2024-03-14T13:12:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 --2- 172.21.15.111:0/94273053 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 0x7f2008106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 --2- 172.21.15.111:0/94273053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fe260 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- 172.21.15.111:0/94273053 >> 172.21.15.111:0/94273053 conn(0x7f2008072bc0 msgr2=0x7f2008072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- 172.21.15.111:0/94273053 shutdown_connections 2024-03-14T13:12:15.303 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- 172.21.15.111:0/94273053 wait complete. 2024-03-14T13:12:15.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 Processor -- start 2024-03-14T13:12:15.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- start start 2024-03-14T13:12:15.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:15.304 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 0x7f20080fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080fc820 con 0x7f20080fde90 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.302+0000 7f200d24e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080fc960 con 0x7f20080fe830 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50890/0 (socket says 172.21.15.111:50890) 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 -- 172.21.15.111:0/16964990 learned_addr learned my addr 172.21.15.111:0/16964990 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f200659c700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 0x7f20080fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 -- 172.21.15.111:0/16964990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 msgr2=0x7f20080fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 0x7f20080fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 -- 172.21.15.111:0/16964990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc005040 con 0x7f20080fde90 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f2006d9d700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc000c00 tx=0x7f1ffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ffc021070 con 0x7f20080fde90 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20080fcbb0 con 0x7f20080fde90 2024-03-14T13:12:15.305 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ffc013e00 con 0x7f20080fde90 2024-03-14T13:12:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ffc00e070 con 0x7f20080fde90 2024-03-14T13:12:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.303+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200811a120 con 0x7f20080fde90 2024-03-14T13:12:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.304+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008065e30 con 0x7f20080fde90 2024-03-14T13:12:15.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.305+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1ffc013950 con 0x7f20080fde90 2024-03-14T13:12:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.305+0000 7f1fef7fe700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 0x7f1ff006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.305+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ffc08ca60 con 0x7f20080fde90 2024-03-14T13:12:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.305+0000 7f200659c700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 0x7f1ff006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:15.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.306+0000 7f200659c700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 0x7f1ff006ec80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20080fd690 tx=0x7f1ff800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:15.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.310+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1ffc056ce0 con 0x7f20080fde90 2024-03-14T13:12:15.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.584+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f200805f5f0 con 0x7f20080fde90 2024-03-14T13:12:15.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.584+0000 7f1fef7fe700 1 -- 172.21.15.111:0/16964990 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1ffc05a300 con 0x7f20080fde90 2024-03-14T13:12:15.585 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:12:15.585 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:12:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:12:15.585 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:12:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 msgr2=0x7f1ff006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 0x7f1ff006ec80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f20080fd690 tx=0x7f1ff800b040 comp rx=0 tx=0).stop 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 msgr2=0x7f20080fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc000c00 tx=0x7f1ffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 shutdown_connections 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ff006c7d0 0x7f1ff006ec80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20080fde90 0x7f20080fbd50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 --2- 172.21.15.111:0/16964990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20080fe830 0x7f20080fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 >> 172.21.15.111:0/16964990 conn(0x7f2008072bc0 msgr2=0x7f20081037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.586+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 shutdown_connections 2024-03-14T13:12:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:15.587+0000 7f200d24e700 1 -- 172.21.15.111:0/16964990 wait complete. 2024-03-14T13:12:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:15 smithi137 ceph-mon[108145]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:15 smithi137 ceph-mon[108145]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:15 smithi137 ceph-mon[108145]: pgmap v244: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:15 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/183575507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:12:15.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:15 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3949179288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:12:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:15 smithi111 ceph-mon[145314]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:15 smithi111 ceph-mon[145314]: from='client.34520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:15 smithi111 ceph-mon[145314]: pgmap v244: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:15 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/183575507' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:12:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:15 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3949179288' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:12:16.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:16 smithi137 ceph-mon[108145]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:16.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:16 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/16964990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:12:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:16 smithi111 ceph-mon[145314]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:16 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/16964990' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:12:17.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:17 smithi137 ceph-mon[108145]: pgmap v245: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:17 smithi111 ceph-mon[145314]: pgmap v245: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:19 smithi137 ceph-mon[108145]: pgmap v246: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:19 smithi111 ceph-mon[145314]: pgmap v246: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:21.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:21 smithi137 ceph-mon[108145]: pgmap v247: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:21 smithi111 ceph-mon[145314]: pgmap v247: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:23.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:23 smithi137 ceph-mon[108145]: pgmap v248: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:23 smithi111 ceph-mon[145314]: pgmap v248: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:25 smithi111 ceph-mon[145314]: pgmap v249: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:25 smithi137 ceph-mon[108145]: pgmap v249: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:27 smithi111 ceph-mon[145314]: pgmap v250: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:27 smithi137 ceph-mon[108145]: pgmap v250: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:29 smithi111 ceph-mon[145314]: pgmap v251: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:29 smithi137 ceph-mon[108145]: pgmap v251: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:31 smithi111 ceph-mon[145314]: pgmap v252: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:31 smithi137 ceph-mon[108145]: pgmap v252: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:33 smithi111 ceph-mon[145314]: pgmap v253: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:33 smithi137 ceph-mon[108145]: pgmap v253: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:35 smithi111 ceph-mon[145314]: pgmap v254: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:35 smithi137 ceph-mon[108145]: pgmap v254: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:37 smithi111 ceph-mon[145314]: pgmap v255: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:37 smithi137 ceph-mon[108145]: pgmap v255: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:38.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:38 smithi111 ceph-mon[145314]: pgmap v256: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:39.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:38 smithi137 ceph-mon[108145]: pgmap v256: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:40.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:40 smithi111 ceph-mon[145314]: pgmap v257: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:40 smithi137 ceph-mon[108145]: pgmap v257: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:43.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:43 smithi111 ceph-mon[145314]: pgmap v258: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:43 smithi137 ceph-mon[108145]: pgmap v258: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:45.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 -- 172.21.15.111:0/3623525242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 msgr2=0x7f242c1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 --2- 172.21.15.111:0/3623525242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c1020d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f241c004940 tx=0x7f241c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 -- 172.21.15.111:0/3623525242 shutdown_connections 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 --2- 172.21.15.111:0/3623525242 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 0x7f242c0fbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 --2- 172.21.15.111:0/3623525242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c1020d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 -- 172.21.15.111:0/3623525242 >> 172.21.15.111:0/3623525242 conn(0x7f242c0f78d0 msgr2=0x7f242c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 -- 172.21.15.111:0/3623525242 shutdown_connections 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.701+0000 7f2433ad8700 1 -- 172.21.15.111:0/3623525242 wait complete. 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 Processor -- start 2024-03-14T13:12:45.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 -- start start 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 0x7f242c10efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10f600 con 0x7f242c101d00 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2433ad8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10f740 con 0x7f242c1026a0 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36096/0 (socket says 172.21.15.111:36096) 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 -- 172.21.15.111:0/2457217944 learned_addr learned my addr 172.21.15.111:0/2457217944 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431073700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 0x7f242c10efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 -- 172.21.15.111:0/2457217944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 msgr2=0x7f242c10efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 0x7f242c10efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 -- 172.21.15.111:0/2457217944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c005040 con 0x7f242c101d00 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.702+0000 7f2431874700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b3b0 tx=0x7f241c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:45.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.703+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c00e040 con 0x7f242c101d00 2024-03-14T13:12:45.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.703+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c117ae0 con 0x7f242c101d00 2024-03-14T13:12:45.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.703+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f241c013420 con 0x7f242c101d00 2024-03-14T13:12:45.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.703+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c01c630 con 0x7f242c101d00 2024-03-14T13:12:45.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.703+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c117ff0 con 0x7f242c101d00 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.704+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c0fd450 con 0x7f242c101d00 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.704+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f241c013970 con 0x7f242c101d00 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.705+0000 7f24227fc700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 0x7f241806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.705+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f241c08cd20 con 0x7f242c101d00 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.705+0000 7f2431073700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 0x7f241806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:45.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.705+0000 7f2431073700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 0x7f241806ec70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2428007fd0 tx=0x7f2428009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:45.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.709+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f241c091070 con 0x7f242c101d00 2024-03-14T13:12:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:45 smithi137 ceph-mon[108145]: pgmap v259: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:45 smithi111 ceph-mon[145314]: pgmap v259: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:45.921 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.922+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f242c04bed0 con 0x7f241806c7c0 2024-03-14T13:12:45.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.923+0000 7f24227fc700 1 -- 172.21.15.111:0/2457217944 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f242c04bed0 con 0x7f241806c7c0 2024-03-14T13:12:45.924 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 msgr2=0x7f241806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 0x7f241806ec70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2428007fd0 tx=0x7f2428009040 comp rx=0 tx=0).stop 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 msgr2=0x7f242c10ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b3b0 tx=0x7f241c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 shutdown_connections 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f241806c7c0 0x7f241806ec70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f242c101d00 0x7f242c10ea70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 --2- 172.21.15.111:0/2457217944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f242c1026a0 0x7f242c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 >> 172.21.15.111:0/2457217944 conn(0x7f242c0f78d0 msgr2=0x7f242c0f89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 shutdown_connections 2024-03-14T13:12:45.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:45.925+0000 7f2433ad8700 1 -- 172.21.15.111:0/2457217944 wait complete. 2024-03-14T13:12:45.935 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:12:46.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.026+0000 7f7534ab9700 1 -- 172.21.15.111:0/2506705889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 msgr2=0x7f75300fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.026+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2506705889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f75300fce20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7524004940 tx=0x7f7524009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 -- 172.21.15.111:0/2506705889 shutdown_connections 2024-03-14T13:12:46.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2506705889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f75300fce20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2506705889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 0x7f75301031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 -- 172.21.15.111:0/2506705889 >> 172.21.15.111:0/2506705889 conn(0x7f75300f8980 msgr2=0x7f75300fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 -- 172.21.15.111:0/2506705889 shutdown_connections 2024-03-14T13:12:46.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 -- 172.21.15.111:0/2506705889 wait complete. 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 Processor -- start 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.027+0000 7f7534ab9700 1 -- start start 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f7534ab9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 0x7f75300702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f7534ab9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f7534ab9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530070d90 con 0x7f75301037b0 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f7534ab9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530070ed0 con 0x7f7530102e10 2024-03-14T13:12:46.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36114/0 (socket says 172.21.15.111:36114) 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 -- 172.21.15.111:0/2995356485 learned_addr learned my addr 172.21.15.111:0/2995356485 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752e59c700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 0x7f75300702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 -- 172.21.15.111:0/2995356485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 msgr2=0x7f75300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 0x7f75300702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 -- 172.21.15.111:0/2995356485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7524005040 con 0x7f75301037b0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f752dd9b700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7524009390 tx=0x7f7524009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752400e040 con 0x7f75301037b0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7524013420 con 0x7f75301037b0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.028+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7530071120 con 0x7f75301037b0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.029+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f752401c630 con 0x7f75301037b0 2024-03-14T13:12:46.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.029+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753011a1f0 con 0x7f75301037b0 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.029+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75300fe560 con 0x7f75301037b0 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.030+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7524013a00 con 0x7f75301037b0 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.030+0000 7f751effd700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 0x7f751806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.031+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f752408cb20 con 0x7f75301037b0 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.031+0000 7f752e59c700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 0x7f751806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.031+0000 7f752e59c700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 0x7f751806ec80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7520004f40 tx=0x7f752000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.034 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.035+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7524056da0 con 0x7f75301037b0 2024-03-14T13:12:46.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.245+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7530109e50 con 0x7f751806c7d0 2024-03-14T13:12:46.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.246+0000 7f751effd700 1 -- 172.21.15.111:0/2995356485 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f752000f040 con 0x7f751806c7d0 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 msgr2=0x7f751806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 0x7f751806ec80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7520004f40 tx=0x7f752000b040 comp rx=0 tx=0).stop 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 msgr2=0x7f7530070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7524009390 tx=0x7f7524009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 shutdown_connections 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f751806c7d0 0x7f751806ec80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7530102e10 0x7f75300702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 --2- 172.21.15.111:0/2995356485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75301037b0 0x7f7530070800 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 >> 172.21.15.111:0/2995356485 conn(0x7f75300f8980 msgr2=0x7f75300f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 shutdown_connections 2024-03-14T13:12:46.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.248+0000 7f7534ab9700 1 -- 172.21.15.111:0/2995356485 wait complete. 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- 172.21.15.111:0/3089191725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f41037b0 msgr2=0x7f18f40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 --2- 172.21.15.111:0/3089191725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f41037b0 0x7f18f40fce20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f18e0004940 tx=0x7f18e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- 172.21.15.111:0/3089191725 shutdown_connections 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 --2- 172.21.15.111:0/3089191725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f41037b0 0x7f18f40fce20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 --2- 172.21.15.111:0/3089191725 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f4102e10 0x7f18f41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- 172.21.15.111:0/3089191725 >> 172.21.15.111:0/3089191725 conn(0x7f18f40f8980 msgr2=0x7f18f40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- 172.21.15.111:0/3089191725 shutdown_connections 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- 172.21.15.111:0/3089191725 wait complete. 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 Processor -- start 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 -- start start 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.350+0000 7f18faa66700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18faa66700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 0x7f18f4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18faa66700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4114680 con 0x7f18f4102e10 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18faa66700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f410da70 con 0x7f18f41037b0 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36130/0 (socket says 172.21.15.111:36130) 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 -- 172.21.15.111:0/1435451512 learned_addr learned my addr 172.21.15.111:0/1435451512 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f3fff700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 0x7f18f4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 -- 172.21.15.111:0/1435451512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 msgr2=0x7f18f4114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 0x7f18f4114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 -- 172.21.15.111:0/1435451512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0005040 con 0x7f18f4102e10 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f3fff700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 0x7f18f4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f8802700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18e8002a50 tx=0x7f18e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e8007cc0 con 0x7f18f4102e10 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18e800f040 con 0x7f18f4102e10 2024-03-14T13:12:46.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f410dd50 con 0x7f18f4102e10 2024-03-14T13:12:46.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18e800da70 con 0x7f18f4102e10 2024-03-14T13:12:46.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.351+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f410e310 con 0x7f18f4102e10 2024-03-14T13:12:46.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.352+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f40fe560 con 0x7f18f4102e10 2024-03-14T13:12:46.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.353+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f18e800dc10 con 0x7f18f4102e10 2024-03-14T13:12:46.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.353+0000 7f18f17fa700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 0x7f18e406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.353+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f18e808b200 con 0x7f18f4102e10 2024-03-14T13:12:46.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.353+0000 7f18f3fff700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 0x7f18e406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.354+0000 7f18f3fff700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 0x7f18e406ec80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b3b0 tx=0x7f18e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.358+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f18e8058e90 con 0x7f18f4102e10 2024-03-14T13:12:46.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.558+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f18f4109dd0 con 0x7f18e406c7d0 2024-03-14T13:12:46.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.568+0000 7f18f17fa700 1 -- 172.21.15.111:0/1435451512 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f18f4109dd0 con 0x7f18e406c7d0 2024-03-14T13:12:46.567 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:12:46.567 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (8m) 2m ago 20m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:12:46.567 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (6m) 2m ago 20m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:12:46.567 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (6m) 118s ago 19m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (8m) 2m ago 19m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (2m) 2m ago 14m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (2m) 2m ago 14m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (2m) 118s ago 14m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (2m) 118s ago 14m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (11m) 2m ago 21m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (10m) 118s ago 19m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (7m) 2m ago 22m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (7m) 118s ago 19m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (9m) 2m ago 20m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (9m) 118s ago 19m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 2m ago 18m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 2m ago 17m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (4m) 2m ago 17m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (4m) 118s ago 16m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:12:46.568 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (3m) 118s ago 16m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:12:46.569 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (3m) 118s ago 16m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:12:46.569 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (9m) 2m ago 19m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:12:46.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 msgr2=0x7f18e406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 0x7f18e406ec80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f18e000b3b0 tx=0x7f18e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 msgr2=0x7f18f4113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f18e8002a50 tx=0x7f18e80046b0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 shutdown_connections 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f18e406c7d0 0x7f18e406ec80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f18f4102e10 0x7f18f4113af0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 --2- 172.21.15.111:0/1435451512 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f18f41037b0 0x7f18f4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 >> 172.21.15.111:0/1435451512 conn(0x7f18f40f8980 msgr2=0x7f18f40f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 shutdown_connections 2024-03-14T13:12:46.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.570+0000 7f18faa66700 1 -- 172.21.15.111:0/1435451512 wait complete. 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 -- 172.21.15.111:0/4281065327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 msgr2=0x7f4d3c106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 --2- 172.21.15.111:0/4281065327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c106f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4d30004940 tx=0x7f4d30009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 -- 172.21.15.111:0/4281065327 shutdown_connections 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 --2- 172.21.15.111:0/4281065327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c106f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 --2- 172.21.15.111:0/4281065327 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 0x7f4d3c0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 -- 172.21.15.111:0/4281065327 >> 172.21.15.111:0/4281065327 conn(0x7f4d3c072bc0 msgr2=0x7f4d3c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 -- 172.21.15.111:0/4281065327 shutdown_connections 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 -- 172.21.15.111:0/4281065327 wait complete. 2024-03-14T13:12:46.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.669+0000 7f4d422a3700 1 Processor -- start 2024-03-14T13:12:46.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d422a3700 1 -- start start 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d422a3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 0x7f4d3c113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d422a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d422a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c114670 con 0x7f4d3c0fe830 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d422a3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3c10da60 con 0x7f4d3c0fde90 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36156/0 (socket says 172.21.15.111:36156) 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 -- 172.21.15.111:0/3127988237 learned_addr learned my addr 172.21.15.111:0/3127988237 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3bfff700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 0x7f4d3c113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 -- 172.21.15.111:0/3127988237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 msgr2=0x7f4d3c113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 0x7f4d3c113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 -- 172.21.15.111:0/3127988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d30005040 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.670+0000 7f4d3b7fe700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000b3b0 tx=0x7f4d30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3000e040 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3c10dce0 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d30013420 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d3001c5f0 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3c10e220 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.671+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3c065e30 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.672+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4d30013590 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.672+0000 7f4d38ff9700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 0x7f4d2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.672+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4d3008cd70 con 0x7f4d3c0fe830 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.673+0000 7f4d3bfff700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 0x7f4d2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:46.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.673+0000 7f4d3bfff700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 0x7f4d2406ec70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c007fd0 tx=0x7f4d2c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:46.676 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.677+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4d3005aa80 con 0x7f4d3c0fe830 2024-03-14T13:12:46.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.947+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4d3c109dd0 con 0x7f4d3c0fe830 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.947+0000 7f4d38ff9700 1 -- 172.21.15.111:0/3127988237 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f4d3005a610 con 0x7f4d3c0fe830 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:46.947 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:12:46.948 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 msgr2=0x7f4d2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 0x7f4d2406ec70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c007fd0 tx=0x7f4d2c009040 comp rx=0 tx=0).stop 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 msgr2=0x7f4d3c114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d3000b3b0 tx=0x7f4d30009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 shutdown_connections 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d2406c7c0 0x7f4d2406ec70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d3c0fde90 0x7f4d3c113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 --2- 172.21.15.111:0/3127988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d3c0fe830 0x7f4d3c114020 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 >> 172.21.15.111:0/3127988237 conn(0x7f4d3c072bc0 msgr2=0x7f4d3c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 shutdown_connections 2024-03-14T13:12:46.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:46.949+0000 7f4d422a3700 1 -- 172.21.15.111:0/3127988237 wait complete. 2024-03-14T13:12:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- 172.21.15.111:0/1347988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9688102e00 msgr2=0x7f96881031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 --2- 172.21.15.111:0/1347988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9688102e00 0x7f96881031d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9678004880 tx=0x7f9678009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- 172.21.15.111:0/1347988237 shutdown_connections 2024-03-14T13:12:47.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 --2- 172.21.15.111:0/1347988237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96881037a0 0x7f96880fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 --2- 172.21.15.111:0/1347988237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9688102e00 0x7f96881031d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- 172.21.15.111:0/1347988237 >> 172.21.15.111:0/1347988237 conn(0x7f96880f89b0 msgr2=0x7f96880fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- 172.21.15.111:0/1347988237 shutdown_connections 2024-03-14T13:12:47.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- 172.21.15.111:0/1347988237 wait complete. 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 Processor -- start 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.050+0000 7f968f665700 1 -- start start 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968f665700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9688102e00 0x7f9688113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968f665700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968f665700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96881145f0 con 0x7f96881037a0 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968f665700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968810d9e0 con 0x7f9688102e00 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36172/0 (socket says 172.21.15.111:36172) 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 -- 172.21.15.111:0/3435690119 learned_addr learned my addr 172.21.15.111:0/3435690119 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968d401700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9688102e00 0x7f9688113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 -- 172.21.15.111:0/3435690119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9688102e00 msgr2=0x7f9688113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9688102e00 0x7f9688113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 -- 172.21.15.111:0/3435690119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678005040 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968cc00700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f967c002f40 tx=0x7f967c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967c007cc0 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968810dc60 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f967c00f040 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.051+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f967c009600 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.052+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968810e220 con 0x7f96881037a0 2024-03-14T13:12:47.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.053+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96880fe550 con 0x7f96881037a0 2024-03-14T13:12:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.053+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f967c009760 con 0x7f96881037a0 2024-03-14T13:12:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.053+0000 7f9685ffb700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 0x7f967006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.053+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f967c08b6b0 con 0x7f96881037a0 2024-03-14T13:12:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.054+0000 7f968d401700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 0x7f967006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.054+0000 7f968d401700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 0x7f967006ec70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9678009fd0 tx=0x7f9678009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.058+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f967c0593c0 con 0x7f96881037a0 2024-03-14T13:12:47.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.297+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9688109de0 con 0x7f96881037a0 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.297+0000 7f9685ffb700 1 -- 172.21.15.111:0/3435690119 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f967c058f50 con 0x7f96881037a0 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:12:47.297 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:12:47.298 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:12:47.299 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:12:47.300 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:47.300 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:47.300 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:47.300 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 msgr2=0x7f967006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 0x7f967006ec70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9678009fd0 tx=0x7f9678009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 msgr2=0x7f9688113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f967c002f40 tx=0x7f967c00b040 comp rx=0 tx=0).stop 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 shutdown_connections 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f967006c7c0 0x7f967006ec70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9688102e00 0x7f9688113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 --2- 172.21.15.111:0/3435690119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96881037a0 0x7f9688113fa0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.299+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 >> 172.21.15.111:0/3435690119 conn(0x7f96880f89b0 msgr2=0x7f96880f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.300+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 shutdown_connections 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.300+0000 7f968f665700 1 -- 172.21.15.111:0/3435690119 wait complete. 2024-03-14T13:12:47.301 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- 172.21.15.111:0/2706533434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c102e70 msgr2=0x7f338c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 --2- 172.21.15.111:0/2706533434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c102e70 0x7f338c103240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f337c004940 tx=0x7f337c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- 172.21.15.111:0/2706533434 shutdown_connections 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 --2- 172.21.15.111:0/2706533434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 --2- 172.21.15.111:0/2706533434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c102e70 0x7f338c103240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- 172.21.15.111:0/2706533434 >> 172.21.15.111:0/2706533434 conn(0x7f338c0714c0 msgr2=0x7f338c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:47.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- 172.21.15.111:0/2706533434 shutdown_connections 2024-03-14T13:12:47.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- 172.21.15.111:0/2706533434 wait complete. 2024-03-14T13:12:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 Processor -- start 2024-03-14T13:12:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- start start 2024-03-14T13:12:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c102e70 0x7f338c0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.405 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c0fd150 con 0x7f338c0fb4a0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.404+0000 7f339422a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c0fd290 con 0x7f338c102e70 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36188/0 (socket says 172.21.15.111:36188) 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 -- 172.21.15.111:0/204268360 learned_addr learned my addr 172.21.15.111:0/204268360 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f33917c5700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c102e70 0x7f338c0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 -- 172.21.15.111:0/204268360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c102e70 msgr2=0x7f338c0fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c102e70 0x7f338c0fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 -- 172.21.15.111:0/204268360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c005040 con 0x7f338c0fb4a0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f3391fc6700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f337c009360 tx=0x7f337c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337c00e040 con 0x7f338c0fb4a0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f337c013420 con 0x7f338c0fb4a0 2024-03-14T13:12:47.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c0fd4e0 con 0x7f338c0fb4a0 2024-03-14T13:12:47.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f337c01c5f0 con 0x7f338c0fb4a0 2024-03-14T13:12:47.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.405+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c11a1a0 con 0x7f338c0fb4a0 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.406+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c100110 con 0x7f338c0fb4a0 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.407+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f337c013590 con 0x7f338c0fb4a0 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.407+0000 7f33827fc700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 0x7f337806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.407+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f337c08cd00 con 0x7f338c0fb4a0 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.407+0000 7f33917c5700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 0x7f337806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.408+0000 7f33917c5700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 0x7f337806ec80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3388007fd0 tx=0x7f3388009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.412+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f337c056f80 con 0x7f338c0fb4a0 2024-03-14T13:12:47.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.622+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f338c04bed0 con 0x7f337806c7d0 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.624+0000 7f33827fc700 1 -- 172.21.15.111:0/204268360 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f338c04bed0 con 0x7f337806c7d0 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:12:47.623 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:12:47.624 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 msgr2=0x7f337806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 0x7f337806ec80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3388007fd0 tx=0x7f3388009040 comp rx=0 tx=0).stop 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 msgr2=0x7f338c0fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f337c009360 tx=0x7f337c009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 shutdown_connections 2024-03-14T13:12:47.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f337806c7d0 0x7f337806ec80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f338c0fb4a0 0x7f338c0fc680 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 --2- 172.21.15.111:0/204268360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f338c102e70 0x7f338c0fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 >> 172.21.15.111:0/204268360 conn(0x7f338c0714c0 msgr2=0x7f338c0fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:47.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 shutdown_connections 2024-03-14T13:12:47.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.626+0000 7f339422a700 1 -- 172.21.15.111:0/204268360 wait complete. 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- 172.21.15.111:0/1703659433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 msgr2=0x7f94500fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 --2- 172.21.15.111:0/1703659433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fb820 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9444004940 tx=0x7f9444009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- 172.21.15.111:0/1703659433 shutdown_connections 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 --2- 172.21.15.111:0/1703659433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fb820 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 --2- 172.21.15.111:0/1703659433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 0x7f9450103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- 172.21.15.111:0/1703659433 >> 172.21.15.111:0/1703659433 conn(0x7f9450072310 msgr2=0x7f9450072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:47.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- 172.21.15.111:0/1703659433 shutdown_connections 2024-03-14T13:12:47.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- 172.21.15.111:0/1703659433 wait complete. 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 Processor -- start 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.726+0000 7f9455728700 1 -- start start 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 0x7f94500fcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94500fe4e0 con 0x7f94500fb3b0 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94500fd010 con 0x7f9450102f70 2024-03-14T13:12:47.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:36206/0 (socket says 172.21.15.111:36206) 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 -- 172.21.15.111:0/438297364 learned_addr learned my addr 172.21.15.111:0/438297364 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944e7fc700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 0x7f94500fcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 -- 172.21.15.111:0/438297364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 msgr2=0x7f94500fcad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 0x7f94500fcad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 -- 172.21.15.111:0/438297364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444005040 con 0x7f94500fb3b0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f944effd700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f944000c830 tx=0x7f9440004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94400106f0 con 0x7f94500fb3b0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94500fd2f0 con 0x7f94500fb3b0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9440010d00 con 0x7f94500fb3b0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9440012e90 con 0x7f94500fb3b0 2024-03-14T13:12:47.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.727+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945011a010 con 0x7f94500fb3b0 2024-03-14T13:12:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.728+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94501053b0 con 0x7f94500fb3b0 2024-03-14T13:12:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.729+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9440010850 con 0x7f94500fb3b0 2024-03-14T13:12:47.729 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.730+0000 7f94377fe700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 0x7f943806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:12:47.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.730+0000 7f944e7fc700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 0x7f943806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:12:47.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.730+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f944008c580 con 0x7f94500fb3b0 2024-03-14T13:12:47.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.730+0000 7f944e7fc700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 0x7f943806ec70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9444004eb0 tx=0x7f9444009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:12:47.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:47.734+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9440056800 con 0x7f94500fb3b0 2024-03-14T13:12:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:47 smithi137 ceph-mon[108145]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:47 smithi137 ceph-mon[108145]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:47 smithi137 ceph-mon[108145]: pgmap v260: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:47 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3127988237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:12:47.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:47 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3435690119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:12:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:47 smithi111 ceph-mon[145314]: from='client.34540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:47 smithi111 ceph-mon[145314]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:47 smithi111 ceph-mon[145314]: pgmap v260: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:47 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3127988237' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:12:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:47 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3435690119' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:12:48.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.002+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f94500fe6d0 con 0x7f94500fb3b0 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.002+0000 7f94377fe700 1 -- 172.21.15.111:0/438297364 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f9440059e20 con 0x7f94500fb3b0 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:12:48.002 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 msgr2=0x7f943806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 0x7f943806ec70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9444004eb0 tx=0x7f9444009e30 comp rx=0 tx=0).stop 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 msgr2=0x7f94500fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f944000c830 tx=0x7f9440004470 comp rx=0 tx=0).stop 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 shutdown_connections 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f943806c7c0 0x7f943806ec70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94500fb3b0 0x7f94500fc590 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 --2- 172.21.15.111:0/438297364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9450102f70 0x7f94500fcad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 >> 172.21.15.111:0/438297364 conn(0x7f9450072310 msgr2=0x7f94500f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 shutdown_connections 2024-03-14T13:12:48.006 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:12:48.004+0000 7f9455728700 1 -- 172.21.15.111:0/438297364 wait complete. 2024-03-14T13:12:48.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:48 smithi137 ceph-mon[108145]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:48.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:48 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/438297364' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:12:48.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:48 smithi111 ceph-mon[145314]: from='client.34548 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:48.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:48 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/438297364' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:12:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:49 smithi137 ceph-mon[108145]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:49 smithi137 ceph-mon[108145]: pgmap v261: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:49 smithi111 ceph-mon[145314]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:12:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:49 smithi111 ceph-mon[145314]: pgmap v261: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:51 smithi137 ceph-mon[108145]: pgmap v262: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:51 smithi111 ceph-mon[145314]: pgmap v262: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:53 smithi137 ceph-mon[108145]: pgmap v263: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:53 smithi111 ceph-mon[145314]: pgmap v263: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:12:55.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:55 smithi137 ceph-mon[108145]: pgmap v264: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:55 smithi111 ceph-mon[145314]: pgmap v264: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:57.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:57 smithi137 ceph-mon[108145]: pgmap v265: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:57 smithi111 ceph-mon[145314]: pgmap v265: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:12:59 smithi137 ceph-mon[108145]: pgmap v266: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:12:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:12:59 smithi111 ceph-mon[145314]: pgmap v266: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:01.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:01 smithi137 ceph-mon[108145]: pgmap v267: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:01 smithi111 ceph-mon[145314]: pgmap v267: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:03.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:03 smithi137 ceph-mon[108145]: pgmap v268: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:03 smithi111 ceph-mon[145314]: pgmap v268: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:05.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:05 smithi137 ceph-mon[108145]: pgmap v269: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:05 smithi111 ceph-mon[145314]: pgmap v269: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:07.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:07 smithi111 ceph-mon[145314]: pgmap v270: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:07 smithi137 ceph-mon[108145]: pgmap v270: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:09 smithi111 ceph-mon[145314]: pgmap v271: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:09 smithi137 ceph-mon[108145]: pgmap v271: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:11 smithi111 ceph-mon[145314]: pgmap v272: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:11 smithi137 ceph-mon[108145]: pgmap v272: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:13 smithi111 ceph-mon[145314]: pgmap v273: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:13 smithi137 ceph-mon[108145]: pgmap v273: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:15 smithi111 ceph-mon[145314]: pgmap v274: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:15 smithi137 ceph-mon[108145]: pgmap v274: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:17 smithi111 ceph-mon[145314]: pgmap v275: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:17 smithi137 ceph-mon[108145]: pgmap v275: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:18.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- 172.21.15.111:0/300142869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d41037b0 msgr2=0x7f05d40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 --2- 172.21.15.111:0/300142869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d41037b0 0x7f05d40fce20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05bc004940 tx=0x7f05bc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- 172.21.15.111:0/300142869 shutdown_connections 2024-03-14T13:13:18.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 --2- 172.21.15.111:0/300142869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d41037b0 0x7f05d40fce20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 --2- 172.21.15.111:0/300142869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d4102e10 0x7f05d41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- 172.21.15.111:0/300142869 >> 172.21.15.111:0/300142869 conn(0x7f05d40f89a0 msgr2=0x7f05d40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- 172.21.15.111:0/300142869 shutdown_connections 2024-03-14T13:13:18.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- 172.21.15.111:0/300142869 wait complete. 2024-03-14T13:13:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 Processor -- start 2024-03-14T13:13:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.118+0000 7f05da87d700 1 -- start start 2024-03-14T13:13:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05da87d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05da87d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d41037b0 0x7f05d4113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05da87d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d41145d0 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05da87d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d410d9c0 con 0x7f05d41037b0 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38892/0 (socket says 172.21.15.111:38892) 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 -- 172.21.15.111:0/2305131906 learned_addr learned my addr 172.21.15.111:0/2305131906 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05cbfff700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d41037b0 0x7f05d4113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 -- 172.21.15.111:0/2305131906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d41037b0 msgr2=0x7f05d4113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d41037b0 0x7f05d4113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 -- 172.21.15.111:0/2305131906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc005040 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.119+0000 7f05d3fff700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f05c400eb40 tx=0x7f05c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.120+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c4007cc0 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.120+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d410dc40 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.120+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05c400f040 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.120+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c400c990 con 0x7f05d4102e10 2024-03-14T13:13:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.120+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d410e1d0 con 0x7f05d4102e10 2024-03-14T13:13:18.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.121+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d40fe560 con 0x7f05d4102e10 2024-03-14T13:13:18.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.121+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f05c400cc00 con 0x7f05d4102e10 2024-03-14T13:13:18.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.122+0000 7f05d17fa700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 0x7f05c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.122+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f05c408b6d0 con 0x7f05d4102e10 2024-03-14T13:13:18.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.122+0000 7f05cbfff700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 0x7f05c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.122+0000 7f05cbfff700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 0x7f05c006ec70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00b0d0 tx=0x7f05bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.126+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f05c40593d0 con 0x7f05d4102e10 2024-03-14T13:13:18.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.336+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f05d4109e00 con 0x7f05c006c7c0 2024-03-14T13:13:18.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.337+0000 7f05d17fa700 1 -- 172.21.15.111:0/2305131906 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f05d4109e00 con 0x7f05c006c7c0 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 msgr2=0x7f05c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 0x7f05c006ec70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00b0d0 tx=0x7f05bc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 msgr2=0x7f05d4113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f05c400eb40 tx=0x7f05c400b040 comp rx=0 tx=0).stop 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 shutdown_connections 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f05c006c7c0 0x7f05c006ec70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d4102e10 0x7f05d4113a40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 --2- 172.21.15.111:0/2305131906 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05d41037b0 0x7f05d4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.339+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 >> 172.21.15.111:0/2305131906 conn(0x7f05d40f89a0 msgr2=0x7f05d40f99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.340+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 shutdown_connections 2024-03-14T13:13:18.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.340+0000 7f05da87d700 1 -- 172.21.15.111:0/2305131906 wait complete. 2024-03-14T13:13:18.349 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- 172.21.15.111:0/341218667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 msgr2=0x7f97ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 --2- 172.21.15.111:0/341218667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec0fbd80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f97d8004940 tx=0x7f97d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- 172.21.15.111:0/341218667 shutdown_connections 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 --2- 172.21.15.111:0/341218667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec0fbd80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 --2- 172.21.15.111:0/341218667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- 172.21.15.111:0/341218667 >> 172.21.15.111:0/341218667 conn(0x7f97ec072bc0 msgr2=0x7f97ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- 172.21.15.111:0/341218667 shutdown_connections 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- 172.21.15.111:0/341218667 wait complete. 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 Processor -- start 2024-03-14T13:13:18.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.441+0000 7f97f2d05700 1 -- start start 2024-03-14T13:13:18.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f2d05700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec1129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f2d05700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f2d05700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec113550 con 0x7f97ec102710 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f2d05700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec10c940 con 0x7f97ec101d70 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38908/0 (socket says 172.21.15.111:38908) 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 -- 172.21.15.111:0/84757193 learned_addr learned my addr 172.21.15.111:0/84757193 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f0aa1700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec1129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 -- 172.21.15.111:0/84757193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 msgr2=0x7f97ec1129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec1129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 -- 172.21.15.111:0/84757193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8005040 con 0x7f97ec102710 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97f0aa1700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec1129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97ebfff700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009380 tx=0x7f97d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.442+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d800e040 con 0x7f97ec102710 2024-03-14T13:13:18.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.443+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec10cbc0 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.443+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97d8013420 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.443+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97d801c630 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.443+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec10d180 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.444+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec0ff080 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.444+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f97d80139e0 con 0x7f97ec102710 2024-03-14T13:13:18.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.444+0000 7f97e97fa700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 0x7f97dc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.444+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f97d808cc60 con 0x7f97ec102710 2024-03-14T13:13:18.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.444+0000 7f97f0aa1700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 0x7f97dc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.445+0000 7f97f0aa1700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 0x7f97dc06ec70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f97e0007fd0 tx=0x7f97e0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.448 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.449+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f97d805a970 con 0x7f97ec102710 2024-03-14T13:13:18.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.659+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ec10e010 con 0x7f97dc06c7c0 2024-03-14T13:13:18.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.660+0000 7f97e97fa700 1 -- 172.21.15.111:0/84757193 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f97ec10e010 con 0x7f97dc06c7c0 2024-03-14T13:13:18.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 msgr2=0x7f97dc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 0x7f97dc06ec70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f97e0007fd0 tx=0x7f97e0009040 comp rx=0 tx=0).stop 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 msgr2=0x7f97ec112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009380 tx=0x7f97d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 shutdown_connections 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f97dc06c7c0 0x7f97dc06ec70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97ec101d70 0x7f97ec1129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 --2- 172.21.15.111:0/84757193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97ec102710 0x7f97ec112f00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 >> 172.21.15.111:0/84757193 conn(0x7f97ec072bc0 msgr2=0x7f97ec0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 shutdown_connections 2024-03-14T13:13:18.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.662+0000 7f97f2d05700 1 -- 172.21.15.111:0/84757193 wait complete. 2024-03-14T13:13:18.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.763+0000 7fd2daf8a700 1 -- 172.21.15.111:0/3789168093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4102660 msgr2=0x7fd2d40fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.763+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/3789168093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4102660 0x7fd2d40fc5b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0004940 tx=0x7fd2c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- 172.21.15.111:0/3789168093 shutdown_connections 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/3789168093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4102660 0x7fd2d40fc5b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/3789168093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- 172.21.15.111:0/3789168093 >> 172.21.15.111:0/3789168093 conn(0x7fd2d40714c0 msgr2=0x7fd2d40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- 172.21.15.111:0/3789168093 shutdown_connections 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- 172.21.15.111:0/3789168093 wait complete. 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 Processor -- start 2024-03-14T13:13:18.764 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- start start 2024-03-14T13:13:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 0x7fd2d4103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4103610 con 0x7fd2d4101d50 2024-03-14T13:13:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.764+0000 7fd2daf8a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4103750 con 0x7fd2d4102660 2024-03-14T13:13:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38936/0 (socket says 172.21.15.111:38936) 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 -- 172.21.15.111:0/1492803427 learned_addr learned my addr 172.21.15.111:0/1492803427 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d3fff700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 0x7fd2d4103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 -- 172.21.15.111:0/1492803427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 msgr2=0x7fd2d4103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 0x7fd2d4103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 -- 172.21.15.111:0/1492803427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0005040 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d3fff700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 0x7fd2d4103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d8d26700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c830 tx=0x7fd2c8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c800a760 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d4119e20 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2c800ad70 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2c80113a0 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.765+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d411a220 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.766+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d40fd5c0 con 0x7fd2d4101d50 2024-03-14T13:13:18.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.767+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd2c8011500 con 0x7fd2d4101d50 2024-03-14T13:13:18.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.767+0000 7fd2d17fa700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 0x7fd2c406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:18.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.767+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd2c808baf0 con 0x7fd2d4101d50 2024-03-14T13:13:18.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.767+0000 7fd2d3fff700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 0x7fd2c406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:18.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.767+0000 7fd2d3fff700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 0x7fd2c406ec70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0000c00 tx=0x7fd2c0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:18.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.771+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd2c8055de0 con 0x7fd2d4101d50 2024-03-14T13:13:18.972 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.972+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2d4104230 con 0x7fd2c406c7c0 2024-03-14T13:13:18.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.982+0000 7fd2d17fa700 1 -- 172.21.15.111:0/1492803427 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd2d4104230 con 0x7fd2c406c7c0 2024-03-14T13:13:18.981 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:13:18.981 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (9m) 3m ago 21m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (7m) 3m ago 21m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (6m) 2m ago 19m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (9m) 3m ago 20m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (3m) 3m ago 14m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (3m) 3m ago 14m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (2m) 2m ago 14m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (2m) 2m ago 14m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (12m) 3m ago 22m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (10m) 2m ago 19m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (8m) 3m ago 22m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (7m) 2m ago 19m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (10m) 3m ago 21m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:13:18.982 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (10m) 2m ago 19m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 3m ago 18m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 3m ago 18m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 3m ago 17m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (4m) 2m ago 17m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (4m) 2m ago 17m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (3m) 2m ago 16m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:13:18.983 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (9m) 3m ago 20m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 msgr2=0x7fd2c406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 0x7fd2c406ec70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0000c00 tx=0x7fd2c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 msgr2=0x7fd2d4102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd2c800c830 tx=0x7fd2c8004470 comp rx=0 tx=0).stop 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 shutdown_connections 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd2c406c7c0 0x7fd2c406ec70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2d4101d50 0x7fd2d4102b40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 --2- 172.21.15.111:0/1492803427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2d4102660 0x7fd2d4103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 >> 172.21.15.111:0/1492803427 conn(0x7fd2d40714c0 msgr2=0x7fd2d4105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 shutdown_connections 2024-03-14T13:13:18.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:18.984+0000 7fd2daf8a700 1 -- 172.21.15.111:0/1492803427 wait complete. 2024-03-14T13:13:19.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 -- 172.21.15.111:0/3027216925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 msgr2=0x7f13a01020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 --2- 172.21.15.111:0/3027216925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a01020e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1394004940 tx=0x7f1394009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:19.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 -- 172.21.15.111:0/3027216925 shutdown_connections 2024-03-14T13:13:19.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 --2- 172.21.15.111:0/3027216925 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 0x7f13a00fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 --2- 172.21.15.111:0/3027216925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a01020e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.082+0000 7f13a6728700 1 -- 172.21.15.111:0/3027216925 >> 172.21.15.111:0/3027216925 conn(0x7f13a00f7880 msgr2=0x7f13a00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 -- 172.21.15.111:0/3027216925 shutdown_connections 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 -- 172.21.15.111:0/3027216925 wait complete. 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 Processor -- start 2024-03-14T13:13:19.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 -- start start 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 0x7f13a01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0103640 con 0x7f13a0101d10 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.083+0000 7f13a6728700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0103780 con 0x7f13a01026b0 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38964/0 (socket says 172.21.15.111:38964) 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 -- 172.21.15.111:0/934693993 learned_addr learned my addr 172.21.15.111:0/934693993 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139f7fe700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 0x7f13a01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 -- 172.21.15.111:0/934693993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 msgr2=0x7f13a01030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 0x7f13a01030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 -- 172.21.15.111:0/934693993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1394005040 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139ffff700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1394009180 tx=0x7f1394009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139400e040 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a01039d0 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1394004290 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1394013e60 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.084+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a011a4f0 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.085+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a00fd460 con 0x7f13a0101d10 2024-03-14T13:13:19.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.086+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f13940136b0 con 0x7f13a0101d10 2024-03-14T13:13:19.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.086+0000 7f139cff9700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 0x7f138806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.086+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f139408ccd0 con 0x7f13a0101d10 2024-03-14T13:13:19.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.086+0000 7f139f7fe700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 0x7f138806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.087+0000 7f139f7fe700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 0x7f138806ec00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f13a01044b0 tx=0x7f13900043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.091+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1394057000 con 0x7f13a0101d10 2024-03-14T13:13:19.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.370+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f13a00fd460 con 0x7f13a0101d10 2024-03-14T13:13:19.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.370+0000 7f139cff9700 1 -- 172.21.15.111:0/934693993 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f139405a620 con 0x7f13a0101d10 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:13:19.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:19.371 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:13:19.371 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:13:19.371 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:13:19.371 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:13:19.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 msgr2=0x7f138806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 0x7f138806ec00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f13a01044b0 tx=0x7f13900043c0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 msgr2=0x7f13a0102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1394009180 tx=0x7f1394009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 shutdown_connections 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138806c750 0x7f138806ec00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a0101d10 0x7f13a0102b70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 --2- 172.21.15.111:0/934693993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a01026b0 0x7f13a01030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 >> 172.21.15.111:0/934693993 conn(0x7f13a00f7880 msgr2=0x7f13a00f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 shutdown_connections 2024-03-14T13:13:19.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.372+0000 7f13a6728700 1 -- 172.21.15.111:0/934693993 wait complete. 2024-03-14T13:13:19.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 -- 172.21.15.111:0/3187476721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 msgr2=0x7f88b80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 --2- 172.21.15.111:0/3187476721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b80fe260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f88a8004940 tx=0x7f88a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:19.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 -- 172.21.15.111:0/3187476721 shutdown_connections 2024-03-14T13:13:19.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 --2- 172.21.15.111:0/3187476721 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88b80fe830 0x7f88b8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 --2- 172.21.15.111:0/3187476721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b80fe260 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 -- 172.21.15.111:0/3187476721 >> 172.21.15.111:0/3187476721 conn(0x7f88b8072bc0 msgr2=0x7f88b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 -- 172.21.15.111:0/3187476721 shutdown_connections 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.472+0000 7f88bf715700 1 -- 172.21.15.111:0/3187476721 wait complete. 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 Processor -- start 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 -- start start 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88b80fe830 0x7f88b8114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b8114710 con 0x7f88b80fde90 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bf715700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b810da70 con 0x7f88b80fe830 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38974/0 (socket says 172.21.15.111:38974) 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 -- 172.21.15.111:0/3159240040 learned_addr learned my addr 172.21.15.111:0/3159240040 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 -- 172.21.15.111:0/3159240040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88b80fe830 msgr2=0x7f88b8114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88b80fe830 0x7f88b8114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.473+0000 7f88bd4b1700 1 -- 172.21.15.111:0/3159240040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a8005040 con 0x7f88b80fde90 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88bd4b1700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009360 tx=0x7f88a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a800e040 con 0x7f88b80fde90 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88a8013420 con 0x7f88b80fde90 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b810dcf0 con 0x7f88b80fde90 2024-03-14T13:13:19.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88a801c630 con 0x7f88b80fde90 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.474+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b810e230 con 0x7f88b80fde90 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.475+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b8065e30 con 0x7f88b80fde90 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.475+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f88a80139e0 con 0x7f88b80fde90 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.476+0000 7f88adffb700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 0x7f88a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.476+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f88a808cbc0 con 0x7f88b80fde90 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.476+0000 7f88bccb0700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 0x7f88a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.476+0000 7f88bccb0700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 0x7f88a406ec70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f88b4006fd0 tx=0x7f88b4006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.480+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f88a805a990 con 0x7f88b80fde90 2024-03-14T13:13:19.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.718+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f88b8109ea0 con 0x7f88b80fde90 2024-03-14T13:13:19.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.719+0000 7f88adffb700 1 -- 172.21.15.111:0/3159240040 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f88a805a520 con 0x7f88b80fde90 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:13:19.719 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:13:19.720 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:19.721 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:19.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 msgr2=0x7f88a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 0x7f88a406ec70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f88b4006fd0 tx=0x7f88b4006db0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 msgr2=0x7f88b8113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009360 tx=0x7f88a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:19.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 shutdown_connections 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88a406c7c0 0x7f88a406ec70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b80fde90 0x7f88b8113af0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 --2- 172.21.15.111:0/3159240040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88b80fe830 0x7f88b8114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 >> 172.21.15.111:0/3159240040 conn(0x7f88b8072bc0 msgr2=0x7f88b81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 shutdown_connections 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.721+0000 7f88bf715700 1 -- 172.21.15.111:0/3159240040 wait complete. 2024-03-14T13:13:19.723 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 -- 172.21.15.111:0/3392641498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 msgr2=0x7f92281031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 --2- 172.21.15.111:0/3392641498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92281031e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f921c004940 tx=0x7f921c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 -- 172.21.15.111:0/3392641498 shutdown_connections 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 --2- 172.21.15.111:0/3392641498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 0x7f92280fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 --2- 172.21.15.111:0/3392641498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92281031e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 -- 172.21.15.111:0/3392641498 >> 172.21.15.111:0/3392641498 conn(0x7f92280f8980 msgr2=0x7f92280fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 -- 172.21.15.111:0/3392641498 shutdown_connections 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 -- 172.21.15.111:0/3392641498 wait complete. 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.822+0000 7f922e93a700 1 Processor -- start 2024-03-14T13:13:19.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 -- start start 2024-03-14T13:13:19.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 0x7f9228070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228070d90 con 0x7f9228102e10 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228070ed0 con 0x7f92281037b0 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38990/0 (socket says 172.21.15.111:38990) 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 -- 172.21.15.111:0/1454532633 learned_addr learned my addr 172.21.15.111:0/1454532633 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f92277fe700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 0x7f9228070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 -- 172.21.15.111:0/1454532633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 msgr2=0x7f9228070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 0x7f9228070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 -- 172.21.15.111:0/1454532633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c005040 con 0x7f9228102e10 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9227fff700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f921c000c00 tx=0x7f921c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921c00e040 con 0x7f9228102e10 2024-03-14T13:13:19.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f921c013420 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9228071120 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f921c01c630 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.823+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922811a240 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.824+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92280fe560 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.825+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f921c013590 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.825+0000 7f9224ff9700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 0x7f921006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.825+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f921c08ccb0 con 0x7f9228102e10 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.825+0000 7f92277fe700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 0x7f921006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:19.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.826+0000 7f92277fe700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 0x7f921006ec70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9228071c00 tx=0x7f9218009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:19.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:19.830+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f921c056f30 con 0x7f9228102e10 2024-03-14T13:13:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:19 smithi111 ceph-mon[145314]: pgmap v276: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:19 smithi111 ceph-mon[145314]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:19 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/934693993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:13:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:19 smithi137 ceph-mon[108145]: pgmap v276: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:19 smithi137 ceph-mon[108145]: from='client.34568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:19 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/934693993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:13:20.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.039+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92280719b0 con 0x7f921006c7c0 2024-03-14T13:13:20.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.040+0000 7f9224ff9700 1 -- 172.21.15.111:0/1454532633 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f92280719b0 con 0x7f921006c7c0 2024-03-14T13:13:20.039 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:13:20.039 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:13:20.039 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:13:20.039 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:13:20.040 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:13:20.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 msgr2=0x7f921006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 0x7f921006ec70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9228071c00 tx=0x7f9218009ee0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 msgr2=0x7f92280702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f921c000c00 tx=0x7f921c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 shutdown_connections 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f921006c7c0 0x7f921006ec70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9228102e10 0x7f92280702c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 --2- 172.21.15.111:0/1454532633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92281037b0 0x7f9228070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 >> 172.21.15.111:0/1454532633 conn(0x7f92280f8980 msgr2=0x7f92280f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 shutdown_connections 2024-03-14T13:13:20.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.042+0000 7f922e93a700 1 -- 172.21.15.111:0/1454532633 wait complete. 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 -- 172.21.15.111:0/3218935831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5500102e70 msgr2=0x7f5500103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 --2- 172.21.15.111:0/3218935831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5500102e70 0x7f5500103240 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f54fc004940 tx=0x7f54fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 -- 172.21.15.111:0/3218935831 shutdown_connections 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 --2- 172.21.15.111:0/3218935831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f55000fb4a0 0x7f55000fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 --2- 172.21.15.111:0/3218935831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5500102e70 0x7f5500103240 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 -- 172.21.15.111:0/3218935831 >> 172.21.15.111:0/3218935831 conn(0x7f55000714c0 msgr2=0x7f55000718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.141+0000 7f55083ea700 1 -- 172.21.15.111:0/3218935831 shutdown_connections 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 -- 172.21.15.111:0/3218935831 wait complete. 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 Processor -- start 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 -- start start 2024-03-14T13:13:20.143 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5500102e70 0x7f5500113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5500114600 con 0x7f55000fb4a0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f55083ea700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550010d9f0 con 0x7f5500102e70 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f5506186700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f5506186700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39020/0 (socket says 172.21.15.111:39020) 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.142+0000 7f5506186700 1 -- 172.21.15.111:0/850904930 learned_addr learned my addr 172.21.15.111:0/850904930 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f5505985700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5500102e70 0x7f5500113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f5506186700 1 -- 172.21.15.111:0/850904930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5500102e70 msgr2=0x7f5500113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f5506186700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5500102e70 0x7f5500113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f5506186700 1 -- 172.21.15.111:0/850904930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc005040 con 0x7f55000fb4a0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f5506186700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009360 tx=0x7f54fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54fc00e040 con 0x7f55000fb4a0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54fc013420 con 0x7f55000fb4a0 2024-03-14T13:13:20.144 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550010dc70 con 0x7f55000fb4a0 2024-03-14T13:13:20.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54fc01c5f0 con 0x7f55000fb4a0 2024-03-14T13:13:20.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.143+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550010e230 con 0x7f55000fb4a0 2024-03-14T13:13:20.145 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.144+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5500100110 con 0x7f55000fb4a0 2024-03-14T13:13:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.144+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f54fc013590 con 0x7f55000fb4a0 2024-03-14T13:13:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.145+0000 7f54f6ffd700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 0x7f54ec06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.145+0000 7f5505985700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 0x7f54ec06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.145+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f54fc08cd60 con 0x7f55000fb4a0 2024-03-14T13:13:20.146 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.145+0000 7f5505985700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 0x7f54ec06ec80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f550010ebb0 tx=0x7f54f0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:20.149 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.149+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f54fc091070 con 0x7f55000fb4a0 2024-03-14T13:13:20.421 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.422+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f550010f0c0 con 0x7f55000fb4a0 2024-03-14T13:13:20.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.422+0000 7f54f6ffd700 1 -- 172.21.15.111:0/850904930 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f54fc05aaa0 con 0x7f55000fb4a0 2024-03-14T13:13:20.590 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:13:20.590 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:13:20.590 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:13:20.590 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:13:20.590 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 msgr2=0x7f54ec06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 0x7f54ec06ec80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f550010ebb0 tx=0x7f54f0009ee0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 msgr2=0x7f5500113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009360 tx=0x7f54fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 shutdown_connections 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f54ec06c7d0 0x7f54ec06ec80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55000fb4a0 0x7f5500113a70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 --2- 172.21.15.111:0/850904930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5500102e70 0x7f5500113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 >> 172.21.15.111:0/850904930 conn(0x7f55000714c0 msgr2=0x7f55000fa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 shutdown_connections 2024-03-14T13:13:20.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:20.424+0000 7f55083ea700 1 -- 172.21.15.111:0/850904930 wait complete. 2024-03-14T13:13:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:20 smithi111 ceph-mon[145314]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:20 smithi111 ceph-mon[145314]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:20 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3159240040' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:13:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:20 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/850904930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:13:21.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:20 smithi137 ceph-mon[108145]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:20 smithi137 ceph-mon[108145]: from='client.34576 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:20 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3159240040' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:13:21.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:20 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/850904930' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:13:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:21 smithi111 ceph-mon[145314]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:21 smithi111 ceph-mon[145314]: pgmap v277: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:21 smithi137 ceph-mon[108145]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:21 smithi137 ceph-mon[108145]: pgmap v277: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:23 smithi111 ceph-mon[145314]: pgmap v278: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:23 smithi137 ceph-mon[108145]: pgmap v278: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:25 smithi111 ceph-mon[145314]: pgmap v279: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:25 smithi137 ceph-mon[108145]: pgmap v279: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:27 smithi111 ceph-mon[145314]: pgmap v280: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:27 smithi137 ceph-mon[108145]: pgmap v280: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:29 smithi111 ceph-mon[145314]: pgmap v281: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:29 smithi137 ceph-mon[108145]: pgmap v281: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:31 smithi111 ceph-mon[145314]: pgmap v282: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:31 smithi137 ceph-mon[108145]: pgmap v282: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:32.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:32 smithi111 ceph-mon[145314]: pgmap v283: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:33.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:32 smithi137 ceph-mon[108145]: pgmap v283: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:35.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:35 smithi137 ceph-mon[108145]: pgmap v284: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:35 smithi111 ceph-mon[145314]: pgmap v284: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:37.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:37 smithi137 ceph-mon[108145]: pgmap v285: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:37 smithi111 ceph-mon[145314]: pgmap v285: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:39 smithi137 ceph-mon[108145]: pgmap v286: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:39.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:39 smithi111 ceph-mon[145314]: pgmap v286: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:41 smithi137 ceph-mon[108145]: pgmap v287: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:41 smithi111 ceph-mon[145314]: pgmap v287: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:43 smithi137 ceph-mon[108145]: pgmap v288: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:43 smithi111 ceph-mon[145314]: pgmap v288: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:45 smithi137 ceph-mon[108145]: pgmap v289: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:45 smithi111 ceph-mon[145314]: pgmap v289: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:47 smithi137 ceph-mon[108145]: pgmap v290: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:47 smithi111 ceph-mon[145314]: pgmap v290: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:49.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:49 smithi137 ceph-mon[108145]: pgmap v291: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:49 smithi111 ceph-mon[145314]: pgmap v291: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:50.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 -- 172.21.15.111:0/296579700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 msgr2=0x7f88d80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 --2- 172.21.15.111:0/296579700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d80fbd80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f88cc004940 tx=0x7f88cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 -- 172.21.15.111:0/296579700 shutdown_connections 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 --2- 172.21.15.111:0/296579700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d80fbd80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 --2- 172.21.15.111:0/296579700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 0x7f88d8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 -- 172.21.15.111:0/296579700 >> 172.21.15.111:0/296579700 conn(0x7f88d8072bc0 msgr2=0x7f88d8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 -- 172.21.15.111:0/296579700 shutdown_connections 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 -- 172.21.15.111:0/296579700 wait complete. 2024-03-14T13:13:50.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.539+0000 7f88de230700 1 Processor -- start 2024-03-14T13:13:50.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88de230700 1 -- start start 2024-03-14T13:13:50.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88de230700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 0x7f88d81129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88de230700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88de230700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8113550 con 0x7f88d8102710 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88de230700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d810c940 con 0x7f88d8101d70 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42008/0 (socket says 172.21.15.111:42008) 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 -- 172.21.15.111:0/4258006109 learned_addr learned my addr 172.21.15.111:0/4258006109 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d77fe700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 0x7f88d81129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 -- 172.21.15.111:0/4258006109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 msgr2=0x7f88d81129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 0x7f88d81129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.540+0000 7f88d6ffd700 1 -- 172.21.15.111:0/4258006109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc005040 con 0x7f88d8102710 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88d6ffd700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009380 tx=0x7f88cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88cc00e040 con 0x7f88d8102710 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88cc013420 con 0x7f88d8102710 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88cc01c630 con 0x7f88d8102710 2024-03-14T13:13:50.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d810cbc0 con 0x7f88d8102710 2024-03-14T13:13:50.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.541+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d810d180 con 0x7f88d8102710 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.542+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d80ff080 con 0x7f88d8102710 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.542+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f88cc0139b0 con 0x7f88d8102710 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.543+0000 7f88bffff700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 0x7f88c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.543+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f88cc08cbe0 con 0x7f88d8102710 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.543+0000 7f88d77fe700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 0x7f88c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.543+0000 7f88d77fe700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 0x7f88c006ec70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f88c8004f40 tx=0x7f88c800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:50.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.547+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f88cc05a8f0 con 0x7f88d8102710 2024-03-14T13:13:50.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.758+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f88d8112690 con 0x7f88c006c7c0 2024-03-14T13:13:50.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.760+0000 7f88bffff700 1 -- 172.21.15.111:0/4258006109 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f88d8112690 con 0x7f88c006c7c0 2024-03-14T13:13:50.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 msgr2=0x7f88c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 0x7f88c006ec70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f88c8004f40 tx=0x7f88c800b040 comp rx=0 tx=0).stop 2024-03-14T13:13:50.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 msgr2=0x7f88d8112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009380 tx=0x7f88cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 shutdown_connections 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f88c006c7c0 0x7f88c006ec70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88d8101d70 0x7f88d81129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 --2- 172.21.15.111:0/4258006109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d8102710 0x7f88d8112f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 >> 172.21.15.111:0/4258006109 conn(0x7f88d8072bc0 msgr2=0x7f88d80fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 shutdown_connections 2024-03-14T13:13:50.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.762+0000 7f88de230700 1 -- 172.21.15.111:0/4258006109 wait complete. 2024-03-14T13:13:50.771 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 -- 172.21.15.111:0/1425995015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 msgr2=0x7f1cb8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1425995015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8106f00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8004940 tx=0x7f1ca8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 -- 172.21.15.111:0/1425995015 shutdown_connections 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1425995015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8106f00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1425995015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 0x7f1cb80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.862+0000 7f1cbf745700 1 -- 172.21.15.111:0/1425995015 >> 172.21.15.111:0/1425995015 conn(0x7f1cb8072bc0 msgr2=0x7f1cb8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 -- 172.21.15.111:0/1425995015 shutdown_connections 2024-03-14T13:13:50.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 -- 172.21.15.111:0/1425995015 wait complete. 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 Processor -- start 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 -- start start 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 0x7f1cb8113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8114670 con 0x7f1cb80fe830 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbf745700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb810da60 con 0x7f1cb80fde90 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbcce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbcce0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42028/0 (socket says 172.21.15.111:42028) 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.863+0000 7f1cbcce0700 1 -- 172.21.15.111:0/1005771795 learned_addr learned my addr 172.21.15.111:0/1005771795 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:50.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbd4e1700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 0x7f1cb8113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbcce0700 1 -- 172.21.15.111:0/1005771795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 msgr2=0x7f1cb8113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbcce0700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 0x7f1cb8113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbcce0700 1 -- 172.21.15.111:0/1005771795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca8005040 con 0x7f1cb80fe830 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbcce0700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800b3b0 tx=0x7f1ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca800e040 con 0x7f1cb80fe830 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ca8013420 con 0x7f1cb80fe830 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb810dce0 con 0x7f1cb80fe830 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca801c5d0 con 0x7f1cb80fe830 2024-03-14T13:13:50.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.864+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb810e220 con 0x7f1cb80fe830 2024-03-14T13:13:50.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.865+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb8065e30 con 0x7f1cb80fe830 2024-03-14T13:13:50.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.865+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1ca8013590 con 0x7f1cb80fe830 2024-03-14T13:13:50.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.866+0000 7f1cadffb700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 0x7f1ca406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:50.867 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.866+0000 7f1cbd4e1700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 0x7f1ca406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:50.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.866+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ca808ccf0 con 0x7f1cb80fe830 2024-03-14T13:13:50.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.866+0000 7f1cbd4e1700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 0x7f1ca406ec70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4004c40 tx=0x7f1cb4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:50.870 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:50.870+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1ca805a980 con 0x7f1cb80fe830 2024-03-14T13:13:51.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.081+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cb8109dd0 con 0x7f1ca406c7c0 2024-03-14T13:13:51.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.082+0000 7f1cadffb700 1 -- 172.21.15.111:0/1005771795 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1cb8109dd0 con 0x7f1ca406c7c0 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 msgr2=0x7f1ca406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 0x7f1ca406ec70 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4004c40 tx=0x7f1cb4009040 comp rx=0 tx=0).stop 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 msgr2=0x7f1cb8114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800b3b0 tx=0x7f1ca8009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 shutdown_connections 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ca406c7c0 0x7f1ca406ec70 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1cb80fde90 0x7f1cb8113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 --2- 172.21.15.111:0/1005771795 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cb80fe830 0x7f1cb8114020 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 >> 172.21.15.111:0/1005771795 conn(0x7f1cb8072bc0 msgr2=0x7f1cb8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 shutdown_connections 2024-03-14T13:13:51.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.084+0000 7f1cbf745700 1 -- 172.21.15.111:0/1005771795 wait complete. 2024-03-14T13:13:51.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 -- 172.21.15.111:0/3113449457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d41037c0 msgr2=0x7f80d40fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 --2- 172.21.15.111:0/3113449457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d41037c0 0x7f80d40fce30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f80c4004940 tx=0x7f80c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 -- 172.21.15.111:0/3113449457 shutdown_connections 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 --2- 172.21.15.111:0/3113449457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d41037c0 0x7f80d40fce30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 --2- 172.21.15.111:0/3113449457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d4102e20 0x7f80d41031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 -- 172.21.15.111:0/3113449457 >> 172.21.15.111:0/3113449457 conn(0x7f80d40f89d0 msgr2=0x7f80d40fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.184+0000 7f80d8858700 1 -- 172.21.15.111:0/3113449457 shutdown_connections 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 -- 172.21.15.111:0/3113449457 wait complete. 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 Processor -- start 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 -- start start 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 0x7f80d4114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d4114660 con 0x7f80d4102e20 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d8858700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d410da50 con 0x7f80d41037c0 2024-03-14T13:13:51.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42040/0 (socket says 172.21.15.111:42040) 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.185+0000 7f80d259c700 1 -- 172.21.15.111:0/1681002324 learned_addr learned my addr 172.21.15.111:0/1681002324 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d1d9b700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 0x7f80d4114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d259c700 1 -- 172.21.15.111:0/1681002324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 msgr2=0x7f80d4114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d259c700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 0x7f80d4114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d259c700 1 -- 172.21.15.111:0/1681002324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c4005040 con 0x7f80d4102e20 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d1d9b700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 0x7f80d4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d259c700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f80c8002a50 tx=0x7f80c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c8007cc0 con 0x7f80d4102e20 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80c800f040 con 0x7f80d4102e20 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d410dd30 con 0x7f80d4102e20 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80c8009540 con 0x7f80d4102e20 2024-03-14T13:13:51.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.186+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d410e2f0 con 0x7f80d4102e20 2024-03-14T13:13:51.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.187+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d40fe570 con 0x7f80d4102e20 2024-03-14T13:13:51.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.187+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f80c80096a0 con 0x7f80d4102e20 2024-03-14T13:13:51.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.188+0000 7f80c2ffd700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 0x7f80bc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.188+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f80c808b8a0 con 0x7f80d4102e20 2024-03-14T13:13:51.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.188+0000 7f80d1d9b700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 0x7f80bc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.188+0000 7f80d1d9b700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 0x7f80bc06ec70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b3b0 tx=0x7f80c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.192+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f80c8059530 con 0x7f80d4102e20 2024-03-14T13:13:51.398 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.398+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f80d4109dc0 con 0x7f80bc06c7c0 2024-03-14T13:13:51.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.408+0000 7f80c2ffd700 1 -- 172.21.15.111:0/1681002324 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f80d4109dc0 con 0x7f80bc06c7c0 2024-03-14T13:13:51.407 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:13:51.407 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (10m) 3m ago 21m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (7m) 3m ago 21m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (7m) 3m ago 20m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (9m) 3m ago 21m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (3m) 3m ago 15m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (3m) 3m ago 15m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (3m) 3m ago 15m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (3m) 3m ago 15m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (12m) 3m ago 23m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (11m) 3m ago 20m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (8m) 3m ago 23m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (8m) 3m ago 20m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (11m) 3m ago 21m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (10m) 3m ago 20m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:13:51.408 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (7m) 3m ago 19m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (6m) 3m ago 18m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 3m ago 18m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (5m) 3m ago 18m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (4m) 3m ago 17m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (4m) 3m ago 17m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:13:51.409 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (10m) 3m ago 20m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 msgr2=0x7f80bc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 0x7f80bc06ec70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f80c400b3b0 tx=0x7f80c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 msgr2=0x7f80d4113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f80c8002a50 tx=0x7f80c80046b0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 shutdown_connections 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f80bc06c7c0 0x7f80bc06ec70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80d4102e20 0x7f80d4113ad0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 --2- 172.21.15.111:0/1681002324 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80d41037c0 0x7f80d4114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 >> 172.21.15.111:0/1681002324 conn(0x7f80d40f89d0 msgr2=0x7f80d40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.410 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 shutdown_connections 2024-03-14T13:13:51.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.410+0000 7f80d8858700 1 -- 172.21.15.111:0/1681002324 wait complete. 2024-03-14T13:13:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 -- 172.21.15.111:0/1635171733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c101d20 msgr2=0x7ff59c1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/1635171733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c101d20 0x7ff59c1020f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff590004940 tx=0x7ff590009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 -- 172.21.15.111:0/1635171733 shutdown_connections 2024-03-14T13:13:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/1635171733 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c1026c0 0x7ff59c0fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/1635171733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c101d20 0x7ff59c1020f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 -- 172.21.15.111:0/1635171733 >> 172.21.15.111:0/1635171733 conn(0x7ff59c0f78d0 msgr2=0x7ff59c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 -- 172.21.15.111:0/1635171733 shutdown_connections 2024-03-14T13:13:51.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.509+0000 7ff5a2eab700 1 -- 172.21.15.111:0/1635171733 wait complete. 2024-03-14T13:13:51.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 Processor -- start 2024-03-14T13:13:51.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 -- start start 2024-03-14T13:13:51.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c101d20 0x7ff59c1129d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c1135f0 con 0x7ff59c1026c0 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff5a2eab700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c10c950 con 0x7ff59c101d20 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff59bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff59bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42062/0 (socket says 172.21.15.111:42062) 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.510+0000 7ff59bfff700 1 -- 172.21.15.111:0/2914624822 learned_addr learned my addr 172.21.15.111:0/2914624822 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5a0c47700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c101d20 0x7ff59c1129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff59bfff700 1 -- 172.21.15.111:0/2914624822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c101d20 msgr2=0x7ff59c1129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff59bfff700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c101d20 0x7ff59c1129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff59bfff700 1 -- 172.21.15.111:0/2914624822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590005040 con 0x7ff59c1026c0 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff59bfff700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b790 tx=0x7ff588009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff58800f6f0 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c10cc30 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff58800fd00 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5880103a0 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.511+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c10d1f0 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.512+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c0fd470 con 0x7ff59c1026c0 2024-03-14T13:13:51.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.513+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff58800f850 con 0x7ff59c1026c0 2024-03-14T13:13:51.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.513+0000 7ff5997fa700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 0x7ff58c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.513+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff58808c4a0 con 0x7ff59c1026c0 2024-03-14T13:13:51.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.513+0000 7ff5a0c47700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 0x7ff58c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.514+0000 7ff5a0c47700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 0x7ff58c06ec70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff590004f90 tx=0x7ff590009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.517+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff58805a1f0 con 0x7ff59c1026c0 2024-03-14T13:13:51.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:51 smithi137 ceph-mon[108145]: pgmap v292: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:51.786 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.787+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff59c04bed0 con 0x7ff59c1026c0 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.787+0000 7ff5997fa700 1 -- 172.21.15.111:0/2914624822 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7ff588060070 con 0x7ff59c1026c0 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:51.787 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:13:51.788 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.789+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 msgr2=0x7ff58c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.789+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 0x7ff58c06ec70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff590004f90 tx=0x7ff590009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.789+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 msgr2=0x7ff59c112f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.789+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff58800b790 tx=0x7ff588009220 comp rx=0 tx=0).stop 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 shutdown_connections 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff58c06c7c0 0x7ff58c06ec70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff59c101d20 0x7ff59c1129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 --2- 172.21.15.111:0/2914624822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff59c1026c0 0x7ff59c112f10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 >> 172.21.15.111:0/2914624822 conn(0x7ff59c0f78d0 msgr2=0x7ff59c0f89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 shutdown_connections 2024-03-14T13:13:51.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.790+0000 7ff5a2eab700 1 -- 172.21.15.111:0/2914624822 wait complete. 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 -- 172.21.15.111:0/2790034164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 msgr2=0x7fc77c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 --2- 172.21.15.111:0/2790034164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c0fbd80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc76c004940 tx=0x7fc76c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 -- 172.21.15.111:0/2790034164 shutdown_connections 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 --2- 172.21.15.111:0/2790034164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c0fbd80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 --2- 172.21.15.111:0/2790034164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 0x7fc77c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 -- 172.21.15.111:0/2790034164 >> 172.21.15.111:0/2790034164 conn(0x7fc77c072bc0 msgr2=0x7fc77c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:51.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 -- 172.21.15.111:0/2790034164 shutdown_connections 2024-03-14T13:13:51.891 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 -- 172.21.15.111:0/2790034164 wait complete. 2024-03-14T13:13:51.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:51 smithi111 ceph-mon[145314]: pgmap v292: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:51.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.890+0000 7fc7809a3700 1 Processor -- start 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 -- start start 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 0x7fc77c113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c114650 con 0x7fc77c102710 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77c10da40 con 0x7fc77c101d70 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42076/0 (socket says 172.21.15.111:42076) 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 -- 172.21.15.111:0/1912845823 learned_addr learned my addr 172.21.15.111:0/1912845823 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc77a59c700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 0x7fc77c113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 -- 172.21.15.111:0/1912845823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 msgr2=0x7fc77c113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:51.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 0x7fc77c113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 -- 172.21.15.111:0/1912845823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc76c005040 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc779d9b700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009390 tx=0x7fc76c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc76c00e040 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.891+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77c10dcc0 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.892+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77c10e280 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.892+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc76c004db0 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.892+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc76c013da0 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.892+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c0052b0 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.893+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc76c013420 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.893+0000 7fc772ffd700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 0x7fc76806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.893+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc76c08cd70 con 0x7fc77c102710 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.893+0000 7fc77a59c700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 0x7fc76806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:51.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.894+0000 7fc77a59c700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 0x7fc76806ec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:51.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:51.898+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc76c05aa80 con 0x7fc77c102710 2024-03-14T13:13:52.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.129+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fc75c005d70 con 0x7fc77c102710 2024-03-14T13:13:52.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.130+0000 7fc772ffd700 1 -- 172.21.15.111:0/1912845823 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fc76c05a610 con 0x7fc77c102710 2024-03-14T13:13:52.129 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:13:52.129 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:13:52.130 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:13:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:52.132 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 msgr2=0x7fc76806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 0x7fc76806ec80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc764007fd0 tx=0x7fc764009040 comp rx=0 tx=0).stop 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 msgr2=0x7fc77c114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009390 tx=0x7fc76c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 shutdown_connections 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc76806c7d0 0x7fc76806ec80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc77c101d70 0x7fc77c113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 --2- 172.21.15.111:0/1912845823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc77c102710 0x7fc77c114000 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 >> 172.21.15.111:0/1912845823 conn(0x7fc77c072bc0 msgr2=0x7fc77c0659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 shutdown_connections 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.132+0000 7fc7809a3700 1 -- 172.21.15.111:0/1912845823 wait complete. 2024-03-14T13:13:52.133 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 -- 172.21.15.111:0/1418804194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 msgr2=0x7fcc5c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1418804194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c102140 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c004940 tx=0x7fcc4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 -- 172.21.15.111:0/1418804194 shutdown_connections 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1418804194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 0x7fcc5c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1418804194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c102140 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 -- 172.21.15.111:0/1418804194 >> 172.21.15.111:0/1418804194 conn(0x7fcc5c072bc0 msgr2=0x7fcc5c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 -- 172.21.15.111:0/1418804194 shutdown_connections 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.233+0000 7fcc627dd700 1 -- 172.21.15.111:0/1418804194 wait complete. 2024-03-14T13:13:52.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 Processor -- start 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 -- start start 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 0x7fcc5c10ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c10f5d0 con 0x7fcc5c101d70 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc627dd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c10f710 con 0x7fcc5c102710 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42094/0 (socket says 172.21.15.111:42094) 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 -- 172.21.15.111:0/1203793270 learned_addr learned my addr 172.21.15.111:0/1203793270 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 -- 172.21.15.111:0/1203793270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 msgr2=0x7fcc5c10ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5b7fe700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 0x7fcc5c10ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 0x7fcc5c10ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.234+0000 7fcc5bfff700 1 -- 172.21.15.111:0/1203793270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c005040 con 0x7fcc5c101d70 2024-03-14T13:13:52.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc5bfff700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009180 tx=0x7fcc4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc4c00e040 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc4c004290 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc4c01c3f0 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c117ab0 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.235+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c117fc0 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.236+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c0ff080 con 0x7fcc5c101d70 2024-03-14T13:13:52.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.236+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcc4c013790 con 0x7fcc5c101d70 2024-03-14T13:13:52.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.237+0000 7fcc58ff9700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 0x7fcc4806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.237+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc4c08cd70 con 0x7fcc5c101d70 2024-03-14T13:13:52.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.237+0000 7fcc5b7fe700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 0x7fcc4806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.237 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.237+0000 7fcc5b7fe700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 0x7fcc4806ec70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c066000 tx=0x7fcc4400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:52.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.241+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcc4c056fb0 con 0x7fcc5c101d70 2024-03-14T13:13:52.451 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.452+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc5c108dd0 con 0x7fcc4806c7c0 2024-03-14T13:13:52.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.453+0000 7fcc58ff9700 1 -- 172.21.15.111:0/1203793270 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc4400f040 con 0x7fcc4806c7c0 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:13:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:13:52.454 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 msgr2=0x7fcc4806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 0x7fcc4806ec70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c066000 tx=0x7fcc4400b040 comp rx=0 tx=0).stop 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 msgr2=0x7fcc5c10ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009180 tx=0x7fcc4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 shutdown_connections 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc4806c7c0 0x7fcc4806ec70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc5c101d70 0x7fcc5c10ea40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 --2- 172.21.15.111:0/1203793270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc5c102710 0x7fcc5c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 >> 172.21.15.111:0/1203793270 conn(0x7fcc5c072bc0 msgr2=0x7fcc5c0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 shutdown_connections 2024-03-14T13:13:52.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.455+0000 7fcc627dd700 1 -- 172.21.15.111:0/1203793270 wait complete. 2024-03-14T13:13:52.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 -- 172.21.15.111:0/843697719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c0702e0 msgr2=0x7f1a7c0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 --2- 172.21.15.111:0/843697719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c0706b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c004940 tx=0x7f1a6c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 -- 172.21.15.111:0/843697719 shutdown_connections 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 --2- 172.21.15.111:0/843697719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 --2- 172.21.15.111:0/843697719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c0706b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 -- 172.21.15.111:0/843697719 >> 172.21.15.111:0/843697719 conn(0x7f1a7c0f8980 msgr2=0x7f1a7c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 -- 172.21.15.111:0/843697719 shutdown_connections 2024-03-14T13:13:52.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.559+0000 7f1a83f78700 1 -- 172.21.15.111:0/843697719 wait complete. 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 Processor -- start 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 -- start start 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c116890 con 0x7f1a7c070bf0 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a83f78700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c10fc60 con 0x7f1a7c0702e0 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.561 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42110/0 (socket says 172.21.15.111:42110) 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 -- 172.21.15.111:0/2623656151 learned_addr learned my addr 172.21.15.111:0/2623656151 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81d14700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 -- 172.21.15.111:0/2623656151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c0702e0 msgr2=0x7f1a7c115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.560+0000 7f1a81513700 1 -- 172.21.15.111:0/2623656151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c005040 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a81513700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800c8c0 tx=0x7f1a780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a7800a760 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a7800ad70 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a780113a0 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c10ff40 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.561+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c110480 con 0x7f1a7c070bf0 2024-03-14T13:13:52.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.562+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c108e80 con 0x7f1a7c070bf0 2024-03-14T13:13:52.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.562+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1a78012400 con 0x7f1a7c070bf0 2024-03-14T13:13:52.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.563+0000 7f1a727fc700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 0x7f1a6806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:13:52.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.563+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1a7808b9a0 con 0x7f1a7c070bf0 2024-03-14T13:13:52.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.563+0000 7f1a81d14700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 0x7f1a6806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:13:52.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.563+0000 7f1a81d14700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 0x7f1a6806ec70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c004f90 tx=0x7f1a6c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:13:52.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.567+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1a78059770 con 0x7f1a7c070bf0 2024-03-14T13:13:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:52 smithi137 ceph-mon[108145]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:52 smithi137 ceph-mon[108145]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:52 smithi137 ceph-mon[108145]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:52 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2914624822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:13:52.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:52 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1912845823' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:13:52.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.839+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1a7c071a20 con 0x7f1a7c070bf0 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.839+0000 7f1a727fc700 1 -- 172.21.15.111:0/2623656151 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f1a78059300 con 0x7f1a7c070bf0 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:13:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 msgr2=0x7f1a6806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 0x7f1a6806ec70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c004f90 tx=0x7f1a6c009e30 comp rx=0 tx=0).stop 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 msgr2=0x7f1a7c116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800c8c0 tx=0x7f1a780044c0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 shutdown_connections 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a6806c7c0 0x7f1a6806ec70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c0702e0 0x7f1a7c115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 --2- 172.21.15.111:0/2623656151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a7c070bf0 0x7f1a7c116240 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 >> 172.21.15.111:0/2623656151 conn(0x7f1a7c0f8980 msgr2=0x7f1a7c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 shutdown_connections 2024-03-14T13:13:52.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:13:52.841+0000 7f1a83f78700 1 -- 172.21.15.111:0/2623656151 wait complete. 2024-03-14T13:13:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:52 smithi111 ceph-mon[145314]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:52 smithi111 ceph-mon[145314]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:52 smithi111 ceph-mon[145314]: from='client.34604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:52 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2914624822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:13:52.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:52 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1912845823' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:13:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:53 smithi137 ceph-mon[108145]: pgmap v293: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:53 smithi137 ceph-mon[108145]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:53.756 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:53 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2623656151' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:13:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:53 smithi111 ceph-mon[145314]: pgmap v293: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:53 smithi111 ceph-mon[145314]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:13:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:53 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2623656151' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:13:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:13:55.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:55 smithi111 ceph-mon[145314]: pgmap v294: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:55 smithi137 ceph-mon[108145]: pgmap v294: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:57 smithi111 ceph-mon[145314]: pgmap v295: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:57 smithi137 ceph-mon[108145]: pgmap v295: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:13:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:13:59 smithi111 ceph-mon[145314]: pgmap v296: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:13:59 smithi137 ceph-mon[108145]: pgmap v296: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:01 smithi111 ceph-mon[145314]: pgmap v297: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:01 smithi137 ceph-mon[108145]: pgmap v297: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:03 smithi111 ceph-mon[145314]: pgmap v298: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:03 smithi137 ceph-mon[108145]: pgmap v298: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:05 smithi111 ceph-mon[145314]: pgmap v299: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:05 smithi137 ceph-mon[108145]: pgmap v299: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:07 smithi111 ceph-mon[145314]: pgmap v300: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:07 smithi137 ceph-mon[108145]: pgmap v300: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:09 smithi111 ceph-mon[145314]: pgmap v301: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:09 smithi137 ceph-mon[108145]: pgmap v301: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:11 smithi111 ceph-mon[145314]: pgmap v302: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:11 smithi137 ceph-mon[108145]: pgmap v302: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:13 smithi111 ceph-mon[145314]: pgmap v303: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:13 smithi137 ceph-mon[108145]: pgmap v303: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:15 smithi111 ceph-mon[145314]: pgmap v304: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:15 smithi137 ceph-mon[108145]: pgmap v304: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:17 smithi111 ceph-mon[145314]: pgmap v305: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:17 smithi137 ceph-mon[108145]: pgmap v305: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:19 smithi111 ceph-mon[145314]: pgmap v306: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:19 smithi137 ceph-mon[108145]: pgmap v306: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:21 smithi111 ceph-mon[145314]: pgmap v307: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:21 smithi137 ceph-mon[108145]: pgmap v307: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 -- 172.21.15.111:0/1415644701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970101d70 msgr2=0x7f6970102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 --2- 172.21.15.111:0/1415644701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970101d70 0x7f6970102140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6964004940 tx=0x7f6964009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 -- 172.21.15.111:0/1415644701 shutdown_connections 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 --2- 172.21.15.111:0/1415644701 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970102710 0x7f69700fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 --2- 172.21.15.111:0/1415644701 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970101d70 0x7f6970102140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 -- 172.21.15.111:0/1415644701 >> 172.21.15.111:0/1415644701 conn(0x7f6970072bc0 msgr2=0x7f6970072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 -- 172.21.15.111:0/1415644701 shutdown_connections 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.954+0000 7f697506d700 1 -- 172.21.15.111:0/1415644701 wait complete. 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 Processor -- start 2024-03-14T13:14:22.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 -- start start 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970101d70 0x7f6970115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970116820 con 0x7f6970102710 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f697506d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697010fc40 con 0x7f6970101d70 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42262/0 (socket says 172.21.15.111:42262) 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 -- 172.21.15.111:0/3939176440 learned_addr learned my addr 172.21.15.111:0/3939176440 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696ed9d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970101d70 0x7f6970115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 -- 172.21.15.111:0/3939176440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970101d70 msgr2=0x7f6970115c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970101d70 0x7f6970115c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 -- 172.21.15.111:0/3939176440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6964005040 con 0x7f6970102710 2024-03-14T13:14:22.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.955+0000 7f696e59c700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f696000c800 tx=0x7f6960004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.956+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69600106f0 con 0x7f6970102710 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.956+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f697010ff20 con 0x7f6970102710 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.956+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6960010d00 con 0x7f6970102710 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.956+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69600113a0 con 0x7f6970102710 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.956+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69701104e0 con 0x7f6970102710 2024-03-14T13:14:22.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.957+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69700fd4c0 con 0x7f6970102710 2024-03-14T13:14:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.957+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6960012400 con 0x7f6970102710 2024-03-14T13:14:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.958+0000 7f69577fe700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 0x7f695806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.958+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f696008ba10 con 0x7f6970102710 2024-03-14T13:14:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.958+0000 7f696ed9d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 0x7f695806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:22.958 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.958+0000 7f696ed9d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 0x7f695806ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6964004f90 tx=0x7f6964009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:22.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:22.962+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6960059720 con 0x7f6970102710 2024-03-14T13:14:23.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.177+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6970108da0 con 0x7f695806c7c0 2024-03-14T13:14:23.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.179+0000 7f69577fe700 1 -- 172.21.15.111:0/3939176440 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6970108da0 con 0x7f695806c7c0 2024-03-14T13:14:23.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 msgr2=0x7f695806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 0x7f695806ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6964004f90 tx=0x7f6964009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 msgr2=0x7f69701161d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f696000c800 tx=0x7f6960004470 comp rx=0 tx=0).stop 2024-03-14T13:14:23.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 shutdown_connections 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f695806c7c0 0x7f695806ec70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6970101d70 0x7f6970115c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 --2- 172.21.15.111:0/3939176440 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970102710 0x7f69701161d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 >> 172.21.15.111:0/3939176440 conn(0x7f6970072bc0 msgr2=0x7f6970065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 shutdown_connections 2024-03-14T13:14:23.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.181+0000 7f697506d700 1 -- 172.21.15.111:0/3939176440 wait complete. 2024-03-14T13:14:23.190 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- 172.21.15.111:0/751930952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0fb3b0 msgr2=0x7faefc0fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 --2- 172.21.15.111:0/751930952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0fb3b0 0x7faefc0fb820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faeec004940 tx=0x7faeec009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- 172.21.15.111:0/751930952 shutdown_connections 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 --2- 172.21.15.111:0/751930952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0fb3b0 0x7faefc0fb820 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 --2- 172.21.15.111:0/751930952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc102f70 0x7faefc103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- 172.21.15.111:0/751930952 >> 172.21.15.111:0/751930952 conn(0x7faefc072310 msgr2=0x7faefc072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- 172.21.15.111:0/751930952 shutdown_connections 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- 172.21.15.111:0/751930952 wait complete. 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 Processor -- start 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.284+0000 7faf0343f700 1 -- start start 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf0343f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 0x7faefc1138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf0343f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf0343f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc114430 con 0x7faefc102f70 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf0343f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc10da30 con 0x7faefc0fb3b0 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42272/0 (socket says 172.21.15.111:42272) 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 -- 172.21.15.111:0/3743423543 learned_addr learned my addr 172.21.15.111:0/3743423543 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf011db700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 0x7faefc1138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 -- 172.21.15.111:0/3743423543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 msgr2=0x7faefc1138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 0x7faefc1138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 -- 172.21.15.111:0/3743423543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec005040 con 0x7faefc102f70 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf011db700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 0x7faefc1138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf009da700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faeec009150 tx=0x7faeec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeec00e040 con 0x7faefc102f70 2024-03-14T13:14:23.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faeec013420 con 0x7faefc102f70 2024-03-14T13:14:23.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faeec01c630 con 0x7faefc102f70 2024-03-14T13:14:23.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.285+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc10dcb0 con 0x7faefc102f70 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.286+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc10e220 con 0x7faefc102f70 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.287+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc1053b0 con 0x7faefc102f70 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.287+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faeec013990 con 0x7faefc102f70 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.287+0000 7faef1ffb700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 0x7faee806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.288+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faeec08ccb0 con 0x7faefc102f70 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.288+0000 7faf011db700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 0x7faee806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.288+0000 7faf011db700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 0x7faee806ec70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faef80041d0 tx=0x7faef8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.292+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faeec05a9c0 con 0x7faefc102f70 2024-03-14T13:14:23.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.508+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7faefc10eef0 con 0x7faee806c7c0 2024-03-14T13:14:23.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.510+0000 7faef1ffb700 1 -- 172.21.15.111:0/3743423543 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7faefc10eef0 con 0x7faee806c7c0 2024-03-14T13:14:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 msgr2=0x7faee806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 0x7faee806ec70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faef80041d0 tx=0x7faef8009040 comp rx=0 tx=0).stop 2024-03-14T13:14:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 msgr2=0x7faefc113de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faeec009150 tx=0x7faeec009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 shutdown_connections 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faee806c7c0 0x7faee806ec70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faefc0fb3b0 0x7faefc1138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 --2- 172.21.15.111:0/3743423543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc102f70 0x7faefc113de0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 >> 172.21.15.111:0/3743423543 conn(0x7faefc072310 msgr2=0x7faefc0f9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 shutdown_connections 2024-03-14T13:14:23.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.512+0000 7faf0343f700 1 -- 172.21.15.111:0/3743423543 wait complete. 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- 172.21.15.111:0/3906949438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 msgr2=0x7fe50c0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 --2- 172.21.15.111:0/3906949438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fe260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc004940 tx=0x7fe4fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- 172.21.15.111:0/3906949438 shutdown_connections 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 --2- 172.21.15.111:0/3906949438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 --2- 172.21.15.111:0/3906949438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fe260 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- 172.21.15.111:0/3906949438 >> 172.21.15.111:0/3906949438 conn(0x7fe50c072bc0 msgr2=0x7fe50c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- 172.21.15.111:0/3906949438 shutdown_connections 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- 172.21.15.111:0/3906949438 wait complete. 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 Processor -- start 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.613+0000 7fe513b22700 1 -- start start 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c0fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c0fc8b0 con 0x7fe50c0fde90 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c0fc9f0 con 0x7fe50c0fe830 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42288/0 (socket says 172.21.15.111:42288) 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 -- 172.21.15.111:0/3362130821 learned_addr learned my addr 172.21.15.111:0/3362130821 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5110bd700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c0fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 -- 172.21.15.111:0/3362130821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 msgr2=0x7fe50c0fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c0fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 -- 172.21.15.111:0/3362130821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc005040 con 0x7fe50c0fde90 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5110bd700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5118be700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0093b0 tx=0x7fe4fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc00e040 con 0x7fe50c0fde90 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc004290 con 0x7fe50c0fde90 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe50c119df0 con 0x7fe50c0fde90 2024-03-14T13:14:23.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4fc01c3f0 con 0x7fe50c0fde90 2024-03-14T13:14:23.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.614+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe50c11a250 con 0x7fe50c0fde90 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.615+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50c065e30 con 0x7fe50c0fde90 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.616+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe4fc013790 con 0x7fe50c0fde90 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.616+0000 7fe5027fc700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 0x7fe4f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.616+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4fc08cd20 con 0x7fe50c0fde90 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.616+0000 7fe5110bd700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 0x7fe4f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.617+0000 7fe5110bd700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 0x7fe4f806ec80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0fd690 tx=0x7fe508008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.621+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe4fc056fa0 con 0x7fe50c0fde90 2024-03-14T13:14:23.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.821+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe50c04bf70 con 0x7fe4f806c7d0 2024-03-14T13:14:23.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.831+0000 7fe5027fc700 1 -- 172.21.15.111:0/3362130821 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fe508009790 con 0x7fe4f806c7d0 2024-03-14T13:14:23.830 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:14:23.830 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (10m) 4m ago 22m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:14:23.830 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (8m) 4m ago 22m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (8m) 3m ago 20m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (10m) 4m ago 21m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (4m) 4m ago 15m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (4m) 4m ago 16m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (3m) 3m ago 15m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (3m) 3m ago 16m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (13m) 4m ago 23m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (11m) 3m ago 20m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (9m) 4m ago 23m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (8m) 3m ago 20m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (11m) 4m ago 22m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (11m) 3m ago 20m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (7m) 4m ago 19m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:14:23.831 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (7m) 4m ago 19m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (6m) 4m ago 18m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (5m) 3m ago 18m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (5m) 3m ago 18m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (4m) 3m ago 17m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (11m) 4m ago 21m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 msgr2=0x7fe4f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.832 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 0x7fe4f806ec80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0fd690 tx=0x7fe508008040 comp rx=0 tx=0).stop 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 msgr2=0x7fe50c0fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0093b0 tx=0x7fe4fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 shutdown_connections 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4f806c7d0 0x7fe4f806ec80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe50c0fde90 0x7fe50c0fbd50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 --2- 172.21.15.111:0/3362130821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe50c0fe830 0x7fe50c0fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 >> 172.21.15.111:0/3362130821 conn(0x7fe50c072bc0 msgr2=0x7fe50c1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 shutdown_connections 2024-03-14T13:14:23.833 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.833+0000 7fe513b22700 1 -- 172.21.15.111:0/3362130821 wait complete. 2024-03-14T13:14:23.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:23 smithi111 ceph-mon[145314]: pgmap v308: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- 172.21.15.111:0/1516120300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 msgr2=0x7f04d4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 --2- 172.21.15.111:0/1516120300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d4102140 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f04d0004940 tx=0x7f04d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- 172.21.15.111:0/1516120300 shutdown_connections 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 --2- 172.21.15.111:0/1516120300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 --2- 172.21.15.111:0/1516120300 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d4102140 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- 172.21.15.111:0/1516120300 >> 172.21.15.111:0/1516120300 conn(0x7f04d4072bc0 msgr2=0x7f04d4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- 172.21.15.111:0/1516120300 shutdown_connections 2024-03-14T13:14:23.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- 172.21.15.111:0/1516120300 wait complete. 2024-03-14T13:14:23.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 Processor -- start 2024-03-14T13:14:23.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 -- start start 2024-03-14T13:14:23.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.933+0000 7f04db7d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04db7d5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d4112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04db7d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d4113530 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04db7d5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d410c920 con 0x7f04d4102710 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42306/0 (socket says 172.21.15.111:42306) 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 -- 172.21.15.111:0/2077652934 learned_addr learned my addr 172.21.15.111:0/2077652934 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d8d70700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d4112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 -- 172.21.15.111:0/2077652934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 msgr2=0x7f04d4112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d4112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 -- 172.21.15.111:0/2077652934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0005040 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d8d70700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d4112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04d9571700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b3b0 tx=0x7f04d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04d000e040 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04d0013420 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d410cba0 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04d001c630 con 0x7f04d4101d70 2024-03-14T13:14:23.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.934+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d410d160 con 0x7f04d4101d70 2024-03-14T13:14:23.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.935+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40ff080 con 0x7f04d4101d70 2024-03-14T13:14:23.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.936+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f04d0013970 con 0x7f04d4101d70 2024-03-14T13:14:23.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.936+0000 7f04c9ffb700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 0x7f04c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:23.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.936+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f04d008cbb0 con 0x7f04d4101d70 2024-03-14T13:14:23.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.936+0000 7f04d8d70700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 0x7f04c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:23.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.937+0000 7f04d8d70700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 0x7f04c006ec70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f04c4007fd0 tx=0x7f04c4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:23.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:23.941+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f04d0093080 con 0x7f04d4101d70 2024-03-14T13:14:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:23 smithi137 ceph-mon[108145]: pgmap v308: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:24.213 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.214+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f04d410dff0 con 0x7f04d4101d70 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.214+0000 7f04c9ffb700 1 -- 172.21.15.111:0/2077652934 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f04d005a980 con 0x7f04d4101d70 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:14:24.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:14:24.215 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 msgr2=0x7f04c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 0x7f04c006ec70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f04c4007fd0 tx=0x7f04c4009040 comp rx=0 tx=0).stop 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 msgr2=0x7f04d41129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f04d000b3b0 tx=0x7f04d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 shutdown_connections 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f04c006c7c0 0x7f04c006ec70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d4101d70 0x7f04d41129a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 --2- 172.21.15.111:0/2077652934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04d4102710 0x7f04d4112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.216+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 >> 172.21.15.111:0/2077652934 conn(0x7f04d4072bc0 msgr2=0x7f04d40fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.217+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 shutdown_connections 2024-03-14T13:14:24.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.217+0000 7f04db7d5700 1 -- 172.21.15.111:0/2077652934 wait complete. 2024-03-14T13:14:24.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- 172.21.15.111:0/2977051467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208101cf0 msgr2=0x7f12081020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 --2- 172.21.15.111:0/2977051467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208101cf0 0x7f12081020c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11fc004940 tx=0x7f11fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- 172.21.15.111:0/2977051467 shutdown_connections 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 --2- 172.21.15.111:0/2977051467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f12080fbca0 0x7f12080fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 --2- 172.21.15.111:0/2977051467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1208101cf0 0x7f12081020c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- 172.21.15.111:0/2977051467 >> 172.21.15.111:0/2977051467 conn(0x7f1208072bc0 msgr2=0x7f1208072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- 172.21.15.111:0/2977051467 shutdown_connections 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- 172.21.15.111:0/2977051467 wait complete. 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 Processor -- start 2024-03-14T13:14:24.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.318+0000 7f120f005700 1 -- start start 2024-03-14T13:14:24.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120f005700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120f005700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1208101cf0 0x7f120810ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120f005700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120810f560 con 0x7f12080fbca0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120f005700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120810f6a0 con 0x7f1208101cf0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42326/0 (socket says 172.21.15.111:42326) 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 -- 172.21.15.111:0/2977433669 learned_addr learned my addr 172.21.15.111:0/2977433669 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f1207fff700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1208101cf0 0x7f120810ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 -- 172.21.15.111:0/2977433669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1208101cf0 msgr2=0x7f120810ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1208101cf0 0x7f120810ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 -- 172.21.15.111:0/2977433669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc005040 con 0x7f12080fbca0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f120cda1700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009180 tx=0x7f11fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.319+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11fc00e040 con 0x7f12080fbca0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.320+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1208117a40 con 0x7f12080fbca0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.320+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11fc013420 con 0x7f12080fbca0 2024-03-14T13:14:24.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.320+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11fc01c630 con 0x7f12080fbca0 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.320+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1208117f50 con 0x7f12080fbca0 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.320+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12080fd390 con 0x7f12080fbca0 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.321+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f11fc0139e0 con 0x7f12080fbca0 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.321+0000 7f12057fa700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 0x7f11f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.321+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f11fc08cc80 con 0x7f12080fbca0 2024-03-14T13:14:24.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.322+0000 7f1207fff700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 0x7f11f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.322+0000 7f1207fff700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 0x7f11f806ec70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1208066000 tx=0x7f11f4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.326+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f11fc056f00 con 0x7f12080fbca0 2024-03-14T13:14:24.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.561+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f120804bed0 con 0x7f12080fbca0 2024-03-14T13:14:24.560 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.561+0000 7f12057fa700 1 -- 172.21.15.111:0/2977433669 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f11fc05a520 con 0x7f12080fbca0 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:14:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:14:24.562 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:24.563 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:24.564 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:24.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 msgr2=0x7f11f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 0x7f11f806ec70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1208066000 tx=0x7f11f4009040 comp rx=0 tx=0).stop 2024-03-14T13:14:24.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 msgr2=0x7f120810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009180 tx=0x7f11fc009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 shutdown_connections 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f11f806c7c0 0x7f11f806ec70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12080fbca0 0x7f120810e9d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 --2- 172.21.15.111:0/2977433669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1208101cf0 0x7f120810ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 >> 172.21.15.111:0/2977433669 conn(0x7f1208072bc0 msgr2=0x7f12081049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 shutdown_connections 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.563+0000 7f120f005700 1 -- 172.21.15.111:0/2977433669 wait complete. 2024-03-14T13:14:24.565 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:14:24.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2499148930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 msgr2=0x7fd4f80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2499148930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f80fe260 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec004940 tx=0x7fd4ec009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2499148930 shutdown_connections 2024-03-14T13:14:24.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2499148930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 0x7fd4f8106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2499148930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f80fe260 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2499148930 >> 172.21.15.111:0/2499148930 conn(0x7fd4f8072bc0 msgr2=0x7fd4f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.664+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2499148930 shutdown_connections 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2499148930 wait complete. 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 Processor -- start 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 -- start start 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 0x7fd4f811a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f811acb0 con 0x7fd4f80fde90 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.665+0000 7fd4ff23d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8114080 con 0x7fd4f80fe830 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42336/0 (socket says 172.21.15.111:42336) 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 -- 172.21.15.111:0/2527127168 learned_addr learned my addr 172.21.15.111:0/2527127168 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:24.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4f7fff700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 0x7fd4f811a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 -- 172.21.15.111:0/2527127168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 msgr2=0x7fd4f811a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 0x7fd4f811a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 -- 172.21.15.111:0/2527127168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec005040 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4fcfd9700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00b3b0 tx=0x7fd4ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4ec00e040 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f8114300 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4ec013420 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4ec01c5d0 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.666+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f8114840 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.667+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8102610 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.668+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd4ec013590 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.668+0000 7fd4f57fa700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 0x7fd4e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.668+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd4ec08cd90 con 0x7fd4f80fde90 2024-03-14T13:14:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.668+0000 7fd4f7fff700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 0x7fd4e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.668+0000 7fd4f7fff700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 0x7fd4e806ec70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4f81159c0 tx=0x7fd4e4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.673+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd4ec05a9a0 con 0x7fd4f80fde90 2024-03-14T13:14:24.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.886+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4f8109e30 con 0x7fd4e806c7c0 2024-03-14T13:14:24.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.887+0000 7fd4f57fa700 1 -- 172.21.15.111:0/2527127168 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fd4f8109e30 con 0x7fd4e806c7c0 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:14:24.887 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:14:24.888 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:14:24.888 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 msgr2=0x7fd4e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 0x7fd4e806ec70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4f81159c0 tx=0x7fd4e4009ee0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 msgr2=0x7fd4f811a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00b3b0 tx=0x7fd4ec009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 shutdown_connections 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4e806c7c0 0x7fd4e806ec70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4f80fde90 0x7fd4f811a120 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 --2- 172.21.15.111:0/2527127168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4f80fe830 0x7fd4f811a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 >> 172.21.15.111:0/2527127168 conn(0x7fd4f8072bc0 msgr2=0x7fd4f80fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 shutdown_connections 2024-03-14T13:14:24.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.889+0000 7fd4ff23d700 1 -- 172.21.15.111:0/2527127168 wait complete. 2024-03-14T13:14:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:24 smithi111 ceph-mon[145314]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:24 smithi111 ceph-mon[145314]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:24 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2077652934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:14:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:24.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- 172.21.15.111:0/3890140683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 msgr2=0x7fd5880fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/3890140683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd5880fbd80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd57c004940 tx=0x7fd57c009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- 172.21.15.111:0/3890140683 shutdown_connections 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/3890140683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd5880fbd80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/3890140683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 0x7fd588102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- 172.21.15.111:0/3890140683 >> 172.21.15.111:0/3890140683 conn(0x7fd588072bc0 msgr2=0x7fd588072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- 172.21.15.111:0/3890140683 shutdown_connections 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- 172.21.15.111:0/3890140683 wait complete. 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 Processor -- start 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- start start 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 0x7fd58810c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58810d210 con 0x7fd588102710 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.989+0000 7fd58cbd1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58810d380 con 0x7fd588101d70 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42362/0 (socket says 172.21.15.111:42362) 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 -- 172.21.15.111:0/1874719651 learned_addr learned my addr 172.21.15.111:0/1874719651 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd58659c700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 0x7fd58810c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 -- 172.21.15.111:0/1874719651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 msgr2=0x7fd58810c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 0x7fd58810c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 -- 172.21.15.111:0/1874719651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c005040 con 0x7fd588102710 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd585d9b700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd57c00e070 con 0x7fd588102710 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd57c0093b0 con 0x7fd588102710 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58810d600 con 0x7fd588102710 2024-03-14T13:14:24.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd57c01ccb0 con 0x7fd588102710 2024-03-14T13:14:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.990+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58811a150 con 0x7fd588102710 2024-03-14T13:14:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.991+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5880ff080 con 0x7fd588102710 2024-03-14T13:14:24.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.992+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd57c01c490 con 0x7fd588102710 2024-03-14T13:14:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.992+0000 7fd576ffd700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 0x7fd57006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.992+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd57c08cd20 con 0x7fd588102710 2024-03-14T13:14:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.992+0000 7fd58659c700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 0x7fd57006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:24.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.992+0000 7fd58659c700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 0x7fd57006ec70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b80 tx=0x7fd57800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:24.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:24.997+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd57c05aa30 con 0x7fd588102710 2024-03-14T13:14:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:24 smithi137 ceph-mon[108145]: from='client.34624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:24 smithi137 ceph-mon[108145]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:24 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2077652934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:14:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.272+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd58810d600 con 0x7fd588102710 2024-03-14T13:14:25.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.272+0000 7fd576ffd700 1 -- 172.21.15.111:0/1874719651 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fd58810d600 con 0x7fd588102710 2024-03-14T13:14:25.272 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:14:25.272 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:14:25.272 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:14:25.272 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:14:25.272 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:14:25.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 msgr2=0x7fd57006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:25.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 0x7fd57006ec70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b80 tx=0x7fd57800b040 comp rx=0 tx=0).stop 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 msgr2=0x7fd58810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd57c000c00 tx=0x7fd57c009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 shutdown_connections 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd57006c7c0 0x7fd57006ec70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd588101d70 0x7fd58810c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 --2- 172.21.15.111:0/1874719651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd588102710 0x7fd58810ccd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 >> 172.21.15.111:0/1874719651 conn(0x7fd588072bc0 msgr2=0x7fd5880faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 shutdown_connections 2024-03-14T13:14:25.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:25.274+0000 7fd58cbd1700 1 -- 172.21.15.111:0/1874719651 wait complete. 2024-03-14T13:14:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:25 smithi111 ceph-mon[145314]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:25 smithi111 ceph-mon[145314]: pgmap v309: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:25 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2977433669' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:14:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:25 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1874719651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:14:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:25 smithi137 ceph-mon[108145]: from='client.34632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:25 smithi137 ceph-mon[108145]: pgmap v309: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:25 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2977433669' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:14:26.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:25 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1874719651' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:14:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:26 smithi111 ceph-mon[145314]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:27.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:26 smithi137 ceph-mon[108145]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:27 smithi111 ceph-mon[145314]: pgmap v310: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:27 smithi137 ceph-mon[108145]: pgmap v310: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:28.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:28 smithi111 ceph-mon[145314]: pgmap v311: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:29.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:28 smithi137 ceph-mon[108145]: pgmap v311: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:31 smithi137 ceph-mon[108145]: pgmap v312: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:31 smithi111 ceph-mon[145314]: pgmap v312: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:33 smithi137 ceph-mon[108145]: pgmap v313: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:33 smithi111 ceph-mon[145314]: pgmap v313: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:35.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:35 smithi137 ceph-mon[108145]: pgmap v314: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:35 smithi111 ceph-mon[145314]: pgmap v314: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:37.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:37 smithi137 ceph-mon[108145]: pgmap v315: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:37 smithi111 ceph-mon[145314]: pgmap v315: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:39 smithi137 ceph-mon[108145]: pgmap v316: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:39.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:39 smithi111 ceph-mon[145314]: pgmap v316: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:41.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:41 smithi137 ceph-mon[108145]: pgmap v317: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:41 smithi111 ceph-mon[145314]: pgmap v317: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:43 smithi137 ceph-mon[108145]: pgmap v318: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:43 smithi111 ceph-mon[145314]: pgmap v318: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:45.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:45 smithi137 ceph-mon[108145]: pgmap v319: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:45 smithi111 ceph-mon[145314]: pgmap v319: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:47.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:47 smithi137 ceph-mon[108145]: pgmap v320: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:47 smithi111 ceph-mon[145314]: pgmap v320: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:49 smithi111 ceph-mon[145314]: pgmap v321: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:49 smithi137 ceph-mon[108145]: pgmap v321: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:51 smithi111 ceph-mon[145314]: pgmap v322: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:51 smithi137 ceph-mon[108145]: pgmap v322: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:53 smithi111 ceph-mon[145314]: pgmap v323: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:53 smithi137 ceph-mon[108145]: pgmap v323: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:14:55.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.391+0000 7f26f5398700 1 -- 172.21.15.111:0/2833670683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 msgr2=0x7f26f01033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 --2- 172.21.15.111:0/2833670683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f01033a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f26e0004940 tx=0x7f26e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 -- 172.21.15.111:0/2833670683 shutdown_connections 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 --2- 172.21.15.111:0/2833670683 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 0x7f26f00fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 --2- 172.21.15.111:0/2833670683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f01033a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 -- 172.21.15.111:0/2833670683 >> 172.21.15.111:0/2833670683 conn(0x7f26f0071530 msgr2=0x7f26f0071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 -- 172.21.15.111:0/2833670683 shutdown_connections 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 -- 172.21.15.111:0/2833670683 wait complete. 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 Processor -- start 2024-03-14T13:14:55.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.392+0000 7f26f5398700 1 -- start start 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26f5398700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 0x7f26f01138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26f5398700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26f5398700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f0114510 con 0x7f26f0102fd0 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26f5398700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f010da80 con 0x7f26f00fb540 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39138/0 (socket says 172.21.15.111:39138) 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 -- 172.21.15.111:0/3840546984 learned_addr learned my addr 172.21.15.111:0/3840546984 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26eeffd700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 0x7f26f01138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 -- 172.21.15.111:0/3840546984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 msgr2=0x7f26f01138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 0x7f26f01138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 -- 172.21.15.111:0/3840546984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0005040 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26ee7fc700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f26e400b790 tx=0x7f26e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.393+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e400f6f0 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.394+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f010dd60 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.394+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26e400fd00 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.394+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26e40103a0 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.394+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f010e2a0 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.395+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f0100270 con 0x7f26f0102fd0 2024-03-14T13:14:55.396 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.395+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f26e4010500 con 0x7f26f0102fd0 2024-03-14T13:14:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.395+0000 7f26d77fe700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 0x7f26d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.395+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f26e408b8e0 con 0x7f26f0102fd0 2024-03-14T13:14:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.395+0000 7f26eeffd700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 0x7f26d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.397 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.396+0000 7f26eeffd700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 0x7f26d806ec70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f26e0004f90 tx=0x7f26e0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:55.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.400+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f26e4059660 con 0x7f26f0102fd0 2024-03-14T13:14:55.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.611+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f26f010ef40 con 0x7f26d806c7c0 2024-03-14T13:14:55.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.612+0000 7f26d77fe700 1 -- 172.21.15.111:0/3840546984 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f26f010ef40 con 0x7f26d806c7c0 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 msgr2=0x7f26d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 0x7f26d806ec70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f26e0004f90 tx=0x7f26e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 msgr2=0x7f26f0113e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f26e400b790 tx=0x7f26e4009250 comp rx=0 tx=0).stop 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 shutdown_connections 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f26d806c7c0 0x7f26d806ec70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26f00fb540 0x7f26f01138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 --2- 172.21.15.111:0/3840546984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f0102fd0 0x7f26f0113e30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 >> 172.21.15.111:0/3840546984 conn(0x7f26f0071530 msgr2=0x7f26f00fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 shutdown_connections 2024-03-14T13:14:55.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.614+0000 7f26f5398700 1 -- 172.21.15.111:0/3840546984 wait complete. 2024-03-14T13:14:55.625 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:14:55.715 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 -- 172.21.15.111:0/1299157036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7468102e00 msgr2=0x7f74681031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.715 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 --2- 172.21.15.111:0/1299157036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7468102e00 0x7f74681031d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7458004940 tx=0x7f7458009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 -- 172.21.15.111:0/1299157036 shutdown_connections 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 --2- 172.21.15.111:0/1299157036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74681037a0 0x7f74680fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 --2- 172.21.15.111:0/1299157036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7468102e00 0x7f74681031d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 -- 172.21.15.111:0/1299157036 >> 172.21.15.111:0/1299157036 conn(0x7f74680f89b0 msgr2=0x7f74680fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 -- 172.21.15.111:0/1299157036 shutdown_connections 2024-03-14T13:14:55.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.715+0000 7f746d2e1700 1 -- 172.21.15.111:0/1299157036 wait complete. 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 Processor -- start 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 -- start start 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7468102e00 0x7f74681138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7468114440 con 0x7f74681037a0 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f746d2e1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746810da40 con 0x7f7468102e00 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39152/0 (socket says 172.21.15.111:39152) 2024-03-14T13:14:55.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 -- 172.21.15.111:0/456707847 learned_addr learned my addr 172.21.15.111:0/456707847 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f7466ffd700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7468102e00 0x7f74681138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 -- 172.21.15.111:0/456707847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7468102e00 msgr2=0x7f74681138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7468102e00 0x7f74681138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 -- 172.21.15.111:0/456707847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7458005040 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.716+0000 7f74667fc700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c800 tx=0x7f745c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c00a760 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f745c00ad70 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746810dd20 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f745c0113a0 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746810e2e0 con 0x7f74681037a0 2024-03-14T13:14:55.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.717+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74680fe550 con 0x7f74681037a0 2024-03-14T13:14:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.718+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f745c012400 con 0x7f74681037a0 2024-03-14T13:14:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.719+0000 7f744f7fe700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 0x7f745006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.719+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f745c08c580 con 0x7f74681037a0 2024-03-14T13:14:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.719+0000 7f7466ffd700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 0x7f745006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:55.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.719+0000 7f7466ffd700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 0x7f745006ec70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7458004f90 tx=0x7f7458009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:55.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.722+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f745c05a350 con 0x7f74681037a0 2024-03-14T13:14:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:55 smithi111 ceph-mon[145314]: pgmap v324: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:55.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.932+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7468109e60 con 0x7f745006c7c0 2024-03-14T13:14:55.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.933+0000 7f744f7fe700 1 -- 172.21.15.111:0/456707847 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7458007e00 con 0x7f745006c7c0 2024-03-14T13:14:55.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 msgr2=0x7f745006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 0x7f745006ec70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7458004f90 tx=0x7f7458009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:55.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 msgr2=0x7f7468113df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:55.935 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f745c00c800 tx=0x7f745c004470 comp rx=0 tx=0).stop 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 shutdown_connections 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f745006c7c0 0x7f745006ec70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7468102e00 0x7f74681138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 --2- 172.21.15.111:0/456707847 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74681037a0 0x7f7468113df0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 >> 172.21.15.111:0/456707847 conn(0x7f74680f89b0 msgr2=0x7f74680f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 shutdown_connections 2024-03-14T13:14:55.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:55.935+0000 7f746d2e1700 1 -- 172.21.15.111:0/456707847 wait complete. 2024-03-14T13:14:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:55 smithi137 ceph-mon[108145]: pgmap v324: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 -- 172.21.15.111:0/3569179167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4000fbb70 msgr2=0x7ff4000fbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 --2- 172.21.15.111:0/3569179167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4000fbb70 0x7ff4000fbfe0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0004940 tx=0x7ff3f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 -- 172.21.15.111:0/3569179167 shutdown_connections 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 --2- 172.21.15.111:0/3569179167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4000fbb70 0x7ff4000fbfe0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 --2- 172.21.15.111:0/3569179167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff400101ba0 0x7ff400101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 -- 172.21.15.111:0/3569179167 >> 172.21.15.111:0/3569179167 conn(0x7ff400071b70 msgr2=0x7ff400071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 -- 172.21.15.111:0/3569179167 shutdown_connections 2024-03-14T13:14:56.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 -- 172.21.15.111:0/3569179167 wait complete. 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.035+0000 7ff3fffff700 1 Processor -- start 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fffff700 1 -- start start 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fffff700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4000fbb70 0x7ff40010c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40010d0d0 con 0x7ff400101ba0 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fffff700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40010d240 con 0x7ff4000fbb70 2024-03-14T13:14:56.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39160/0 (socket says 172.21.15.111:39160) 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 -- 172.21.15.111:0/2402509089 learned_addr learned my addr 172.21.15.111:0/2402509089 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3feffd700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4000fbb70 0x7ff40010c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 -- 172.21.15.111:0/2402509089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4000fbb70 msgr2=0x7ff40010c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4000fbb70 0x7ff40010c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 -- 172.21.15.111:0/2402509089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0005040 con 0x7ff400101ba0 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.036+0000 7ff3fe7fc700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009180 tx=0x7ff3f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.037+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f000e040 con 0x7ff400101ba0 2024-03-14T13:14:56.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.037+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3f0004290 con 0x7ff400101ba0 2024-03-14T13:14:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.037+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40010d470 con 0x7ff400101ba0 2024-03-14T13:14:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.037+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f0013e20 con 0x7ff400101ba0 2024-03-14T13:14:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.037+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40011a040 con 0x7ff400101ba0 2024-03-14T13:14:56.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.038+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4000fd200 con 0x7ff400101ba0 2024-03-14T13:14:56.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.038+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff3f0025400 con 0x7ff400101ba0 2024-03-14T13:14:56.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.038+0000 7ff3e77fe700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 0x7ff3e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.039+0000 7ff3feffd700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 0x7ff3e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.039+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff3f008cb40 con 0x7ff400101ba0 2024-03-14T13:14:56.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.039+0000 7ff3feffd700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 0x7ff3e806ec70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40041d0 tx=0x7ff3f40043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.043+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff3f005a980 con 0x7ff400101ba0 2024-03-14T13:14:56.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.243+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff40004bed0 con 0x7ff3e806c7c0 2024-03-14T13:14:56.253 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.253+0000 7ff3e77fe700 1 -- 172.21.15.111:0/2402509089 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7ff40004bed0 con 0x7ff3e806c7c0 2024-03-14T13:14:56.253 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:14:56.253 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (11m) 4m ago 22m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (8m) 4m ago 22m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (8m) 4m ago 21m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (10m) 4m ago 22m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (4m) 4m ago 16m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (5m) 4m ago 16m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (4m) 4m ago 16m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (4m) 4m ago 16m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (13m) 4m ago 24m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (12m) 4m ago 21m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (9m) 4m ago 24m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (9m) 4m ago 21m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (12m) 4m ago 22m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (11m) 4m ago 21m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:14:56.254 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (8m) 4m ago 20m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (7m) 4m ago 20m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (7m) 4m ago 19m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (6m) 4m ago 19m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (5m) 4m ago 18m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (5m) 4m ago 18m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:14:56.255 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (11m) 4m ago 21m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 msgr2=0x7ff3e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 0x7ff3e806ec70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40041d0 tx=0x7ff3f40043c0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 msgr2=0x7ff40010cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009180 tx=0x7ff3f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 shutdown_connections 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff3e806c7c0 0x7ff3e806ec70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4000fbb70 0x7ff40010c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 --2- 172.21.15.111:0/2402509089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff400101ba0 0x7ff40010cb90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.255+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 >> 172.21.15.111:0/2402509089 conn(0x7ff400071b70 msgr2=0x7ff400104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.256+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 shutdown_connections 2024-03-14T13:14:56.256 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.256+0000 7ff3fffff700 1 -- 172.21.15.111:0/2402509089 wait complete. 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 -- 172.21.15.111:0/92711202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 msgr2=0x7f1af40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 --2- 172.21.15.111:0/92711202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af40fbd80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0004940 tx=0x7f1ae0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 -- 172.21.15.111:0/92711202 shutdown_connections 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 --2- 172.21.15.111:0/92711202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af40fbd80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 --2- 172.21.15.111:0/92711202 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 0x7f1af4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 -- 172.21.15.111:0/92711202 >> 172.21.15.111:0/92711202 conn(0x7f1af4072bc0 msgr2=0x7f1af4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 -- 172.21.15.111:0/92711202 shutdown_connections 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 -- 172.21.15.111:0/92711202 wait complete. 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.355+0000 7f1afabbc700 1 Processor -- start 2024-03-14T13:14:56.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1afabbc700 1 -- start start 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1afabbc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 0x7f1af41129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1afabbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1afabbc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4113550 con 0x7f1af4102710 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1afabbc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af410c940 con 0x7f1af4101d70 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39178/0 (socket says 172.21.15.111:39178) 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 -- 172.21.15.111:0/2318905036 learned_addr learned my addr 172.21.15.111:0/2318905036 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af8958700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 0x7f1af41129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 -- 172.21.15.111:0/2318905036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 msgr2=0x7f1af41129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 0x7f1af41129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 -- 172.21.15.111:0/2318905036 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0005040 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.356+0000 7f1af3fff700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00093b0 tx=0x7f1ae0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.357+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae000e040 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.357+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ae0013420 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.357+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ae001c630 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.357+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af410cbc0 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.357+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af410d180 con 0x7f1af4102710 2024-03-14T13:14:56.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.358+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af40ff080 con 0x7f1af4102710 2024-03-14T13:14:56.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.358+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1ae00139e0 con 0x7f1af4102710 2024-03-14T13:14:56.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.359+0000 7f1af17fa700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 0x7f1ae406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.359+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1ae008cb70 con 0x7f1af4102710 2024-03-14T13:14:56.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.359+0000 7f1af8958700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 0x7f1ae406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.359+0000 7f1af8958700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 0x7f1ae406ec70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8007fd0 tx=0x7f1ae8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.363+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1ae005a880 con 0x7f1af4102710 2024-03-14T13:14:56.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.640+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1af410e010 con 0x7f1af4102710 2024-03-14T13:14:56.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.640+0000 7f1af17fa700 1 -- 172.21.15.111:0/2318905036 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f1ae005a410 con 0x7f1af4102710 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:14:56.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:14:56.642 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:14:56.642 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:14:56.642 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:14:56.642 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.642+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 msgr2=0x7f1ae406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.642+0000 7f1afabbc700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 0x7f1ae406ec70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8007fd0 tx=0x7f1ae8009040 comp rx=0 tx=0).stop 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.642+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 msgr2=0x7f1af4112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.642+0000 7f1afabbc700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00093b0 tx=0x7f1ae0009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 shutdown_connections 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1ae406c7c0 0x7f1ae406ec70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af4101d70 0x7f1af41129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 --2- 172.21.15.111:0/2318905036 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1af4102710 0x7f1af4112f00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 >> 172.21.15.111:0/2318905036 conn(0x7f1af4072bc0 msgr2=0x7f1af40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 shutdown_connections 2024-03-14T13:14:56.644 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.643+0000 7f1afabbc700 1 -- 172.21.15.111:0/2318905036 wait complete. 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 -- 172.21.15.111:0/381787467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070c10 msgr2=0x7f7fa0109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/381787467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070c10 0x7f7fa0109180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c004970 tx=0x7f7f8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 -- 172.21.15.111:0/381787467 shutdown_connections 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/381787467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070c10 0x7f7fa0109180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/381787467 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070300 0x7f7fa00706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 -- 172.21.15.111:0/381787467 >> 172.21.15.111:0/381787467 conn(0x7f7fa00f89a0 msgr2=0x7f7fa00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 -- 172.21.15.111:0/381787467 shutdown_connections 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.742+0000 7f7fa6f84700 1 -- 172.21.15.111:0/381787467 wait complete. 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 Processor -- start 2024-03-14T13:14:56.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 -- start start 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070c10 0x7f7fa01161d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa0116820 con 0x7f7fa0070300 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa6f84700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa010fbf0 con 0x7f7fa0070c10 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa4d20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa4d20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39190/0 (socket says 172.21.15.111:39190) 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.743+0000 7f7fa4d20700 1 -- 172.21.15.111:0/595442643 learned_addr learned my addr 172.21.15.111:0/595442643 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7f9ffff700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070c10 0x7f7fa01161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa4d20700 1 -- 172.21.15.111:0/595442643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070c10 msgr2=0x7f7fa01161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa4d20700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070c10 0x7f7fa01161d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa4d20700 1 -- 172.21.15.111:0/595442643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f8c005040 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa4d20700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c810 tx=0x7f7f94004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f94012810 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f94012e20 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9400a370 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa010fed0 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.744+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa0110410 con 0x7f7fa0070300 2024-03-14T13:14:56.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.745+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0071a40 con 0x7f7fa0070300 2024-03-14T13:14:56.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.745+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7f94011780 con 0x7f7fa0070300 2024-03-14T13:14:56.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.746+0000 7f7f9d7fa700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 0x7f7f9006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:56.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.746+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7f9408bea0 con 0x7f7fa0070300 2024-03-14T13:14:56.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.746+0000 7f7f9ffff700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 0x7f7f9006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:56.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.746+0000 7f7f9ffff700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 0x7f7f9006ec70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c004540 tx=0x7f7f8c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:56.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.750+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7f94059b70 con 0x7f7fa0070300 2024-03-14T13:14:56.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.986+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7fa004bed0 con 0x7f7fa0070300 2024-03-14T13:14:56.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.986+0000 7f7f9d7fa700 1 -- 172.21.15.111:0/595442643 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f7f94016750 con 0x7f7fa0070300 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:14:56.987 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:14:56.988 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:56.989 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:14:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 msgr2=0x7f7f9006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 0x7f7f9006ec70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c004540 tx=0x7f7f8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 msgr2=0x7f7fa0115c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c810 tx=0x7f7f94004470 comp rx=0 tx=0).stop 2024-03-14T13:14:56.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 shutdown_connections 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7f9006c7c0 0x7f7f9006ec70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7fa0070300 0x7f7fa0115c90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 --2- 172.21.15.111:0/595442643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa0070c10 0x7f7fa01161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 >> 172.21.15.111:0/595442643 conn(0x7f7fa00f89a0 msgr2=0x7f7fa00f9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 shutdown_connections 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:56.988+0000 7f7fa6f84700 1 -- 172.21.15.111:0/595442643 wait complete. 2024-03-14T13:14:56.991 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:14:57.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 -- 172.21.15.111:0/2398668525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 msgr2=0x7f3a381031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 --2- 172.21.15.111:0/2398668525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a381031f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3a28004940 tx=0x7f3a28009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 -- 172.21.15.111:0/2398668525 shutdown_connections 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 --2- 172.21.15.111:0/2398668525 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a380fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 --2- 172.21.15.111:0/2398668525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a381031f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 -- 172.21.15.111:0/2398668525 >> 172.21.15.111:0/2398668525 conn(0x7f3a380f89d0 msgr2=0x7f3a380fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 -- 172.21.15.111:0/2398668525 shutdown_connections 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 -- 172.21.15.111:0/2398668525 wait complete. 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.089+0000 7f3a40614700 1 Processor -- start 2024-03-14T13:14:57.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 -- start start 2024-03-14T13:14:57.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a38070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38070d90 con 0x7f3a38102e20 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38070ed0 con 0x7f3a381037c0 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39206/0 (socket says 172.21.15.111:39206) 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 -- 172.21.15.111:0/296520636 learned_addr learned my addr 172.21.15.111:0/296520636 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3dbaf700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a38070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 -- 172.21.15.111:0/296520636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 msgr2=0x7f3a38070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a38070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 -- 172.21.15.111:0/296520636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a28005040 con 0x7f3a38102e20 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3dbaf700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a3e3b0700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009180 tx=0x7f3a28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:57.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a2800e040 con 0x7f3a38102e20 2024-03-14T13:14:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38071120 con 0x7f3a38102e20 2024-03-14T13:14:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a28004290 con 0x7f3a38102e20 2024-03-14T13:14:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.090+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a2801c3f0 con 0x7f3a38102e20 2024-03-14T13:14:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.091+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3811a1b0 con 0x7f3a38102e20 2024-03-14T13:14:57.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.091+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a380fe570 con 0x7f3a38102e20 2024-03-14T13:14:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.092+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3a28013790 con 0x7f3a38102e20 2024-03-14T13:14:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.092+0000 7f3a2effd700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 0x7f3a2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.092+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3a2808cd30 con 0x7f3a38102e20 2024-03-14T13:14:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.092+0000 7f3a3dbaf700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 0x7f3a2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.093+0000 7f3a3dbaf700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 0x7f3a2406ec70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a34007fd0 tx=0x7f3a34009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:57.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.097+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3a28056fb0 con 0x7f3a38102e20 2024-03-14T13:14:57.306 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.306+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a380fe570 con 0x7f3a2406c7c0 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.307+0000 7f3a2effd700 1 -- 172.21.15.111:0/296520636 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f3a380fe570 con 0x7f3a2406c7c0 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:14:57.307 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:14:57.308 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:14:57.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 msgr2=0x7f3a2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 0x7f3a2406ec70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3a34007fd0 tx=0x7f3a34009f60 comp rx=0 tx=0).stop 2024-03-14T13:14:57.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 msgr2=0x7f3a380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009180 tx=0x7f3a28009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 shutdown_connections 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f3a2406c7c0 0x7f3a2406ec70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a38102e20 0x7f3a380702c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 --2- 172.21.15.111:0/296520636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a381037c0 0x7f3a38070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 >> 172.21.15.111:0/296520636 conn(0x7f3a380f89d0 msgr2=0x7f3a38105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 shutdown_connections 2024-03-14T13:14:57.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.309+0000 7f3a40614700 1 -- 172.21.15.111:0/296520636 wait complete. 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 -- 172.21.15.111:0/3811686379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 msgr2=0x7fdd2c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 --2- 172.21.15.111:0/3811686379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c103320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd14004940 tx=0x7fdd14009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 -- 172.21.15.111:0/3811686379 shutdown_connections 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 --2- 172.21.15.111:0/3811686379 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 --2- 172.21.15.111:0/3811686379 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c103320 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 -- 172.21.15.111:0/3811686379 >> 172.21.15.111:0/3811686379 conn(0x7fdd2c0714c0 msgr2=0x7fdd2c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 -- 172.21.15.111:0/3811686379 shutdown_connections 2024-03-14T13:14:57.411 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.409+0000 7fdd317b6700 1 -- 172.21.15.111:0/3811686379 wait complete. 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 Processor -- start 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 -- start start 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c1138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c114470 con 0x7fdd2c102f50 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd317b6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c10da70 con 0x7fdd2c0fb4d0 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39234/0 (socket says 172.21.15.111:39234) 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 -- 172.21.15.111:0/427469485 learned_addr learned my addr 172.21.15.111:0/427469485 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2affd700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c1138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 -- 172.21.15.111:0/427469485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 msgr2=0x7fdd2c1138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c1138e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 -- 172.21.15.111:0/427469485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14005040 con 0x7fdd2c102f50 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2affd700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c1138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:14:57.412 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.410+0000 7fdd2a7fc700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c002a50 tx=0x7fdd1c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c007cc0 con 0x7fdd2c102f50 2024-03-14T13:14:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd1c00f040 con 0x7fdd2c102f50 2024-03-14T13:14:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c10dd50 con 0x7fdd2c102f50 2024-03-14T13:14:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd1c009540 con 0x7fdd2c102f50 2024-03-14T13:14:57.413 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c10e290 con 0x7fdd2c102f50 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.411+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c100200 con 0x7fdd2c102f50 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.412+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fdd1c0096a0 con 0x7fdd2c102f50 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.412+0000 7fdd237fe700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 0x7fdd1806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.412+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdd1c08b7a0 con 0x7fdd2c102f50 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.412+0000 7fdd2affd700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 0x7fdd1806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:14:57.414 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.413+0000 7fdd2affd700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 0x7fdd1806ec70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400b3b0 tx=0x7fdd14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:14:57.417 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.417+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdd1c0594b0 con 0x7fdd2c102f50 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.690+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdd2c10ef30 con 0x7fdd2c102f50 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.691+0000 7fdd237fe700 1 -- 172.21.15.111:0/427469485 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fdd1c059040 con 0x7fdd2c102f50 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:14:57.691 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:14:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 msgr2=0x7fdd1806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 0x7fdd1806ec70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400b3b0 tx=0x7fdd14009e30 comp rx=0 tx=0).stop 2024-03-14T13:14:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 msgr2=0x7fdd2c113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:14:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c002a50 tx=0x7fdd1c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 shutdown_connections 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdd1806c7c0 0x7fdd1806ec70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd2c0fb4d0 0x7fdd2c1138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 --2- 172.21.15.111:0/427469485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c102f50 0x7fdd2c113e20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 >> 172.21.15.111:0/427469485 conn(0x7fdd2c0714c0 msgr2=0x7fdd2c0fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 shutdown_connections 2024-03-14T13:14:57.694 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:14:57.693+0000 7fdd317b6700 1 -- 172.21.15.111:0/427469485 wait complete. 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: from='client.34660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: pgmap v325: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2318905036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:14:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:57 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/595442643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: from='client.34652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: from='client.34660 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: pgmap v325: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2318905036' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:14:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:57 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/595442643' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:14:58.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:58 smithi111 ceph-mon[145314]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:58.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:58 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/427469485' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:14:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:58 smithi137 ceph-mon[108145]: from='client.34672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:14:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:58 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/427469485' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:14:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:14:59 smithi111 ceph-mon[145314]: pgmap v326: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:14:59 smithi137 ceph-mon[108145]: pgmap v326: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:01 smithi111 ceph-mon[145314]: pgmap v327: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:01 smithi137 ceph-mon[108145]: pgmap v327: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:03 smithi111 ceph-mon[145314]: pgmap v328: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:03 smithi137 ceph-mon[108145]: pgmap v328: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:05 smithi111 ceph-mon[145314]: pgmap v329: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:05 smithi137 ceph-mon[108145]: pgmap v329: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:07.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:07 smithi137 ceph-mon[108145]: pgmap v330: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:07 smithi111 ceph-mon[145314]: pgmap v330: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:09 smithi111 ceph-mon[145314]: pgmap v331: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:09 smithi137 ceph-mon[108145]: pgmap v331: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:11 smithi111 ceph-mon[145314]: pgmap v332: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:11 smithi137 ceph-mon[108145]: pgmap v332: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:13 smithi111 ceph-mon[145314]: pgmap v333: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:13 smithi137 ceph-mon[108145]: pgmap v333: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:15 smithi111 ceph-mon[145314]: pgmap v334: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:15 smithi137 ceph-mon[108145]: pgmap v334: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:17 smithi111 ceph-mon[145314]: pgmap v335: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:17 smithi137 ceph-mon[108145]: pgmap v335: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:19 smithi111 ceph-mon[145314]: pgmap v336: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:19 smithi137 ceph-mon[108145]: pgmap v336: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:21 smithi111 ceph-mon[145314]: pgmap v337: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:21 smithi137 ceph-mon[108145]: pgmap v337: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:23 smithi111 ceph-mon[145314]: pgmap v338: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:23 smithi137 ceph-mon[108145]: pgmap v338: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:24 smithi111 ceph-mon[145314]: pgmap v339: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:25.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:24 smithi137 ceph-mon[108145]: pgmap v339: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:27.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:27 smithi137 ceph-mon[108145]: pgmap v340: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:27.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 -- 172.21.15.111:0/1280638932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8102660 msgr2=0x7f8af80fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 --2- 172.21.15.111:0/1280638932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8102660 0x7f8af80fc5b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4004940 tx=0x7f8ae4009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 -- 172.21.15.111:0/1280638932 shutdown_connections 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 --2- 172.21.15.111:0/1280638932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8102660 0x7f8af80fc5b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 --2- 172.21.15.111:0/1280638932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8101d50 0x7f8af8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.806+0000 7f8afee38700 1 -- 172.21.15.111:0/1280638932 >> 172.21.15.111:0/1280638932 conn(0x7f8af80714c0 msgr2=0x7f8af80718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.807+0000 7f8afee38700 1 -- 172.21.15.111:0/1280638932 shutdown_connections 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.807+0000 7f8afee38700 1 -- 172.21.15.111:0/1280638932 wait complete. 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.807+0000 7f8afee38700 1 Processor -- start 2024-03-14T13:15:27.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.807+0000 7f8afee38700 1 -- start start 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afee38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afee38700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8102660 0x7f8af8116230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afee38700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8116880 con 0x7f8af8101d50 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afee38700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af810fc50 con 0x7f8af8102660 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48466/0 (socket says 172.21.15.111:48466) 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 -- 172.21.15.111:0/2161429536 learned_addr learned my addr 172.21.15.111:0/2161429536 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8af7fff700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8102660 0x7f8af8116230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 -- 172.21.15.111:0/2161429536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8102660 msgr2=0x7f8af8116230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:27.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8102660 0x7f8af8116230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 -- 172.21.15.111:0/2161429536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae4005040 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8afcbd4700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c830 tx=0x7f8aec004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.808+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aec0106f0 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.809+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af810ff30 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.809+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aec010d00 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.809+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8aec012e90 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.809+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af8110470 con 0x7f8af8101d50 2024-03-14T13:15:27.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.809+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af80fd5c0 con 0x7f8af8101d50 2024-03-14T13:15:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.810+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8aec010850 con 0x7f8af8101d50 2024-03-14T13:15:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.811+0000 7f8af57fa700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 0x7f8ae806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.811+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8aec08c5c0 con 0x7f8af8101d50 2024-03-14T13:15:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.811+0000 7f8af7fff700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 0x7f8ae806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:27.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.811+0000 7f8af7fff700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 0x7f8ae806ec70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8af8111590 tx=0x7f8ae4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:27.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:27.815+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8aec05a2d0 con 0x7f8af8101d50 2024-03-14T13:15:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:27 smithi111 ceph-mon[145314]: pgmap v340: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:28.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.026+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8af804bf70 con 0x7f8ae806c7c0 2024-03-14T13:15:28.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.028+0000 7f8af57fa700 1 -- 172.21.15.111:0/2161429536 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8ae4007e20 con 0x7f8ae806c7c0 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 msgr2=0x7f8ae806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 0x7f8ae806ec70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8af8111590 tx=0x7f8ae4009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 msgr2=0x7f8af8115cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c830 tx=0x7f8aec004470 comp rx=0 tx=0).stop 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 shutdown_connections 2024-03-14T13:15:28.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8ae806c7c0 0x7f8ae806ec70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af8101d50 0x7f8af8115cf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 --2- 172.21.15.111:0/2161429536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8af8102660 0x7f8af8116230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 >> 172.21.15.111:0/2161429536 conn(0x7f8af80714c0 msgr2=0x7f8af80fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 shutdown_connections 2024-03-14T13:15:28.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.030+0000 7f8afee38700 1 -- 172.21.15.111:0/2161429536 wait complete. 2024-03-14T13:15:28.039 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:15:28.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 -- 172.21.15.111:0/4238687092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc102710 msgr2=0x7f90dc0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 --2- 172.21.15.111:0/4238687092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc102710 0x7f90dc0fbd80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f90cc004940 tx=0x7f90cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 -- 172.21.15.111:0/4238687092 shutdown_connections 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 --2- 172.21.15.111:0/4238687092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc102710 0x7f90dc0fbd80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 --2- 172.21.15.111:0/4238687092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc101d70 0x7f90dc102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 -- 172.21.15.111:0/4238687092 >> 172.21.15.111:0/4238687092 conn(0x7f90dc072bc0 msgr2=0x7f90dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 -- 172.21.15.111:0/4238687092 shutdown_connections 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 -- 172.21.15.111:0/4238687092 wait complete. 2024-03-14T13:15:28.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.131+0000 7f90e1b30700 1 Processor -- start 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 -- start start 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 0x7f90dc10efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc10f5f0 con 0x7f90dc101d70 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90dc10f730 con 0x7f90dc102710 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48496/0 (socket says 172.21.15.111:48496) 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 -- 172.21.15.111:0/3366765766 learned_addr learned my addr 172.21.15.111:0/3366765766 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90daffd700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 0x7f90dc10efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 -- 172.21.15.111:0/3366765766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 msgr2=0x7f90dc10efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 0x7f90dc10efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 -- 172.21.15.111:0/3366765766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc005040 con 0x7f90dc101d70 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90daffd700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 0x7f90dc10efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90db7fe700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f90d000c830 tx=0x7f90d0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d000a760 con 0x7f90dc101d70 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.132+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90dc117ad0 con 0x7f90dc101d70 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.133+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90d000ad70 con 0x7f90dc101d70 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.133+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f90d00113a0 con 0x7f90dc101d70 2024-03-14T13:15:28.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.133+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90dc118030 con 0x7f90dc101d70 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.134+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f90d0012400 con 0x7f90dc101d70 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.134+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90dc0ff080 con 0x7f90dc101d70 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.134+0000 7f90c3fff700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 0x7f90c406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.134+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f90d008b990 con 0x7f90dc101d70 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.134+0000 7f90daffd700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 0x7f90c406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.135+0000 7f90daffd700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 0x7f90c406ec70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00b3b0 tx=0x7f90cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.139+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f90d0055b90 con 0x7f90dc101d70 2024-03-14T13:15:28.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.348+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90dc04bed0 con 0x7f90c406c7c0 2024-03-14T13:15:28.348 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.349+0000 7f90c3fff700 1 -- 172.21.15.111:0/3366765766 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f90dc04bed0 con 0x7f90c406c7c0 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 msgr2=0x7f90c406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 0x7f90c406ec70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00b3b0 tx=0x7f90cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 msgr2=0x7f90dc10ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f90d000c830 tx=0x7f90d0004470 comp rx=0 tx=0).stop 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 shutdown_connections 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f90c406c7c0 0x7f90c406ec70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90dc101d70 0x7f90dc10ea60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 --2- 172.21.15.111:0/3366765766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90dc102710 0x7f90dc10efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 >> 172.21.15.111:0/3366765766 conn(0x7f90dc072bc0 msgr2=0x7f90dc0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 shutdown_connections 2024-03-14T13:15:28.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.351+0000 7f90e1b30700 1 -- 172.21.15.111:0/3366765766 wait complete. 2024-03-14T13:15:28.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- 172.21.15.111:0/1995488428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 msgr2=0x7f86d01020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 --2- 172.21.15.111:0/1995488428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d01020f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004940 tx=0x7f86c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- 172.21.15.111:0/1995488428 shutdown_connections 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 --2- 172.21.15.111:0/1995488428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d00fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 --2- 172.21.15.111:0/1995488428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d01020f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- 172.21.15.111:0/1995488428 >> 172.21.15.111:0/1995488428 conn(0x7f86d00f7870 msgr2=0x7f86d00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- 172.21.15.111:0/1995488428 shutdown_connections 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- 172.21.15.111:0/1995488428 wait complete. 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 Processor -- start 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.452+0000 7f86d7ead700 1 -- start start 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d010f500 con 0x7f86d0101d20 2024-03-14T13:15:28.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d010f640 con 0x7f86d01026c0 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48520/0 (socket says 172.21.15.111:48520) 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 -- 172.21.15.111:0/4288365625 learned_addr learned my addr 172.21.15.111:0/4288365625 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5448700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 -- 172.21.15.111:0/4288365625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 msgr2=0x7f86d010efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d010efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 -- 172.21.15.111:0/4288365625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8005040 con 0x7f86d0101d20 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5448700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d5c49700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b3b0 tx=0x7f86c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c800e040 con 0x7f86d0101d20 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c8013420 con 0x7f86d0101d20 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d010f8c0 con 0x7f86d0101d20 2024-03-14T13:15:28.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86c801c630 con 0x7f86d0101d20 2024-03-14T13:15:28.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.453+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d0117e40 con 0x7f86d0101d20 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.454+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d00fd470 con 0x7f86d0101d20 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.455+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f86c8013970 con 0x7f86d0101d20 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.455+0000 7f86ce7fc700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 0x7f86b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.455+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86c808ccc0 con 0x7f86d0101d20 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.455+0000 7f86d5448700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 0x7f86b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.455+0000 7f86d5448700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 0x7f86b806ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009fd0 tx=0x7f86c4009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.459+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f86c8056f40 con 0x7f86d0101d20 2024-03-14T13:15:28.658 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.659+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f86d00f9010 con 0x7f86b806c7c0 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.669+0000 7f86ce7fc700 1 -- 172.21.15.111:0/4288365625 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f86d00f9010 con 0x7f86b806c7c0 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (11m) 5m ago 23m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (9m) 5m ago 23m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (9m) 4m ago 21m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (11m) 5m ago 22m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (5m) 5m ago 17m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (5m) 5m ago 17m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:15:28.668 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (4m) 4m ago 17m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (4m) 4m ago 17m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (14m) 5m ago 24m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (12m) 4m ago 21m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (10m) 5m ago 24m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (9m) 4m ago 21m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (12m) 5m ago 23m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (12m) 4m ago 21m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (8m) 5m ago 21m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (8m) 5m ago 20m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (7m) 5m ago 20m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (6m) 4m ago 19m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (6m) 4m ago 19m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:15:28.669 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (5m) 4m ago 18m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:15:28.670 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (12m) 5m ago 22m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:15:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 msgr2=0x7f86b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 0x7f86b806ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009fd0 tx=0x7f86c4009f20 comp rx=0 tx=0).stop 2024-03-14T13:15:28.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 msgr2=0x7f86d010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b3b0 tx=0x7f86c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 shutdown_connections 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86b806c7c0 0x7f86b806ec70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d0101d20 0x7f86d010ea80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 --2- 172.21.15.111:0/4288365625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d01026c0 0x7f86d010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 >> 172.21.15.111:0/4288365625 conn(0x7f86d00f7870 msgr2=0x7f86d00f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 shutdown_connections 2024-03-14T13:15:28.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.671+0000 7f86d7ead700 1 -- 172.21.15.111:0/4288365625 wait complete. 2024-03-14T13:15:28.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- 172.21.15.111:0/3360725169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 msgr2=0x7fe5540fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 --2- 172.21.15.111:0/3360725169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe5540fe260 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe548004940 tx=0x7fe548009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- 172.21.15.111:0/3360725169 shutdown_connections 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 --2- 172.21.15.111:0/3360725169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 0x7fe554106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 --2- 172.21.15.111:0/3360725169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe5540fe260 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- 172.21.15.111:0/3360725169 >> 172.21.15.111:0/3360725169 conn(0x7fe554072bc0 msgr2=0x7fe554072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- 172.21.15.111:0/3360725169 shutdown_connections 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- 172.21.15.111:0/3360725169 wait complete. 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 Processor -- start 2024-03-14T13:15:28.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.770+0000 7fe55a084700 1 -- start start 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 0x7fe554116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554116850 con 0x7fe5540fde90 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55410fcb0 con 0x7fe5540fe830 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48540/0 (socket says 172.21.15.111:48540) 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 -- 172.21.15.111:0/137307161 learned_addr learned my addr 172.21.15.111:0/137307161 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe552ffd700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 0x7fe554116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 -- 172.21.15.111:0/137307161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 msgr2=0x7fe554116200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 0x7fe554116200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 -- 172.21.15.111:0/137307161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548005040 con 0x7fe5540fde90 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe5537fe700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe548009360 tx=0x7fe548009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54800e040 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe54800bed0 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55410ff30 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54801c3f0 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.771+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe554110470 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.772+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe554106a20 con 0x7fe5540fde90 2024-03-14T13:15:28.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.773+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe548013790 con 0x7fe5540fde90 2024-03-14T13:15:28.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.773+0000 7fe53bfff700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 0x7fe53c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:28.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.773+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe54808cec0 con 0x7fe5540fde90 2024-03-14T13:15:28.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.773+0000 7fe552ffd700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 0x7fe53c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:28.774 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.774+0000 7fe552ffd700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 0x7fe53c06ec70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5541115a0 tx=0x7fe54400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:28.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:28.778+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe548057040 con 0x7fe5540fde90 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.051+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe554109df0 con 0x7fe5540fde90 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.051+0000 7fe53bfff700 1 -- 172.21.15.111:0/137307161 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fe548013420 con 0x7fe5540fde90 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:15:29.051 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:15:29.052 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.053+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 msgr2=0x7fe53c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.053+0000 7fe55a084700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 0x7fe53c06ec70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5541115a0 tx=0x7fe54400b040 comp rx=0 tx=0).stop 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.053+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 msgr2=0x7fe554115cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.053+0000 7fe55a084700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe548009360 tx=0x7fe548009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 shutdown_connections 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53c06c7c0 0x7fe53c06ec70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5540fde90 0x7fe554115cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 --2- 172.21.15.111:0/137307161 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5540fe830 0x7fe554116200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 >> 172.21.15.111:0/137307161 conn(0x7fe554072bc0 msgr2=0x7fe5540fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 shutdown_connections 2024-03-14T13:15:29.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.054+0000 7fe55a084700 1 -- 172.21.15.111:0/137307161 wait complete. 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 -- 172.21.15.111:0/3226253181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fde90 msgr2=0x7fca240fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 --2- 172.21.15.111:0/3226253181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fde90 0x7fca240fe260 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fca18004940 tx=0x7fca18009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 -- 172.21.15.111:0/3226253181 shutdown_connections 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 --2- 172.21.15.111:0/3226253181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fe830 0x7fca24106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 --2- 172.21.15.111:0/3226253181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fde90 0x7fca240fe260 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 -- 172.21.15.111:0/3226253181 >> 172.21.15.111:0/3226253181 conn(0x7fca24072bc0 msgr2=0x7fca24072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 -- 172.21.15.111:0/3226253181 shutdown_connections 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 -- 172.21.15.111:0/3226253181 wait complete. 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.153+0000 7fca2addc700 1 Processor -- start 2024-03-14T13:15:29.153 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca2addc700 1 -- start start 2024-03-14T13:15:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca2addc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 0x7fca24113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca2addc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca2addc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24114680 con 0x7fca240fe830 2024-03-14T13:15:29.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca2addc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2410da70 con 0x7fca240fde90 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48562/0 (socket says 172.21.15.111:48562) 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 -- 172.21.15.111:0/49432068 learned_addr learned my addr 172.21.15.111:0/49432068 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca28b78700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 0x7fca24113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 -- 172.21.15.111:0/49432068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 msgr2=0x7fca24113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 0x7fca24113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 -- 172.21.15.111:0/49432068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca18005040 con 0x7fca240fe830 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca28b78700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 0x7fca24113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.154+0000 7fca23fff700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c830 tx=0x7fca10004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.155+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca1000a760 con 0x7fca240fe830 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.155+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca1000ad70 con 0x7fca240fe830 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.155+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2410dd50 con 0x7fca240fe830 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.155+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca100113a0 con 0x7fca240fe830 2024-03-14T13:15:29.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.155+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2410e290 con 0x7fca240fe830 2024-03-14T13:15:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.156+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca241019b0 con 0x7fca240fe830 2024-03-14T13:15:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.156+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fca10012400 con 0x7fca240fe830 2024-03-14T13:15:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.157+0000 7fca217fa700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 0x7fca1406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.157+0000 7fca28b78700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 0x7fca1406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.157+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fca1008bae0 con 0x7fca240fe830 2024-03-14T13:15:29.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.157+0000 7fca28b78700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 0x7fca1406ec70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fca1800b3b0 tx=0x7fca18009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.161+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fca100597f0 con 0x7fca240fe830 2024-03-14T13:15:29.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.396+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fca2410f140 con 0x7fca240fe830 2024-03-14T13:15:29.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.396+0000 7fca217fa700 1 -- 172.21.15.111:0/49432068 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fca10059380 con 0x7fca240fe830 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:15:29.396 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:15:29.397 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:15:29.398 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 msgr2=0x7fca1406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 0x7fca1406ec70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fca1800b3b0 tx=0x7fca18009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 msgr2=0x7fca24114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c830 tx=0x7fca10004470 comp rx=0 tx=0).stop 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 shutdown_connections 2024-03-14T13:15:29.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fca1406c7c0 0x7fca1406ec70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca240fde90 0x7fca24113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 --2- 172.21.15.111:0/49432068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca240fe830 0x7fca24114030 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 >> 172.21.15.111:0/49432068 conn(0x7fca24072bc0 msgr2=0x7fca240ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 shutdown_connections 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.398+0000 7fca2addc700 1 -- 172.21.15.111:0/49432068 wait complete. 2024-03-14T13:15:29.400 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- 172.21.15.111:0/1500644850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 msgr2=0x7f38b8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/1500644850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b8102140 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f38a8004940 tx=0x7f38a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- 172.21.15.111:0/1500644850 shutdown_connections 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/1500644850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 0x7f38b80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/1500644850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b8102140 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- 172.21.15.111:0/1500644850 >> 172.21.15.111:0/1500644850 conn(0x7f38b8072bc0 msgr2=0x7f38b8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- 172.21.15.111:0/1500644850 shutdown_connections 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- 172.21.15.111:0/1500644850 wait complete. 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 Processor -- start 2024-03-14T13:15:29.503 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.502+0000 7f38bf9d0700 1 -- start start 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bf9d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bf9d0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 0x7f38b810ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bf9d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b810f5d0 con 0x7f38b8101d70 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bf9d0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b810f710 con 0x7f38b8102710 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48586/0 (socket says 172.21.15.111:48586) 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 -- 172.21.15.111:0/2352203391 learned_addr learned my addr 172.21.15.111:0/2352203391 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bcf6b700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 0x7f38b810ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 -- 172.21.15.111:0/2352203391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 msgr2=0x7f38b810ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 0x7f38b810ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 -- 172.21.15.111:0/2352203391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8005040 con 0x7f38b8101d70 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bd76c700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f38a800b3b0 tx=0x7f38a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a800e040 con 0x7f38b8101d70 2024-03-14T13:15:29.504 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a8013420 con 0x7f38b8101d70 2024-03-14T13:15:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b8117ab0 con 0x7f38b8101d70 2024-03-14T13:15:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.503+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38a801c610 con 0x7f38b8101d70 2024-03-14T13:15:29.505 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.504+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b8117fc0 con 0x7f38b8101d70 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.504+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b80ff080 con 0x7f38b8101d70 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.505+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f38a8013a00 con 0x7f38b8101d70 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.505+0000 7f38adffb700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 0x7f38a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.505+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f38a808cac0 con 0x7f38b8101d70 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.505+0000 7f38bcf6b700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 0x7f38a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.506+0000 7f38bcf6b700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 0x7f38a406ec70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f38b8066000 tx=0x7f38b400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.510+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f38a8056e00 con 0x7f38b8101d70 2024-03-14T13:15:29.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.719+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f38b804bed0 con 0x7f38a406c7c0 2024-03-14T13:15:29.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.720+0000 7f38adffb700 1 -- 172.21.15.111:0/2352203391 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f38b804bed0 con 0x7f38a406c7c0 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:15:29.720 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:15:29.721 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:15:29.721 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 msgr2=0x7f38a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 0x7f38a406ec70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f38b8066000 tx=0x7f38b400b040 comp rx=0 tx=0).stop 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 msgr2=0x7f38b810ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f38a800b3b0 tx=0x7f38a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 shutdown_connections 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f38a406c7c0 0x7f38a406ec70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38b8101d70 0x7f38b810ea40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 --2- 172.21.15.111:0/2352203391 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38b8102710 0x7f38b810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.722+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 >> 172.21.15.111:0/2352203391 conn(0x7f38b8072bc0 msgr2=0x7f38b80fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.723+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 shutdown_connections 2024-03-14T13:15:29.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.723+0000 7f38bf9d0700 1 -- 172.21.15.111:0/2352203391 wait complete. 2024-03-14T13:15:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:29 smithi137 ceph-mon[108145]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:29 smithi137 ceph-mon[108145]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:29 smithi137 ceph-mon[108145]: pgmap v341: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:29 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/137307161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:15:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:29 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/49432068' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 -- 172.21.15.111:0/3615125562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fe830 msgr2=0x7f5f10106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 --2- 172.21.15.111:0/3615125562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fe830 0x7f5f10106f00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5f00004940 tx=0x7f5f00009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 -- 172.21.15.111:0/3615125562 shutdown_connections 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 --2- 172.21.15.111:0/3615125562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fe830 0x7f5f10106f00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 --2- 172.21.15.111:0/3615125562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fde90 0x7f5f100fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 -- 172.21.15.111:0/3615125562 >> 172.21.15.111:0/3615125562 conn(0x7f5f10072bc0 msgr2=0x7f5f10072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 -- 172.21.15.111:0/3615125562 shutdown_connections 2024-03-14T13:15:29.822 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.822+0000 7f5f17652700 1 -- 172.21.15.111:0/3615125562 wait complete. 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 Processor -- start 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 -- start start 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fe830 0x7f5f10113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f101146d0 con 0x7f5f100fde90 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f17652700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1010da30 con 0x7f5f100fe830 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f153ee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f153ee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48596/0 (socket says 172.21.15.111:48596) 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.823+0000 7f5f153ee700 1 -- 172.21.15.111:0/1794190426 learned_addr learned my addr 172.21.15.111:0/1794190426 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:15:29.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f14bed700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fe830 0x7f5f10113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f153ee700 1 -- 172.21.15.111:0/1794190426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fe830 msgr2=0x7f5f10113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f153ee700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fe830 0x7f5f10113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f153ee700 1 -- 172.21.15.111:0/1794190426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00005040 con 0x7f5f100fde90 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f153ee700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00b790 tx=0x7f5f0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c00f770 con 0x7f5f100fde90 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f0c00fd80 con 0x7f5f100fde90 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1010dd10 con 0x7f5f100fde90 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0c00d310 con 0x7f5f100fde90 2024-03-14T13:15:29.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.824+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1010e250 con 0x7f5f100fde90 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.825+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f10065e30 con 0x7f5f100fde90 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.825+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5f0c010420 con 0x7f5f100fde90 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.826+0000 7f5f05ffb700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 0x7f5efc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.826+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5f0c08b960 con 0x7f5f100fde90 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.826+0000 7f5f14bed700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 0x7f5efc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:15:29.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.826+0000 7f5f14bed700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 0x7f5efc06ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:15:29.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:29.830+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5f0c059730 con 0x7f5f100fde90 2024-03-14T13:15:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:29 smithi111 ceph-mon[145314]: from='client.34680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:29 smithi111 ceph-mon[145314]: from='client.34684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:29 smithi111 ceph-mon[145314]: pgmap v341: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:29 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/137307161' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:15:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:29 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/49432068' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.102+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f5f1005f5f0 con 0x7f5f100fde90 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.103+0000 7f5f05ffb700 1 -- 172.21.15.111:0/1794190426 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f5f0c0592c0 con 0x7f5f100fde90 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:15:30.102 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 msgr2=0x7f5efc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 0x7f5efc06ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00009e30 comp rx=0 tx=0).stop 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 msgr2=0x7f5f10113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00b790 tx=0x7f5f0c009250 comp rx=0 tx=0).stop 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 shutdown_connections 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5efc06c7c0 0x7f5efc06ec70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:30.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f100fde90 0x7f5f10113ab0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:30.105 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 --2- 172.21.15.111:0/1794190426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f100fe830 0x7f5f10113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:15:30.105 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 >> 172.21.15.111:0/1794190426 conn(0x7f5f10072bc0 msgr2=0x7f5f101037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:15:30.105 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 shutdown_connections 2024-03-14T13:15:30.105 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:15:30.105+0000 7f5f17652700 1 -- 172.21.15.111:0/1794190426 wait complete. 2024-03-14T13:15:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:30 smithi137 ceph-mon[108145]: from='client.34688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:30.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:30 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1794190426' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:15:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:30 smithi111 ceph-mon[145314]: from='client.34688 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:30 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1794190426' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:15:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:31 smithi137 ceph-mon[108145]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:31.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:31 smithi137 ceph-mon[108145]: pgmap v342: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:31 smithi111 ceph-mon[145314]: from='client.34700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:15:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:31 smithi111 ceph-mon[145314]: pgmap v342: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:33 smithi137 ceph-mon[108145]: pgmap v343: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:33 smithi111 ceph-mon[145314]: pgmap v343: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:35.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:35 smithi137 ceph-mon[108145]: pgmap v344: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:35 smithi111 ceph-mon[145314]: pgmap v344: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:37.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:37 smithi137 ceph-mon[108145]: pgmap v345: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:37 smithi111 ceph-mon[145314]: pgmap v345: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:39 smithi111 ceph-mon[145314]: pgmap v346: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:39 smithi137 ceph-mon[108145]: pgmap v346: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:41 smithi111 ceph-mon[145314]: pgmap v347: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:41 smithi137 ceph-mon[108145]: pgmap v347: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:43 smithi111 ceph-mon[145314]: pgmap v348: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:43 smithi137 ceph-mon[108145]: pgmap v348: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:45 smithi111 ceph-mon[145314]: pgmap v349: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:45 smithi137 ceph-mon[108145]: pgmap v349: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:47 smithi111 ceph-mon[145314]: pgmap v350: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:47 smithi137 ceph-mon[108145]: pgmap v350: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:49 smithi111 ceph-mon[145314]: pgmap v351: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:49 smithi137 ceph-mon[108145]: pgmap v351: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:51 smithi111 ceph-mon[145314]: pgmap v352: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:51 smithi137 ceph-mon[108145]: pgmap v352: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:53 smithi111 ceph-mon[145314]: pgmap v353: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:53 smithi137 ceph-mon[108145]: pgmap v353: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:15:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:55 smithi111 ceph-mon[145314]: pgmap v354: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:55 smithi137 ceph-mon[108145]: pgmap v354: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:57 smithi111 ceph-mon[145314]: pgmap v355: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:57 smithi137 ceph-mon[108145]: pgmap v355: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:15:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:15:59 smithi111 ceph-mon[145314]: pgmap v356: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:15:59 smithi137 ceph-mon[108145]: pgmap v356: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 -- 172.21.15.111:0/1870311860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b98101d70 msgr2=0x7f8b98102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/1870311860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b98101d70 0x7f8b98102140 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c004940 tx=0x7f8b8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 -- 172.21.15.111:0/1870311860 shutdown_connections 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/1870311860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b980fc5b0 0x7f8b980fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/1870311860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b98101d70 0x7f8b98102140 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 -- 172.21.15.111:0/1870311860 >> 172.21.15.111:0/1870311860 conn(0x7f8b980714c0 msgr2=0x7f8b980718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 -- 172.21.15.111:0/1870311860 shutdown_connections 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.217+0000 7f8b9efb2700 1 -- 172.21.15.111:0/1870311860 wait complete. 2024-03-14T13:16:00.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 Processor -- start 2024-03-14T13:16:00.219 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 -- start start 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b98101d70 0x7f8b9810f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9810f6a0 con 0x7f8b980fc5b0 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9efb2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9810f7e0 con 0x7f8b98101d70 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9cd4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9cd4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:37532/0 (socket says 172.21.15.111:37532) 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b9cd4e700 1 -- 172.21.15.111:0/2151815913 learned_addr learned my addr 172.21.15.111:0/2151815913 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:00.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.218+0000 7f8b97fff700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b98101d70 0x7f8b9810f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9cd4e700 1 -- 172.21.15.111:0/2151815913 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b98101d70 msgr2=0x7f8b9810f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9cd4e700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b98101d70 0x7f8b9810f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9cd4e700 1 -- 172.21.15.111:0/2151815913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c005040 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9cd4e700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00b3b0 tx=0x7f8b8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b8c00e040 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b8c009b10 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98117ad0 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b8c013420 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.219+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b98118060 con 0x7f8b980fc5b0 2024-03-14T13:16:00.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.220+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b980fd5c0 con 0x7f8b980fc5b0 2024-03-14T13:16:00.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.220+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8b8c01c430 con 0x7f8b980fc5b0 2024-03-14T13:16:00.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.221+0000 7f8b957fa700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.221+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b8c08c660 con 0x7f8b980fc5b0 2024-03-14T13:16:00.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.221+0000 7f8b97fff700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.221+0000 7f8b97fff700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b980fbec0 tx=0x7f8b84009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.225+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8b8c056990 con 0x7f8b980fc5b0 2024-03-14T13:16:00.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.436+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b9804bf70 con 0x7f8b8806c7c0 2024-03-14T13:16:00.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.437+0000 7f8b957fa700 1 -- 172.21.15.111:0/2151815913 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8b84004e30 con 0x7f8b8806c7c0 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 msgr2=0x7f8b8806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8b980fbec0 tx=0x7f8b84009ee0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 msgr2=0x7f8b9810ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00b3b0 tx=0x7f8b8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 shutdown_connections 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b980fc5b0 0x7f8b9810ec20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 --2- 172.21.15.111:0/2151815913 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b98101d70 0x7f8b9810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 >> 172.21.15.111:0/2151815913 conn(0x7f8b980714c0 msgr2=0x7f8b98104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 shutdown_connections 2024-03-14T13:16:00.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.439+0000 7f8b9efb2700 1 -- 172.21.15.111:0/2151815913 wait complete. 2024-03-14T13:16:00.450 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:16:00.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- 172.21.15.111:0/3576094366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 msgr2=0x7f03380fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 --2- 172.21.15.111:0/3576094366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03380fbd30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0334004940 tx=0x7f0334009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- 172.21.15.111:0/3576094366 shutdown_connections 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 --2- 172.21.15.111:0/3576094366 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03380fbd30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 --2- 172.21.15.111:0/3576094366 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0338101d20 0x7f03381020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- 172.21.15.111:0/3576094366 >> 172.21.15.111:0/3576094366 conn(0x7f03380f7870 msgr2=0x7f03380f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- 172.21.15.111:0/3576094366 shutdown_connections 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- 172.21.15.111:0/3576094366 wait complete. 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 Processor -- start 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.541+0000 7f033fe1f700 1 -- start start 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0338101d20 0x7f0338102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338104ad0 con 0x7f03381026c0 2024-03-14T13:16:00.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0338103600 con 0x7f0338101d20 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58260/0 (socket says 172.21.15.111:58260) 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 -- 172.21.15.111:0/140058582 learned_addr learned my addr 172.21.15.111:0/140058582 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 -- 172.21.15.111:0/140058582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0338101d20 msgr2=0x7f0338102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0338101d20 0x7f0338102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 -- 172.21.15.111:0/140058582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334005040 con 0x7f03381026c0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033d3ba700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03340093b0 tx=0x7f0334009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033400e040 con 0x7f03381026c0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0334013420 con 0x7f03381026c0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f033401c630 con 0x7f03381026c0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0338103880 con 0x7f03381026c0 2024-03-14T13:16:00.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.542+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033811a090 con 0x7f03381026c0 2024-03-14T13:16:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.543+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03380fd470 con 0x7f03381026c0 2024-03-14T13:16:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.544+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f03340139e0 con 0x7f03381026c0 2024-03-14T13:16:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.544+0000 7f032e7fc700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 0x7f032406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.544+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f033408cc20 con 0x7f03381026c0 2024-03-14T13:16:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.544+0000 7f033dbbb700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 0x7f032406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.545+0000 7f033dbbb700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 0x7f032406ec70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0328004d00 tx=0x7f0328009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.549+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0334056ea0 con 0x7f03381026c0 2024-03-14T13:16:00.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.757+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f03380f9740 con 0x7f032406c7c0 2024-03-14T13:16:00.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.759+0000 7f032e7fc700 1 -- 172.21.15.111:0/140058582 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f03380f9740 con 0x7f032406c7c0 2024-03-14T13:16:00.761 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 msgr2=0x7f032406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 0x7f032406ec70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0328004d00 tx=0x7f0328009ce0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 msgr2=0x7f03381030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03340093b0 tx=0x7f0334009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 shutdown_connections 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f032406c7c0 0x7f032406ec70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0338101d20 0x7f0338102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 --2- 172.21.15.111:0/140058582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03381026c0 0x7f03381030c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 >> 172.21.15.111:0/140058582 conn(0x7f03380f7870 msgr2=0x7f03380f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 shutdown_connections 2024-03-14T13:16:00.762 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.761+0000 7f033fe1f700 1 -- 172.21.15.111:0/140058582 wait complete. 2024-03-14T13:16:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 -- 172.21.15.111:0/1825229046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fde90 msgr2=0x7fd4100fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 --2- 172.21.15.111:0/1825229046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fde90 0x7fd4100fe260 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd404004940 tx=0x7fd404009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 -- 172.21.15.111:0/1825229046 shutdown_connections 2024-03-14T13:16:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 --2- 172.21.15.111:0/1825229046 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fe830 0x7fd410106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.863 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 --2- 172.21.15.111:0/1825229046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fde90 0x7fd4100fe260 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 -- 172.21.15.111:0/1825229046 >> 172.21.15.111:0/1825229046 conn(0x7fd410072bc0 msgr2=0x7fd410072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 -- 172.21.15.111:0/1825229046 shutdown_connections 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.861+0000 7fd416fe6700 1 -- 172.21.15.111:0/1825229046 wait complete. 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 Processor -- start 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 -- start start 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fde90 0x7fd4100fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100fc820 con 0x7fd4100fe830 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd416fe6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100fc960 con 0x7fd4100fde90 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd40ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd40ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58276/0 (socket says 172.21.15.111:58276) 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd40ffff700 1 -- 172.21.15.111:0/2251600731 learned_addr learned my addr 172.21.15.111:0/2251600731 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.862+0000 7fd414d82700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fde90 0x7fd4100fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40ffff700 1 -- 172.21.15.111:0/2251600731 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fde90 msgr2=0x7fd4100fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40ffff700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fde90 0x7fd4100fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40ffff700 1 -- 172.21.15.111:0/2251600731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd404005040 con 0x7fd4100fe830 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40ffff700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c830 tx=0x7fd3fc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3fc0106f0 con 0x7fd4100fe830 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd410119df0 con 0x7fd4100fe830 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3fc010d00 con 0x7fd4100fe830 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3fc012e90 con 0x7fd4100fe830 2024-03-14T13:16:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.863+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41011a1f0 con 0x7fd4100fe830 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.864+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd410065e30 con 0x7fd4100fe830 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.864+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd3fc010850 con 0x7fd4100fe830 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.865+0000 7fd40d7fa700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.865+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd3fc08ba20 con 0x7fd4100fe830 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.865+0000 7fd414d82700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.865+0000 7fd414d82700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd40400b7e0 tx=0x7fd404009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:00.869 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:00.869+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd3fc055ca0 con 0x7fd4100fe830 2024-03-14T13:16:01.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.070+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd41004bed0 con 0x7fd40006c7c0 2024-03-14T13:16:01.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.080+0000 7fd40d7fa700 1 -- 172.21.15.111:0/2251600731 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fd41004bed0 con 0x7fd40006c7c0 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (12m) 5m ago 23m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (9m) 5m ago 24m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (9m) 5m ago 22m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (11m) 5m ago 23m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (5m) 5m ago 17m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (6m) 5m ago 17m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (5m) 5m ago 17m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:16:01.081 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (5m) 5m ago 17m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (14m) 5m ago 25m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (13m) 5m ago 22m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (10m) 5m ago 25m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (10m) 5m ago 22m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (13m) 5m ago 23m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (13m) 5m ago 22m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (9m) 5m ago 21m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (8m) 5m ago 21m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (8m) 5m ago 20m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (7m) 5m ago 20m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (6m) 5m ago 19m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (6m) 5m ago 19m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:16:01.082 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (12m) 5m ago 22m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 msgr2=0x7fd40006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd40400b7e0 tx=0x7fd404009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 msgr2=0x7fd4100fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00c830 tx=0x7fd3fc004470 comp rx=0 tx=0).stop 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 shutdown_connections 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.083 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4100fde90 0x7fd4100fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 --2- 172.21.15.111:0/2251600731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4100fe830 0x7fd4100fc290 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 >> 172.21.15.111:0/2251600731 conn(0x7fd410072bc0 msgr2=0x7fd4100ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.082+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 shutdown_connections 2024-03-14T13:16:01.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.083+0000 7fd416fe6700 1 -- 172.21.15.111:0/2251600731 wait complete. 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- 172.21.15.111:0/2128851199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fde70 msgr2=0x7f842c0fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 --2- 172.21.15.111:0/2128851199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fde70 0x7f842c0fe240 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f841c004940 tx=0x7f841c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- 172.21.15.111:0/2128851199 shutdown_connections 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 --2- 172.21.15.111:0/2128851199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fe810 0x7f842c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 --2- 172.21.15.111:0/2128851199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fde70 0x7f842c0fe240 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- 172.21.15.111:0/2128851199 >> 172.21.15.111:0/2128851199 conn(0x7f842c0f7880 msgr2=0x7f842c0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- 172.21.15.111:0/2128851199 shutdown_connections 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- 172.21.15.111:0/2128851199 wait complete. 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 Processor -- start 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.182+0000 7f8433e99700 1 -- start start 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 0x7f842c0fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0fc770 con 0x7f842c0fe810 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0fc8b0 con 0x7f842c0fde70 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58298/0 (socket says 172.21.15.111:58298) 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 -- 172.21.15.111:0/1319558174 learned_addr learned my addr 172.21.15.111:0/1319558174 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431c35700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 0x7f842c0fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 -- 172.21.15.111:0/1319558174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 msgr2=0x7f842c0fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 0x7f842c0fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 -- 172.21.15.111:0/1319558174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c005040 con 0x7f842c0fe810 2024-03-14T13:16:01.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431c35700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 0x7f842c0fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8431434700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8428002a50 tx=0x7f84280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8428007cc0 con 0x7f842c0fe810 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f842800f040 con 0x7f842c0fe810 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8428009540 con 0x7f842c0fe810 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c11a070 con 0x7f842c0fe810 2024-03-14T13:16:01.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.183+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c11a410 con 0x7f842c0fe810 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.184+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c066710 con 0x7f842c0fe810 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.185+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f84280096a0 con 0x7f842c0fe810 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.185+0000 7f84227fc700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 0x7f841806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.185+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f842808b8a0 con 0x7f842c0fe810 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.185+0000 7f8431c35700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 0x7f841806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.186+0000 7f8431c35700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 0x7f841806ec70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.190+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8428055aa0 con 0x7f842c0fe810 2024-03-14T13:16:01.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.464+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f842c04bed0 con 0x7f842c0fe810 2024-03-14T13:16:01.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.464+0000 7f84227fc700 1 -- 172.21.15.111:0/1319558174 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f84280590c0 con 0x7f842c0fe810 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:16:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:16:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:16:01.466 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:16:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 msgr2=0x7f841806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 0x7f841806ec70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f841c000c00 tx=0x7f841c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 msgr2=0x7f842c0fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8428002a50 tx=0x7f84280046b0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 shutdown_connections 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f841806c7c0 0x7f841806ec70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f842c0fde70 0x7f842c0fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 --2- 172.21.15.111:0/1319558174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0fe810 0x7f842c0fc1e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 >> 172.21.15.111:0/1319558174 conn(0x7f842c0f7880 msgr2=0x7f842c0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 shutdown_connections 2024-03-14T13:16:01.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.466+0000 7f8433e99700 1 -- 172.21.15.111:0/1319558174 wait complete. 2024-03-14T13:16:01.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 -- 172.21.15.111:0/3506943476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 msgr2=0x7f0d600fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 --2- 172.21.15.111:0/3506943476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d600fbd80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0d54004940 tx=0x7f0d54009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 -- 172.21.15.111:0/3506943476 shutdown_connections 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 --2- 172.21.15.111:0/3506943476 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d600fbd80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 --2- 172.21.15.111:0/3506943476 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 -- 172.21.15.111:0/3506943476 >> 172.21.15.111:0/3506943476 conn(0x7f0d60072bc0 msgr2=0x7f0d60072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 -- 172.21.15.111:0/3506943476 shutdown_connections 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.566+0000 7f0d65044700 1 -- 172.21.15.111:0/3506943476 wait complete. 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 Processor -- start 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 -- start start 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d601036a0 con 0x7f0d60102710 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d65044700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d601037e0 con 0x7f0d60101d70 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d5e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d5e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58316/0 (socket says 172.21.15.111:58316) 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.567+0000 7f0d5e59c700 1 -- 172.21.15.111:0/1025727118 learned_addr learned my addr 172.21.15.111:0/1025727118 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5ed9d700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5e59c700 1 -- 172.21.15.111:0/1025727118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 msgr2=0x7f0d60102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5e59c700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5e59c700 1 -- 172.21.15.111:0/1025727118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54005040 con 0x7f0d60102710 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5ed9d700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d5e59c700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009390 tx=0x7f0d54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5400e040 con 0x7f0d60102710 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d60103a30 con 0x7f0d60102710 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d54013420 con 0x7f0d60102710 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5401c630 con 0x7f0d60102710 2024-03-14T13:16:01.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.568+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6011a1f0 con 0x7f0d60102710 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.569+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d600fd4c0 con 0x7f0d60102710 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.569+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0d54013a00 con 0x7f0d60102710 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.570+0000 7f0d477fe700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 0x7f0d4806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.570+0000 7f0d5ed9d700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 0x7f0d4806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.570+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d5408cd20 con 0x7f0d60102710 2024-03-14T13:16:01.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.570+0000 7f0d5ed9d700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 0x7f0d4806ec80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009fd0 tx=0x7f0d5000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.574+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0d54056fa0 con 0x7f0d60102710 2024-03-14T13:16:01.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.807+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0d60108da0 con 0x7f0d60102710 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.807+0000 7f0d477fe700 1 -- 172.21.15.111:0/1025727118 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f0d5405a5c0 con 0x7f0d60102710 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:16:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:16:01.809 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:01.810 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 msgr2=0x7f0d4806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 0x7f0d4806ec80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009fd0 tx=0x7f0d5000b040 comp rx=0 tx=0).stop 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 msgr2=0x7f0d60103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009390 tx=0x7f0d54009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 shutdown_connections 2024-03-14T13:16:01.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0d4806c7d0 0x7f0d4806ec80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d60101d70 0x7f0d60102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 --2- 172.21.15.111:0/1025727118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d60102710 0x7f0d60103110 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 >> 172.21.15.111:0/1025727118 conn(0x7f0d60072bc0 msgr2=0x7f0d60065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 shutdown_connections 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.809+0000 7f0d65044700 1 -- 172.21.15.111:0/1025727118 wait complete. 2024-03-14T13:16:01.812 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:16:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:01 smithi111 ceph-mon[145314]: pgmap v357: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:01 smithi111 ceph-mon[145314]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:01 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1319558174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 -- 172.21.15.111:0/2173608171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 msgr2=0x7fb378102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 --2- 172.21.15.111:0/2173608171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378102140 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb368004940 tx=0x7fb368009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 -- 172.21.15.111:0/2173608171 shutdown_connections 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 --2- 172.21.15.111:0/2173608171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 0x7fb3780fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 --2- 172.21.15.111:0/2173608171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378102140 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 -- 172.21.15.111:0/2173608171 >> 172.21.15.111:0/2173608171 conn(0x7fb378072bc0 msgr2=0x7fb378072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 -- 172.21.15.111:0/2173608171 shutdown_connections 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.910+0000 7fb37e59c700 1 -- 172.21.15.111:0/2173608171 wait complete. 2024-03-14T13:16:01.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 Processor -- start 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 -- start start 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 0x7fb378103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb378103df0 con 0x7fb378101d70 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb37e59c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb378103f30 con 0x7fb378102710 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb377fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb377fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58334/0 (socket says 172.21.15.111:58334) 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb377fff700 1 -- 172.21.15.111:0/4000764699 learned_addr learned my addr 172.21.15.111:0/4000764699 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.911+0000 7fb3777fe700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 0x7fb378103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb377fff700 1 -- 172.21.15.111:0/4000764699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 msgr2=0x7fb378103860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:01.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb377fff700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 0x7fb378103860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb377fff700 1 -- 172.21.15.111:0/4000764699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb368005040 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb377fff700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb36800b3b0 tx=0x7fb368009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36800e040 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb368013420 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb378104180 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36801c630 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.912+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37811a1b0 con 0x7fb378101d70 2024-03-14T13:16:01.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.913+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3780fd4c0 con 0x7fb378101d70 2024-03-14T13:16:01.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.913+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb368013970 con 0x7fb378101d70 2024-03-14T13:16:01.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.914+0000 7fb374ff9700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 0x7fb36006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:01.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.914+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb36808ccc0 con 0x7fb378101d70 2024-03-14T13:16:01.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.914+0000 7fb3777fe700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 0x7fb36006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:01.915 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.914+0000 7fb3777fe700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 0x7fb36006ec70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007fd0 tx=0x7fb36c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:01.918 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:01.918+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb368056f40 con 0x7fb378101d70 2024-03-14T13:16:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:01 smithi137 ceph-mon[108145]: pgmap v357: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:01 smithi137 ceph-mon[108145]: from='client.34708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:02.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:01 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1319558174' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:16:02.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.127+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb378104a10 con 0x7fb36006c7c0 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.129+0000 7fb374ff9700 1 -- 172.21.15.111:0/4000764699 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb378104a10 con 0x7fb36006c7c0 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:16:02.129 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:16:02.130 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 msgr2=0x7fb36006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 0x7fb36006ec70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb36c007fd0 tx=0x7fb36c009040 comp rx=0 tx=0).stop 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 msgr2=0x7fb378103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb36800b3b0 tx=0x7fb368009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 shutdown_connections 2024-03-14T13:16:02.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb36006c7c0 0x7fb36006ec70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb378101d70 0x7fb378103320 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 --2- 172.21.15.111:0/4000764699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb378102710 0x7fb378103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 >> 172.21.15.111:0/4000764699 conn(0x7fb378072bc0 msgr2=0x7fb378105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 shutdown_connections 2024-03-14T13:16:02.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.131+0000 7fb37e59c700 1 -- 172.21.15.111:0/4000764699 wait complete. 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 -- 172.21.15.111:0/3464700468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 msgr2=0x7fc8a81031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 --2- 172.21.15.111:0/3464700468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a81031d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc89c004940 tx=0x7fc89c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 -- 172.21.15.111:0/3464700468 shutdown_connections 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 --2- 172.21.15.111:0/3464700468 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 0x7fc8a80fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 --2- 172.21.15.111:0/3464700468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a81031d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 -- 172.21.15.111:0/3464700468 >> 172.21.15.111:0/3464700468 conn(0x7fc8a80f89b0 msgr2=0x7fc8a80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 -- 172.21.15.111:0/3464700468 shutdown_connections 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 -- 172.21.15.111:0/3464700468 wait complete. 2024-03-14T13:16:02.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.231+0000 7fc8aef86700 1 Processor -- start 2024-03-14T13:16:02.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8aef86700 1 -- start start 2024-03-14T13:16:02.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8aef86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8aef86700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 0x7fc8a8070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8aef86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8070d90 con 0x7fc8a8102e00 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8aef86700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8070ed0 con 0x7fc8a81037a0 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58352/0 (socket says 172.21.15.111:58352) 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 -- 172.21.15.111:0/2503120782 learned_addr learned my addr 172.21.15.111:0/2503120782 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8a7fff700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 0x7fc8a8070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 -- 172.21.15.111:0/2503120782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 msgr2=0x7fc8a8070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 0x7fc8a8070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 -- 172.21.15.111:0/2503120782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c005040 con 0x7fc8a8102e00 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.232+0000 7fc8acd22700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b3b0 tx=0x7fc89c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:02.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.233+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc89c00e040 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.233+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a8071120 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.233+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc89c013420 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.233+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc89c01c630 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.233+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a811a270 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.234+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a80fe550 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.234+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc89c013970 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.234+0000 7fc8a57fa700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.234+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc89c08ccc0 con 0x7fc8a8102e00 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.234+0000 7fc8a7fff700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:02.236 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.235+0000 7fc8a7fff700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 0x7fc89806ec70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc894009fd0 tx=0x7fc894009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:02.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.239+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc89c056f40 con 0x7fc8a8102e00 2024-03-14T13:16:02.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.510+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc8a80719b0 con 0x7fc8a8102e00 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.510+0000 7fc8a57fa700 1 -- 172.21.15.111:0/2503120782 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc89c05a560 con 0x7fc8a8102e00 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:16:02.511 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.512+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 msgr2=0x7fc89806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.512+0000 7fc8aef86700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 0x7fc89806ec70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc894009fd0 tx=0x7fc894009f20 comp rx=0 tx=0).stop 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.512+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 msgr2=0x7fc8a80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.512+0000 7fc8aef86700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b3b0 tx=0x7fc89c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 shutdown_connections 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc89806c7c0 0x7fc89806ec70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8a8102e00 0x7fc8a80702c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 --2- 172.21.15.111:0/2503120782 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8a81037a0 0x7fc8a8070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:02.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 >> 172.21.15.111:0/2503120782 conn(0x7fc8a80f89b0 msgr2=0x7fc8a80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:02.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 shutdown_connections 2024-03-14T13:16:02.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:02.513+0000 7fc8aef86700 1 -- 172.21.15.111:0/2503120782 wait complete. 2024-03-14T13:16:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:02 smithi111 ceph-mon[145314]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:02 smithi111 ceph-mon[145314]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:02 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1025727118' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:16:02.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:02 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2503120782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:16:03.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:02 smithi137 ceph-mon[108145]: from='client.34712 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:03.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:02 smithi137 ceph-mon[108145]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:03.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:02 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1025727118' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:16:03.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:02 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2503120782' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:16:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:03 smithi111 ceph-mon[145314]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:03 smithi111 ceph-mon[145314]: pgmap v358: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:03 smithi137 ceph-mon[108145]: from='client.34728 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:03 smithi137 ceph-mon[108145]: pgmap v358: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:05 smithi111 ceph-mon[145314]: pgmap v359: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:05 smithi137 ceph-mon[108145]: pgmap v359: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:07 smithi111 ceph-mon[145314]: pgmap v360: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:07 smithi137 ceph-mon[108145]: pgmap v360: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:09 smithi111 ceph-mon[145314]: pgmap v361: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:09 smithi137 ceph-mon[108145]: pgmap v361: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:10.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:10 smithi111 ceph-mon[145314]: pgmap v362: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:10 smithi137 ceph-mon[108145]: pgmap v362: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:13 smithi137 ceph-mon[108145]: pgmap v363: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:13 smithi111 ceph-mon[145314]: pgmap v363: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:15.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:15 smithi137 ceph-mon[108145]: pgmap v364: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:15 smithi111 ceph-mon[145314]: pgmap v364: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:17 smithi111 ceph-mon[145314]: pgmap v365: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:17 smithi137 ceph-mon[108145]: pgmap v365: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:19 smithi111 ceph-mon[145314]: pgmap v366: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:19 smithi137 ceph-mon[108145]: pgmap v366: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:21 smithi111 ceph-mon[145314]: pgmap v367: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:21 smithi137 ceph-mon[108145]: pgmap v367: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:23 smithi111 ceph-mon[145314]: pgmap v368: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:23 smithi137 ceph-mon[108145]: pgmap v368: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:25 smithi111 ceph-mon[145314]: pgmap v369: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:25 smithi137 ceph-mon[108145]: pgmap v369: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:27 smithi111 ceph-mon[145314]: pgmap v370: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:27 smithi137 ceph-mon[108145]: pgmap v370: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:29 smithi111 ceph-mon[145314]: pgmap v371: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:29 smithi137 ceph-mon[108145]: pgmap v371: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:31 smithi111 ceph-mon[145314]: pgmap v372: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:31 smithi137 ceph-mon[108145]: pgmap v372: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- 172.21.15.111:0/1289103267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 msgr2=0x7f6294102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 --2- 172.21.15.111:0/1289103267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294102140 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6290004940 tx=0x7f6290009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- 172.21.15.111:0/1289103267 shutdown_connections 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 --2- 172.21.15.111:0/1289103267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6294102710 0x7f62940fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 --2- 172.21.15.111:0/1289103267 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294102140 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- 172.21.15.111:0/1289103267 >> 172.21.15.111:0/1289103267 conn(0x7f6294072bc0 msgr2=0x7f6294072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- 172.21.15.111:0/1289103267 shutdown_connections 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- 172.21.15.111:0/1289103267 wait complete. 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 Processor -- start 2024-03-14T13:16:32.628 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- start start 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6294102710 0x7f6294113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294114640 con 0x7f6294101d70 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.627+0000 7f629bdf6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629410da80 con 0x7f6294102710 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47708/0 (socket says 172.21.15.111:47708) 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 -- 172.21.15.111:0/2647970817 learned_addr learned my addr 172.21.15.111:0/2647970817 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 -- 172.21.15.111:0/2647970817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6294102710 msgr2=0x7f6294113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6294102710 0x7f6294113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 -- 172.21.15.111:0/2647970817 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290005040 con 0x7f6294101d70 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f6299b92700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f629000b3b0 tx=0x7f6290009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f629000e040 con 0x7f6294101d70 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6290013420 con 0x7f6294101d70 2024-03-14T13:16:32.629 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629410dd00 con 0x7f6294101d70 2024-03-14T13:16:32.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f629001c630 con 0x7f6294101d70 2024-03-14T13:16:32.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.628+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629410e2c0 con 0x7f6294101d70 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.629+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62940fd4c0 con 0x7f6294101d70 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.630+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6290013970 con 0x7f6294101d70 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.630+0000 7f628a7fc700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 0x7f628006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.630+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f629008cce0 con 0x7f6294101d70 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.630+0000 7f6299391700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 0x7f628006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:32.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.631+0000 7f6299391700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 0x7f628006ec70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f629410ebf0 tx=0x7f6284009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:32.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.635+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6290091070 con 0x7f6294101d70 2024-03-14T13:16:32.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.845+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6294108dc0 con 0x7f628006c7c0 2024-03-14T13:16:32.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.846+0000 7f628a7fc700 1 -- 172.21.15.111:0/2647970817 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6294108dc0 con 0x7f628006c7c0 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 msgr2=0x7f628006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 0x7f628006ec70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f629410ebf0 tx=0x7f6284009ce0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 msgr2=0x7f6294113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f629000b3b0 tx=0x7f6290009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 shutdown_connections 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f628006c7c0 0x7f628006ec70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6294101d70 0x7f6294113ab0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 --2- 172.21.15.111:0/2647970817 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6294102710 0x7f6294113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 >> 172.21.15.111:0/2647970817 conn(0x7f6294072bc0 msgr2=0x7f6294065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 shutdown_connections 2024-03-14T13:16:32.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.848+0000 7f629bdf6700 1 -- 172.21.15.111:0/2647970817 wait complete. 2024-03-14T13:16:32.858 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.949+0000 7f4c119e4700 1 -- 172.21.15.111:0/136251846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c0fb4d0 msgr2=0x7f4c0c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.949+0000 7f4c119e4700 1 --2- 172.21.15.111:0/136251846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c0fb940 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc004940 tx=0x7f4bfc009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- 172.21.15.111:0/136251846 shutdown_connections 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 --2- 172.21.15.111:0/136251846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c0fb940 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 --2- 172.21.15.111:0/136251846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c102f50 0x7f4c0c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- 172.21.15.111:0/136251846 >> 172.21.15.111:0/136251846 conn(0x7f4c0c0714c0 msgr2=0x7f4c0c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- 172.21.15.111:0/136251846 shutdown_connections 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- 172.21.15.111:0/136251846 wait complete. 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 Processor -- start 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- start start 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c113920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c1144b0 con 0x7f4c0c102f50 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.950+0000 7f4c119e4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c10dab0 con 0x7f4c0c0fb4d0 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47728/0 (socket says 172.21.15.111:47728) 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 -- 172.21.15.111:0/956812008 learned_addr learned my addr 172.21.15.111:0/956812008 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 -- 172.21.15.111:0/956812008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 msgr2=0x7f4c0c113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0affd700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 -- 172.21.15.111:0/956812008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc005040 con 0x7f4c0c102f50 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0affd700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c0a7fc700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00b3b0 tx=0x7f4bfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bfc00e040 con 0x7f4c0c102f50 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c10dd30 con 0x7f4c0c102f50 2024-03-14T13:16:32.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bfc013420 con 0x7f4c0c102f50 2024-03-14T13:16:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bfc01c630 con 0x7f4c0c102f50 2024-03-14T13:16:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.951+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c10e270 con 0x7f4c0c102f50 2024-03-14T13:16:32.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.952+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c105330 con 0x7f4c0c102f50 2024-03-14T13:16:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.953+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4bfc013970 con 0x7f4c0c102f50 2024-03-14T13:16:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.953+0000 7f4bf37fe700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 0x7f4bf406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.953+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4bfc08cce0 con 0x7f4c0c102f50 2024-03-14T13:16:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.953+0000 7f4c0affd700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 0x7f4bf406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:32.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.954+0000 7f4c0affd700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 0x7f4bf406ec70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c00007fd0 tx=0x7f4c00009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:32.957 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:32.958+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4bfc091070 con 0x7f4c0c102f50 2024-03-14T13:16:33.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.173+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c0c10ef70 con 0x7f4bf406c7c0 2024-03-14T13:16:33.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.174+0000 7f4bf37fe700 1 -- 172.21.15.111:0/956812008 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4c0c10ef70 con 0x7f4bf406c7c0 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 msgr2=0x7f4bf406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 0x7f4bf406ec70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4c00007fd0 tx=0x7f4c00009040 comp rx=0 tx=0).stop 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 msgr2=0x7f4c0c113e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00b3b0 tx=0x7f4bfc009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 shutdown_connections 2024-03-14T13:16:33.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bf406c7c0 0x7f4bf406ec70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c0c0fb4d0 0x7f4c0c113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 --2- 172.21.15.111:0/956812008 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c0c102f50 0x7f4c0c113e60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 >> 172.21.15.111:0/956812008 conn(0x7f4c0c0714c0 msgr2=0x7f4c0c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 shutdown_connections 2024-03-14T13:16:33.176 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.176+0000 7f4c119e4700 1 -- 172.21.15.111:0/956812008 wait complete. 2024-03-14T13:16:33.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.276+0000 7f4074420700 1 -- 172.21.15.111:0/2413877458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c102710 msgr2=0x7f406c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.276+0000 7f4074420700 1 --2- 172.21.15.111:0/2413877458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c102710 0x7f406c0fbd80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f405c004940 tx=0x7f405c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.276+0000 7f4074420700 1 -- 172.21.15.111:0/2413877458 shutdown_connections 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.276+0000 7f4074420700 1 --2- 172.21.15.111:0/2413877458 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c102710 0x7f406c0fbd80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 --2- 172.21.15.111:0/2413877458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c101d70 0x7f406c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- 172.21.15.111:0/2413877458 >> 172.21.15.111:0/2413877458 conn(0x7f406c072bc0 msgr2=0x7f406c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- 172.21.15.111:0/2413877458 shutdown_connections 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- 172.21.15.111:0/2413877458 wait complete. 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 Processor -- start 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- start start 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c102710 0x7f406c10f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c10f680 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f4074420700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c10f7c0 con 0x7f406c102710 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f40721bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f40721bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47754/0 (socket says 172.21.15.111:47754) 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.277+0000 7f40721bc700 1 -- 172.21.15.111:0/267580377 learned_addr learned my addr 172.21.15.111:0/267580377 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f40719bb700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c102710 0x7f406c10f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f40721bc700 1 -- 172.21.15.111:0/267580377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c102710 msgr2=0x7f406c10f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f40721bc700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c102710 0x7f406c10f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f40721bc700 1 -- 172.21.15.111:0/267580377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c005040 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f40721bc700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f406800c830 tx=0x7f4068004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f406800a760 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f406800ad70 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c117ab0 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40680113a0 con 0x7f406c101d70 2024-03-14T13:16:33.279 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.278+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c118040 con 0x7f406c101d70 2024-03-14T13:16:33.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.279+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c0ff080 con 0x7f406c101d70 2024-03-14T13:16:33.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.279+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4068012400 con 0x7f406c101d70 2024-03-14T13:16:33.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.280+0000 7f4062ffd700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 0x7f405806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.280+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f406808bb60 con 0x7f406c101d70 2024-03-14T13:16:33.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.280+0000 7f40719bb700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 0x7f405806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.280+0000 7f40719bb700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 0x7f405806ec70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b3b0 tx=0x7f405c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.283 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.284+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4068055de0 con 0x7f406c101d70 2024-03-14T13:16:33.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.484+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f406c108db0 con 0x7f405806c7c0 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.494+0000 7f4062ffd700 1 -- 172.21.15.111:0/267580377 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f405c007d50 con 0x7f405806c7c0 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (12m) 6m ago 24m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (10m) 6m ago 24m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (10m) 5m ago 23m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:16:33.493 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (12m) 6m ago 23m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (6m) 6m ago 18m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (6m) 6m ago 18m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (5m) 5m ago 18m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (6m) 5m ago 18m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (15m) 6m ago 25m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (13m) 5m ago 22m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (11m) 6m ago 25m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (10m) 5m ago 22m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (13m) 6m ago 24m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (13m) 5m ago 23m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (9m) 6m ago 22m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (9m) 6m ago 21m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:16:33.494 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (8m) 6m ago 21m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (8m) 5m ago 20m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (7m) 5m ago 20m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (6m) 5m ago 19m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (13m) 6m ago 23m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 msgr2=0x7f405806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.495 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 0x7f405806ec70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f405c00b3b0 tx=0x7f405c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 msgr2=0x7f406c10ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f406800c830 tx=0x7f4068004470 comp rx=0 tx=0).stop 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 shutdown_connections 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f405806c7c0 0x7f405806ec70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f406c101d70 0x7f406c10ec00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 --2- 172.21.15.111:0/267580377 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f406c102710 0x7f406c10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 >> 172.21.15.111:0/267580377 conn(0x7f406c072bc0 msgr2=0x7f406c0fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 shutdown_connections 2024-03-14T13:16:33.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.496+0000 7f4074420700 1 -- 172.21.15.111:0/267580377 wait complete. 2024-03-14T13:16:33.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 -- 172.21.15.111:0/2404227428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 msgr2=0x7f647c0fe1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 --2- 172.21.15.111:0/2404227428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fe1d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f646c004940 tx=0x7f646c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 -- 172.21.15.111:0/2404227428 shutdown_connections 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 --2- 172.21.15.111:0/2404227428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 0x7f647c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 --2- 172.21.15.111:0/2404227428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fe1d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 -- 172.21.15.111:0/2404227428 >> 172.21.15.111:0/2404227428 conn(0x7f647c0f78d0 msgr2=0x7f647c0f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 -- 172.21.15.111:0/2404227428 shutdown_connections 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.596+0000 7f6481afb700 1 -- 172.21.15.111:0/2404227428 wait complete. 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 Processor -- start 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 -- start start 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 0x7f647c0fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0fc790 con 0x7f647c0fde00 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f6481afb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0fc8d0 con 0x7f647c0fe7a0 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f647b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f647b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47782/0 (socket says 172.21.15.111:47782) 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.597+0000 7f647b7fe700 1 -- 172.21.15.111:0/1687066468 learned_addr learned my addr 172.21.15.111:0/1687066468 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f647b7fe700 1 -- 172.21.15.111:0/1687066468 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 msgr2=0x7f647c0fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f647affd700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 0x7f647c0fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f647b7fe700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 0x7f647c0fc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f647b7fe700 1 -- 172.21.15.111:0/1687066468 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c005040 con 0x7f647c0fde00 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f647b7fe700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f646c009380 tx=0x7f646c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646c00e040 con 0x7f647c0fde00 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c119dc0 con 0x7f647c0fde00 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f646c004290 con 0x7f647c0fde00 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646c01c3f0 con 0x7f647c0fde00 2024-03-14T13:16:33.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.598+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c11a220 con 0x7f647c0fde00 2024-03-14T13:16:33.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.599+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c066770 con 0x7f647c0fde00 2024-03-14T13:16:33.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.600+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f646c013790 con 0x7f647c0fde00 2024-03-14T13:16:33.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.600+0000 7f6473fff700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 0x7f646806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.600+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f646c08cd90 con 0x7f647c0fde00 2024-03-14T13:16:33.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.600+0000 7f647affd700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 0x7f646806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.600+0000 7f647affd700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 0x7f646806ec70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6464006fd0 tx=0x7f6464008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.604+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f646c057010 con 0x7f647c0fde00 2024-03-14T13:16:33.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.877+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f647c0fd320 con 0x7f647c0fde00 2024-03-14T13:16:33.877 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.878+0000 7f6473fff700 1 -- 172.21.15.111:0/1687066468 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f646c013420 con 0x7f647c0fde00 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:16:33.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 msgr2=0x7f646806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 0x7f646806ec70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6464006fd0 tx=0x7f6464008040 comp rx=0 tx=0).stop 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 msgr2=0x7f647c0fbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f646c009380 tx=0x7f646c009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 shutdown_connections 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f646806c7c0 0x7f646806ec70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f647c0fde00 0x7f647c0fbc30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 --2- 172.21.15.111:0/1687066468 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f647c0fe7a0 0x7f647c0fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 >> 172.21.15.111:0/1687066468 conn(0x7f647c0f78d0 msgr2=0x7f647c0f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 shutdown_connections 2024-03-14T13:16:33.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.880+0000 7f6481afb700 1 -- 172.21.15.111:0/1687066468 wait complete. 2024-03-14T13:16:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:33 smithi111 ceph-mon[145314]: pgmap v373: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:33.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3496926430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 msgr2=0x7f7cb00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3496926430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb00fbd80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7cac004940 tx=0x7f7cac009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:33.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3496926430 shutdown_connections 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3496926430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb00fbd80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3496926430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 0x7f7cb0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3496926430 >> 172.21.15.111:0/3496926430 conn(0x7f7cb0072bc0 msgr2=0x7f7cb0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3496926430 shutdown_connections 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.980+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3496926430 wait complete. 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 Processor -- start 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 -- start start 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 0x7f7cb0115cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0116850 con 0x7f7cb0102710 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb73f2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb010fc20 con 0x7f7cb0101d70 2024-03-14T13:16:33.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47800/0 (socket says 172.21.15.111:47800) 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 -- 172.21.15.111:0/3265933439 learned_addr learned my addr 172.21.15.111:0/3265933439 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb518e700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 0x7f7cb0115cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 -- 172.21.15.111:0/3265933439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 msgr2=0x7f7cb0115cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 0x7f7cb0115cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.981+0000 7f7cb498d700 1 -- 172.21.15.111:0/3265933439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac005040 con 0x7f7cb0102710 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7cb498d700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009380 tx=0x7f7cac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cac00e040 con 0x7f7cb0102710 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cac013420 con 0x7f7cb0102710 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7cac01c630 con 0x7f7cb0102710 2024-03-14T13:16:33.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb010fea0 con 0x7f7cb0102710 2024-03-14T13:16:33.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.982+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb0110460 con 0x7f7cb0102710 2024-03-14T13:16:33.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.983+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb00fd4c0 con 0x7f7cb0102710 2024-03-14T13:16:33.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.983+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7cac0139b0 con 0x7f7cb0102710 2024-03-14T13:16:33.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.984+0000 7f7ca1ffb700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 0x7f7c9c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:33.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.984+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7cac08caf0 con 0x7f7cb0102710 2024-03-14T13:16:33.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.984+0000 7f7cb518e700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 0x7f7c9c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:33.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.984+0000 7f7cb518e700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 0x7f7c9c06ec70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4004f40 tx=0x7f7ca400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:33.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:33.988+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7cac05a8c0 con 0x7f7cb0102710 2024-03-14T13:16:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:33 smithi137 ceph-mon[108145]: pgmap v373: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.222+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7cb004bed0 con 0x7f7cb0102710 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.223+0000 7f7ca1ffb700 1 -- 172.21.15.111:0/3265933439 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f7cac05a450 con 0x7f7cb0102710 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:16:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:16:34.223 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:16:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:34.225 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 msgr2=0x7f7c9c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 0x7f7c9c06ec70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4004f40 tx=0x7f7ca400b040 comp rx=0 tx=0).stop 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 msgr2=0x7f7cb0116200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009380 tx=0x7f7cac009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 shutdown_connections 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c9c06c7c0 0x7f7c9c06ec70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cb0101d70 0x7f7cb0115cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 --2- 172.21.15.111:0/3265933439 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cb0102710 0x7f7cb0116200 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 >> 172.21.15.111:0/3265933439 conn(0x7f7cb0072bc0 msgr2=0x7f7cb00659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:34.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 shutdown_connections 2024-03-14T13:16:34.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.225+0000 7f7cb73f2700 1 -- 172.21.15.111:0/3265933439 wait complete. 2024-03-14T13:16:34.227 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:16:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 -- 172.21.15.111:0/49801978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 msgr2=0x7ff3880fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/49801978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3880fe1f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff378004940 tx=0x7ff378009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 -- 172.21.15.111:0/49801978 shutdown_connections 2024-03-14T13:16:34.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/49801978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 0x7ff388106e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/49801978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3880fe1f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 -- 172.21.15.111:0/49801978 >> 172.21.15.111:0/49801978 conn(0x7ff3880f7870 msgr2=0x7ff3880f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 -- 172.21.15.111:0/49801978 shutdown_connections 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.325+0000 7ff38d2f9700 1 -- 172.21.15.111:0/49801978 wait complete. 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 Processor -- start 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 -- start start 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 0x7ff388113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3881144c0 con 0x7ff3880fde20 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff38d2f9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38810da30 con 0x7ff3880fe7c0 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff386ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff386ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47818/0 (socket says 172.21.15.111:47818) 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff386ffd700 1 -- 172.21.15.111:0/1738969492 learned_addr learned my addr 172.21.15.111:0/1738969492 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:34.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.326+0000 7ff3867fc700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 0x7ff388113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff386ffd700 1 -- 172.21.15.111:0/1738969492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 msgr2=0x7ff388113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff386ffd700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 0x7ff388113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff386ffd700 1 -- 172.21.15.111:0/1738969492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378005040 con 0x7ff3880fde20 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff386ffd700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff378004890 tx=0x7ff378009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff37800e040 con 0x7ff3880fde20 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff378009af0 con 0x7ff3880fde20 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38810dcb0 con 0x7ff3880fde20 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff37801c3f0 con 0x7ff3880fde20 2024-03-14T13:16:34.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.327+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38810e1f0 con 0x7ff3880fde20 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.328+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff388066b70 con 0x7ff3880fde20 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.328+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff378013420 con 0x7ff3880fde20 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.329+0000 7ff36f7fe700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 0x7ff37006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.329+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff37808ce30 con 0x7ff3880fde20 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.329+0000 7ff3867fc700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 0x7ff37006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.329+0000 7ff3867fc700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 0x7ff37006ec70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006fd0 tx=0x7ff37c008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:34.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.333+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff37805ab40 con 0x7ff3880fde20 2024-03-14T13:16:34.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.551+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff388109e60 con 0x7ff37006c7c0 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.553+0000 7ff36f7fe700 1 -- 172.21.15.111:0/1738969492 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff388109e60 con 0x7ff37006c7c0 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:16:34.552 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:16:34.553 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 msgr2=0x7ff37006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 0x7ff37006ec70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff37c006fd0 tx=0x7ff37c008040 comp rx=0 tx=0).stop 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 msgr2=0x7ff3881138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff378004890 tx=0x7ff378009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 shutdown_connections 2024-03-14T13:16:34.554 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff37006c7c0 0x7ff37006ec70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3880fde20 0x7ff3881138a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 --2- 172.21.15.111:0/1738969492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3880fe7c0 0x7ff388113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 >> 172.21.15.111:0/1738969492 conn(0x7ff3880f7870 msgr2=0x7ff3880f8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 shutdown_connections 2024-03-14T13:16:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.555+0000 7ff38d2f9700 1 -- 172.21.15.111:0/1738969492 wait complete. 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- 172.21.15.111:0/2209023332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9218102e10 msgr2=0x7f92181031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 --2- 172.21.15.111:0/2209023332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9218102e10 0x7f92181031e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9210004940 tx=0x7f9210009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- 172.21.15.111:0/2209023332 shutdown_connections 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 --2- 172.21.15.111:0/2209023332 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92181037b0 0x7f92180fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 --2- 172.21.15.111:0/2209023332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9218102e10 0x7f92181031e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- 172.21.15.111:0/2209023332 >> 172.21.15.111:0/2209023332 conn(0x7f92180f8980 msgr2=0x7f92180fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- 172.21.15.111:0/2209023332 shutdown_connections 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- 172.21.15.111:0/2209023332 wait complete. 2024-03-14T13:16:34.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 Processor -- start 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.655+0000 7f922034f700 1 -- start start 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9218102e10 0x7f92181138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9218114450 con 0x7f92181037b0 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921810da50 con 0x7f9218102e10 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:47836/0 (socket says 172.21.15.111:47836) 2024-03-14T13:16:34.659 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 -- 172.21.15.111:0/4248955146 learned_addr learned my addr 172.21.15.111:0/4248955146 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921e0eb700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9218102e10 0x7f92181138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 -- 172.21.15.111:0/4248955146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9218102e10 msgr2=0x7f92181138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9218102e10 0x7f92181138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 -- 172.21.15.111:0/4248955146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9210005040 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f921d8ea700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f920800c800 tx=0x7f9208004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f920800a760 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f920800ad70 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921810dd30 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92080113a0 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.656+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921810e2f0 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.657+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92180fe560 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.658+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9208012400 con 0x7f92181037b0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.658+0000 7f920effd700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 0x7f920406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.658+0000 7f921e0eb700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 0x7f920406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:16:34.660 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.658+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f920808ba90 con 0x7f92181037b0 2024-03-14T13:16:34.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.659+0000 7f921e0eb700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 0x7f920406ec70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9210004f90 tx=0x7f9210009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:16:34.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.663+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f92080597a0 con 0x7f92181037b0 2024-03-14T13:16:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:34 smithi111 ceph-mon[145314]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:34 smithi111 ceph-mon[145314]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:34 smithi111 ceph-mon[145314]: from='client.34744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:34 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1687066468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:16:34.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:34 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3265933439' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:16:34.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.934+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9218109e00 con 0x7f92181037b0 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.934+0000 7f920effd700 1 -- 172.21.15.111:0/4248955146 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f9208059330 con 0x7f92181037b0 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:16:34.934 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 msgr2=0x7f920406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 0x7f920406ec70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9210004f90 tx=0x7f9210009e30 comp rx=0 tx=0).stop 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 msgr2=0x7f9218113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f920800c800 tx=0x7f9208004470 comp rx=0 tx=0).stop 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 shutdown_connections 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f920406c7c0 0x7f920406ec70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9218102e10 0x7f92181138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 --2- 172.21.15.111:0/4248955146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92181037b0 0x7f9218113e00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 >> 172.21.15.111:0/4248955146 conn(0x7f92180f8980 msgr2=0x7f92180f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:16:34.936 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 shutdown_connections 2024-03-14T13:16:34.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:16:34.936+0000 7f922034f700 1 -- 172.21.15.111:0/4248955146 wait complete. 2024-03-14T13:16:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:34 smithi137 ceph-mon[108145]: from='client.34736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:34 smithi137 ceph-mon[108145]: from='client.34740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:35.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:34 smithi137 ceph-mon[108145]: from='client.34744 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:35.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:34 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1687066468' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:16:35.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:34 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3265933439' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:16:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:35 smithi111 ceph-mon[145314]: pgmap v374: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:35 smithi111 ceph-mon[145314]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:35 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4248955146' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:16:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:35 smithi137 ceph-mon[108145]: pgmap v374: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:35 smithi137 ceph-mon[108145]: from='client.34756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:16:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:35 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4248955146' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:16:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:37 smithi111 ceph-mon[145314]: pgmap v375: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:37 smithi137 ceph-mon[108145]: pgmap v375: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:39 smithi111 ceph-mon[145314]: pgmap v376: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:39 smithi137 ceph-mon[108145]: pgmap v376: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:41 smithi111 ceph-mon[145314]: pgmap v377: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:41 smithi137 ceph-mon[108145]: pgmap v377: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:43 smithi111 ceph-mon[145314]: pgmap v378: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:43 smithi137 ceph-mon[108145]: pgmap v378: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:45 smithi111 ceph-mon[145314]: pgmap v379: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:45 smithi137 ceph-mon[108145]: pgmap v379: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:47 smithi111 ceph-mon[145314]: pgmap v380: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:47 smithi137 ceph-mon[108145]: pgmap v380: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:49 smithi111 ceph-mon[145314]: pgmap v381: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:49 smithi137 ceph-mon[108145]: pgmap v381: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:51 smithi111 ceph-mon[145314]: pgmap v382: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:51 smithi137 ceph-mon[108145]: pgmap v382: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:53 smithi111 ceph-mon[145314]: pgmap v383: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:53 smithi137 ceph-mon[108145]: pgmap v383: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:16:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:55 smithi111 ceph-mon[145314]: pgmap v384: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:55 smithi137 ceph-mon[108145]: pgmap v384: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:56.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:56 smithi111 ceph-mon[145314]: pgmap v385: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:57.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:56 smithi137 ceph-mon[108145]: pgmap v385: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:59.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:16:59 smithi137 ceph-mon[108145]: pgmap v386: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:16:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:16:59 smithi111 ceph-mon[145314]: pgmap v386: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:01 smithi111 ceph-mon[145314]: pgmap v387: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:01 smithi137 ceph-mon[108145]: pgmap v387: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:03 smithi111 ceph-mon[145314]: pgmap v388: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:03 smithi137 ceph-mon[108145]: pgmap v388: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 -- 172.21.15.111:0/1068444591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 msgr2=0x7fea581020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 --2- 172.21.15.111:0/1068444591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea581020e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea4c004940 tx=0x7fea4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 -- 172.21.15.111:0/1068444591 shutdown_connections 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 --2- 172.21.15.111:0/1068444591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 0x7fea580fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 --2- 172.21.15.111:0/1068444591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea581020e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 -- 172.21.15.111:0/1068444591 >> 172.21.15.111:0/1068444591 conn(0x7fea580f7880 msgr2=0x7fea580f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 -- 172.21.15.111:0/1068444591 shutdown_connections 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.049+0000 7fea5dd70700 1 -- 172.21.15.111:0/1068444591 wait complete. 2024-03-14T13:17:05.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 Processor -- start 2024-03-14T13:17:05.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 -- start start 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 0x7fea5810f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5810f950 con 0x7fea58101d10 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea5dd70700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5810fa90 con 0x7fea581026b0 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41684/0 (socket says 172.21.15.111:41684) 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea577fe700 1 -- 172.21.15.111:0/206192081 learned_addr learned my addr 172.21.15.111:0/206192081 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea56ffd700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 0x7fea5810f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea577fe700 1 -- 172.21.15.111:0/206192081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 msgr2=0x7fea5810f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.050+0000 7fea577fe700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 0x7fea5810f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea577fe700 1 -- 172.21.15.111:0/206192081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c005040 con 0x7fea58101d10 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea577fe700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00b3b0 tx=0x7fea4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea4c00e040 con 0x7fea58101d10 2024-03-14T13:17:05.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea58117e30 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea4c013420 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea4c01c610 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.051+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea581182e0 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.052+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580fd460 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.052+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fea4c00ba80 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.053+0000 7fea3ffff700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 0x7fea4006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.053+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fea4c08cbe0 con 0x7fea58101d10 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.053+0000 7fea56ffd700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 0x7fea4006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.053+0000 7fea56ffd700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 0x7fea4006ec70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fea58066890 tx=0x7fea4800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.057+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fea4c056e60 con 0x7fea58101d10 2024-03-14T13:17:05.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.268+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fea580fd460 con 0x7fea4006c7c0 2024-03-14T13:17:05.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.269+0000 7fea3ffff700 1 -- 172.21.15.111:0/206192081 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fea580fd460 con 0x7fea4006c7c0 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 msgr2=0x7fea4006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 0x7fea4006ec70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fea58066890 tx=0x7fea4800b040 comp rx=0 tx=0).stop 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 msgr2=0x7fea5810edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00b3b0 tx=0x7fea4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 shutdown_connections 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fea4006c7c0 0x7fea4006ec70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea58101d10 0x7fea5810edc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 --2- 172.21.15.111:0/206192081 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea581026b0 0x7fea5810f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 >> 172.21.15.111:0/206192081 conn(0x7fea580f7880 msgr2=0x7fea580f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 shutdown_connections 2024-03-14T13:17:05.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.272+0000 7fea5dd70700 1 -- 172.21.15.111:0/206192081 wait complete. 2024-03-14T13:17:05.282 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:17:05.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 -- 172.21.15.111:0/1471077637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 msgr2=0x7fefe00fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 --2- 172.21.15.111:0/1471077637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe00fce20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fefd0004940 tx=0x7fefd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 -- 172.21.15.111:0/1471077637 shutdown_connections 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 --2- 172.21.15.111:0/1471077637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe00fce20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 --2- 172.21.15.111:0/1471077637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 0x7fefe01031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 -- 172.21.15.111:0/1471077637 >> 172.21.15.111:0/1471077637 conn(0x7fefe00f8980 msgr2=0x7fefe00fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 -- 172.21.15.111:0/1471077637 shutdown_connections 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.373+0000 7fefe5caa700 1 -- 172.21.15.111:0/1471077637 wait complete. 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 Processor -- start 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 -- start start 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 0x7fefe0113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.376 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01145f0 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefe5caa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe010d9e0 con 0x7fefe0102e10 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41706/0 (socket says 172.21.15.111:41706) 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdeffd700 1 -- 172.21.15.111:0/2889785428 learned_addr learned my addr 172.21.15.111:0/2889785428 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 0x7fefe0113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdeffd700 1 -- 172.21.15.111:0/2889785428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 msgr2=0x7fefe0113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.374+0000 7fefdeffd700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 0x7fefe0113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefdeffd700 1 -- 172.21.15.111:0/2889785428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0005040 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefdeffd700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000c00 tx=0x7fefd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefd000e070 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefd0013de0 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe010dc60 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefd001cde0 con 0x7fefe01037b0 2024-03-14T13:17:05.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.375+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe010e1c0 con 0x7fefe01037b0 2024-03-14T13:17:05.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.376+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe00fe560 con 0x7fefe01037b0 2024-03-14T13:17:05.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.376+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fefd0026460 con 0x7fefe01037b0 2024-03-14T13:17:05.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.377+0000 7fefd7fff700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 0x7fefcc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.377+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fefd008cff0 con 0x7fefe01037b0 2024-03-14T13:17:05.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.377+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 0x7fefcc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.377+0000 7fefdf7fe700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 0x7fefcc06ec70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fefc8004600 tx=0x7fefc800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.382+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fefd005ad70 con 0x7fefe01037b0 2024-03-14T13:17:05.592 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.591+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefe0113730 con 0x7fefcc06c7c0 2024-03-14T13:17:05.593 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.593+0000 7fefd7fff700 1 -- 172.21.15.111:0/2889785428 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fefe0113730 con 0x7fefcc06c7c0 2024-03-14T13:17:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 msgr2=0x7fefcc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 0x7fefcc06ec70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fefc8004600 tx=0x7fefc800b040 comp rx=0 tx=0).stop 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 msgr2=0x7fefe0113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fefd0000c00 tx=0x7fefd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 shutdown_connections 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fefcc06c7c0 0x7fefcc06ec70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fefe0102e10 0x7fefe0113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 --2- 172.21.15.111:0/2889785428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fefe01037b0 0x7fefe0113fa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 >> 172.21.15.111:0/2889785428 conn(0x7fefe00f8980 msgr2=0x7fefe00f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 shutdown_connections 2024-03-14T13:17:05.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.595+0000 7fefe5caa700 1 -- 172.21.15.111:0/2889785428 wait complete. 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- 172.21.15.111:0/2106493801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa01026b0 msgr2=0x7faaa00fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 --2- 172.21.15.111:0/2106493801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa01026b0 0x7faaa00fbd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004940 tx=0x7faa9c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- 172.21.15.111:0/2106493801 shutdown_connections 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 --2- 172.21.15.111:0/2106493801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa01026b0 0x7faaa00fbd20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 --2- 172.21.15.111:0/2106493801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa0101d10 0x7faaa01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- 172.21.15.111:0/2106493801 >> 172.21.15.111:0/2106493801 conn(0x7faaa00f7880 msgr2=0x7faaa00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- 172.21.15.111:0/2106493801 shutdown_connections 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- 172.21.15.111:0/2106493801 wait complete. 2024-03-14T13:17:05.698 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 Processor -- start 2024-03-14T13:17:05.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.696+0000 7faaa860a700 1 -- start start 2024-03-14T13:17:05.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 0x7faaa01164f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa0116b40 con 0x7faaa0101d10 2024-03-14T13:17:05.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa010ff10 con 0x7faaa01026b0 2024-03-14T13:17:05.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41714/0 (socket says 172.21.15.111:41714) 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 -- 172.21.15.111:0/1348556801 learned_addr learned my addr 172.21.15.111:0/1348556801 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 -- 172.21.15.111:0/1348556801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 msgr2=0x7faaa01164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa5ba5700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 0x7faaa01164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 0x7faaa01164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 -- 172.21.15.111:0/1348556801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c005040 con 0x7faaa0101d10 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa5ba5700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 0x7faaa01164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa63a6700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c8c0 tx=0x7faa900044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa9000a980 con 0x7faaa0101d10 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa01101f0 con 0x7faaa0101d10 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa9000ce60 con 0x7faaa0101d10 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa900113a0 con 0x7faaa0101d10 2024-03-14T13:17:05.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.697+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa01107b0 con 0x7faaa0101d10 2024-03-14T13:17:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.698+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa00fd460 con 0x7faaa0101d10 2024-03-14T13:17:05.732 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.699+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faa90012400 con 0x7faaa0101d10 2024-03-14T13:17:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.699+0000 7faa96ffd700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 0x7faa8c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.699+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa9008bb90 con 0x7faaa0101d10 2024-03-14T13:17:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.699+0000 7faaa5ba5700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 0x7faa8c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.700+0000 7faaa5ba5700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 0x7faa8c06ec70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c00 tx=0x7faa9c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:05.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.705+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faa900598a0 con 0x7faaa0101d10 2024-03-14T13:17:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:05 smithi111 ceph-mon[145314]: pgmap v389: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:05.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.907+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7faaa004bed0 con 0x7faa8c06c7c0 2024-03-14T13:17:05.916 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.916+0000 7faa96ffd700 1 -- 172.21.15.111:0/1348556801 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7faaa004bed0 con 0x7faa8c06c7c0 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (13m) 6m ago 24m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (10m) 6m ago 25m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (10m) 6m ago 23m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (12m) 6m ago 24m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (6m) 6m ago 18m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (7m) 6m ago 18m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (6m) 6m ago 18m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (6m) 6m ago 18m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (15m) 6m ago 26m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (14m) 6m ago 23m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (11m) 6m ago 26m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (11m) 6m ago 23m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:17:05.917 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (14m) 6m ago 25m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (14m) 6m ago 23m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (10m) 6m ago 22m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (9m) 6m ago 22m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (9m) 6m ago 21m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (8m) 6m ago 21m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (8m) 6m ago 20m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (7m) 6m ago 20m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:17:05.918 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (13m) 6m ago 24m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 msgr2=0x7faa8c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 0x7faa8c06ec70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c00 tx=0x7faa9c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 msgr2=0x7faaa0115fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c8c0 tx=0x7faa900044c0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 shutdown_connections 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa8c06c7c0 0x7faa8c06ec70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faaa0101d10 0x7faaa0115fb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 --2- 172.21.15.111:0/1348556801 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faaa01026b0 0x7faaa01164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.918+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 >> 172.21.15.111:0/1348556801 conn(0x7faaa00f7880 msgr2=0x7faaa00f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.919+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 shutdown_connections 2024-03-14T13:17:05.919 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:05.919+0000 7faaa860a700 1 -- 172.21.15.111:0/1348556801 wait complete. 2024-03-14T13:17:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:05 smithi137 ceph-mon[108145]: pgmap v389: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- 172.21.15.111:0/2410201636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 msgr2=0x7ff014102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 --2- 172.21.15.111:0/2410201636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff014102140 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff004004940 tx=0x7ff004009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- 172.21.15.111:0/2410201636 shutdown_connections 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 --2- 172.21.15.111:0/2410201636 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff014102710 0x7ff0140fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 --2- 172.21.15.111:0/2410201636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff014102140 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- 172.21.15.111:0/2410201636 >> 172.21.15.111:0/2410201636 conn(0x7ff014072bc0 msgr2=0x7ff014072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- 172.21.15.111:0/2410201636 shutdown_connections 2024-03-14T13:17:06.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- 172.21.15.111:0/2410201636 wait complete. 2024-03-14T13:17:06.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 Processor -- start 2024-03-14T13:17:06.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.018+0000 7ff018975700 1 -- start start 2024-03-14T13:17:06.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff014102710 0x7ff014112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff014113550 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01410c940 con 0x7ff014102710 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41738/0 (socket says 172.21.15.111:41738) 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 -- 172.21.15.111:0/410479187 learned_addr learned my addr 172.21.15.111:0/410479187 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 -- 172.21.15.111:0/410479187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff014102710 msgr2=0x7ff014112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff014102710 0x7ff014112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 -- 172.21.15.111:0/410479187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff004005040 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff01259c700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff00400b3b0 tx=0x7ff004009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff00400e040 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01410cbc0 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff004013420 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff00401c630 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.019+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01410d180 con 0x7ff014101d70 2024-03-14T13:17:06.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.020+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0140ff080 con 0x7ff014101d70 2024-03-14T13:17:06.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.021+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff004013970 con 0x7ff014101d70 2024-03-14T13:17:06.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.021+0000 7ff002ffd700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 0x7feffc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.021+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff00408cce0 con 0x7ff014101d70 2024-03-14T13:17:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.021+0000 7ff011d9b700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 0x7feffc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.022+0000 7ff011d9b700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 0x7feffc06ec70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff01410db00 tx=0x7ff008009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.026+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff004091070 con 0x7ff014101d70 2024-03-14T13:17:06.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.296+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff014108d50 con 0x7ff014101d70 2024-03-14T13:17:06.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.298+0000 7ff002ffd700 1 -- 172.21.15.111:0/410479187 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7ff00405a9f0 con 0x7ff014101d70 2024-03-14T13:17:06.297 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:06.298 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:17:06.299 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:17:06.299 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:17:06.299 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:17:06.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 msgr2=0x7feffc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 0x7feffc06ec70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff01410db00 tx=0x7ff008009ce0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 msgr2=0x7ff0141129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff00400b3b0 tx=0x7ff004009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 shutdown_connections 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feffc06c7c0 0x7feffc06ec70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff014101d70 0x7ff0141129c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 --2- 172.21.15.111:0/410479187 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff014102710 0x7ff014112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 >> 172.21.15.111:0/410479187 conn(0x7ff014072bc0 msgr2=0x7ff0140fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 shutdown_connections 2024-03-14T13:17:06.300 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.299+0000 7ff018975700 1 -- 172.21.15.111:0/410479187 wait complete. 2024-03-14T13:17:06.399 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 -- 172.21.15.111:0/1286273357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 msgr2=0x7f6820106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 --2- 172.21.15.111:0/1286273357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f6820106f00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6810004940 tx=0x7f6810009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 -- 172.21.15.111:0/1286273357 shutdown_connections 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 --2- 172.21.15.111:0/1286273357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f6820106f00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 --2- 172.21.15.111:0/1286273357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 0x7f68200fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 -- 172.21.15.111:0/1286273357 >> 172.21.15.111:0/1286273357 conn(0x7f6820072bc0 msgr2=0x7f6820072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 -- 172.21.15.111:0/1286273357 shutdown_connections 2024-03-14T13:17:06.400 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.400+0000 7f6825268700 1 -- 172.21.15.111:0/1286273357 wait complete. 2024-03-14T13:17:06.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 Processor -- start 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 -- start start 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 0x7f68200fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200fc820 con 0x7f68200fe830 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f6825268700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200fc960 con 0x7f68200fde90 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41762/0 (socket says 172.21.15.111:41762) 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 -- 172.21.15.111:0/1221586176 learned_addr learned my addr 172.21.15.111:0/1221586176 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681effd700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 0x7f68200fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 -- 172.21.15.111:0/1221586176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 msgr2=0x7f68200fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 0x7f68200fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.401+0000 7f681e7fc700 1 -- 172.21.15.111:0/1221586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810005040 con 0x7f68200fe830 2024-03-14T13:17:06.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f681e7fc700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6810009390 tx=0x7f6810009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f681000e040 con 0x7f68200fe830 2024-03-14T13:17:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68200fcbb0 con 0x7f68200fe830 2024-03-14T13:17:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6810004290 con 0x7f68200fe830 2024-03-14T13:17:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f681001c3f0 con 0x7f68200fe830 2024-03-14T13:17:06.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.402+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682011a190 con 0x7f68200fe830 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.403+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6820065e30 con 0x7f68200fe830 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.403+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6810013420 con 0x7f68200fe830 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.404+0000 7f68077fe700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 0x7f680806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.404+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f681008cc70 con 0x7f68200fe830 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.404+0000 7f681effd700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 0x7f680806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.404+0000 7f681effd700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 0x7f680806ec70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6814004990 tx=0x7f6814009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.408+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6810056ef0 con 0x7f68200fe830 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.643+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6820109e00 con 0x7f68200fe830 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.644+0000 7f68077fe700 1 -- 172.21.15.111:0/1221586176 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f681005a510 con 0x7f68200fe830 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:17:06.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:17:06.644 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:06.645 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:06.646 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 msgr2=0x7f680806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 0x7f680806ec70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6814004990 tx=0x7f6814009ee0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 msgr2=0x7f68200fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6810009390 tx=0x7f6810009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 shutdown_connections 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f680806c7c0 0x7f680806ec70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68200fde90 0x7f68200fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 --2- 172.21.15.111:0/1221586176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68200fe830 0x7f68200fc290 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 >> 172.21.15.111:0/1221586176 conn(0x7f6820072bc0 msgr2=0x7f6820103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 shutdown_connections 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.646+0000 7f6825268700 1 -- 172.21.15.111:0/1221586176 wait complete. 2024-03-14T13:17:06.647 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:17:06.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.746+0000 7eff9c3f2700 1 -- 172.21.15.111:0/1441885287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070360 msgr2=0x7eff94070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.746+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/1441885287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070360 0x7eff94070730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7eff84004970 tx=0x7eff84009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 -- 172.21.15.111:0/1441885287 shutdown_connections 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/1441885287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070c70 0x7eff94109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/1441885287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070360 0x7eff94070730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 -- 172.21.15.111:0/1441885287 >> 172.21.15.111:0/1441885287 conn(0x7eff940f89b0 msgr2=0x7eff940fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 -- 172.21.15.111:0/1441885287 shutdown_connections 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 -- 172.21.15.111:0/1441885287 wait complete. 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 Processor -- start 2024-03-14T13:17:06.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.747+0000 7eff9c3f2700 1 -- start start 2024-03-14T13:17:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070360 0x7eff94115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff94116920 con 0x7eff94070c70 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9410fc60 con 0x7eff94070360 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41768/0 (socket says 172.21.15.111:41768) 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 -- 172.21.15.111:0/62371023 learned_addr learned my addr 172.21.15.111:0/62371023 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9a18e700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070360 0x7eff94115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 -- 172.21.15.111:0/62371023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070360 msgr2=0x7eff94115d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070360 0x7eff94115d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 -- 172.21.15.111:0/62371023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84005040 con 0x7eff94070c70 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9998d700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff9000b790 tx=0x7eff90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9000f770 con 0x7eff94070c70 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9410fee0 con 0x7eff94070c70 2024-03-14T13:17:06.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff9000fd80 con 0x7eff94070c70 2024-03-14T13:17:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9000d310 con 0x7eff94070c70 2024-03-14T13:17:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.748+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff94110420 con 0x7eff94070c70 2024-03-14T13:17:06.750 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.749+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94071aa0 con 0x7eff94070c70 2024-03-14T13:17:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.750+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7eff90010420 con 0x7eff94070c70 2024-03-14T13:17:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.750+0000 7eff8affd700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 0x7eff8006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.750+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7eff9008ba80 con 0x7eff94070c70 2024-03-14T13:17:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.750+0000 7eff9a18e700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 0x7eff8006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:06.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.751+0000 7eff9a18e700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 0x7eff8006ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7eff84004ee0 tx=0x7eff84009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:06.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.755+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7eff90059790 con 0x7eff94070c70 2024-03-14T13:17:06.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:06 smithi111 ceph-mon[145314]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:06.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:06 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/410479187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:17:06.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.966+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff9410c0a0 con 0x7eff8006c7d0 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.968+0000 7eff8affd700 1 -- 172.21.15.111:0/62371023 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7eff9410c0a0 con 0x7eff8006c7d0 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:17:06.967 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:17:06.968 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:17:06.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 msgr2=0x7eff8006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 0x7eff8006ec80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7eff84004ee0 tx=0x7eff84009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:06.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 msgr2=0x7eff94116240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:06.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff9000b790 tx=0x7eff90009250 comp rx=0 tx=0).stop 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 shutdown_connections 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7eff8006c7d0 0x7eff8006ec80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff94070360 0x7eff94115d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 --2- 172.21.15.111:0/62371023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff94070c70 0x7eff94116240 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 >> 172.21.15.111:0/62371023 conn(0x7eff940f89b0 msgr2=0x7eff940f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 shutdown_connections 2024-03-14T13:17:06.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:06.970+0000 7eff9c3f2700 1 -- 172.21.15.111:0/62371023 wait complete. 2024-03-14T13:17:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:06 smithi137 ceph-mon[108145]: from='client.34764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:07.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:06 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/410479187' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- 172.21.15.111:0/1597877005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd364102e10 msgr2=0x7fd3641031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 --2- 172.21.15.111:0/1597877005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd364102e10 0x7fd3641031e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd354004940 tx=0x7fd354009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- 172.21.15.111:0/1597877005 shutdown_connections 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 --2- 172.21.15.111:0/1597877005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3641037b0 0x7fd3640fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 --2- 172.21.15.111:0/1597877005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd364102e10 0x7fd3641031e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- 172.21.15.111:0/1597877005 >> 172.21.15.111:0/1597877005 conn(0x7fd3640f8980 msgr2=0x7fd3640fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- 172.21.15.111:0/1597877005 shutdown_connections 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- 172.21.15.111:0/1597877005 wait complete. 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 Processor -- start 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- start start 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd364102e10 0x7fd364113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd364114680 con 0x7fd3641037b0 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.071+0000 7fd36c60b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36410da70 con 0x7fd364102e10 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41790/0 (socket says 172.21.15.111:41790) 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 -- 172.21.15.111:0/2938166797 learned_addr learned my addr 172.21.15.111:0/2938166797 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd36a3a7700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd364102e10 0x7fd364113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 -- 172.21.15.111:0/2938166797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd364102e10 msgr2=0x7fd364113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:07.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd364102e10 0x7fd364113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 -- 172.21.15.111:0/2938166797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd354005040 con 0x7fd3641037b0 2024-03-14T13:17:07.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd369ba6700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c7e0 tx=0x7fd360007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:07.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd360012810 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36410dd50 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd360012e20 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36000a310 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.072+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36410e310 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.073+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3640fe560 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.073+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd360011380 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.074+0000 7fd35affd700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 0x7fd35006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.074+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd36008beb0 con 0x7fd3641037b0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.074+0000 7fd36a3a7700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 0x7fd35006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:07.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.074+0000 7fd36a3a7700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 0x7fd35006ec70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd354004f90 tx=0x7fd354009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:07.077 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.078+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd360059b40 con 0x7fd3641037b0 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.352+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fd36405f5f0 con 0x7fd3641037b0 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.352+0000 7fd35affd700 1 -- 172.21.15.111:0/2938166797 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fd3600596d0 con 0x7fd3641037b0 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:17:07.352 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:17:07.353 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 msgr2=0x7fd35006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 0x7fd35006ec70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd354004f90 tx=0x7fd354009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 msgr2=0x7fd364114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c7e0 tx=0x7fd360007f90 comp rx=0 tx=0).stop 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 shutdown_connections 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35006c7c0 0x7fd35006ec70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd364102e10 0x7fd364113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 --2- 172.21.15.111:0/2938166797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3641037b0 0x7fd364114030 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.354+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 >> 172.21.15.111:0/2938166797 conn(0x7fd3640f8980 msgr2=0x7fd3640f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:07.354 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.355+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 shutdown_connections 2024-03-14T13:17:07.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:07.355+0000 7fd36c60b700 1 -- 172.21.15.111:0/2938166797 wait complete. 2024-03-14T13:17:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:07 smithi111 ceph-mon[145314]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:07 smithi111 ceph-mon[145314]: from='client.34772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:07 smithi111 ceph-mon[145314]: pgmap v390: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:07 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1221586176' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:17:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:07 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2938166797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:17:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:07 smithi137 ceph-mon[108145]: from='client.34768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:07 smithi137 ceph-mon[108145]: from='client.34772 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:07 smithi137 ceph-mon[108145]: pgmap v390: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:07 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1221586176' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:17:08.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:07 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2938166797' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:17:08.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:08 smithi111 ceph-mon[145314]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:09.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:08 smithi137 ceph-mon[108145]: from='client.34784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:09 smithi111 ceph-mon[145314]: pgmap v391: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:09 smithi137 ceph-mon[108145]: pgmap v391: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:11 smithi111 ceph-mon[145314]: pgmap v392: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:11 smithi137 ceph-mon[108145]: pgmap v392: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:13 smithi111 ceph-mon[145314]: pgmap v393: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:13 smithi137 ceph-mon[108145]: pgmap v393: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:15 smithi111 ceph-mon[145314]: pgmap v394: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:15 smithi137 ceph-mon[108145]: pgmap v394: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:17 smithi111 ceph-mon[145314]: pgmap v395: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:17 smithi137 ceph-mon[108145]: pgmap v395: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:19 smithi111 ceph-mon[145314]: pgmap v396: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:19 smithi137 ceph-mon[108145]: pgmap v396: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:21 smithi111 ceph-mon[145314]: pgmap v397: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:21 smithi137 ceph-mon[108145]: pgmap v397: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:23 smithi111 ceph-mon[145314]: pgmap v398: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:23 smithi137 ceph-mon[108145]: pgmap v398: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:25 smithi111 ceph-mon[145314]: pgmap v399: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:25 smithi137 ceph-mon[108145]: pgmap v399: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:27 smithi111 ceph-mon[145314]: pgmap v400: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:27 smithi137 ceph-mon[108145]: pgmap v400: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:29 smithi111 ceph-mon[145314]: pgmap v401: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:29 smithi137 ceph-mon[108145]: pgmap v401: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:31 smithi111 ceph-mon[145314]: pgmap v402: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:31 smithi137 ceph-mon[108145]: pgmap v402: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:33 smithi111 ceph-mon[145314]: pgmap v403: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:33 smithi137 ceph-mon[108145]: pgmap v403: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:35 smithi111 ceph-mon[145314]: pgmap v404: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:35 smithi137 ceph-mon[108145]: pgmap v404: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:37.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 -- 172.21.15.111:0/4063858027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 msgr2=0x7f4db40fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 --2- 172.21.15.111:0/4063858027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db40fe260 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4da8004940 tx=0x7f4da8009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:37.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 -- 172.21.15.111:0/4063858027 shutdown_connections 2024-03-14T13:17:37.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 --2- 172.21.15.111:0/4063858027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 0x7f4db4106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 --2- 172.21.15.111:0/4063858027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db40fe260 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 -- 172.21.15.111:0/4063858027 >> 172.21.15.111:0/4063858027 conn(0x7f4db4072bc0 msgr2=0x7f4db4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 -- 172.21.15.111:0/4063858027 shutdown_connections 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.468+0000 7f4db8c56700 1 -- 172.21.15.111:0/4063858027 wait complete. 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 Processor -- start 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 -- start start 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 0x7f4db4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db4114680 con 0x7f4db40fde90 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db8c56700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db410da70 con 0x7f4db40fe830 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53340/0 (socket says 172.21.15.111:53340) 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 -- 172.21.15.111:0/3295417804 learned_addr learned my addr 172.21.15.111:0/3295417804 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:37.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db1d9b700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 0x7f4db4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 -- 172.21.15.111:0/3295417804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 msgr2=0x7f4db4114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 0x7f4db4114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.469+0000 7f4db259c700 1 -- 172.21.15.111:0/3295417804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da8005040 con 0x7f4db40fde90 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4db259c700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4da80093b0 tx=0x7f4da8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da800e040 con 0x7f4db40fde90 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4da800bed0 con 0x7f4db40fde90 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db410dcf0 con 0x7f4db40fde90 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4da801c3f0 con 0x7f4db40fde90 2024-03-14T13:17:37.471 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.470+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db410e230 con 0x7f4db40fde90 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.471+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db41019b0 con 0x7f4db40fde90 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.471+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4da8013790 con 0x7f4db40fde90 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.472+0000 7f4da2ffd700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 0x7f4d9c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.472+0000 7f4db1d9b700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 0x7f4d9c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.472+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4da808cd90 con 0x7f4db40fde90 2024-03-14T13:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.472+0000 7f4db1d9b700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 0x7f4d9c06ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4db410ec30 tx=0x7f4da4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:37.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.477+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4da805aaa0 con 0x7f4db40fde90 2024-03-14T13:17:37.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.689+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db410f140 con 0x7f4d9c06c7d0 2024-03-14T13:17:37.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.690+0000 7f4da2ffd700 1 -- 172.21.15.111:0/3295417804 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4db410f140 con 0x7f4d9c06c7d0 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 msgr2=0x7f4d9c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 0x7f4d9c06ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4db410ec30 tx=0x7f4da4009040 comp rx=0 tx=0).stop 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 msgr2=0x7f4db4113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4da80093b0 tx=0x7f4da8009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 shutdown_connections 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4d9c06c7d0 0x7f4d9c06ec80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4db40fde90 0x7f4db4113af0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 --2- 172.21.15.111:0/3295417804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4db40fe830 0x7f4db4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.692+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 >> 172.21.15.111:0/3295417804 conn(0x7f4db4072bc0 msgr2=0x7f4db40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.693+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 shutdown_connections 2024-03-14T13:17:37.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.693+0000 7f4db8c56700 1 -- 172.21.15.111:0/3295417804 wait complete. 2024-03-14T13:17:37.702 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:17:37.795 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 -- 172.21.15.111:0/4118636765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c101d70 msgr2=0x7f5b8c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 --2- 172.21.15.111:0/4118636765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102140 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5b74004940 tx=0x7f5b74009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 -- 172.21.15.111:0/4118636765 shutdown_connections 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 --2- 172.21.15.111:0/4118636765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102710 0x7f5b8c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 --2- 172.21.15.111:0/4118636765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102140 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 -- 172.21.15.111:0/4118636765 >> 172.21.15.111:0/4118636765 conn(0x7f5b8c072bc0 msgr2=0x7f5b8c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 -- 172.21.15.111:0/4118636765 shutdown_connections 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.793+0000 7f5b90a93700 1 -- 172.21.15.111:0/4118636765 wait complete. 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 Processor -- start 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 -- start start 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c103730 con 0x7f5b8c102710 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b90a93700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c103870 con 0x7f5b8c101d70 2024-03-14T13:17:37.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53346/0 (socket says 172.21.15.111:53346) 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 -- 172.21.15.111:0/3291871453 learned_addr learned my addr 172.21.15.111:0/3291871453 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 -- 172.21.15.111:0/3291871453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c101d70 msgr2=0x7f5b8c102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 -- 172.21.15.111:0/3291871453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74005040 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.794+0000 7f5b89d9b700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b790 tx=0x7f5b7c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.795+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b7c00f6f0 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.795+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b7c00fd00 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.795+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c119db0 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.795+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b7c0103a0 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.795+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c11a2c0 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.796+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c0fd4c0 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.796+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5b7c00f850 con 0x7f5b8c102710 2024-03-14T13:17:37.797 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.796+0000 7f5b82ffd700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 0x7f5b7806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:37.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.796+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b7c08b780 con 0x7f5b8c102710 2024-03-14T13:17:37.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.796+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 0x7f5b7806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:37.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.797+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 0x7f5b7806ec70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5b74004f90 tx=0x7f5b74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:37.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:37.801+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5b7c055ac0 con 0x7f5b8c102710 2024-03-14T13:17:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:37 smithi111 ceph-mon[145314]: pgmap v405: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:37 smithi137 ceph-mon[108145]: pgmap v405: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:38.009 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.010+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b8c04bed0 con 0x7f5b7806c7c0 2024-03-14T13:17:38.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.011+0000 7f5b82ffd700 1 -- 172.21.15.111:0/3291871453 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5b8c04bed0 con 0x7f5b7806c7c0 2024-03-14T13:17:38.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 msgr2=0x7f5b7806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 0x7f5b7806ec70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5b74004f90 tx=0x7f5b74009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 msgr2=0x7f5b8c103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b790 tx=0x7f5b7c009220 comp rx=0 tx=0).stop 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 shutdown_connections 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7c0 0x7f5b7806ec70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c101d70 0x7f5b8c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 --2- 172.21.15.111:0/3291871453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102710 0x7f5b8c103110 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 >> 172.21.15.111:0/3291871453 conn(0x7f5b8c072bc0 msgr2=0x7f5b8c0659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 shutdown_connections 2024-03-14T13:17:38.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.013+0000 7f5b90a93700 1 -- 172.21.15.111:0/3291871453 wait complete. 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 -- 172.21.15.111:0/4261180751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c102e10 msgr2=0x7f192c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 --2- 172.21.15.111:0/4261180751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c102e10 0x7f192c1031e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f191c004940 tx=0x7f191c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 -- 172.21.15.111:0/4261180751 shutdown_connections 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 --2- 172.21.15.111:0/4261180751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c1037b0 0x7f192c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 --2- 172.21.15.111:0/4261180751 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c102e10 0x7f192c1031e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 -- 172.21.15.111:0/4261180751 >> 172.21.15.111:0/4261180751 conn(0x7f192c0f8980 msgr2=0x7f192c0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 -- 172.21.15.111:0/4261180751 shutdown_connections 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 -- 172.21.15.111:0/4261180751 wait complete. 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.113+0000 7f19328eb700 1 Processor -- start 2024-03-14T13:17:38.113 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f19328eb700 1 -- start start 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f19328eb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c102e10 0x7f192c113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f19328eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f19328eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c1146d0 con 0x7f192c1037b0 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f19328eb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c10da30 con 0x7f192c102e10 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53362/0 (socket says 172.21.15.111:53362) 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 -- 172.21.15.111:0/2579006540 learned_addr learned my addr 172.21.15.111:0/2579006540 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192bfff700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c102e10 0x7f192c113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 -- 172.21.15.111:0/2579006540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c102e10 msgr2=0x7f192c113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c102e10 0x7f192c113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 -- 172.21.15.111:0/2579006540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c005040 con 0x7f192c1037b0 2024-03-14T13:17:38.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f192b7fe700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f192000b790 tx=0x7f1920009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.114+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192000f6f0 con 0x7f192c1037b0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.115+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c10dd10 con 0x7f192c1037b0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.115+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f192000fd00 con 0x7f192c1037b0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.115+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19200103a0 con 0x7f192c1037b0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.115+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c10e2d0 con 0x7f192c1037b0 2024-03-14T13:17:38.116 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.116+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c0fe560 con 0x7f192c1037b0 2024-03-14T13:17:38.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.116+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f192000f850 con 0x7f192c1037b0 2024-03-14T13:17:38.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.116+0000 7f1928ff9700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 0x7f191406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.116+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f192008b860 con 0x7f192c1037b0 2024-03-14T13:17:38.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.116+0000 7f192bfff700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 0x7f191406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.117 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.117+0000 7f192bfff700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 0x7f191406ec70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f191c004f90 tx=0x7f191c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.121+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1920059570 con 0x7f192c1037b0 2024-03-14T13:17:38.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.322+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f192c109e40 con 0x7f191406c7c0 2024-03-14T13:17:38.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.332+0000 7f1928ff9700 1 -- 172.21.15.111:0/2579006540 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f192c109e40 con 0x7f191406c7c0 2024-03-14T13:17:38.331 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:17:38.331 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (13m) 7m ago 25m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:17:38.331 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (11m) 7m ago 25m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:17:38.331 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (11m) 6m ago 24m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (13m) 7m ago 24m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (7m) 7m ago 19m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (7m) 7m ago 19m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (6m) 6m ago 19m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (7m) 6m ago 19m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (16m) 7m ago 26m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (15m) 6m ago 24m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (12m) 7m ago 26m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (11m) 6m ago 23m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (14m) 7m ago 25m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (14m) 6m ago 24m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (10m) 7m ago 23m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (10m) 7m ago 22m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (9m) 7m ago 22m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:17:38.332 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (9m) 6m ago 21m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:17:38.333 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (8m) 6m ago 21m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:17:38.333 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (8m) 6m ago 20m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:17:38.333 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (14m) 7m ago 24m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:17:38.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 msgr2=0x7f191406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 0x7f191406ec70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f191c004f90 tx=0x7f191c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 msgr2=0x7f192c113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f192000b790 tx=0x7f1920009220 comp rx=0 tx=0).stop 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 shutdown_connections 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f191406c7c0 0x7f191406ec70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c102e10 0x7f192c113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 --2- 172.21.15.111:0/2579006540 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f192c1037b0 0x7f192c113ff0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 >> 172.21.15.111:0/2579006540 conn(0x7f192c0f8980 msgr2=0x7f192c0f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 shutdown_connections 2024-03-14T13:17:38.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.334+0000 7f19328eb700 1 -- 172.21.15.111:0/2579006540 wait complete. 2024-03-14T13:17:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 -- 172.21.15.111:0/3218507892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fe870 msgr2=0x7f11ac106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3218507892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fe870 0x7f11ac106f40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f11a0004940 tx=0x7f11a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 -- 172.21.15.111:0/3218507892 shutdown_connections 2024-03-14T13:17:38.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3218507892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fe870 0x7f11ac106f40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3218507892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fded0 0x7f11ac0fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 -- 172.21.15.111:0/3218507892 >> 172.21.15.111:0/3218507892 conn(0x7f11ac072bc0 msgr2=0x7f11ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 -- 172.21.15.111:0/3218507892 shutdown_connections 2024-03-14T13:17:38.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 -- 172.21.15.111:0/3218507892 wait complete. 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.433+0000 7f11b2f20700 1 Processor -- start 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b2f20700 1 -- start start 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b2f20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b2f20700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fe870 0x7f11ac113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b2f20700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac1145f0 con 0x7f11ac0fded0 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b2f20700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac10d9e0 con 0x7f11ac0fe870 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53392/0 (socket says 172.21.15.111:53392) 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 -- 172.21.15.111:0/3153248018 learned_addr learned my addr 172.21.15.111:0/3153248018 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11abfff700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fe870 0x7f11ac113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 -- 172.21.15.111:0/3153248018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fe870 msgr2=0x7f11ac113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fe870 0x7f11ac113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.434+0000 7f11b0cbc700 1 -- 172.21.15.111:0/3153248018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0005040 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11b0cbc700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f119c00c830 tx=0x7f119c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119c00a760 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac10dcc0 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f119c00ad70 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f119c0113a0 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.435+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac10e200 con 0x7f11ac0fded0 2024-03-14T13:17:38.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.436+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac1019a0 con 0x7f11ac0fded0 2024-03-14T13:17:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.436+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f119c012400 con 0x7f11ac0fded0 2024-03-14T13:17:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.437+0000 7f11a97fa700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 0x7f119406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.437+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f119c08bae0 con 0x7f11ac0fded0 2024-03-14T13:17:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.437+0000 7f11abfff700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 0x7f119406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.437+0000 7f11abfff700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 0x7f119406ec80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11ac10eba0 tx=0x7f11a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.441+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f119c0597f0 con 0x7f11ac0fded0 2024-03-14T13:17:38.713 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.714+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f11ac10f0b0 con 0x7f11ac0fded0 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.715+0000 7f11a97fa700 1 -- 172.21.15.111:0/3153248018 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f119c059380 con 0x7f11ac0fded0 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:38.714 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:17:38.715 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 msgr2=0x7f119406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 0x7f119406ec80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f11ac10eba0 tx=0x7f11a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 msgr2=0x7f11ac113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f119c00c830 tx=0x7f119c004470 comp rx=0 tx=0).stop 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 shutdown_connections 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f119406c7d0 0x7f119406ec80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f11ac0fded0 0x7f11ac113a60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 --2- 172.21.15.111:0/3153248018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11ac0fe870 0x7f11ac113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 >> 172.21.15.111:0/3153248018 conn(0x7f11ac072bc0 msgr2=0x7f11ac0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.716 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 shutdown_connections 2024-03-14T13:17:38.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.717+0000 7f11b2f20700 1 -- 172.21.15.111:0/3153248018 wait complete. 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 -- 172.21.15.111:0/3973945591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800102710 msgr2=0x7f18000fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 --2- 172.21.15.111:0/3973945591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800102710 0x7f18000fbd80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f17f4004940 tx=0x7f17f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 -- 172.21.15.111:0/3973945591 shutdown_connections 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 --2- 172.21.15.111:0/3973945591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800102710 0x7f18000fbd80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 --2- 172.21.15.111:0/3973945591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800101d70 0x7f1800102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 -- 172.21.15.111:0/3973945591 >> 172.21.15.111:0/3973945591 conn(0x7f1800072bc0 msgr2=0x7f1800072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:38.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 -- 172.21.15.111:0/3973945591 shutdown_connections 2024-03-14T13:17:38.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.816+0000 7f1805c95700 1 -- 172.21.15.111:0/3973945591 wait complete. 2024-03-14T13:17:38.817 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 Processor -- start 2024-03-14T13:17:38.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 -- start start 2024-03-14T13:17:38.818 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800102710 0x7f1800114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1800114710 con 0x7f1800101d70 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f1805c95700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180010da70 con 0x7f1800102710 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f17ff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f17ff7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53412/0 (socket says 172.21.15.111:53412) 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.817+0000 7f17ff7fe700 1 -- 172.21.15.111:0/1252845362 learned_addr learned my addr 172.21.15.111:0/1252845362 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17ff7fe700 1 -- 172.21.15.111:0/1252845362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800102710 msgr2=0x7f1800114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17feffd700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800102710 0x7f1800114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17ff7fe700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800102710 0x7f1800114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17ff7fe700 1 -- 172.21.15.111:0/1252845362 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f4005040 con 0x7f1800101d70 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17ff7fe700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f17f000b790 tx=0x7f17f0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f000f770 con 0x7f1800101d70 2024-03-14T13:17:38.819 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180010dd50 con 0x7f1800101d70 2024-03-14T13:17:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17f000fd80 con 0x7f1800101d70 2024-03-14T13:17:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17f000d310 con 0x7f1800101d70 2024-03-14T13:17:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.818+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180010e310 con 0x7f1800101d70 2024-03-14T13:17:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.819+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18000fd4c0 con 0x7f1800101d70 2024-03-14T13:17:38.820 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.820+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f17f0010420 con 0x7f1800101d70 2024-03-14T13:17:38.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.820+0000 7f17e7fff700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 0x7f17e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:38.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.820+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f17f008ba80 con 0x7f1800101d70 2024-03-14T13:17:38.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.820+0000 7f17feffd700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 0x7f17e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:38.821 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.820+0000 7f17feffd700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 0x7f17e806ec70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009fd0 tx=0x7f17f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:38.825 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:38.825+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f17f0059790 con 0x7f1800101d70 2024-03-14T13:17:39.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.057+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f180004bed0 con 0x7f1800101d70 2024-03-14T13:17:39.056 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.057+0000 7f17e7fff700 1 -- 172.21.15.111:0/1252845362 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f17f0059320 con 0x7f1800101d70 2024-03-14T13:17:39.056 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:17:39.056 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:17:39.057 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:17:39.058 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:39.059 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 msgr2=0x7f17e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 0x7f17e806ec70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f17f4009fd0 tx=0x7f17f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 msgr2=0x7f1800113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f17f000b790 tx=0x7f17f0009250 comp rx=0 tx=0).stop 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 shutdown_connections 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f17e806c7c0 0x7f17e806ec70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1800101d70 0x7f1800113af0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 --2- 172.21.15.111:0/1252845362 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1800102710 0x7f1800114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.060 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 >> 172.21.15.111:0/1252845362 conn(0x7f1800072bc0 msgr2=0x7f18000659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:39.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 shutdown_connections 2024-03-14T13:17:39.061 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.059+0000 7f1805c95700 1 -- 172.21.15.111:0/1252845362 wait complete. 2024-03-14T13:17:39.061 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 -- 172.21.15.111:0/1904148926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 msgr2=0x7f47200fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 --2- 172.21.15.111:0/1904148926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f47200fbd80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f471c004940 tx=0x7f471c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 -- 172.21.15.111:0/1904148926 shutdown_connections 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 --2- 172.21.15.111:0/1904148926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f47200fbd80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 --2- 172.21.15.111:0/1904148926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 0x7f4720102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 -- 172.21.15.111:0/1904148926 >> 172.21.15.111:0/1904148926 conn(0x7f4720072bc0 msgr2=0x7f4720072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.160+0000 7f4727e02700 1 -- 172.21.15.111:0/1904148926 shutdown_connections 2024-03-14T13:17:39.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 -- 172.21.15.111:0/1904148926 wait complete. 2024-03-14T13:17:39.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 Processor -- start 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 -- start start 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 0x7f472010ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472010f680 con 0x7f4720102710 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f4727e02700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472010f7c0 con 0x7f4720101d70 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f472539d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f472539d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53436/0 (socket says 172.21.15.111:53436) 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.161+0000 7f472539d700 1 -- 172.21.15.111:0/4208058365 learned_addr learned my addr 172.21.15.111:0/4208058365 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f4725b9e700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 0x7f472010ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f472539d700 1 -- 172.21.15.111:0/4208058365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 msgr2=0x7f472010ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f472539d700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 0x7f472010ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f472539d700 1 -- 172.21.15.111:0/4208058365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c005040 con 0x7f4720102710 2024-03-14T13:17:39.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f472539d700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f471c009390 tx=0x7f471c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:39.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f471c00e040 con 0x7f4720102710 2024-03-14T13:17:39.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4720117ab0 con 0x7f4720102710 2024-03-14T13:17:39.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f471c013420 con 0x7f4720102710 2024-03-14T13:17:39.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f471c01c630 con 0x7f4720102710 2024-03-14T13:17:39.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.162+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4720118040 con 0x7f4720102710 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.163+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47200ff080 con 0x7f4720102710 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.163+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f471c0139e0 con 0x7f4720102710 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.164+0000 7f47167fc700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 0x7f470c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.164+0000 7f4725b9e700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 0x7f470c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.164+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f471c08cd20 con 0x7f4720102710 2024-03-14T13:17:39.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.164+0000 7f4725b9e700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 0x7f470c06ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4710007fd0 tx=0x7f4710009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:39.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.168+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f471c056fa0 con 0x7f4720102710 2024-03-14T13:17:39.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.381+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4720065dd0 con 0x7f470c06c7d0 2024-03-14T13:17:39.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.382+0000 7f47167fc700 1 -- 172.21.15.111:0/4208058365 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4720065dd0 con 0x7f470c06c7d0 2024-03-14T13:17:39.381 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:17:39.382 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:17:39.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 msgr2=0x7f470c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 0x7f470c06ec80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4710007fd0 tx=0x7f4710009040 comp rx=0 tx=0).stop 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 msgr2=0x7f472010f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f471c009390 tx=0x7f471c009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 shutdown_connections 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f470c06c7d0 0x7f470c06ec80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4720101d70 0x7f472010ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 --2- 172.21.15.111:0/4208058365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4720102710 0x7f472010f140 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 >> 172.21.15.111:0/4208058365 conn(0x7f4720072bc0 msgr2=0x7f47200fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 shutdown_connections 2024-03-14T13:17:39.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.384+0000 7f4727e02700 1 -- 172.21.15.111:0/4208058365 wait complete. 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 -- 172.21.15.111:0/3613501874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4102710 msgr2=0x7ff4f40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 --2- 172.21.15.111:0/3613501874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4102710 0x7ff4f40fbd80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4004940 tx=0x7ff4e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 -- 172.21.15.111:0/3613501874 shutdown_connections 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 --2- 172.21.15.111:0/3613501874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4102710 0x7ff4f40fbd80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 --2- 172.21.15.111:0/3613501874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4101d70 0x7ff4f4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 -- 172.21.15.111:0/3613501874 >> 172.21.15.111:0/3613501874 conn(0x7ff4f4072bc0 msgr2=0x7ff4f4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:39.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 -- 172.21.15.111:0/3613501874 shutdown_connections 2024-03-14T13:17:39.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 -- 172.21.15.111:0/3613501874 wait complete. 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.485+0000 7ff4f9835700 1 Processor -- start 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 -- start start 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4102710 0x7ff4f4112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4113550 con 0x7ff4f4101d70 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f410c940 con 0x7ff4f4102710 2024-03-14T13:17:39.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53456/0 (socket says 172.21.15.111:53456) 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 -- 172.21.15.111:0/1050786070 learned_addr learned my addr 172.21.15.111:0/1050786070 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f27fc700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4102710 0x7ff4f4112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 -- 172.21.15.111:0/1050786070 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4102710 msgr2=0x7ff4f4112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4102710 0x7ff4f4112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 -- 172.21.15.111:0/1050786070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4005040 con 0x7ff4f4101d70 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f2ffd700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c8c0 tx=0x7ff4e80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e800a980 con 0x7ff4f4101d70 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e800ce60 con 0x7ff4f4101d70 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f410cc20 con 0x7ff4f4101d70 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.486+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e80113a0 con 0x7ff4f4101d70 2024-03-14T13:17:39.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.487+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f410d1e0 con 0x7ff4f4101d70 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.487+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f40ff080 con 0x7ff4f4101d70 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.488+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff4e800aae0 con 0x7ff4f4101d70 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.488+0000 7ff4db7fe700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 0x7ff4dc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.488+0000 7ff4f27fc700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 0x7ff4dc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.488+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4e808bb50 con 0x7ff4f4101d70 2024-03-14T13:17:39.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.489+0000 7ff4f27fc700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 0x7ff4dc06ec70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fd0 tx=0x7ff4e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:17:39.492 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.493+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff4e8059860 con 0x7ff4f4101d70 2024-03-14T13:17:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.765+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff4f410e010 con 0x7ff4f4101d70 2024-03-14T13:17:39.765 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.766+0000 7ff4db7fe700 1 -- 172.21.15.111:0/1050786070 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff4e80593f0 con 0x7ff4f4101d70 2024-03-14T13:17:39.765 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:17:39.765 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:17:39.765 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:17:39.766 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:17:39.766 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:17:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 msgr2=0x7ff4dc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.767 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 0x7ff4dc06ec70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4009fd0 tx=0x7ff4e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:17:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 msgr2=0x7ff4f41129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:17:39.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c8c0 tx=0x7ff4e80044c0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 shutdown_connections 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4dc06c7c0 0x7ff4dc06ec70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f4101d70 0x7ff4f41129c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 --2- 172.21.15.111:0/1050786070 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f4102710 0x7ff4f4112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 >> 172.21.15.111:0/1050786070 conn(0x7ff4f4072bc0 msgr2=0x7ff4f40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 shutdown_connections 2024-03-14T13:17:39.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:17:39.768+0000 7ff4f9835700 1 -- 172.21.15.111:0/1050786070 wait complete. 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: pgmap v406: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3153248018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1252845362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:17:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='client.34792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='client.34796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: pgmap v406: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3153248018' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1252845362' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:17:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:40 smithi111 ceph-mon[145314]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:40 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1050786070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:17:40.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:40 smithi111 ceph-mon[145314]: pgmap v407: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:40 smithi137 ceph-mon[108145]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:17:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:40 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1050786070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:17:41.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:40 smithi137 ceph-mon[108145]: pgmap v407: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:43.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:43 smithi137 ceph-mon[108145]: pgmap v408: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:43 smithi111 ceph-mon[145314]: pgmap v408: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:45 smithi111 ceph-mon[145314]: pgmap v409: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:45 smithi137 ceph-mon[108145]: pgmap v409: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:47 smithi111 ceph-mon[145314]: pgmap v410: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:47 smithi137 ceph-mon[108145]: pgmap v410: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:49 smithi111 ceph-mon[145314]: pgmap v411: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:49 smithi137 ceph-mon[108145]: pgmap v411: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:51 smithi111 ceph-mon[145314]: pgmap v412: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:51 smithi137 ceph-mon[108145]: pgmap v412: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:53 smithi111 ceph-mon[145314]: pgmap v413: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:53 smithi137 ceph-mon[108145]: pgmap v413: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:17:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:55 smithi111 ceph-mon[145314]: pgmap v414: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:55 smithi137 ceph-mon[108145]: pgmap v414: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:57 smithi111 ceph-mon[145314]: pgmap v415: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:57 smithi137 ceph-mon[108145]: pgmap v415: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:17:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:17:59 smithi111 ceph-mon[145314]: pgmap v416: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:17:59 smithi137 ceph-mon[108145]: pgmap v416: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:01 smithi111 ceph-mon[145314]: pgmap v417: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:01 smithi137 ceph-mon[108145]: pgmap v417: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:03 smithi111 ceph-mon[145314]: pgmap v418: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:03 smithi137 ceph-mon[108145]: pgmap v418: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:05 smithi111 ceph-mon[145314]: pgmap v419: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:05 smithi137 ceph-mon[108145]: pgmap v419: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:07 smithi111 ceph-mon[145314]: pgmap v420: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:07 smithi137 ceph-mon[108145]: pgmap v420: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:09.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 -- 172.21.15.111:0/1881237271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78102e10 msgr2=0x7f8e781031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:09.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1881237271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78102e10 0x7f8e781031e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004940 tx=0x7f8e74009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:09.880 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 -- 172.21.15.111:0/1881237271 shutdown_connections 2024-03-14T13:18:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1881237271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e781037b0 0x7f8e780fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1881237271 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e78102e10 0x7f8e781031e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 -- 172.21.15.111:0/1881237271 >> 172.21.15.111:0/1881237271 conn(0x7f8e780f89a0 msgr2=0x7f8e780fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 -- 172.21.15.111:0/1881237271 shutdown_connections 2024-03-14T13:18:09.881 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.881+0000 7f8e7f945700 1 -- 172.21.15.111:0/1881237271 wait complete. 2024-03-14T13:18:09.882 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 Processor -- start 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 -- start start 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e78102e10 0x7f8e78113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781146a0 con 0x7f8e781037b0 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7f945700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7810da90 con 0x7f8e78102e10 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59134/0 (socket says 172.21.15.111:59134) 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 -- 172.21.15.111:0/1504925471 learned_addr learned my addr 172.21.15.111:0/1504925471 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7d6e1700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e78102e10 0x7f8e78113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 -- 172.21.15.111:0/1504925471 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e78102e10 msgr2=0x7f8e78113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e78102e10 0x7f8e78113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.882+0000 7f8e7cee0700 1 -- 172.21.15.111:0/1504925471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74005040 con 0x7f8e781037b0 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e7cee0700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8e68002a80 tx=0x7f8e68004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:09.883 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e68004e80 con 0x7f8e781037b0 2024-03-14T13:18:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7810dd70 con 0x7f8e781037b0 2024-03-14T13:18:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e6800f040 con 0x7f8e781037b0 2024-03-14T13:18:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8e6800da00 con 0x7f8e781037b0 2024-03-14T13:18:09.884 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.883+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7810e330 con 0x7f8e781037b0 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.884+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e780fe560 con 0x7f8e781037b0 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.884+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8e68007cc0 con 0x7f8e781037b0 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.885+0000 7f8e6dffb700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 0x7f8e6406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.885+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8e6808b0d0 con 0x7f8e781037b0 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.885+0000 7f8e7d6e1700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 0x7f8e6406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:09.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.885+0000 7f8e7d6e1700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 0x7f8e6406ec70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004f90 tx=0x7f8e74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:09.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:09.889+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8e68058d60 con 0x7f8e781037b0 2024-03-14T13:18:09.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:09 smithi111 ceph-mon[145314]: pgmap v421: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:09.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:09 smithi137 ceph-mon[108145]: pgmap v421: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:10.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.100+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e781137e0 con 0x7f8e6406c7c0 2024-03-14T13:18:10.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.101+0000 7f8e6dffb700 1 -- 172.21.15.111:0/1504925471 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8e781137e0 con 0x7f8e6406c7c0 2024-03-14T13:18:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 msgr2=0x7f8e6406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 0x7f8e6406ec70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8e74004f90 tx=0x7f8e74009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 msgr2=0x7f8e78114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8e68002a80 tx=0x7f8e68004630 comp rx=0 tx=0).stop 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 shutdown_connections 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8e6406c7c0 0x7f8e6406ec70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e78102e10 0x7f8e78113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 --2- 172.21.15.111:0/1504925471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e781037b0 0x7f8e78114050 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 >> 172.21.15.111:0/1504925471 conn(0x7f8e780f89a0 msgr2=0x7f8e780f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 shutdown_connections 2024-03-14T13:18:10.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.103+0000 7f8e7f945700 1 -- 172.21.15.111:0/1504925471 wait complete. 2024-03-14T13:18:10.112 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:18:10.205 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 -- 172.21.15.111:0/673726119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94102710 msgr2=0x7fba940fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 --2- 172.21.15.111:0/673726119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94102710 0x7fba940fbd80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fba90004940 tx=0x7fba90009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 -- 172.21.15.111:0/673726119 shutdown_connections 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 --2- 172.21.15.111:0/673726119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94102710 0x7fba940fbd80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 --2- 172.21.15.111:0/673726119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94101d70 0x7fba94102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.204+0000 7fba9bff6700 1 -- 172.21.15.111:0/673726119 >> 172.21.15.111:0/673726119 conn(0x7fba94072bc0 msgr2=0x7fba94072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 -- 172.21.15.111:0/673726119 shutdown_connections 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 -- 172.21.15.111:0/673726119 wait complete. 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 Processor -- start 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 -- start start 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94102710 0x7fba9410f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9410f650 con 0x7fba94101d70 2024-03-14T13:18:10.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba9bff6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9410f790 con 0x7fba94102710 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba99d92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba99d92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59142/0 (socket says 172.21.15.111:59142) 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.205+0000 7fba99d92700 1 -- 172.21.15.111:0/3665740963 learned_addr learned my addr 172.21.15.111:0/3665740963 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba99591700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94102710 0x7fba9410f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba99d92700 1 -- 172.21.15.111:0/3665740963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94102710 msgr2=0x7fba9410f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba99d92700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94102710 0x7fba9410f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba99d92700 1 -- 172.21.15.111:0/3665740963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba90005040 con 0x7fba94101d70 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba99d92700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba84002a50 tx=0x7fba840046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba84007cc0 con 0x7fba94101d70 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba94117b30 con 0x7fba94101d70 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba8400f040 con 0x7fba94101d70 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8400da70 con 0x7fba94101d70 2024-03-14T13:18:10.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.206+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba94118090 con 0x7fba94101d70 2024-03-14T13:18:10.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.207+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba94104240 con 0x7fba94101d70 2024-03-14T13:18:10.208 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.208+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fba8400dbd0 con 0x7fba94101d70 2024-03-14T13:18:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.208+0000 7fba8a7fc700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 0x7fba8006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.208+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fba8408b3d0 con 0x7fba94101d70 2024-03-14T13:18:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.208+0000 7fba99591700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 0x7fba8006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.208+0000 7fba99591700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 0x7fba8006ec70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba90000c00 tx=0x7fba90009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.212+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fba84055650 con 0x7fba94101d70 2024-03-14T13:18:10.423 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.424+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fba9404bed0 con 0x7fba8006c7c0 2024-03-14T13:18:10.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.425+0000 7fba8a7fc700 1 -- 172.21.15.111:0/3665740963 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fba9404bed0 con 0x7fba8006c7c0 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 msgr2=0x7fba8006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 0x7fba8006ec70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba90000c00 tx=0x7fba90009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 msgr2=0x7fba9410eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fba84002a50 tx=0x7fba840046b0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 shutdown_connections 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba8006c7c0 0x7fba8006ec70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba94101d70 0x7fba9410eac0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 --2- 172.21.15.111:0/3665740963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba94102710 0x7fba9410f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.427+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 >> 172.21.15.111:0/3665740963 conn(0x7fba94072bc0 msgr2=0x7fba940fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.428+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 shutdown_connections 2024-03-14T13:18:10.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.428+0000 7fba9bff6700 1 -- 172.21.15.111:0/3665740963 wait complete. 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- 172.21.15.111:0/2312498425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c102e00 msgr2=0x7fde5c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 --2- 172.21.15.111:0/2312498425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c102e00 0x7fde5c1031d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde58004940 tx=0x7fde58009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- 172.21.15.111:0/2312498425 shutdown_connections 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 --2- 172.21.15.111:0/2312498425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c1037a0 0x7fde5c0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 --2- 172.21.15.111:0/2312498425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c102e00 0x7fde5c1031d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- 172.21.15.111:0/2312498425 >> 172.21.15.111:0/2312498425 conn(0x7fde5c0f89b0 msgr2=0x7fde5c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- 172.21.15.111:0/2312498425 shutdown_connections 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- 172.21.15.111:0/2312498425 wait complete. 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 Processor -- start 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.528+0000 7fde64126700 1 -- start start 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c102e00 0x7fde5c1138d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.529 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c114460 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c10da60 con 0x7fde5c102e00 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44580/0 (socket says 172.21.15.111:44580) 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 -- 172.21.15.111:0/3603202127 learned_addr learned my addr 172.21.15.111:0/3603202127 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde61ec2700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c102e00 0x7fde5c1138d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 -- 172.21.15.111:0/3603202127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c102e00 msgr2=0x7fde5c1138d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c102e00 0x7fde5c1138d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 -- 172.21.15.111:0/3603202127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58005040 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde616c1700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fde4c002a80 tx=0x7fde4c004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde4c004e80 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c10dd40 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde4c00f040 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde4c00da00 con 0x7fde5c1037a0 2024-03-14T13:18:10.530 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.529+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c10e300 con 0x7fde5c1037a0 2024-03-14T13:18:10.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c0fe550 con 0x7fde5c1037a0 2024-03-14T13:18:10.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fde4c007cc0 con 0x7fde5c1037a0 2024-03-14T13:18:10.531 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde527fc700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 0x7fde4806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde4c08b0f0 con 0x7fde5c1037a0 2024-03-14T13:18:10.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde61ec2700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 0x7fde4806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.531+0000 7fde61ec2700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 0x7fde4806ec70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde58004f90 tx=0x7fde58009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.536+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fde4c058d80 con 0x7fde5c1037a0 2024-03-14T13:18:10.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.736+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fde5c1135a0 con 0x7fde4806c7c0 2024-03-14T13:18:10.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.745+0000 7fde527fc700 1 -- 172.21.15.111:0/3603202127 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fde5c1135a0 con 0x7fde4806c7c0 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (14m) 7m ago 25m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (11m) 7m ago 26m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (11m) 7m ago 24m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (14m) 7m ago 25m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (7m) 7m ago 19m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (8m) 7m ago 19m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (7m) 7m ago 19m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (7m) 7m ago 19m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (16m) 7m ago 27m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (15m) 7m ago 24m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:18:10.745 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (12m) 7m ago 27m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (12m) 7m ago 24m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (15m) 7m ago 26m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (15m) 7m ago 24m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (11m) 7m ago 23m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (10m) 7m ago 23m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (10m) 7m ago 22m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (9m) 7m ago 22m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (9m) 7m ago 21m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (8m) 7m ago 21m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:18:10.746 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (14m) 7m ago 25m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.747+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 msgr2=0x7fde4806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.747+0000 7fde64126700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 0x7fde4806ec70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fde58004f90 tx=0x7fde58009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.747+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 msgr2=0x7fde5c113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.747+0000 7fde64126700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fde4c002a80 tx=0x7fde4c004630 comp rx=0 tx=0).stop 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.747+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 shutdown_connections 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde4806c7c0 0x7fde4806ec70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5c102e00 0x7fde5c1138d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 --2- 172.21.15.111:0/3603202127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde5c1037a0 0x7fde5c113e10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 >> 172.21.15.111:0/3603202127 conn(0x7fde5c0f89b0 msgr2=0x7fde5c0f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 shutdown_connections 2024-03-14T13:18:10.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.748+0000 7fde64126700 1 -- 172.21.15.111:0/3603202127 wait complete. 2024-03-14T13:18:10.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 -- 172.21.15.111:0/2106561416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 msgr2=0x7f83f0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 --2- 172.21.15.111:0/2106561416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f83e0004940 tx=0x7f83e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 -- 172.21.15.111:0/2106561416 shutdown_connections 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 --2- 172.21.15.111:0/2106561416 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 --2- 172.21.15.111:0/2106561416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102140 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 -- 172.21.15.111:0/2106561416 >> 172.21.15.111:0/2106561416 conn(0x7f83f0072bc0 msgr2=0x7f83f0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 -- 172.21.15.111:0/2106561416 shutdown_connections 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 -- 172.21.15.111:0/2106561416 wait complete. 2024-03-14T13:18:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.846+0000 7f83f6081700 1 Processor -- start 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83f6081700 1 -- start start 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83f6081700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83f6081700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83f6081700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f0103650 con 0x7f83f0101d70 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83f6081700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f0103790 con 0x7f83f0102710 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44594/0 (socket says 172.21.15.111:44594) 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 -- 172.21.15.111:0/3680174934 learned_addr learned my addr 172.21.15.111:0/3680174934 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83eeffd700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 -- 172.21.15.111:0/3680174934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 msgr2=0x7f83f0103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f0103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 -- 172.21.15.111:0/3680174934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0005040 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83eeffd700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83ef7fe700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009180 tx=0x7f83e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.847+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e000e040 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.848+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f01039e0 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.848+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83e0004290 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.848+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83e001c3f0 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.848+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f011a1b0 con 0x7f83f0101d70 2024-03-14T13:18:10.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.849+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f00fd4c0 con 0x7f83f0101d70 2024-03-14T13:18:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.849+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f83e0013790 con 0x7f83f0101d70 2024-03-14T13:18:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.849+0000 7f83e7fff700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 0x7f83dc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.850+0000 7f83eeffd700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 0x7f83dc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.850+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f83e008ccb0 con 0x7f83f0101d70 2024-03-14T13:18:10.850 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.850+0000 7f83eeffd700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 0x7f83dc06ec70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009fd0 tx=0x7f83d8009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:10.853 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:10.854+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f83e0056f30 con 0x7f83f0101d70 2024-03-14T13:18:11.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.126+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f83f0104270 con 0x7f83f0101d70 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.127+0000 7f83e7fff700 1 -- 172.21.15.111:0/3680174934 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f83e0013420 con 0x7f83f0101d70 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:18:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:18:11.127 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 msgr2=0x7f83dc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 0x7f83dc06ec70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009fd0 tx=0x7f83d8009ee0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 msgr2=0x7f83f0102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009180 tx=0x7f83e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 shutdown_connections 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f83dc06c7c0 0x7f83dc06ec70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83f0101d70 0x7f83f0102bd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.128 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 --2- 172.21.15.111:0/3680174934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83f0102710 0x7f83f0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 >> 172.21.15.111:0/3680174934 conn(0x7f83f0072bc0 msgr2=0x7f83f0065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 shutdown_connections 2024-03-14T13:18:11.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.129+0000 7f83f6081700 1 -- 172.21.15.111:0/3680174934 wait complete. 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 -- 172.21.15.111:0/2644195055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 msgr2=0x7fada41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 --2- 172.21.15.111:0/2644195055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada41031d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fad94004940 tx=0x7fad94009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 -- 172.21.15.111:0/2644195055 shutdown_connections 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 --2- 172.21.15.111:0/2644195055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 --2- 172.21.15.111:0/2644195055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada41031d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 -- 172.21.15.111:0/2644195055 >> 172.21.15.111:0/2644195055 conn(0x7fada40f89b0 msgr2=0x7fada40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 -- 172.21.15.111:0/2644195055 shutdown_connections 2024-03-14T13:18:11.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.228+0000 7fada9126700 1 -- 172.21.15.111:0/2644195055 wait complete. 2024-03-14T13:18:11.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 Processor -- start 2024-03-14T13:18:11.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 -- start start 2024-03-14T13:18:11.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4070d90 con 0x7fada4102e00 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada9126700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada4070ed0 con 0x7fada41037a0 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada2d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44606/0 (socket says 172.21.15.111:44606) 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.229+0000 7fada2d9d700 1 -- 172.21.15.111:0/1936811613 learned_addr learned my addr 172.21.15.111:0/1936811613 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada259c700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada2d9d700 1 -- 172.21.15.111:0/1936811613 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 msgr2=0x7fada4070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada2d9d700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada4070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada2d9d700 1 -- 172.21.15.111:0/1936811613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94005040 con 0x7fada4102e00 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada259c700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada2d9d700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad94009360 tx=0x7fad94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad9400e040 con 0x7fada4102e00 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad94013420 con 0x7fada4102e00 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada4071120 con 0x7fada4102e00 2024-03-14T13:18:11.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad9401c5f0 con 0x7fada4102e00 2024-03-14T13:18:11.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.230+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada411a270 con 0x7fada4102e00 2024-03-14T13:18:11.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.231+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada40fe550 con 0x7fada4102e00 2024-03-14T13:18:11.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.232+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fad94013590 con 0x7fada4102e00 2024-03-14T13:18:11.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.232+0000 7fad8b7fe700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 0x7fad8c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.232+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fad9408cd00 con 0x7fada4102e00 2024-03-14T13:18:11.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.232+0000 7fada259c700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 0x7fad8c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.232+0000 7fada259c700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 0x7fad8c06ec80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fad98009fd0 tx=0x7fad9800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.236+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fad94056f80 con 0x7fada4102e00 2024-03-14T13:18:11.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.471+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fada40719b0 con 0x7fada4102e00 2024-03-14T13:18:11.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.471+0000 7fad8b7fe700 1 -- 172.21.15.111:0/1936811613 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fad9405a5a0 con 0x7fada4102e00 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:18:11.471 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:18:11.472 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:11.473 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.473+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 msgr2=0x7fad8c06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.473+0000 7fada9126700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 0x7fad8c06ec80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fad98009fd0 tx=0x7fad9800b040 comp rx=0 tx=0).stop 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.473+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 msgr2=0x7fada40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad94009360 tx=0x7fad94009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 shutdown_connections 2024-03-14T13:18:11.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fad8c06c7d0 0x7fad8c06ec80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fada4102e00 0x7fada40702c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 --2- 172.21.15.111:0/1936811613 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fada41037a0 0x7fada4070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 >> 172.21.15.111:0/1936811613 conn(0x7fada40f89b0 msgr2=0x7fada40f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 shutdown_connections 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.474+0000 7fada9126700 1 -- 172.21.15.111:0/1936811613 wait complete. 2024-03-14T13:18:11.475 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:18:11.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/2119110746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fe820 msgr2=0x7f6adc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.576 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/2119110746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fe820 0x7f6adc106e60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6acc004940 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/2119110746 shutdown_connections 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/2119110746 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fe820 0x7f6adc106e60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/2119110746 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/2119110746 >> 172.21.15.111:0/2119110746 conn(0x7f6adc0f7870 msgr2=0x7f6adc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/2119110746 shutdown_connections 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/2119110746 wait complete. 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 Processor -- start 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- start start 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fe820 0x7f6adc0fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0fdc90 con 0x7f6adc0fde80 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.575+0000 7f6ae3ee5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc0fc730 con 0x7f6adc0fe820 2024-03-14T13:18:11.577 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44618/0 (socket says 172.21.15.111:44618) 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 -- 172.21.15.111:0/1474523858 learned_addr learned my addr 172.21.15.111:0/1474523858 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1480700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fe820 0x7f6adc0fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 -- 172.21.15.111:0/1474523858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fe820 msgr2=0x7f6adc0fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fe820 0x7f6adc0fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 -- 172.21.15.111:0/1474523858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc005040 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae1c81700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800b790 tx=0x7f6ad8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad800f6f0 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ad800fd00 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc0fca10 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ad80103a0 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.576+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc11a020 con 0x7f6adc0fde80 2024-03-14T13:18:11.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.577+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc066b10 con 0x7f6adc0fde80 2024-03-14T13:18:11.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.578+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6ad800f850 con 0x7f6adc0fde80 2024-03-14T13:18:11.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.578+0000 7f6ad27fc700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 0x7f6ac806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.578+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ad808b970 con 0x7f6adc0fde80 2024-03-14T13:18:11.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.578+0000 7f6ae1480700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 0x7f6ac806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.579+0000 7f6ae1480700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 0x7f6ac806ec70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0fd3e0 tx=0x7f6acc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.583+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ad8055b70 con 0x7f6adc0fde80 2024-03-14T13:18:11.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.793+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6adc0fd200 con 0x7f6ac806c7c0 2024-03-14T13:18:11.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.794+0000 7f6ad27fc700 1 -- 172.21.15.111:0/1474523858 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6adc0fd200 con 0x7f6ac806c7c0 2024-03-14T13:18:11.793 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:18:11.793 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:18:11.794 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:18:11.795 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 msgr2=0x7f6ac806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 0x7f6ac806ec70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6adc0fd3e0 tx=0x7f6acc009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 msgr2=0x7f6adc0fbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6ad800b790 tx=0x7f6ad8009250 comp rx=0 tx=0).stop 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 shutdown_connections 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ac806c7c0 0x7f6ac806ec70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6adc0fde80 0x7f6adc0fbcb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 --2- 172.21.15.111:0/1474523858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6adc0fe820 0x7f6adc0fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 >> 172.21.15.111:0/1474523858 conn(0x7f6adc0f7870 msgr2=0x7f6adc0f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 shutdown_connections 2024-03-14T13:18:11.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.796+0000 7f6ae3ee5700 1 -- 172.21.15.111:0/1474523858 wait complete. 2024-03-14T13:18:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:11 smithi111 ceph-mon[145314]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:11 smithi111 ceph-mon[145314]: pgmap v422: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:11 smithi111 ceph-mon[145314]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:11 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3680174934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:18:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:11 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1936811613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/564159615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fde90 msgr2=0x7f7ac00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/564159615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fe260 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4004940 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/564159615 shutdown_connections 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/564159615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fe830 0x7f7ac0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/564159615 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fe260 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.896+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/564159615 >> 172.21.15.111:0/564159615 conn(0x7f7ac0072bc0 msgr2=0x7f7ac0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:11.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/564159615 shutdown_connections 2024-03-14T13:18:11.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/564159615 wait complete. 2024-03-14T13:18:11.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 Processor -- start 2024-03-14T13:18:11.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 -- start start 2024-03-14T13:18:11.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00fc820 con 0x7f7ac00fe830 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7ac6ac2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00fc960 con 0x7f7ac00fde90 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.897+0000 7f7abffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44622/0 (socket says 172.21.15.111:44622) 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 -- 172.21.15.111:0/2470229528 learned_addr learned my addr 172.21.15.111:0/2470229528 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7ac485e700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 -- 172.21.15.111:0/2470229528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fde90 msgr2=0x7f7ac00fbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 -- 172.21.15.111:0/2470229528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab4005040 con 0x7f7ac00fe830 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abffff700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c8c0 tx=0x7f7aac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aac00a760 con 0x7f7ac00fe830 2024-03-14T13:18:11.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0119df0 con 0x7f7ac00fe830 2024-03-14T13:18:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7aac00ad70 con 0x7f7ac00fe830 2024-03-14T13:18:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7aac0113a0 con 0x7f7ac00fe830 2024-03-14T13:18:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.898+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac011a1f0 con 0x7f7ac00fe830 2024-03-14T13:18:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.899+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac0065e30 con 0x7f7ac00fe830 2024-03-14T13:18:11.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.900+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7aac012400 con 0x7f7ac00fe830 2024-03-14T13:18:11.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.900+0000 7f7abd7fa700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 0x7f7ab006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:11.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.900+0000 7f7ac485e700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 0x7f7ab006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:11.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.900+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7aac08bab0 con 0x7f7ac00fe830 2024-03-14T13:18:11.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.901+0000 7f7ac485e700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 0x7f7ab006ec70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400b7e0 tx=0x7f7ab4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:11.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:11.905+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7aac055d30 con 0x7f7ac00fe830 2024-03-14T13:18:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:11 smithi137 ceph-mon[108145]: from='client.34820 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:11 smithi137 ceph-mon[108145]: pgmap v422: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:11 smithi137 ceph-mon[108145]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:11 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3680174934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:18:12.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:11 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1936811613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:18:12.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.175+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f7ac005f5f0 con 0x7f7ac00fe830 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.176+0000 7f7abd7fa700 1 -- 172.21.15.111:0/2470229528 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f7aac059350 con 0x7f7ac00fe830 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:18:12.175 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 msgr2=0x7f7ab006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 0x7f7ab006ec70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400b7e0 tx=0x7f7ab4009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 msgr2=0x7f7ac00fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c8c0 tx=0x7f7aac0044c0 comp rx=0 tx=0).stop 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 shutdown_connections 2024-03-14T13:18:12.177 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab006c7c0 0x7f7ab006ec70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac00fde90 0x7f7ac00fbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 --2- 172.21.15.111:0/2470229528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac00fe830 0x7f7ac00fc290 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 >> 172.21.15.111:0/2470229528 conn(0x7f7ac0072bc0 msgr2=0x7f7ac00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 shutdown_connections 2024-03-14T13:18:12.178 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:12.178+0000 7f7ac6ac2700 1 -- 172.21.15.111:0/2470229528 wait complete. 2024-03-14T13:18:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:12 smithi111 ceph-mon[145314]: from='client.34828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:12.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:12 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2470229528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:18:13.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:12 smithi137 ceph-mon[108145]: from='client.34828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:13.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:12 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2470229528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:18:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:13 smithi111 ceph-mon[145314]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:13 smithi111 ceph-mon[145314]: pgmap v423: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:13 smithi137 ceph-mon[108145]: from='client.34840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:13 smithi137 ceph-mon[108145]: pgmap v423: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:15 smithi111 ceph-mon[145314]: pgmap v424: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:15 smithi137 ceph-mon[108145]: pgmap v424: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:17 smithi111 ceph-mon[145314]: pgmap v425: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:17 smithi137 ceph-mon[108145]: pgmap v425: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:19 smithi111 ceph-mon[145314]: pgmap v426: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:19 smithi137 ceph-mon[108145]: pgmap v426: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:21 smithi111 ceph-mon[145314]: pgmap v427: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:21 smithi137 ceph-mon[108145]: pgmap v427: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:23 smithi111 ceph-mon[145314]: pgmap v428: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:23 smithi137 ceph-mon[108145]: pgmap v428: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:25 smithi111 ceph-mon[145314]: pgmap v429: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:25 smithi137 ceph-mon[108145]: pgmap v429: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:26 smithi111 ceph-mon[145314]: pgmap v430: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:27.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:26 smithi137 ceph-mon[108145]: pgmap v430: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:29.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:29 smithi137 ceph-mon[108145]: pgmap v431: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:29 smithi111 ceph-mon[145314]: pgmap v431: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:31 smithi111 ceph-mon[145314]: pgmap v432: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:31 smithi137 ceph-mon[108145]: pgmap v432: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:33 smithi111 ceph-mon[145314]: pgmap v433: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:33 smithi137 ceph-mon[108145]: pgmap v433: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:35 smithi111 ceph-mon[145314]: pgmap v434: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:35 smithi137 ceph-mon[108145]: pgmap v434: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:37 smithi111 ceph-mon[145314]: pgmap v435: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:37 smithi137 ceph-mon[108145]: pgmap v435: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:39 smithi111 ceph-mon[145314]: pgmap v436: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:39 smithi137 ceph-mon[108145]: pgmap v436: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:41 smithi111 ceph-mon[145314]: pgmap v437: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:41 smithi137 ceph-mon[108145]: pgmap v437: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 -- 172.21.15.111:0/4171609002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fe870 msgr2=0x7fa27c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 --2- 172.21.15.111:0/4171609002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fe870 0x7fa27c106f40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa26c004940 tx=0x7fa26c009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 -- 172.21.15.111:0/4171609002 shutdown_connections 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 --2- 172.21.15.111:0/4171609002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fe870 0x7fa27c106f40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 --2- 172.21.15.111:0/4171609002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fded0 0x7fa27c0fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 -- 172.21.15.111:0/4171609002 >> 172.21.15.111:0/4171609002 conn(0x7fa27c072bc0 msgr2=0x7fa27c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 -- 172.21.15.111:0/4171609002 shutdown_connections 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 -- 172.21.15.111:0/4171609002 wait complete. 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.291+0000 7fa282564700 1 Processor -- start 2024-03-14T13:18:42.292 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa282564700 1 -- start start 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa282564700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa282564700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fe870 0x7fa27c113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa282564700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c114610 con 0x7fa27c0fded0 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa282564700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27c10da00 con 0x7fa27c0fe870 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39728/0 (socket says 172.21.15.111:39728) 2024-03-14T13:18:42.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 -- 172.21.15.111:0/3685624119 learned_addr learned my addr 172.21.15.111:0/3685624119 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fe870 0x7fa27c113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 -- 172.21.15.111:0/3685624119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fe870 msgr2=0x7fa27c113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fe870 0x7fa27c113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 -- 172.21.15.111:0/3685624119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c005040 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.292+0000 7fa27bfff700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa270002a50 tx=0x7fa2700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.293+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa270007cc0 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.293+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa27000f040 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.293+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27c10dce0 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.293+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa27000da70 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.293+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27c10e220 con 0x7fa27c0fded0 2024-03-14T13:18:42.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.294+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa27c1019a0 con 0x7fa27c0fded0 2024-03-14T13:18:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.294+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa27000dc10 con 0x7fa27c0fded0 2024-03-14T13:18:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.294+0000 7fa278ff9700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 0x7fa26406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.295+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa27008b1d0 con 0x7fa27c0fded0 2024-03-14T13:18:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.295+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 0x7fa26406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.295+0000 7fa27b7fe700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 0x7fa26406ec70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa27c10ebc0 tx=0x7fa26c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.299+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa270058e60 con 0x7fa27c0fded0 2024-03-14T13:18:42.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.513+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa27c10f0d0 con 0x7fa26406c7c0 2024-03-14T13:18:42.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.514+0000 7fa278ff9700 1 -- 172.21.15.111:0/3685624119 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fa27c10f0d0 con 0x7fa26406c7c0 2024-03-14T13:18:42.516 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 msgr2=0x7fa26406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 0x7fa26406ec70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa27c10ebc0 tx=0x7fa26c009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 msgr2=0x7fa27c113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa270002a50 tx=0x7fa2700046b0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 shutdown_connections 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa26406c7c0 0x7fa26406ec70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa27c0fded0 0x7fa27c113a80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 --2- 172.21.15.111:0/3685624119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa27c0fe870 0x7fa27c113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 >> 172.21.15.111:0/3685624119 conn(0x7fa27c072bc0 msgr2=0x7fa27c0ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 shutdown_connections 2024-03-14T13:18:42.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.516+0000 7fa282564700 1 -- 172.21.15.111:0/3685624119 wait complete. 2024-03-14T13:18:42.527 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:18:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 -- 172.21.15.111:0/847360190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d81026c0 msgr2=0x7f86d80fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 --2- 172.21.15.111:0/847360190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d81026c0 0x7f86d80fbd30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004940 tx=0x7f86c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 -- 172.21.15.111:0/847360190 shutdown_connections 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 --2- 172.21.15.111:0/847360190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d81026c0 0x7f86d80fbd30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 --2- 172.21.15.111:0/847360190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d8101d20 0x7f86d81020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 -- 172.21.15.111:0/847360190 >> 172.21.15.111:0/847360190 conn(0x7f86d80f7870 msgr2=0x7f86d80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 -- 172.21.15.111:0/847360190 shutdown_connections 2024-03-14T13:18:42.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.618+0000 7f86dda2c700 1 -- 172.21.15.111:0/847360190 wait complete. 2024-03-14T13:18:42.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 Processor -- start 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 -- start start 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d81026c0 0x7f86d81030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8104ad0 con 0x7f86d8101d20 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86dda2c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d8103600 con 0x7f86d81026c0 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39746/0 (socket says 172.21.15.111:39746) 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 -- 172.21.15.111:0/1721680732 learned_addr learned my addr 172.21.15.111:0/1721680732 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d67fc700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d81026c0 0x7f86d81030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 -- 172.21.15.111:0/1721680732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d81026c0 msgr2=0x7f86d81030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d81026c0 0x7f86d81030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 -- 172.21.15.111:0/1721680732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8005040 con 0x7f86d8101d20 2024-03-14T13:18:42.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86d6ffd700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c8c0 tx=0x7f86cc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.619+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86cc00a760 con 0x7f86d8101d20 2024-03-14T13:18:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.620+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86cc00ad70 con 0x7f86d8101d20 2024-03-14T13:18:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.620+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86cc0113a0 con 0x7f86d8101d20 2024-03-14T13:18:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.620+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d81038e0 con 0x7f86d8101d20 2024-03-14T13:18:42.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.620+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d811a000 con 0x7f86d8101d20 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.621+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d80fd470 con 0x7f86d8101d20 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.621+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f86cc012400 con 0x7f86d8101d20 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.621+0000 7f86bf7fe700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 0x7f86c006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.622+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f86cc08ba60 con 0x7f86d8101d20 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.622+0000 7f86d67fc700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 0x7f86c006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.622+0000 7f86d67fc700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 0x7f86c006ed40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004890 tx=0x7f86c8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.626+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f86cc055ce0 con 0x7f86d8101d20 2024-03-14T13:18:42.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.836+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f86d8104100 con 0x7f86c006c890 2024-03-14T13:18:42.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.837+0000 7f86bf7fe700 1 -- 172.21.15.111:0/1721680732 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f86d8104100 con 0x7f86c006c890 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 msgr2=0x7f86c006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 0x7f86c006ed40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86c8004890 tx=0x7f86c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 msgr2=0x7f86d8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00c8c0 tx=0x7f86cc0044c0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 shutdown_connections 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f86c006c890 0x7f86c006ed40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86d8101d20 0x7f86d8102b80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 --2- 172.21.15.111:0/1721680732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86d81026c0 0x7f86d81030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 >> 172.21.15.111:0/1721680732 conn(0x7f86d80f7870 msgr2=0x7f86d80f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 shutdown_connections 2024-03-14T13:18:42.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.839+0000 7f86dda2c700 1 -- 172.21.15.111:0/1721680732 wait complete. 2024-03-14T13:18:42.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 -- 172.21.15.111:0/3991649804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f00fb3b0 msgr2=0x7f72f00fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 --2- 172.21.15.111:0/3991649804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f00fb3b0 0x7f72f00fb820 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f72e0004940 tx=0x7f72e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 -- 172.21.15.111:0/3991649804 shutdown_connections 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 --2- 172.21.15.111:0/3991649804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f00fb3b0 0x7f72f00fb820 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 --2- 172.21.15.111:0/3991649804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f0102f70 0x7f72f0103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 -- 172.21.15.111:0/3991649804 >> 172.21.15.111:0/3991649804 conn(0x7f72f0072310 msgr2=0x7f72f0072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 -- 172.21.15.111:0/3991649804 shutdown_connections 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.939+0000 7f72f5791700 1 -- 172.21.15.111:0/3991649804 wait complete. 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 Processor -- start 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 -- start start 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f00fb3b0 0x7f72f0113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f01143f0 con 0x7f72f0102f70 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72f5791700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f010d9f0 con 0x7f72f00fb3b0 2024-03-14T13:18:42.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39772/0 (socket says 172.21.15.111:39772) 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 -- 172.21.15.111:0/4095426132 learned_addr learned my addr 172.21.15.111:0/4095426132 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72eeffd700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f00fb3b0 0x7f72f0113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 -- 172.21.15.111:0/4095426132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f00fb3b0 msgr2=0x7f72f0113860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f00fb3b0 0x7f72f0113860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 -- 172.21.15.111:0/4095426132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e0005040 con 0x7f72f0102f70 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.940+0000 7f72ee7fc700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009150 tx=0x7f72e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e000e040 con 0x7f72f0102f70 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72e0013420 con 0x7f72f0102f70 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f010dc70 con 0x7f72f0102f70 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72e001c630 con 0x7f72f0102f70 2024-03-14T13:18:42.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f010e230 con 0x7f72f0102f70 2024-03-14T13:18:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.941+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f01053b0 con 0x7f72f0102f70 2024-03-14T13:18:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.942+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f72e0013990 con 0x7f72f0102f70 2024-03-14T13:18:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.942+0000 7f72d77fe700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 0x7f72d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.943+0000 7f72eeffd700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 0x7f72d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:42.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.943+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f72e008ccb0 con 0x7f72f0102f70 2024-03-14T13:18:42.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.943+0000 7f72eeffd700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 0x7f72d806ec70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72e4004990 tx=0x7f72e4009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:42.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:42.947+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f72e005a9c0 con 0x7f72f0102f70 2024-03-14T13:18:43.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.159+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f010eeb0 con 0x7f72d806c7c0 2024-03-14T13:18:43.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.168+0000 7f72d77fe700 1 -- 172.21.15.111:0/4095426132 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f72f010eeb0 con 0x7f72d806c7c0 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (14m) 8m ago 26m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (12m) 8m ago 26m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (12m) 7m ago 25m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (14m) 8m ago 25m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (8m) 8m ago 20m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (8m) 8m ago 20m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (7m) 7m ago 20m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (8m) 7m ago 20m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (17m) 8m ago 27m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (16m) 7m ago 25m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:18:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (13m) 8m ago 28m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (13m) 7m ago 25m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (15m) 8m ago 26m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (15m) 7m ago 25m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (12m) 8m ago 24m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (11m) 8m ago 23m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (10m) 8m ago 23m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (10m) 7m ago 22m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (9m) 7m ago 22m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (9m) 7m ago 21m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:18:43.170 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (15m) 8m ago 25m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.170+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 msgr2=0x7f72d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.170+0000 7f72f5791700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 0x7f72d806ec70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f72e4004990 tx=0x7f72e4009ee0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.170+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 msgr2=0x7f72f0113da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.170+0000 7f72f5791700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f72e0009150 tx=0x7f72e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 shutdown_connections 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f72d806c7c0 0x7f72d806ec70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72f00fb3b0 0x7f72f0113860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 --2- 172.21.15.111:0/4095426132 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f0102f70 0x7f72f0113da0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.171 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 >> 172.21.15.111:0/4095426132 conn(0x7f72f0072310 msgr2=0x7f72f00f9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 shutdown_connections 2024-03-14T13:18:43.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.171+0000 7f72f5791700 1 -- 172.21.15.111:0/4095426132 wait complete. 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.269+0000 7f410354e700 1 -- 172.21.15.111:0/1529453390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 msgr2=0x7f40fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.269+0000 7f410354e700 1 --2- 172.21.15.111:0/1529453390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc102140 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f40f8004940 tx=0x7f40f8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.269+0000 7f410354e700 1 -- 172.21.15.111:0/1529453390 shutdown_connections 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.269+0000 7f410354e700 1 --2- 172.21.15.111:0/1529453390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 0x7f40fc0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.269+0000 7f410354e700 1 --2- 172.21.15.111:0/1529453390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc102140 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- 172.21.15.111:0/1529453390 >> 172.21.15.111:0/1529453390 conn(0x7f40fc072bc0 msgr2=0x7f40fc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- 172.21.15.111:0/1529453390 shutdown_connections 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- 172.21.15.111:0/1529453390 wait complete. 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 Processor -- start 2024-03-14T13:18:43.272 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- start start 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 0x7f40fc10ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc10f5d0 con 0x7f40fc101d70 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f410354e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc10f710 con 0x7f40fc102710 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f41012ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f41012ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39790/0 (socket says 172.21.15.111:39790) 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.270+0000 7f41012ea700 1 -- 172.21.15.111:0/47298694 learned_addr learned my addr 172.21.15.111:0/47298694 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f41012ea700 1 -- 172.21.15.111:0/47298694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 msgr2=0x7f40fc10ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f4100ae9700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 0x7f40fc10ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f41012ea700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 0x7f40fc10ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f41012ea700 1 -- 172.21.15.111:0/47298694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8005040 con 0x7f40fc101d70 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f41012ea700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009180 tx=0x7f40f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:43.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f800e040 con 0x7f40fc101d70 2024-03-14T13:18:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40f8004290 con 0x7f40fc101d70 2024-03-14T13:18:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc117ab0 con 0x7f40fc101d70 2024-03-14T13:18:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40f801c3f0 con 0x7f40fc101d70 2024-03-14T13:18:43.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.271+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc117fc0 con 0x7f40fc101d70 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.272+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc0ff080 con 0x7f40fc101d70 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.273+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f40f8013790 con 0x7f40fc101d70 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.273+0000 7f40f1ffb700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 0x7f40e806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.273+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f40f808cc80 con 0x7f40fc101d70 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.273+0000 7f4100ae9700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 0x7f40e806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.274+0000 7f4100ae9700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 0x7f40e806ec70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f40fc066000 tx=0x7f40ec00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:43.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.277+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f40f8057070 con 0x7f40fc101d70 2024-03-14T13:18:43.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.559+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f40fc065db0 con 0x7f40fc101d70 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.560+0000 7f40f1ffb700 1 -- 172.21.15.111:0/47298694 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f40f8013420 con 0x7f40fc101d70 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:18:43.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:18:43.565 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 msgr2=0x7f40e806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 0x7f40e806ec70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f40fc066000 tx=0x7f40ec00b040 comp rx=0 tx=0).stop 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 msgr2=0x7f40fc10ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009180 tx=0x7f40f8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 shutdown_connections 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f40e806c7c0 0x7f40e806ec70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40fc101d70 0x7f40fc10ea40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 --2- 172.21.15.111:0/47298694 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40fc102710 0x7f40fc10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 >> 172.21.15.111:0/47298694 conn(0x7f40fc072bc0 msgr2=0x7f40fc0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 shutdown_connections 2024-03-14T13:18:43.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.562+0000 7f410354e700 1 -- 172.21.15.111:0/47298694 wait complete. 2024-03-14T13:18:43.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 -- 172.21.15.111:0/3511959580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 msgr2=0x7f6a300fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 --2- 172.21.15.111:0/3511959580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a300fbd20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6a20004940 tx=0x7f6a20009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:43.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 -- 172.21.15.111:0/3511959580 shutdown_connections 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 --2- 172.21.15.111:0/3511959580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a300fbd20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 --2- 172.21.15.111:0/3511959580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 0x7f6a301020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 -- 172.21.15.111:0/3511959580 >> 172.21.15.111:0/3511959580 conn(0x7f6a300f7880 msgr2=0x7f6a300f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 -- 172.21.15.111:0/3511959580 shutdown_connections 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.661+0000 7f6a34b48700 1 -- 172.21.15.111:0/3511959580 wait complete. 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 Processor -- start 2024-03-14T13:18:43.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 -- start start 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 0x7f6a30102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30103640 con 0x7f6a301026b0 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a34b48700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30103780 con 0x7f6a30101d10 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39804/0 (socket says 172.21.15.111:39804) 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 -- 172.21.15.111:0/914356342 learned_addr learned my addr 172.21.15.111:0/914356342 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2e59c700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 0x7f6a30102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 -- 172.21.15.111:0/914356342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 msgr2=0x7f6a30102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 0x7f6a30102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.662+0000 7f6a2dd9b700 1 -- 172.21.15.111:0/914356342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20005040 con 0x7f6a301026b0 2024-03-14T13:18:43.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a2dd9b700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009380 tx=0x7f6a20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2000e040 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a20013420 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a301039d0 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a2001c630 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.663+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3011a550 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.664+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a300fd460 con 0x7f6a301026b0 2024-03-14T13:18:43.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.664+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6a200139b0 con 0x7f6a301026b0 2024-03-14T13:18:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.665+0000 7f6a1effd700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 0x7f6a1806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.665+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6a2008cc60 con 0x7f6a301026b0 2024-03-14T13:18:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.665+0000 7f6a2e59c700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 0x7f6a1806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:43.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.665+0000 7f6a2e59c700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 0x7f6a1806ec70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6a24007fd0 tx=0x7f6a24009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:43.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.669+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6a20056e60 con 0x7f6a301026b0 2024-03-14T13:18:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:43 smithi111 ceph-mon[145314]: pgmap v438: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:43 smithi111 ceph-mon[145314]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:43.906 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.905+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6a300fd460 con 0x7f6a301026b0 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.906+0000 7f6a1effd700 1 -- 172.21.15.111:0/914356342 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f6a2005a480 con 0x7f6a301026b0 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:43.907 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:18:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:18:43.909 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:18:43.910 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:43.911 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:43.912 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:43.912 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 msgr2=0x7f6a1806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 0x7f6a1806ec70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6a24007fd0 tx=0x7f6a24009040 comp rx=0 tx=0).stop 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 msgr2=0x7f6a301030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009380 tx=0x7f6a20009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 shutdown_connections 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6a1806c7c0 0x7f6a1806ec70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a30101d10 0x7f6a30102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 --2- 172.21.15.111:0/914356342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a301026b0 0x7f6a301030b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:43.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 >> 172.21.15.111:0/914356342 conn(0x7f6a300f7880 msgr2=0x7f6a300f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:43.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 shutdown_connections 2024-03-14T13:18:43.914 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:43.908+0000 7f6a34b48700 1 -- 172.21.15.111:0/914356342 wait complete. 2024-03-14T13:18:43.914 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:18:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:43 smithi137 ceph-mon[108145]: pgmap v438: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:44.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:43 smithi137 ceph-mon[108145]: from='client.34848 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 -- 172.21.15.111:0/3206326102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 msgr2=0x7f09d00fb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.010 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 --2- 172.21.15.111:0/3206326102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d00fb820 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f09b8004940 tx=0x7f09b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 -- 172.21.15.111:0/3206326102 shutdown_connections 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 --2- 172.21.15.111:0/3206326102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d00fb820 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 --2- 172.21.15.111:0/3206326102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 0x7f09d0103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 -- 172.21.15.111:0/3206326102 >> 172.21.15.111:0/3206326102 conn(0x7f09d0072310 msgr2=0x7f09d0072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 -- 172.21.15.111:0/3206326102 shutdown_connections 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.009+0000 7f09d669f700 1 -- 172.21.15.111:0/3206326102 wait complete. 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 Processor -- start 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 -- start start 2024-03-14T13:18:44.011 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 0x7f09d0113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0114430 con 0x7f09d00fb3b0 2024-03-14T13:18:44.012 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09d669f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d010da30 con 0x7f09d0102f70 2024-03-14T13:18:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39818/0 (socket says 172.21.15.111:39818) 2024-03-14T13:18:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 -- 172.21.15.111:0/1599079674 learned_addr learned my addr 172.21.15.111:0/1599079674 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cf7fe700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 0x7f09d0113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 -- 172.21.15.111:0/1599079674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 msgr2=0x7f09d0113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 0x7f09d0113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 -- 172.21.15.111:0/1599079674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8005040 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.010+0000 7f09cffff700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c830 tx=0x7f09c0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.011+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09c00106f0 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.011+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09c0010d00 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.011+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d010dd10 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.011+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09c0012e90 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.011+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d010e2a0 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.012+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d01053b0 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.012+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f09c0010850 con 0x7f09d00fb3b0 2024-03-14T13:18:44.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.013+0000 7f09ccff9700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 0x7f09bc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.013+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f09c008b9a0 con 0x7f09d00fb3b0 2024-03-14T13:18:44.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.013+0000 7f09cf7fe700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 0x7f09bc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.013+0000 7f09cf7fe700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 0x7f09bc06ec70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09d010f1d0 tx=0x7f09b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:44.017 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.017+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f09c0059770 con 0x7f09d00fb3b0 2024-03-14T13:18:44.227 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.227+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f09d010ef60 con 0x7f09bc06c7c0 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.228+0000 7f09ccff9700 1 -- 172.21.15.111:0/1599079674 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f09d010ef60 con 0x7f09bc06c7c0 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:18:44.228 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:18:44.229 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:18:44.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 msgr2=0x7f09bc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 0x7f09bc06ec70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09d010f1d0 tx=0x7f09b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 msgr2=0x7f09d01138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c830 tx=0x7f09c0004470 comp rx=0 tx=0).stop 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 shutdown_connections 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f09bc06c7c0 0x7f09bc06ec70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09d00fb3b0 0x7f09d01138a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 --2- 172.21.15.111:0/1599079674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d0102f70 0x7f09d0113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 >> 172.21.15.111:0/1599079674 conn(0x7f09d0072310 msgr2=0x7f09d00f9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 shutdown_connections 2024-03-14T13:18:44.231 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.230+0000 7f09d669f700 1 -- 172.21.15.111:0/1599079674 wait complete. 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- 172.21.15.111:0/508802734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4102710 msgr2=0x7f35a40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 --2- 172.21.15.111:0/508802734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4102710 0x7f35a40fbd80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f35a0004940 tx=0x7f35a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- 172.21.15.111:0/508802734 shutdown_connections 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 --2- 172.21.15.111:0/508802734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4102710 0x7f35a40fbd80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 --2- 172.21.15.111:0/508802734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4101d70 0x7f35a4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- 172.21.15.111:0/508802734 >> 172.21.15.111:0/508802734 conn(0x7f35a4072bc0 msgr2=0x7f35a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- 172.21.15.111:0/508802734 shutdown_connections 2024-03-14T13:18:44.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- 172.21.15.111:0/508802734 wait complete. 2024-03-14T13:18:44.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 Processor -- start 2024-03-14T13:18:44.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.330+0000 7f35ac215700 1 -- start start 2024-03-14T13:18:44.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4102710 0x7f35a4112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4113550 con 0x7f35a4101d70 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a410c940 con 0x7f35a4102710 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39834/0 (socket says 172.21.15.111:39834) 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 -- 172.21.15.111:0/385063745 learned_addr learned my addr 172.21.15.111:0/385063745 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a97b0700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4102710 0x7f35a4112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 -- 172.21.15.111:0/385063745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4102710 msgr2=0x7f35a4112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4102710 0x7f35a4112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 -- 172.21.15.111:0/385063745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0005040 con 0x7f35a4101d70 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35a9fb1700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f359400c830 tx=0x7f3594004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f359400a760 con 0x7f35a4101d70 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a410cc20 con 0x7f35a4101d70 2024-03-14T13:18:44.333 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f359400ad70 con 0x7f35a4101d70 2024-03-14T13:18:44.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35940113a0 con 0x7f35a4101d70 2024-03-14T13:18:44.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.331+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a410d1e0 con 0x7f35a4101d70 2024-03-14T13:18:44.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.332+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a40ff080 con 0x7f35a4101d70 2024-03-14T13:18:44.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.333+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3594012400 con 0x7f35a4101d70 2024-03-14T13:18:44.334 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.333+0000 7f359a7fc700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 0x7f359006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:18:44.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.333+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f359408bae0 con 0x7f35a4101d70 2024-03-14T13:18:44.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.333+0000 7f35a97b0700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 0x7f359006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:18:44.335 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.334+0000 7f35a97b0700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 0x7f359006ec70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009fd0 tx=0x7f35a0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:18:44.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.338+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f35940597f0 con 0x7f35a4101d70 2024-03-14T13:18:44.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.615+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f35a410e010 con 0x7f35a4101d70 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.615+0000 7f359a7fc700 1 -- 172.21.15.111:0/385063745 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f3594059380 con 0x7f35a4101d70 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:18:44.616 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:18:44.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 msgr2=0x7f359006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 0x7f359006ec70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f35a0009fd0 tx=0x7f35a0009e30 comp rx=0 tx=0).stop 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 msgr2=0x7f35a41129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f359400c830 tx=0x7f3594004470 comp rx=0 tx=0).stop 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 shutdown_connections 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f359006c7c0 0x7f359006ec70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a4101d70 0x7f35a41129c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 --2- 172.21.15.111:0/385063745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35a4102710 0x7f35a4112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 >> 172.21.15.111:0/385063745 conn(0x7f35a4072bc0 msgr2=0x7f35a40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 shutdown_connections 2024-03-14T13:18:44.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:18:44.617+0000 7f35ac215700 1 -- 172.21.15.111:0/385063745 wait complete. 2024-03-14T13:18:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:44 smithi111 ceph-mon[145314]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:44 smithi111 ceph-mon[145314]: from='client.34856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:44 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/47298694' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:18:44.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:44 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/914356342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:18:45.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:44 smithi137 ceph-mon[108145]: from='client.34852 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:44 smithi137 ceph-mon[108145]: from='client.34856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:44 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/47298694' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:18:45.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:44 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/914356342' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:18:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:45 smithi111 ceph-mon[145314]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:45 smithi111 ceph-mon[145314]: pgmap v439: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:45 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/385063745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:18:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:45 smithi137 ceph-mon[108145]: from='client.34868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:18:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:45 smithi137 ceph-mon[108145]: pgmap v439: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:46.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:45 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/385063745' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:18:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:47 smithi111 ceph-mon[145314]: pgmap v440: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:47 smithi137 ceph-mon[108145]: pgmap v440: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:49 smithi111 ceph-mon[145314]: pgmap v441: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:49 smithi137 ceph-mon[108145]: pgmap v441: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:51 smithi111 ceph-mon[145314]: pgmap v442: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:51 smithi137 ceph-mon[108145]: pgmap v442: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:53 smithi111 ceph-mon[145314]: pgmap v443: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:54.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:53 smithi137 ceph-mon[108145]: pgmap v443: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:18:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:55 smithi111 ceph-mon[145314]: pgmap v444: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:55 smithi137 ceph-mon[108145]: pgmap v444: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:57 smithi111 ceph-mon[145314]: pgmap v445: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:57 smithi137 ceph-mon[108145]: pgmap v445: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:18:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:18:59 smithi111 ceph-mon[145314]: pgmap v446: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:18:59 smithi137 ceph-mon[108145]: pgmap v446: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:01 smithi111 ceph-mon[145314]: pgmap v447: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:01 smithi137 ceph-mon[108145]: pgmap v447: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:03 smithi111 ceph-mon[145314]: pgmap v448: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:03 smithi137 ceph-mon[108145]: pgmap v448: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:05 smithi111 ceph-mon[145314]: pgmap v449: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:05 smithi137 ceph-mon[108145]: pgmap v449: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:07 smithi111 ceph-mon[145314]: pgmap v450: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:07 smithi137 ceph-mon[108145]: pgmap v450: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:09 smithi111 ceph-mon[145314]: pgmap v451: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:09 smithi137 ceph-mon[108145]: pgmap v451: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:11.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:10 smithi137 ceph-mon[108145]: pgmap v452: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:11.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:10 smithi111 ceph-mon[145314]: pgmap v452: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:13.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:13 smithi137 ceph-mon[108145]: pgmap v453: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:13 smithi111 ceph-mon[145314]: pgmap v453: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 -- 172.21.15.111:0/3749103034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fde90 msgr2=0x7fe5bc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3749103034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc0fe260 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac004940 tx=0x7fe5ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 -- 172.21.15.111:0/3749103034 shutdown_connections 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3749103034 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3749103034 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc0fe260 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 -- 172.21.15.111:0/3749103034 >> 172.21.15.111:0/3749103034 conn(0x7fe5bc072bc0 msgr2=0x7fe5bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 -- 172.21.15.111:0/3749103034 shutdown_connections 2024-03-14T13:19:14.731 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.731+0000 7fe5c335e700 1 -- 172.21.15.111:0/3749103034 wait complete. 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 Processor -- start 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 -- start start 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc114680 con 0x7fe5bc0fe830 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c335e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc10da70 con 0x7fe5bc0fde90 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c10fa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54708/0 (socket says 172.21.15.111:54708) 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 -- 172.21.15.111:0/3990586094 learned_addr learned my addr 172.21.15.111:0/3990586094 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 -- 172.21.15.111:0/3990586094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fde90 msgr2=0x7fe5bc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.732+0000 7fe5c08f9700 1 -- 172.21.15.111:0/3990586094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac005040 con 0x7fe5bc0fe830 2024-03-14T13:19:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5c08f9700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800c830 tx=0x7fe5b8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b80106f0 con 0x7fe5bc0fe830 2024-03-14T13:19:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc10dd50 con 0x7fe5bc0fe830 2024-03-14T13:19:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5b8010d00 con 0x7fe5bc0fe830 2024-03-14T13:19:14.734 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5b8012e90 con 0x7fe5bc0fe830 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.733+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc10e290 con 0x7fe5bc0fe830 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.734+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc1019b0 con 0x7fe5bc0fe830 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.734+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe5b8010850 con 0x7fe5bc0fe830 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.735+0000 7fe5b1ffb700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 0x7fe5a806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.735+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe5b808ba30 con 0x7fe5bc0fe830 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.735+0000 7fe5c10fa700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 0x7fe5a806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:14.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.735+0000 7fe5c10fa700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 0x7fe5a806ec70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00b7e0 tx=0x7fe5ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:14.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.739+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe5b808de60 con 0x7fe5bc0fe830 2024-03-14T13:19:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.949+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5bc1137c0 con 0x7fe5a806c7c0 2024-03-14T13:19:14.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.950+0000 7fe5b1ffb700 1 -- 172.21.15.111:0/3990586094 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe5bc1137c0 con 0x7fe5a806c7c0 2024-03-14T13:19:14.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.952+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 msgr2=0x7fe5a806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:14.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.952+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 0x7fe5a806ec70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00b7e0 tx=0x7fe5ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:14.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.952+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 msgr2=0x7fe5bc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:14.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.952+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800c830 tx=0x7fe5b8004470 comp rx=0 tx=0).stop 2024-03-14T13:19:14.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 shutdown_connections 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5a806c7c0 0x7fe5a806ec70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5bc0fde90 0x7fe5bc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 --2- 172.21.15.111:0/3990586094 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5bc0fe830 0x7fe5bc114030 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 >> 172.21.15.111:0/3990586094 conn(0x7fe5bc072bc0 msgr2=0x7fe5bc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 shutdown_connections 2024-03-14T13:19:14.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:14.953+0000 7fe5c335e700 1 -- 172.21.15.111:0/3990586094 wait complete. 2024-03-14T13:19:14.963 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:19:15.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.054+0000 7f13a7924700 1 -- 172.21.15.111:0/3805566764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 msgr2=0x7f13a0106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.054+0000 7f13a7924700 1 --2- 172.21.15.111:0/3805566764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0106f00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1390004940 tx=0x7f1390009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 -- 172.21.15.111:0/3805566764 shutdown_connections 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 --2- 172.21.15.111:0/3805566764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0106f00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 --2- 172.21.15.111:0/3805566764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 0x7f13a00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 -- 172.21.15.111:0/3805566764 >> 172.21.15.111:0/3805566764 conn(0x7f13a0072bc0 msgr2=0x7f13a0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 -- 172.21.15.111:0/3805566764 shutdown_connections 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 -- 172.21.15.111:0/3805566764 wait complete. 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 Processor -- start 2024-03-14T13:19:15.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.055+0000 7f13a7924700 1 -- start start 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a7924700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 0x7f13a0113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a7924700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a7924700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0114670 con 0x7f13a00fe830 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a7924700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a010da60 con 0x7f13a00fde90 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54726/0 (socket says 172.21.15.111:54726) 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 -- 172.21.15.111:0/3773654519 learned_addr learned my addr 172.21.15.111:0/3773654519 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a56c0700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 0x7f13a0113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 -- 172.21.15.111:0/3773654519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 msgr2=0x7f13a0113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 0x7f13a0113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 -- 172.21.15.111:0/3773654519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390005040 con 0x7f13a00fe830 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a4ebf700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f139000b3b0 tx=0x7f1390009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139000e040 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1390013420 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a010dce0 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.056+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139001c5f0 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.057+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a010e220 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.057+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a0065e30 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.058+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1390013590 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.058+0000 7f1395ffb700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 0x7f138c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.058+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f139008cd70 con 0x7f13a00fe830 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.058+0000 7f13a56c0700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 0x7f138c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.059 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.059+0000 7f13a56c0700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 0x7f138c06ec70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f139c004f40 tx=0x7f139c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.063+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f139005aa80 con 0x7f13a00fe830 2024-03-14T13:19:15.273 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.274+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f13a0109dd0 con 0x7f138c06c7c0 2024-03-14T13:19:15.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.275+0000 7f1395ffb700 1 -- 172.21.15.111:0/3773654519 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f13a0109dd0 con 0x7f138c06c7c0 2024-03-14T13:19:15.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 msgr2=0x7f138c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 0x7f138c06ec70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f139c004f40 tx=0x7f139c00b040 comp rx=0 tx=0).stop 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 msgr2=0x7f13a0114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f139000b3b0 tx=0x7f1390009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 shutdown_connections 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f138c06c7c0 0x7f138c06ec70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13a00fde90 0x7f13a0113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 --2- 172.21.15.111:0/3773654519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13a00fe830 0x7f13a0114020 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 >> 172.21.15.111:0/3773654519 conn(0x7f13a0072bc0 msgr2=0x7f13a0103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 shutdown_connections 2024-03-14T13:19:15.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.277+0000 7f13a7924700 1 -- 172.21.15.111:0/3773654519 wait complete. 2024-03-14T13:19:15.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- 172.21.15.111:0/992627864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc102e10 msgr2=0x7fa7fc1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 --2- 172.21.15.111:0/992627864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc102e10 0x7fa7fc1031e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0004940 tx=0x7fa7f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- 172.21.15.111:0/992627864 shutdown_connections 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 --2- 172.21.15.111:0/992627864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 --2- 172.21.15.111:0/992627864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc102e10 0x7fa7fc1031e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- 172.21.15.111:0/992627864 >> 172.21.15.111:0/992627864 conn(0x7fa7fc0f8980 msgr2=0x7fa7fc0fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- 172.21.15.111:0/992627864 shutdown_connections 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- 172.21.15.111:0/992627864 wait complete. 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 Processor -- start 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- start start 2024-03-14T13:19:15.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc102e10 0x7fa7fc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc070d90 con 0x7fa7fc1037b0 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.378+0000 7fa800fb8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc070ed0 con 0x7fa7fc102e10 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54754/0 (socket says 172.21.15.111:54754) 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 -- 172.21.15.111:0/3668439743 learned_addr learned my addr 172.21.15.111:0/3668439743 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7fa59c700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc102e10 0x7fa7fc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 -- 172.21.15.111:0/3668439743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc102e10 msgr2=0x7fa7fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc102e10 0x7fa7fc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 -- 172.21.15.111:0/3668439743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0005040 con 0x7fa7fc1037b0 2024-03-14T13:19:15.380 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7f9d9b700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec002a50 tx=0x7fa7ec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7ec007cc0 con 0x7fa7fc1037b0 2024-03-14T13:19:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7ec00f040 con 0x7fa7fc1037b0 2024-03-14T13:19:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc119e10 con 0x7fa7fc1037b0 2024-03-14T13:19:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa7ec00d920 con 0x7fa7fc1037b0 2024-03-14T13:19:15.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.379+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc11a290 con 0x7fa7fc1037b0 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.380+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc0fe560 con 0x7fa7fc1037b0 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.381+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa7ec009680 con 0x7fa7fc1037b0 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.381+0000 7fa7eaffd700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 0x7fa7e406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.381+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa7ec08b720 con 0x7fa7fc1037b0 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.381+0000 7fa7fa59c700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 0x7fa7e406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.381+0000 7fa7fa59c700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 0x7fa7e406ec70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0004f90 tx=0x7fa7f0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.386+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa7ec0559a0 con 0x7fa7fc1037b0 2024-03-14T13:19:15.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.588+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7fc04bed0 con 0x7fa7e406c7c0 2024-03-14T13:19:15.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.597+0000 7fa7eaffd700 1 -- 172.21.15.111:0/3668439743 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fa7fc04bed0 con 0x7fa7e406c7c0 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (15m) 8m ago 27m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (12m) 8m ago 27m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (12m) 8m ago 25m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (15m) 8m ago 26m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (9m) 8m ago 20m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (9m) 8m ago 20m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (8m) 8m ago 20m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (8m) 8m ago 20m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (18m) 8m ago 28m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (16m) 8m ago 25m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (14m) 8m ago 28m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:19:15.597 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (13m) 8m ago 25m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (16m) 8m ago 27m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (16m) 8m ago 25m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (12m) 8m ago 24m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (11m) 8m ago 24m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (11m) 8m ago 23m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (10m) 8m ago 23m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (10m) 8m ago 22m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (9m) 8m ago 22m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:19:15.598 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (15m) 8m ago 26m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 msgr2=0x7fa7e406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 0x7fa7e406ec70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0004f90 tx=0x7fa7f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 msgr2=0x7fa7fc070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec002a50 tx=0x7fa7ec004600 comp rx=0 tx=0).stop 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 shutdown_connections 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa7e406c7c0 0x7fa7e406ec70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc102e10 0x7fa7fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 --2- 172.21.15.111:0/3668439743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7fc1037b0 0x7fa7fc070800 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 >> 172.21.15.111:0/3668439743 conn(0x7fa7fc0f8980 msgr2=0x7fa7fc0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.599+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 shutdown_connections 2024-03-14T13:19:15.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.600+0000 7fa800fb8700 1 -- 172.21.15.111:0/3668439743 wait complete. 2024-03-14T13:19:15.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 -- 172.21.15.111:0/1813249692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 msgr2=0x7f5e300fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 --2- 172.21.15.111:0/1813249692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e300fbd80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c004940 tx=0x7f5e2c009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 -- 172.21.15.111:0/1813249692 shutdown_connections 2024-03-14T13:19:15.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 --2- 172.21.15.111:0/1813249692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e300fbd80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 --2- 172.21.15.111:0/1813249692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e30101d70 0x7f5e30102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 -- 172.21.15.111:0/1813249692 >> 172.21.15.111:0/1813249692 conn(0x7f5e30072bc0 msgr2=0x7f5e30072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 -- 172.21.15.111:0/1813249692 shutdown_connections 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.698+0000 7f5e378b3700 1 -- 172.21.15.111:0/1813249692 wait complete. 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 Processor -- start 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 -- start start 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e30101d70 0x7f5e30113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30114650 con 0x7f5e30102710 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e378b3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3010da40 con 0x7f5e30101d70 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54766/0 (socket says 172.21.15.111:54766) 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 -- 172.21.15.111:0/354829278 learned_addr learned my addr 172.21.15.111:0/354829278 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 -- 172.21.15.111:0/354829278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e30101d70 msgr2=0x7f5e30113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e30101d70 0x7f5e30113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 -- 172.21.15.111:0/354829278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c005040 con 0x7f5e30102710 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.699+0000 7f5e34e4e700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009380 tx=0x7f5e2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.700+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e2c00e040 con 0x7f5e30102710 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.700+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e2c013420 con 0x7f5e30102710 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.700+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3010dcc0 con 0x7f5e30102710 2024-03-14T13:19:15.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.700+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e2c01c630 con 0x7f5e30102710 2024-03-14T13:19:15.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.700+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3010e280 con 0x7f5e30102710 2024-03-14T13:19:15.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.701+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e300fd4c0 con 0x7f5e30102710 2024-03-14T13:19:15.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.701+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5e2c0139e0 con 0x7f5e30102710 2024-03-14T13:19:15.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.702+0000 7f5e21ffb700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 0x7f5e1c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:15.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.702+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e2c08cc60 con 0x7f5e30102710 2024-03-14T13:19:15.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.702+0000 7f5e3564f700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 0x7f5e1c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:15.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.702+0000 7f5e3564f700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 0x7f5e1c06ec70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e24004ad0 tx=0x7f5e24007d40 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:15.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.706+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e2c05a970 con 0x7f5e30102710 2024-03-14T13:19:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:15 smithi111 ceph-mon[145314]: pgmap v454: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:15.978 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.979+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e30108d10 con 0x7f5e30102710 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.979+0000 7f5e21ffb700 1 -- 172.21.15.111:0/354829278 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f5e2c05a500 con 0x7f5e30102710 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:19:15.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:19:15.980 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 msgr2=0x7f5e1c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 0x7f5e1c06ec70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5e24004ad0 tx=0x7f5e24007d40 comp rx=0 tx=0).stop 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 msgr2=0x7f5e30114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009380 tx=0x7f5e2c009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 shutdown_connections 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e1c06c7c0 0x7f5e1c06ec70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e30101d70 0x7f5e30113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 --2- 172.21.15.111:0/354829278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e30102710 0x7f5e30114000 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 >> 172.21.15.111:0/354829278 conn(0x7f5e30072bc0 msgr2=0x7f5e30065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:15.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 shutdown_connections 2024-03-14T13:19:15.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:15.982+0000 7f5e378b3700 1 -- 172.21.15.111:0/354829278 wait complete. 2024-03-14T13:19:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:15 smithi137 ceph-mon[108145]: pgmap v454: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 -- 172.21.15.111:0/525948946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 msgr2=0x7f77f40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/525948946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f40fbd80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f77e8004940 tx=0x7f77e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 -- 172.21.15.111:0/525948946 shutdown_connections 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/525948946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f40fbd80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/525948946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 0x7f77f4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 -- 172.21.15.111:0/525948946 >> 172.21.15.111:0/525948946 conn(0x7f77f4072bc0 msgr2=0x7f77f4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:16.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 -- 172.21.15.111:0/525948946 shutdown_connections 2024-03-14T13:19:16.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.081+0000 7f77f9b2c700 1 -- 172.21.15.111:0/525948946 wait complete. 2024-03-14T13:19:16.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 Processor -- start 2024-03-14T13:19:16.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 -- start start 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 0x7f77f41129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f4113550 con 0x7f77f4102710 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f9b2c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410c940 con 0x7f77f4101d70 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54784/0 (socket says 172.21.15.111:54784) 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.082+0000 7f77f2ffd700 1 -- 172.21.15.111:0/1682894354 learned_addr learned my addr 172.21.15.111:0/1682894354 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f37fe700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 0x7f77f41129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f2ffd700 1 -- 172.21.15.111:0/1682894354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 msgr2=0x7f77f41129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f2ffd700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 0x7f77f41129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f2ffd700 1 -- 172.21.15.111:0/1682894354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e8005040 con 0x7f77f4102710 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f2ffd700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009390 tx=0x7f77e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e800e040 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f410cbc0 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77e8013420 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77e801c630 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.083+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f410d180 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.084+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f40ff080 con 0x7f77f4102710 2024-03-14T13:19:16.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.084+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f77e8013a00 con 0x7f77f4102710 2024-03-14T13:19:16.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.085+0000 7f77dbfff700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 0x7f77dc06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.085+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f77e808cd50 con 0x7f77f4102710 2024-03-14T13:19:16.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.085+0000 7f77f37fe700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 0x7f77dc06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.085+0000 7f77f37fe700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 0x7f77dc06ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77e4007fd0 tx=0x7f77e4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.089+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f77e805aa60 con 0x7f77f4102710 2024-03-14T13:19:16.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.328+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77f410e010 con 0x7f77f4102710 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.328+0000 7f77dbfff700 1 -- 172.21.15.111:0/1682894354 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f77e805a5f0 con 0x7f77f4102710 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:19:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:19:16.329 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:16.330 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 msgr2=0x7f77dc06ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 0x7f77dc06ec80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f77e4007fd0 tx=0x7f77e4009040 comp rx=0 tx=0).stop 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 msgr2=0x7f77f4112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009390 tx=0x7f77e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:16.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 shutdown_connections 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f77dc06c7d0 0x7f77dc06ec80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77f4101d70 0x7f77f41129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 --2- 172.21.15.111:0/1682894354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f4102710 0x7f77f4112f00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.330+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 >> 172.21.15.111:0/1682894354 conn(0x7f77f4072bc0 msgr2=0x7f77f40fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.331+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 shutdown_connections 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.331+0000 7f77f9b2c700 1 -- 172.21.15.111:0/1682894354 wait complete. 2024-03-14T13:19:16.332 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 -- 172.21.15.111:0/3203704600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102e20 msgr2=0x7f5b8c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/3203704600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102e20 0x7f5b8c1031f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5b74004940 tx=0x7f5b74009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 -- 172.21.15.111:0/3203704600 shutdown_connections 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/3203704600 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/3203704600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c102e20 0x7f5b8c1031f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 -- 172.21.15.111:0/3203704600 >> 172.21.15.111:0/3203704600 conn(0x7f5b8c0f89d0 msgr2=0x7f5b8c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 -- 172.21.15.111:0/3203704600 shutdown_connections 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.434+0000 7f5b90dc7700 1 -- 172.21.15.111:0/3203704600 wait complete. 2024-03-14T13:19:16.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 Processor -- start 2024-03-14T13:19:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 -- start start 2024-03-14T13:19:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102e20 0x7f5b8c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c114710 con 0x7f5b8c1037c0 2024-03-14T13:19:16.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b90dc7700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c10da70 con 0x7f5b8c102e20 2024-03-14T13:19:16.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.436 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54794/0 (socket says 172.21.15.111:54794) 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 -- 172.21.15.111:0/2601618016 learned_addr learned my addr 172.21.15.111:0/2601618016 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102e20 0x7f5b8c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 -- 172.21.15.111:0/2601618016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102e20 msgr2=0x7f5b8c113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102e20 0x7f5b8c113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 -- 172.21.15.111:0/2601618016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74005040 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b89d9b700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b790 tx=0x7f5b7c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.435+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b7c00f6f0 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.436+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b7c00fd00 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.436+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c10dd50 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.436+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b7c0103a0 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.436+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c10e310 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.437+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c0fe570 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.437+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5b7c00f850 con 0x7f5b8c1037c0 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.437+0000 7f5b82ffd700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 0x7f5b7806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.437 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.437+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5b7c08b880 con 0x7f5b8c1037c0 2024-03-14T13:19:16.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.437+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 0x7f5b7806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.438 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.438+0000 7f5b8a59c700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 0x7f5b7806ec80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009fd0 tx=0x7f5b74009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.442+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5b7c059590 con 0x7f5b8c1037c0 2024-03-14T13:19:16.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.653+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b8c109e50 con 0x7f5b7806c7d0 2024-03-14T13:19:16.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.654+0000 7f5b82ffd700 1 -- 172.21.15.111:0/2601618016 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5b8c109e50 con 0x7f5b7806c7d0 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:19:16.654 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:19:16.655 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:19:16.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 msgr2=0x7f5b7806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 0x7f5b7806ec80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009fd0 tx=0x7f5b74009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 msgr2=0x7f5b8c114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00b790 tx=0x7f5b7c009220 comp rx=0 tx=0).stop 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 shutdown_connections 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5b7806c7d0 0x7f5b7806ec80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8c102e20 0x7f5b8c113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 --2- 172.21.15.111:0/2601618016 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b8c1037c0 0x7f5b8c114030 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 >> 172.21.15.111:0/2601618016 conn(0x7f5b8c0f89d0 msgr2=0x7f5b8c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 shutdown_connections 2024-03-14T13:19:16.656 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.656+0000 7f5b90dc7700 1 -- 172.21.15.111:0/2601618016 wait complete. 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 -- 172.21.15.111:0/1409367585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 msgr2=0x7f25481021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 --2- 172.21.15.111:0/1409367585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f25481021a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2530004940 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 -- 172.21.15.111:0/1409367585 shutdown_connections 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 --2- 172.21.15.111:0/1409367585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 0x7f25480fc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 --2- 172.21.15.111:0/1409367585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f25481021a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 -- 172.21.15.111:0/1409367585 >> 172.21.15.111:0/1409367585 conn(0x7f25480714c0 msgr2=0x7f25480718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 -- 172.21.15.111:0/1409367585 shutdown_connections 2024-03-14T13:19:16.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.756+0000 7f254d334700 1 -- 172.21.15.111:0/1409367585 wait complete. 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 Processor -- start 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 -- start start 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 0x7f254810efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254810f500 con 0x7f2548101dd0 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f254d334700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254810f640 con 0x7f2548102770 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:54810/0 (socket says 172.21.15.111:54810) 2024-03-14T13:19:16.758 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 -- 172.21.15.111:0/3434057550 learned_addr learned my addr 172.21.15.111:0/3434057550 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f25467fc700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 0x7f254810efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 -- 172.21.15.111:0/3434057550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 msgr2=0x7f254810efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 0x7f254810efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.757+0000 7f2546ffd700 1 -- 172.21.15.111:0/3434057550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530005040 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f2546ffd700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f253000b3b0 tx=0x7f2530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f253000e040 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2530013420 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254810f8c0 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f253001c630 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.758+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2548117e40 con 0x7f2548101dd0 2024-03-14T13:19:16.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.759+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25480ff0e0 con 0x7f2548101dd0 2024-03-14T13:19:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.759+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2530013970 con 0x7f2548101dd0 2024-03-14T13:19:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.759+0000 7f253f7fe700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 0x7f253406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.760+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f253008ccc0 con 0x7f2548101dd0 2024-03-14T13:19:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.760+0000 7f25467fc700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 0x7f253406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:16.760 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.760+0000 7f25467fc700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 0x7f253406ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2538009fd0 tx=0x7f2538009f20 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:16.763 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:16.764+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2530056f40 con 0x7f2548101dd0 2024-03-14T13:19:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:16 smithi111 ceph-mon[145314]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:16 smithi111 ceph-mon[145314]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:16 smithi111 ceph-mon[145314]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:16 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/354829278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:19:16.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:16 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1682894354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:19:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:16 smithi137 ceph-mon[108145]: from='client.34876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:16 smithi137 ceph-mon[108145]: from='client.34880 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:16 smithi137 ceph-mon[108145]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:16 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/354829278' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:19:17.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:16 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1682894354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.038+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f25480fd5f0 con 0x7f2548101dd0 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.038+0000 7f253f7fe700 1 -- 172.21.15.111:0/3434057550 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f253005a560 con 0x7f2548101dd0 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:19:17.038 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 msgr2=0x7f253406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 0x7f253406ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2538009fd0 tx=0x7f2538009f20 comp rx=0 tx=0).stop 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 msgr2=0x7f254810ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f253000b3b0 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 shutdown_connections 2024-03-14T13:19:17.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f253406c7c0 0x7f253406ec70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:17.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2548101dd0 0x7f254810ea80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:17.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 --2- 172.21.15.111:0/3434057550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2548102770 0x7f254810efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:17.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 >> 172.21.15.111:0/3434057550 conn(0x7f25480714c0 msgr2=0x7f25480fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:17.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 shutdown_connections 2024-03-14T13:19:17.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:17.041+0000 7f254d334700 1 -- 172.21.15.111:0/3434057550 wait complete. 2024-03-14T13:19:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:17 smithi111 ceph-mon[145314]: pgmap v455: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:17 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3434057550' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:19:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:17 smithi137 ceph-mon[108145]: pgmap v455: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:17 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3434057550' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:19:18.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:18 smithi111 ceph-mon[145314]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:19.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:18 smithi137 ceph-mon[108145]: from='client.34896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:19 smithi111 ceph-mon[145314]: pgmap v456: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:20.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:19 smithi137 ceph-mon[108145]: pgmap v456: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:21 smithi111 ceph-mon[145314]: pgmap v457: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:21 smithi137 ceph-mon[108145]: pgmap v457: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:23 smithi111 ceph-mon[145314]: pgmap v458: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:23 smithi137 ceph-mon[108145]: pgmap v458: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:25 smithi111 ceph-mon[145314]: pgmap v459: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:25 smithi137 ceph-mon[108145]: pgmap v459: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:27 smithi111 ceph-mon[145314]: pgmap v460: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:27 smithi137 ceph-mon[108145]: pgmap v460: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:29 smithi111 ceph-mon[145314]: pgmap v461: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:30.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:29 smithi137 ceph-mon[108145]: pgmap v461: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:31 smithi111 ceph-mon[145314]: pgmap v462: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:31 smithi137 ceph-mon[108145]: pgmap v462: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:33 smithi111 ceph-mon[145314]: pgmap v463: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:33 smithi137 ceph-mon[108145]: pgmap v463: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:35 smithi111 ceph-mon[145314]: pgmap v464: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:35 smithi137 ceph-mon[108145]: pgmap v464: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:37 smithi111 ceph-mon[145314]: pgmap v465: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:37 smithi137 ceph-mon[108145]: pgmap v465: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:39 smithi111 ceph-mon[145314]: pgmap v466: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:39 smithi137 ceph-mon[108145]: pgmap v466: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:41 smithi111 ceph-mon[145314]: pgmap v467: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:41 smithi137 ceph-mon[108145]: pgmap v467: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:43 smithi111 ceph-mon[145314]: pgmap v468: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:43 smithi137 ceph-mon[108145]: pgmap v468: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:45 smithi111 ceph-mon[145314]: pgmap v469: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:45 smithi137 ceph-mon[108145]: pgmap v469: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:47.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 -- 172.21.15.111:0/4212074944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0101d70 msgr2=0x7f36c0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 --2- 172.21.15.111:0/4212074944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0101d70 0x7f36c0102140 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004940 tx=0x7f36b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:47.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 -- 172.21.15.111:0/4212074944 shutdown_connections 2024-03-14T13:19:47.154 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 --2- 172.21.15.111:0/4212074944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0102710 0x7f36c00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 --2- 172.21.15.111:0/4212074944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0101d70 0x7f36c0102140 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 -- 172.21.15.111:0/4212074944 >> 172.21.15.111:0/4212074944 conn(0x7f36c0072bc0 msgr2=0x7f36c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 -- 172.21.15.111:0/4212074944 shutdown_connections 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.153+0000 7f36c67aa700 1 -- 172.21.15.111:0/4212074944 wait complete. 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 Processor -- start 2024-03-14T13:19:47.155 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 -- start start 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 0x7f36c0113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c0114610 con 0x7f36c0102710 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36c67aa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c010da00 con 0x7f36c0101d70 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36bf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36bf7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48386/0 (socket says 172.21.15.111:48386) 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.154+0000 7f36bf7fe700 1 -- 172.21.15.111:0/174078006 learned_addr learned my addr 172.21.15.111:0/174078006 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bffff700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 0x7f36c0113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bf7fe700 1 -- 172.21.15.111:0/174078006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 msgr2=0x7f36c0113a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bf7fe700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 0x7f36c0113a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bf7fe700 1 -- 172.21.15.111:0/174078006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4005040 con 0x7f36c0102710 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bffff700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 0x7f36c0113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:19:47.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bf7fe700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f36b0002a50 tx=0x7f36b0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b0007cc0 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c010dce0 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f36b000f040 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36b000d9a0 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.155+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c010e2a0 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.156+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c00fd4c0 con 0x7f36c0102710 2024-03-14T13:19:47.158 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.157+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f36b000dc60 con 0x7f36c0102710 2024-03-14T13:19:47.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.157+0000 7f36bcff9700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 0x7f36a806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.157+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f36b008b8c0 con 0x7f36c0102710 2024-03-14T13:19:47.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.157+0000 7f36bffff700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 0x7f36a806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.157+0000 7f36bffff700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 0x7f36a806ec70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004560 tx=0x7f36b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.161+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f36b0059550 con 0x7f36c0102710 2024-03-14T13:19:47.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.374+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f36c00666a0 con 0x7f36a806c7c0 2024-03-14T13:19:47.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.375+0000 7f36bcff9700 1 -- 172.21.15.111:0/174078006 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f36c00666a0 con 0x7f36a806c7c0 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 msgr2=0x7f36a806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 0x7f36a806ec70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f36b4004560 tx=0x7f36b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 msgr2=0x7f36c0113fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f36b0002a50 tx=0x7f36b0004600 comp rx=0 tx=0).stop 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 shutdown_connections 2024-03-14T13:19:47.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f36a806c7c0 0x7f36a806ec70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f36c0101d70 0x7f36c0113a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 --2- 172.21.15.111:0/174078006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f36c0102710 0x7f36c0113fc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 >> 172.21.15.111:0/174078006 conn(0x7f36c0072bc0 msgr2=0x7f36c0105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:47.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 shutdown_connections 2024-03-14T13:19:47.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.377+0000 7f36c67aa700 1 -- 172.21.15.111:0/174078006 wait complete. 2024-03-14T13:19:47.387 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/4016944770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fde90 msgr2=0x7ff5c00fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/4016944770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fde90 0x7ff5c00fe260 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc004940 tx=0x7ff5bc009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/4016944770 shutdown_connections 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/4016944770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fe830 0x7ff5c0106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/4016944770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fde90 0x7ff5c00fe260 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.478+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/4016944770 >> 172.21.15.111:0/4016944770 conn(0x7ff5c0072bc0 msgr2=0x7ff5c0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/4016944770 shutdown_connections 2024-03-14T13:19:47.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/4016944770 wait complete. 2024-03-14T13:19:47.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 Processor -- start 2024-03-14T13:19:47.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 -- start start 2024-03-14T13:19:47.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fde90 0x7ff5c0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c0114680 con 0x7ff5c00fe830 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c7ff2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c010da70 con 0x7ff5c00fde90 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c558d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c558d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48404/0 (socket says 172.21.15.111:48404) 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c558d700 1 -- 172.21.15.111:0/328741860 learned_addr learned my addr 172.21.15.111:0/328741860 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.479+0000 7ff5c5d8e700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fde90 0x7ff5c0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c558d700 1 -- 172.21.15.111:0/328741860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fde90 msgr2=0x7ff5c0113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c558d700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fde90 0x7ff5c0113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c558d700 1 -- 172.21.15.111:0/328741860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc005040 con 0x7ff5c00fe830 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c558d700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0002a50 tx=0x7ff5b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b0007cc0 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c010dd50 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5b000f040 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5b0009540 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.480+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c010e290 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.481+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c01019b0 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.481+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff5b00096a0 con 0x7ff5c00fe830 2024-03-14T13:19:47.483 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.482+0000 7ff5b67fc700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 0x7ff5ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.482+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff5b008b8a0 con 0x7ff5c00fe830 2024-03-14T13:19:47.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.482+0000 7ff5c5d8e700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 0x7ff5ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.482+0000 7ff5c5d8e700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 0x7ff5ac06ec70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b3b0 tx=0x7ff5bc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.486+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff5b0059530 con 0x7ff5c00fe830 2024-03-14T13:19:47.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.696+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5c01137c0 con 0x7ff5ac06c7c0 2024-03-14T13:19:47.697 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.697+0000 7ff5b67fc700 1 -- 172.21.15.111:0/328741860 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff5c01137c0 con 0x7ff5ac06c7c0 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 msgr2=0x7ff5ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 0x7ff5ac06ec70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00b3b0 tx=0x7ff5bc009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 msgr2=0x7ff5c0114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0002a50 tx=0x7ff5b00046b0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 shutdown_connections 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff5ac06c7c0 0x7ff5ac06ec70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5c00fde90 0x7ff5c0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 --2- 172.21.15.111:0/328741860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5c00fe830 0x7ff5c0114030 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.699+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 >> 172.21.15.111:0/328741860 conn(0x7ff5c0072bc0 msgr2=0x7ff5c00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:47.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.700+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 shutdown_connections 2024-03-14T13:19:47.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.700+0000 7ff5c7ff2700 1 -- 172.21.15.111:0/328741860 wait complete. 2024-03-14T13:19:47.801 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 -- 172.21.15.111:0/1599542117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 msgr2=0x7fdde0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/1599542117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde0102140 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fddd4004940 tx=0x7fddd4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 -- 172.21.15.111:0/1599542117 shutdown_connections 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/1599542117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/1599542117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde0102140 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 -- 172.21.15.111:0/1599542117 >> 172.21.15.111:0/1599542117 conn(0x7fdde0072bc0 msgr2=0x7fdde0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 -- 172.21.15.111:0/1599542117 shutdown_connections 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.799+0000 7fdde6ca2700 1 -- 172.21.15.111:0/1599542117 wait complete. 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 Processor -- start 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 -- start start 2024-03-14T13:19:47.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde010ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010f5d0 con 0x7fdde0101d70 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde6ca2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010f710 con 0x7fdde0102710 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde4a3e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde4a3e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48424/0 (socket says 172.21.15.111:48424) 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdde4a3e700 1 -- 172.21.15.111:0/2366174854 learned_addr learned my addr 172.21.15.111:0/2366174854 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.800+0000 7fdddffff700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde010ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde4a3e700 1 -- 172.21.15.111:0/2366174854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 msgr2=0x7fdde010ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde4a3e700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde010ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde4a3e700 1 -- 172.21.15.111:0/2366174854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4005040 con 0x7fdde0101d70 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdddffff700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde4a3e700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fddd400b3b0 tx=0x7fddd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddd400e040 con 0x7fdde0101d70 2024-03-14T13:19:47.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddd4013420 con 0x7fdde0101d70 2024-03-14T13:19:47.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde0117ab0 con 0x7fdde0101d70 2024-03-14T13:19:47.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fddd401c630 con 0x7fdde0101d70 2024-03-14T13:19:47.804 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.801+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde0117fc0 con 0x7fdde0101d70 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.802+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00ff080 con 0x7fdde0101d70 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.802+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fddd4013970 con 0x7fdde0101d70 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.803+0000 7fdddd7fa700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 0x7fddd006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.803+0000 7fdddffff700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 0x7fddd006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.803+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fddd408cca0 con 0x7fdde0101d70 2024-03-14T13:19:47.805 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.803+0000 7fdddffff700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 0x7fddd006ec70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009fd0 tx=0x7fddcc009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:47.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:47.807+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fddd4056f20 con 0x7fdde0101d70 2024-03-14T13:19:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:47 smithi111 ceph-mon[145314]: pgmap v470: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:47 smithi137 ceph-mon[108145]: pgmap v470: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:48.008 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.008+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdde004bed0 con 0x7fddd006c7c0 2024-03-14T13:19:48.018 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.017+0000 7fdddd7fa700 1 -- 172.21.15.111:0/2366174854 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fdde004bed0 con 0x7fddd006c7c0 2024-03-14T13:19:48.018 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (16m) 9m ago 27m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (13m) 9m ago 27m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (13m) 8m ago 26m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (15m) 9m ago 26m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (9m) 9m ago 21m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (9m) 9m ago 21m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (9m) 8m ago 21m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (9m) 8m ago 21m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (18m) 9m ago 29m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (17m) 8m ago 26m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (14m) 9m ago 29m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (14m) 8m ago 26m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:19:48.019 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (16m) 9m ago 27m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (16m) 8m ago 26m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (13m) 9m ago 25m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (12m) 9m ago 24m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (11m) 9m ago 24m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (11m) 8m ago 23m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (10m) 8m ago 23m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (10m) 8m ago 23m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:19:48.020 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (16m) 9m ago 26m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.019+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 msgr2=0x7fddd006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.019+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 0x7fddd006ec70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009fd0 tx=0x7fddcc009ee0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.019+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 msgr2=0x7fdde010ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.019+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fddd400b3b0 tx=0x7fddd4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 shutdown_connections 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fddd006c7c0 0x7fddd006ec70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde0101d70 0x7fdde010ea40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 --2- 172.21.15.111:0/2366174854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdde0102710 0x7fdde010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 >> 172.21.15.111:0/2366174854 conn(0x7fdde0072bc0 msgr2=0x7fdde00fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 shutdown_connections 2024-03-14T13:19:48.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.020+0000 7fdde6ca2700 1 -- 172.21.15.111:0/2366174854 wait complete. 2024-03-14T13:19:48.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 -- 172.21.15.111:0/666049395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008102710 msgr2=0x7fb0080fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 --2- 172.21.15.111:0/666049395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008102710 0x7fb0080fbd80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7faff4004940 tx=0x7faff4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 -- 172.21.15.111:0/666049395 shutdown_connections 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 --2- 172.21.15.111:0/666049395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008102710 0x7fb0080fbd80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 --2- 172.21.15.111:0/666049395 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008101d70 0x7fb008102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 -- 172.21.15.111:0/666049395 >> 172.21.15.111:0/666049395 conn(0x7fb008072bc0 msgr2=0x7fb008072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 -- 172.21.15.111:0/666049395 shutdown_connections 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 -- 172.21.15.111:0/666049395 wait complete. 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.119+0000 7fb00f10e700 1 Processor -- start 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00f10e700 1 -- start start 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00f10e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00f10e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008102710 0x7fb008112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00f10e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb008113550 con 0x7fb008101d70 2024-03-14T13:19:48.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00f10e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00810c940 con 0x7fb008102710 2024-03-14T13:19:48.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48452/0 (socket says 172.21.15.111:48452) 2024-03-14T13:19:48.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 -- 172.21.15.111:0/371836838 learned_addr learned my addr 172.21.15.111:0/371836838 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:48.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb007fff700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008102710 0x7fb008112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 -- 172.21.15.111:0/371836838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008102710 msgr2=0x7fb008112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008102710 0x7fb008112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 -- 172.21.15.111:0/371836838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff4005040 con 0x7fb008101d70 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.120+0000 7fb00ceaa700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faffc00fce0 tx=0x7faffc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc007ea0 con 0x7fb008101d70 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faffc00dd80 con 0x7fb008101d70 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00810cbc0 con 0x7fb008101d70 2024-03-14T13:19:48.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc012460 con 0x7fb008101d70 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00810d150 con 0x7fb008101d70 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.121+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0080fd4c0 con 0x7fb008101d70 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.122+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faffc009360 con 0x7fb008101d70 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.122+0000 7fb0057fa700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 0x7faff806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.122+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faffc08e170 con 0x7fb008101d70 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.122+0000 7fb007fff700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 0x7faff806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.123+0000 7fb007fff700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 0x7faff806ec70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faff4004890 tx=0x7faff4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.127+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faffc05be00 con 0x7fb008101d70 2024-03-14T13:19:48.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.402+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb00804bed0 con 0x7fb008101d70 2024-03-14T13:19:48.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.403+0000 7fb0057fa700 1 -- 172.21.15.111:0/371836838 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7faffc05b990 con 0x7fb008101d70 2024-03-14T13:19:48.404 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:19:48.404 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:19:48.404 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:19:48.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:48.404 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:19:48.405 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:19:48.406 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:19:48.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:19:48.406 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:19:48.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 msgr2=0x7faff806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 0x7faff806ec70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faff4004890 tx=0x7faff4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 msgr2=0x7fb0081129c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faffc00fce0 tx=0x7faffc0046b0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 shutdown_connections 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faff806c7c0 0x7faff806ec70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb008101d70 0x7fb0081129c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 --2- 172.21.15.111:0/371836838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb008102710 0x7fb008112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 >> 172.21.15.111:0/371836838 conn(0x7fb008072bc0 msgr2=0x7fb0080fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.408 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 shutdown_connections 2024-03-14T13:19:48.409 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.405+0000 7fb00f10e700 1 -- 172.21.15.111:0/371836838 wait complete. 2024-03-14T13:19:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 -- 172.21.15.111:0/3954833918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c1026c0 msgr2=0x7fd94c0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 --2- 172.21.15.111:0/3954833918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c1026c0 0x7fd94c0fbd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd940004940 tx=0x7fd940009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 -- 172.21.15.111:0/3954833918 shutdown_connections 2024-03-14T13:19:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 --2- 172.21.15.111:0/3954833918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c1026c0 0x7fd94c0fbd30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.506 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 --2- 172.21.15.111:0/3954833918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd94c101d20 0x7fd94c1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 -- 172.21.15.111:0/3954833918 >> 172.21.15.111:0/3954833918 conn(0x7fd94c0f7870 msgr2=0x7fd94c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 -- 172.21.15.111:0/3954833918 shutdown_connections 2024-03-14T13:19:48.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.505+0000 7fd951618700 1 -- 172.21.15.111:0/3954833918 wait complete. 2024-03-14T13:19:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 Processor -- start 2024-03-14T13:19:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 -- start start 2024-03-14T13:19:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd94c1026c0 0x7fd94c112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c113350 con 0x7fd94c101d20 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd951618700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c10c950 con 0x7fd94c1026c0 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.506+0000 7fd94affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48464/0 (socket says 172.21.15.111:48464) 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 -- 172.21.15.111:0/1405887710 learned_addr learned my addr 172.21.15.111:0/1405887710 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 -- 172.21.15.111:0/1405887710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd94c1026c0 msgr2=0x7fd94c112d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd94c1026c0 0x7fd94c112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 -- 172.21.15.111:0/1405887710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940005040 con 0x7fd94c101d20 2024-03-14T13:19:48.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd94affd700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd93c002a50 tx=0x7fd93c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93c007cc0 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c10cc30 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd93c00f040 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd93c009540 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.507+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c10d1f0 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.508+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94c0fd470 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.509+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd93c0096a0 con 0x7fd94c101d20 2024-03-14T13:19:48.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.509+0000 7fd9337fe700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 0x7fd93406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.509+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd93c08b8a0 con 0x7fd94c101d20 2024-03-14T13:19:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.509+0000 7fd94a7fc700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 0x7fd93406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.509+0000 7fd94a7fc700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 0x7fd93406ec70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd940000c00 tx=0x7fd940009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.513+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd93c059530 con 0x7fd94c101d20 2024-03-14T13:19:48.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.752+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd94c108cc0 con 0x7fd94c101d20 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.752+0000 7fd9337fe700 1 -- 172.21.15.111:0/1405887710 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fd93c0590c0 con 0x7fd94c101d20 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:19:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:19:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:48.755 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:48.756 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:48.756 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:19:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.754+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 msgr2=0x7fd93406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.754+0000 7fd951618700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 0x7fd93406ec70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd940000c00 tx=0x7fd940009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.754+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 msgr2=0x7fd94c1127c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.754+0000 7fd951618700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd93c002a50 tx=0x7fd93c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 shutdown_connections 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd93406c7c0 0x7fd93406ec70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd94c101d20 0x7fd94c1127c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 --2- 172.21.15.111:0/1405887710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd94c1026c0 0x7fd94c112d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 >> 172.21.15.111:0/1405887710 conn(0x7fd94c0f7870 msgr2=0x7fd94c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 shutdown_connections 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.755+0000 7fd951618700 1 -- 172.21.15.111:0/1405887710 wait complete. 2024-03-14T13:19:48.757 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/249506971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4101d30 msgr2=0x7fcdc4102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/249506971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4004940 tx=0x7fcdb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/249506971 shutdown_connections 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/249506971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc41026d0 0x7fcdc40fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/249506971 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102100 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/249506971 >> 172.21.15.111:0/249506971 conn(0x7fcdc40f7860 msgr2=0x7fcdc40f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/249506971 shutdown_connections 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/249506971 wait complete. 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 Processor -- start 2024-03-14T13:19:48.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.856+0000 7fcdcb4d1700 1 -- start start 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdcb4d1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdcb4d1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdcb4d1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4104b30 con 0x7fcdc41026d0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdcb4d1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4103610 con 0x7fcdc4101d30 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48478/0 (socket says 172.21.15.111:48478) 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 -- 172.21.15.111:0/3080211399 learned_addr learned my addr 172.21.15.111:0/3080211399 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc926d700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 -- 172.21.15.111:0/3080211399 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc4101d30 msgr2=0x7fcdc4102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 -- 172.21.15.111:0/3080211399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4005040 con 0x7fcdc41026d0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdc8a6c700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0002a50 tx=0x7fcdc0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdc0007cc0 con 0x7fcdc41026d0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdc000f040 con 0x7fcdc41026d0 2024-03-14T13:19:48.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc41038c0 con 0x7fcdc41026d0 2024-03-14T13:19:48.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.857+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdc000d920 con 0x7fcdc41026d0 2024-03-14T13:19:48.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.858+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc411a070 con 0x7fcdc41026d0 2024-03-14T13:19:48.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.858+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc40fd480 con 0x7fcdc41026d0 2024-03-14T13:19:48.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.859+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcdc0009680 con 0x7fcdc41026d0 2024-03-14T13:19:48.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.859+0000 7fcdb9ffb700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 0x7fcdb006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:48.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.859+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdc008b740 con 0x7fcdc41026d0 2024-03-14T13:19:48.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.859+0000 7fcdc926d700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 0x7fcdb006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:48.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.859+0000 7fcdc926d700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 0x7fcdb006ec70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4004f90 tx=0x7fcdb4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:48.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:48.864+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcdc00559c0 con 0x7fcdc41026d0 2024-03-14T13:19:48.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:48 smithi111 ceph-mon[145314]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:48.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:48 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/371836838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:19:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:48 smithi137 ceph-mon[108145]: from='client.34904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:49.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:48 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/371836838' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:19:49.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.074+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc4066a20 con 0x7fcdb006c7c0 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.076+0000 7fcdb9ffb700 1 -- 172.21.15.111:0/3080211399 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcdc4066a20 con 0x7fcdb006c7c0 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:19:49.076 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:19:49.077 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 msgr2=0x7fcdb006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 0x7fcdb006ec70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4004f90 tx=0x7fcdb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 msgr2=0x7fcdc41030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0002a50 tx=0x7fcdc0004600 comp rx=0 tx=0).stop 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 shutdown_connections 2024-03-14T13:19:49.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcdb006c7c0 0x7fcdb006ec70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdc4101d30 0x7fcdc4102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 --2- 172.21.15.111:0/3080211399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdc41026d0 0x7fcdc41030d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 >> 172.21.15.111:0/3080211399 conn(0x7fcdc40f7860 msgr2=0x7fcdc4066310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 shutdown_connections 2024-03-14T13:19:49.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.078+0000 7fcdcb4d1700 1 -- 172.21.15.111:0/3080211399 wait complete. 2024-03-14T13:19:49.180 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.177+0000 7fc5a9880700 1 -- 172.21.15.111:0/1769155742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 msgr2=0x7fc5a40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 --2- 172.21.15.111:0/1769155742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a40fbd80 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc598004940 tx=0x7fc598009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 -- 172.21.15.111:0/1769155742 shutdown_connections 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 --2- 172.21.15.111:0/1769155742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a40fbd80 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 --2- 172.21.15.111:0/1769155742 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 -- 172.21.15.111:0/1769155742 >> 172.21.15.111:0/1769155742 conn(0x7fc5a4072bc0 msgr2=0x7fc5a4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 -- 172.21.15.111:0/1769155742 shutdown_connections 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 -- 172.21.15.111:0/1769155742 wait complete. 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 Processor -- start 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.178+0000 7fc5a9880700 1 -- start start 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a41036a0 con 0x7fc5a4102710 2024-03-14T13:19:49.181 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a41037e0 con 0x7fc5a4101d70 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:48498/0 (socket says 172.21.15.111:48498) 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 -- 172.21.15.111:0/4109148003 learned_addr learned my addr 172.21.15.111:0/4109148003 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 -- 172.21.15.111:0/4109148003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 msgr2=0x7fc5a4102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a2ffd700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 -- 172.21.15.111:0/4109148003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598005040 con 0x7fc5a4102710 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a2ffd700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a27fc700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc598009380 tx=0x7fc598009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59800e040 con 0x7fc5a4102710 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc598013420 con 0x7fc5a4102710 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a4103a30 con 0x7fc5a4102710 2024-03-14T13:19:49.182 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc59801c630 con 0x7fc5a4102710 2024-03-14T13:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.179+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a411a1f0 con 0x7fc5a4102710 2024-03-14T13:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.180+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a40fd4c0 con 0x7fc5a4102710 2024-03-14T13:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.181+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc5980139e0 con 0x7fc5a4102710 2024-03-14T13:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.181+0000 7fc58b7fe700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 0x7fc58c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:19:49.183 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.181+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc59808cc90 con 0x7fc5a4102710 2024-03-14T13:19:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.181+0000 7fc5a2ffd700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 0x7fc58c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:19:49.184 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.182+0000 7fc5a2ffd700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 0x7fc58c06ec70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc594009fd0 tx=0x7fc59400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:19:49.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.186+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc598056ed0 con 0x7fc5a4102710 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.459+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc5a4108d40 con 0x7fc5a4102710 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.459+0000 7fc58b7fe700 1 -- 172.21.15.111:0/4109148003 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc59805a4f0 con 0x7fc5a4102710 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:19:49.460 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:19:49.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.461+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 msgr2=0x7fc58c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.461+0000 7fc5a9880700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 0x7fc58c06ec70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc594009fd0 tx=0x7fc59400b040 comp rx=0 tx=0).stop 2024-03-14T13:19:49.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.461+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 msgr2=0x7fc5a4103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.461+0000 7fc5a9880700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc598009380 tx=0x7fc598009e30 comp rx=0 tx=0).stop 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 shutdown_connections 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc58c06c7c0 0x7fc58c06ec70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5a4101d70 0x7fc5a4102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 --2- 172.21.15.111:0/4109148003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5a4102710 0x7fc5a4103110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 >> 172.21.15.111:0/4109148003 conn(0x7fc5a4072bc0 msgr2=0x7fc5a4065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 shutdown_connections 2024-03-14T13:19:49.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:19:49.462+0000 7fc5a9880700 1 -- 172.21.15.111:0/4109148003 wait complete. 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: from='client.34912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: pgmap v471: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1405887710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:49 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4109148003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:19:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: from='client.34912 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:50.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: pgmap v471: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:50.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1405887710' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:19:50.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: from='client.34924 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:19:50.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:49 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4109148003' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:19:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:50 smithi111 ceph-mon[145314]: pgmap v472: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:51.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:50 smithi137 ceph-mon[108145]: pgmap v472: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:53.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:53 smithi137 ceph-mon[108145]: pgmap v473: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:53 smithi111 ceph-mon[145314]: pgmap v473: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:54.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:19:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:55 smithi111 ceph-mon[145314]: pgmap v474: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:55 smithi137 ceph-mon[108145]: pgmap v474: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:57 smithi111 ceph-mon[145314]: pgmap v475: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:57 smithi137 ceph-mon[108145]: pgmap v475: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:19:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:19:59 smithi111 ceph-mon[145314]: pgmap v476: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:19:59 smithi137 ceph-mon[108145]: pgmap v476: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:00.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:00 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:19:59.999+0000 7eff54e8f700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:20:00.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:00 smithi111 ceph-mon[145314]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:20:01.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:00 smithi137 ceph-mon[108145]: overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:20:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:01 smithi111 ceph-mon[145314]: pgmap v477: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:01 smithi137 ceph-mon[108145]: pgmap v477: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:03 smithi111 ceph-mon[145314]: pgmap v478: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:03 smithi137 ceph-mon[108145]: pgmap v478: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:05 smithi111 ceph-mon[145314]: pgmap v479: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:05 smithi137 ceph-mon[108145]: pgmap v479: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:07 smithi111 ceph-mon[145314]: pgmap v480: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:07 smithi137 ceph-mon[108145]: pgmap v480: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:09 smithi111 ceph-mon[145314]: pgmap v481: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:09 smithi137 ceph-mon[108145]: pgmap v481: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:11 smithi111 ceph-mon[145314]: pgmap v482: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:11 smithi137 ceph-mon[108145]: pgmap v482: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:13 smithi111 ceph-mon[145314]: pgmap v483: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:13 smithi137 ceph-mon[108145]: pgmap v483: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:15 smithi111 ceph-mon[145314]: pgmap v484: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:15 smithi137 ceph-mon[108145]: pgmap v484: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:17.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:17 smithi111 ceph-mon[145314]: pgmap v485: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:18.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:17 smithi137 ceph-mon[108145]: pgmap v485: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:19.581 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 -- 172.21.15.111:0/3373430355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101d70 msgr2=0x7fc0b4102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/3373430355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101d70 0x7fc0b4102140 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009ce0 tx=0x7fc0a4004d10 comp rx=0 tx=0).stop 2024-03-14T13:20:19.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 -- 172.21.15.111:0/3373430355 shutdown_connections 2024-03-14T13:20:19.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/3373430355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4102710 0x7fc0b40fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/3373430355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4101d70 0x7fc0b4102140 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 -- 172.21.15.111:0/3373430355 >> 172.21.15.111:0/3373430355 conn(0x7fc0b4072bc0 msgr2=0x7fc0b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 -- 172.21.15.111:0/3373430355 shutdown_connections 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.577+0000 7fc0bb88a700 1 -- 172.21.15.111:0/3373430355 wait complete. 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 Processor -- start 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 -- start start 2024-03-14T13:20:19.583 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4101d70 0x7fc0b410eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b410f730 con 0x7fc0b4102710 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0bb88a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b410f870 con 0x7fc0b4101d70 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0b8e25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0b8e25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60112/0 (socket says 172.21.15.111:60112) 2024-03-14T13:20:19.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0b8e25700 1 -- 172.21.15.111:0/2630702810 learned_addr learned my addr 172.21.15.111:0/2630702810 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.578+0000 7fc0b9626700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4101d70 0x7fc0b410eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b8e25700 1 -- 172.21.15.111:0/2630702810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4101d70 msgr2=0x7fc0b410eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b8e25700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4101d70 0x7fc0b410eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b8e25700 1 -- 172.21.15.111:0/2630702810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4009940 con 0x7fc0b4102710 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b8e25700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c800 tx=0x7fc0a8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:19.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a800a760 con 0x7fc0b4102710 2024-03-14T13:20:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0a800ad70 con 0x7fc0b4102710 2024-03-14T13:20:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b4117c10 con 0x7fc0b4102710 2024-03-14T13:20:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0a80113a0 con 0x7fc0b4102710 2024-03-14T13:20:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.579+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b41180f0 con 0x7fc0b4102710 2024-03-14T13:20:19.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.580+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b40ff080 con 0x7fc0b4102710 2024-03-14T13:20:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.581+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc0a8012400 con 0x7fc0b4102710 2024-03-14T13:20:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.581+0000 7fc0b1ffb700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 0x7fc09c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.581+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc0a808b9a0 con 0x7fc0b4102710 2024-03-14T13:20:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.581+0000 7fc0b9626700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 0x7fc09c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.582+0000 7fc0b9626700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 0x7fc09c06ec70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400b740 tx=0x7fc0a4004d10 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:19.588 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.586+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc0a8055ce0 con 0x7fc0b4102710 2024-03-14T13:20:19.795 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.795+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0b4108d30 con 0x7fc09c06c7c0 2024-03-14T13:20:19.796 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.796+0000 7fc0b1ffb700 1 -- 172.21.15.111:0/2630702810 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fc0b4108d30 con 0x7fc09c06c7c0 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.798+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 msgr2=0x7fc09c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.798+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 0x7fc09c06ec70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400b740 tx=0x7fc0a4004d10 comp rx=0 tx=0).stop 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.798+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 msgr2=0x7fc0b410f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.798+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800c800 tx=0x7fc0a8004470 comp rx=0 tx=0).stop 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 shutdown_connections 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc09c06c7c0 0x7fc09c06ec70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.798 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0b4101d70 0x7fc0b410eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 --2- 172.21.15.111:0/2630702810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc0b4102710 0x7fc0b410f0e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 >> 172.21.15.111:0/2630702810 conn(0x7fc0b4072bc0 msgr2=0x7fc0b40fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 shutdown_connections 2024-03-14T13:20:19.799 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.799+0000 7fc0bb88a700 1 -- 172.21.15.111:0/2630702810 wait complete. 2024-03-14T13:20:19.808 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:20:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:19 smithi111 ceph-mon[145314]: pgmap v486: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.899+0000 7f280ee38700 1 -- 172.21.15.111:0/2118912240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fc4f0 msgr2=0x7f2808109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.899+0000 7f280ee38700 1 --2- 172.21.15.111:0/2118912240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fc4f0 0x7f2808109250 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f27f4004940 tx=0x7f27f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 -- 172.21.15.111:0/2118912240 shutdown_connections 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 --2- 172.21.15.111:0/2118912240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fc4f0 0x7f2808109250 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 --2- 172.21.15.111:0/2118912240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fbbe0 0x7f28080fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 -- 172.21.15.111:0/2118912240 >> 172.21.15.111:0/2118912240 conn(0x7f28080f7870 msgr2=0x7f28080f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:19.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 -- 172.21.15.111:0/2118912240 shutdown_connections 2024-03-14T13:20:19.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 -- 172.21.15.111:0/2118912240 wait complete. 2024-03-14T13:20:19.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 Processor -- start 2024-03-14T13:20:19.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.900+0000 7f280ee38700 1 -- start start 2024-03-14T13:20:19.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280ee38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280ee38700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fc4f0 0x7f2808115f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280ee38700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808116590 con 0x7f28080fbbe0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280ee38700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280810fb90 con 0x7f28080fc4f0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60120/0 (socket says 172.21.15.111:60120) 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 -- 172.21.15.111:0/2916490700 learned_addr learned my addr 172.21.15.111:0/2916490700 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f2807fff700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fc4f0 0x7f2808115f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 -- 172.21.15.111:0/2916490700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fc4f0 msgr2=0x7f2808115f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fc4f0 0x7f2808115f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 -- 172.21.15.111:0/2916490700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4005040 con 0x7f28080fbbe0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.901+0000 7f280cbd4700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f27fc002a50 tx=0x7f27fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.902+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27fc007cc0 con 0x7f28080fbbe0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.902+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280810fe70 con 0x7f28080fbbe0 2024-03-14T13:20:19.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.902+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27fc00f040 con 0x7f28080fbbe0 2024-03-14T13:20:19.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.902+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27fc00d9c0 con 0x7f28080fbbe0 2024-03-14T13:20:19.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.902+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808110400 con 0x7f28080fbbe0 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.903+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2808066b10 con 0x7f28080fbbe0 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.903+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f27fc00db20 con 0x7f28080fbbe0 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.903+0000 7f28057fa700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 0x7f27f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.903+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f27fc08b7a0 con 0x7f28080fbbe0 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.904+0000 7f2807fff700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 0x7f27f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:19.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.904+0000 7f2807fff700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 0x7f27f806ec70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2808110e70 tx=0x7f27f4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:19.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:19.908+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f27fc059430 con 0x7f28080fbbe0 2024-03-14T13:20:20.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:19 smithi137 ceph-mon[108145]: pgmap v486: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:20.118 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.118+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2808111050 con 0x7f27f806c7c0 2024-03-14T13:20:20.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.120+0000 7f28057fa700 1 -- 172.21.15.111:0/2916490700 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2808111050 con 0x7f27f806c7c0 2024-03-14T13:20:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 msgr2=0x7f27f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 0x7f27f806ec70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2808110e70 tx=0x7f27f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 msgr2=0x7f2808115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f27fc002a50 tx=0x7f27fc0046b0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 shutdown_connections 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f27f806c7c0 0x7f27f806ec70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28080fbbe0 0x7f2808115a00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 --2- 172.21.15.111:0/2916490700 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28080fc4f0 0x7f2808115f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 >> 172.21.15.111:0/2916490700 conn(0x7f28080f7870 msgr2=0x7f28080f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 shutdown_connections 2024-03-14T13:20:20.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.122+0000 7f280ee38700 1 -- 172.21.15.111:0/2916490700 wait complete. 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 -- 172.21.15.111:0/1540727786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fc590 msgr2=0x7f548c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 --2- 172.21.15.111:0/1540727786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fc590 0x7f548c1092f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f547c004940 tx=0x7f547c009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 -- 172.21.15.111:0/1540727786 shutdown_connections 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 --2- 172.21.15.111:0/1540727786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fc590 0x7f548c1092f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 --2- 172.21.15.111:0/1540727786 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fbc80 0x7f548c0fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.221+0000 7f5493d56700 1 -- 172.21.15.111:0/1540727786 >> 172.21.15.111:0/1540727786 conn(0x7f548c072bc0 msgr2=0x7f548c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 -- 172.21.15.111:0/1540727786 shutdown_connections 2024-03-14T13:20:20.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 -- 172.21.15.111:0/1540727786 wait complete. 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 Processor -- start 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 -- start start 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fc590 0x7f548c1161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c116800 con 0x7f548c0fbc80 2024-03-14T13:20:20.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5493d56700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10fbd0 con 0x7f548c0fc590 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5491af2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5491af2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:60138/0 (socket says 172.21.15.111:60138) 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.222+0000 7f5491af2700 1 -- 172.21.15.111:0/2957822846 learned_addr learned my addr 172.21.15.111:0/2957822846 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f54912f1700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fc590 0x7f548c1161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5491af2700 1 -- 172.21.15.111:0/2957822846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fc590 msgr2=0x7f548c1161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5491af2700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fc590 0x7f548c1161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5491af2700 1 -- 172.21.15.111:0/2957822846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c005040 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5491af2700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5488002a50 tx=0x7f54880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5488007cc0 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f548800f040 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c10feb0 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5488009540 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.223+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c110470 con 0x7f548c0fbc80 2024-03-14T13:20:20.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.224+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c065e30 con 0x7f548c0fbc80 2024-03-14T13:20:20.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.224+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f54880096a0 con 0x7f548c0fbc80 2024-03-14T13:20:20.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.225+0000 7f54827fc700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 0x7f547806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.225+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f548808b7a0 con 0x7f548c0fbc80 2024-03-14T13:20:20.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.225+0000 7f54912f1700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 0x7f547806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.225+0000 7f54912f1700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 0x7f547806ec70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f547c004f90 tx=0x7f547c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.229+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f54880594b0 con 0x7f548c0fbc80 2024-03-14T13:20:20.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.431+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f548c04bf40 con 0x7f547806c7c0 2024-03-14T13:20:20.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.441+0000 7f54827fc700 1 -- 172.21.15.111:0/2957822846 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f548c04bf40 con 0x7f547806c7c0 2024-03-14T13:20:20.440 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:20:20.440 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (16m) 10m ago 28m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:20:20.440 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (14m) 10m ago 28m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (13m) 9m ago 26m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (16m) 10m ago 27m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (10m) 10m ago 21m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (10m) 10m ago 22m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (9m) 9m ago 21m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (9m) 9m ago 21m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (19m) 10m ago 29m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (17m) 9m ago 26m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (15m) 10m ago 29m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (14m) 9m ago 26m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (17m) 10m ago 28m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (17m) 9m ago 26m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (13m) 10m ago 25m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (13m) 10m ago 25m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:20:20.441 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (12m) 10m ago 24m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:20:20.442 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (11m) 9m ago 24m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:20:20.442 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (11m) 9m ago 24m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:20:20.442 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (10m) 9m ago 23m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:20:20.442 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (16m) 10m ago 27m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:20:20.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 msgr2=0x7f547806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 0x7f547806ec70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f547c004f90 tx=0x7f547c009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 msgr2=0x7f548c115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5488002a50 tx=0x7f54880046b0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 shutdown_connections 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f547806c7c0 0x7f547806ec70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0fbc80 0x7f548c115c70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 --2- 172.21.15.111:0/2957822846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f548c0fc590 0x7f548c1161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 >> 172.21.15.111:0/2957822846 conn(0x7f548c072bc0 msgr2=0x7f548c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 shutdown_connections 2024-03-14T13:20:20.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.443+0000 7f5493d56700 1 -- 172.21.15.111:0/2957822846 wait complete. 2024-03-14T13:20:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 -- 172.21.15.111:0/604891863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444102710 msgr2=0x7fc4440fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.543 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 --2- 172.21.15.111:0/604891863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444102710 0x7fc4440fbd80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc434004940 tx=0x7fc434009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 -- 172.21.15.111:0/604891863 shutdown_connections 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 --2- 172.21.15.111:0/604891863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444102710 0x7fc4440fbd80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 --2- 172.21.15.111:0/604891863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444101d70 0x7fc444102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 -- 172.21.15.111:0/604891863 >> 172.21.15.111:0/604891863 conn(0x7fc444072bc0 msgr2=0x7fc444072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 -- 172.21.15.111:0/604891863 shutdown_connections 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.542+0000 7fc448fa6700 1 -- 172.21.15.111:0/604891863 wait complete. 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 Processor -- start 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 -- start start 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444102710 0x7fc444112e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc444113490 con 0x7fc444101d70 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc448fa6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44410c880 con 0x7fc444102710 2024-03-14T13:20:20.544 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc44259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc44259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38752/0 (socket says 172.21.15.111:38752) 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc44259c700 1 -- 172.21.15.111:0/2651757915 learned_addr learned my addr 172.21.15.111:0/2651757915 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.543+0000 7fc441d9b700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444102710 0x7fc444112e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc44259c700 1 -- 172.21.15.111:0/2651757915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444102710 msgr2=0x7fc444112e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc44259c700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444102710 0x7fc444112e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc44259c700 1 -- 172.21.15.111:0/2651757915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434005040 con 0x7fc444101d70 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc44259c700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc438002a80 tx=0x7fc4380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc438007cc0 con 0x7fc444101d70 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc43800f040 con 0x7fc444101d70 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44410cb60 con 0x7fc444101d70 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43800da60 con 0x7fc444101d70 2024-03-14T13:20:20.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.544+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44410d0f0 con 0x7fc444101d70 2024-03-14T13:20:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.545+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4440fd4c0 con 0x7fc444101d70 2024-03-14T13:20:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.545+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc43800dbc0 con 0x7fc444101d70 2024-03-14T13:20:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.546+0000 7fc432ffd700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 0x7fc42c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.546 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.546+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc43808b630 con 0x7fc444101d70 2024-03-14T13:20:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.546+0000 7fc441d9b700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 0x7fc42c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.547 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.546+0000 7fc441d9b700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 0x7fc42c06ec70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc44410db60 tx=0x7fc434009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.549 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.550+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc438059340 con 0x7fc444101d70 2024-03-14T13:20:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.826+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc44404bed0 con 0x7fc444101d70 2024-03-14T13:20:20.826 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.827+0000 7fc432ffd700 1 -- 172.21.15.111:0/2651757915 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fc438058ed0 con 0x7fc444101d70 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:20.827 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:20:20.828 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:20:20.828 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:20:20.828 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:20:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 msgr2=0x7fc42c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 0x7fc42c06ec70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc44410db60 tx=0x7fc434009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 msgr2=0x7fc444112900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc438002a80 tx=0x7fc4380046b0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 shutdown_connections 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc42c06c7c0 0x7fc42c06ec70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc444101d70 0x7fc444112900 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 --2- 172.21.15.111:0/2651757915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc444102710 0x7fc444112e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 >> 172.21.15.111:0/2651757915 conn(0x7fc444072bc0 msgr2=0x7fc4440faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 shutdown_connections 2024-03-14T13:20:20.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.829+0000 7fc448fa6700 1 -- 172.21.15.111:0/2651757915 wait complete. 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 -- 172.21.15.111:0/1570176268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038101d70 msgr2=0x7f1038102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 --2- 172.21.15.111:0/1570176268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038101d70 0x7f1038102140 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f1028004940 tx=0x7f1028009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 -- 172.21.15.111:0/1570176268 shutdown_connections 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 --2- 172.21.15.111:0/1570176268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038102710 0x7f10380fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 --2- 172.21.15.111:0/1570176268 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038101d70 0x7f1038102140 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 -- 172.21.15.111:0/1570176268 >> 172.21.15.111:0/1570176268 conn(0x7f1038072bc0 msgr2=0x7f1038072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:20.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 -- 172.21.15.111:0/1570176268 shutdown_connections 2024-03-14T13:20:20.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.929+0000 7f103e1ed700 1 -- 172.21.15.111:0/1570176268 wait complete. 2024-03-14T13:20:20.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 Processor -- start 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 -- start start 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038101d70 0x7f10381129a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1038113530 con 0x7f1038102710 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f103e1ed700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103810c920 con 0x7f1038101d70 2024-03-14T13:20:20.931 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.930+0000 7f1036ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38770/0 (socket says 172.21.15.111:38770) 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 -- 172.21.15.111:0/2329347478 learned_addr learned my addr 172.21.15.111:0/2329347478 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f10377fe700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038101d70 0x7f10381129a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 -- 172.21.15.111:0/2329347478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038101d70 msgr2=0x7f10381129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038101d70 0x7f10381129a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 -- 172.21.15.111:0/2329347478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028005040 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f1036ffd700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f102c00fce0 tx=0x7f102c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102c007ea0 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f102c00dcd0 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103810cba0 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102c012460 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.931+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103810d160 con 0x7f1038102710 2024-03-14T13:20:20.932 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.932+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10380fd4c0 con 0x7f1038102710 2024-03-14T13:20:20.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.933+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f102c00d820 con 0x7f1038102710 2024-03-14T13:20:20.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.933+0000 7f101ffff700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 0x7f102006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:20.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.933+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f102c08e1c0 con 0x7f1038102710 2024-03-14T13:20:20.933 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.933+0000 7f10377fe700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 0x7f102006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:20.934 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.933+0000 7f10377fe700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 0x7f102006ec70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1028004f90 tx=0x7f1028009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:20.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:20.938+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f102c05be50 con 0x7f1038102710 2024-03-14T13:20:21.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.170+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f103804bed0 con 0x7f1038102710 2024-03-14T13:20:21.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.170+0000 7f101ffff700 1 -- 172.21.15.111:0/2329347478 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f102c05b9e0 con 0x7f1038102710 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:20:21.170 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:20:21.171 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:21.172 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 msgr2=0x7f102006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 0x7f102006ec70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1028004f90 tx=0x7f1028009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 msgr2=0x7f1038112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f102c00fce0 tx=0x7f102c004600 comp rx=0 tx=0).stop 2024-03-14T13:20:21.173 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 shutdown_connections 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102006c7c0 0x7f102006ec70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038101d70 0x7f10381129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 --2- 172.21.15.111:0/2329347478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038102710 0x7f1038112ee0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 >> 172.21.15.111:0/2329347478 conn(0x7f1038072bc0 msgr2=0x7f10380fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 shutdown_connections 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.172+0000 7f103e1ed700 1 -- 172.21.15.111:0/2329347478 wait complete. 2024-03-14T13:20:21.174 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- 172.21.15.111:0/3052313887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 msgr2=0x7f78880fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 --2- 172.21.15.111:0/3052313887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f78880fce20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7878004940 tx=0x7f7878009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- 172.21.15.111:0/3052313887 shutdown_connections 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 --2- 172.21.15.111:0/3052313887 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f78880fce20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 --2- 172.21.15.111:0/3052313887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 0x7f78881031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- 172.21.15.111:0/3052313887 >> 172.21.15.111:0/3052313887 conn(0x7f78880f8980 msgr2=0x7f78880fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- 172.21.15.111:0/3052313887 shutdown_connections 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- 172.21.15.111:0/3052313887 wait complete. 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 Processor -- start 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.273+0000 7f788cbe9700 1 -- start start 2024-03-14T13:20:21.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 0x7f7888113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888114680 con 0x7f78881037b0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788810da70 con 0x7f7888102e10 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38790/0 (socket says 172.21.15.111:38790) 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 -- 172.21.15.111:0/1011370365 learned_addr learned my addr 172.21.15.111:0/1011370365 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788659c700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 0x7f7888113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 -- 172.21.15.111:0/1011370365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 msgr2=0x7f7888113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 0x7f7888113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 -- 172.21.15.111:0/1011370365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878005040 con 0x7f78881037b0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7885d9b700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7878009380 tx=0x7f7878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787800e040 con 0x7f78881037b0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788810dcf0 con 0x7f78881037b0 2024-03-14T13:20:21.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7878013420 con 0x7f78881037b0 2024-03-14T13:20:21.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787801c630 con 0x7f78881037b0 2024-03-14T13:20:21.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.274+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788810e2b0 con 0x7f78881037b0 2024-03-14T13:20:21.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.275+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78880fe560 con 0x7f78881037b0 2024-03-14T13:20:21.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.276+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f78780139b0 con 0x7f78881037b0 2024-03-14T13:20:21.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.276+0000 7f7876ffd700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 0x7f787006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.276+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f787808cbe0 con 0x7f78881037b0 2024-03-14T13:20:21.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.276+0000 7f788659c700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 0x7f787006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.276+0000 7f788659c700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 0x7f787006ec70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f787c0041d0 tx=0x7f787c0043c0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:21.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.281+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f787805a8f0 con 0x7f78881037b0 2024-03-14T13:20:21.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.490+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f78880fe560 con 0x7f787006c7c0 2024-03-14T13:20:21.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.491+0000 7f7876ffd700 1 -- 172.21.15.111:0/1011370365 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f78880fe560 con 0x7f787006c7c0 2024-03-14T13:20:21.490 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:20:21.491 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.493+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 msgr2=0x7f787006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.493+0000 7f788cbe9700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 0x7f787006ec70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f787c0041d0 tx=0x7f787c0043c0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.493+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 msgr2=0x7f7888114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.493+0000 7f788cbe9700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7878009380 tx=0x7f7878009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 shutdown_connections 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f787006c7c0 0x7f787006ec70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7888102e10 0x7f7888113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 --2- 172.21.15.111:0/1011370365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78881037b0 0x7f7888114030 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 >> 172.21.15.111:0/1011370365 conn(0x7f78880f8980 msgr2=0x7f78880f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:21.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 shutdown_connections 2024-03-14T13:20:21.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.494+0000 7f788cbe9700 1 -- 172.21.15.111:0/1011370365 wait complete. 2024-03-14T13:20:21.595 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- 172.21.15.111:0/370094844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 msgr2=0x7faeac0fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 --2- 172.21.15.111:0/370094844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac0fce10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7faea8004940 tx=0x7faea8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- 172.21.15.111:0/370094844 shutdown_connections 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 --2- 172.21.15.111:0/370094844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac0fce10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 --2- 172.21.15.111:0/370094844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- 172.21.15.111:0/370094844 >> 172.21.15.111:0/370094844 conn(0x7faeac0f89b0 msgr2=0x7faeac0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- 172.21.15.111:0/370094844 shutdown_connections 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- 172.21.15.111:0/370094844 wait complete. 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 Processor -- start 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 -- start start 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.595+0000 7faeb4047700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb4047700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb4047700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac114450 con 0x7faeac1037a0 2024-03-14T13:20:21.596 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb4047700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac10da50 con 0x7faeac102e00 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38810/0 (socket says 172.21.15.111:38810) 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 -- 172.21.15.111:0/2335893637 learned_addr learned my addr 172.21.15.111:0/2335893637 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb1de3700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 -- 172.21.15.111:0/2335893637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 msgr2=0x7faeac1138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 -- 172.21.15.111:0/2335893637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea8005040 con 0x7faeac1037a0 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb1de3700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:20:21.597 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.596+0000 7faeb15e2700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faea80093b0 tx=0x7faea8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:21.598 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.597+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea800e040 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.597+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faea8013420 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.597+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac10dcd0 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.597+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faea801c630 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.597+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac10e290 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.598+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac0fe550 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.599+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faea80139e0 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.599+0000 7faea27fc700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 0x7fae9806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.599+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faea808ccc0 con 0x7faeac1037a0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.599+0000 7faeb1de3700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 0x7fae9806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:21.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.599+0000 7faeb1de3700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 0x7fae9806ec70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009fd0 tx=0x7fae9c009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:21.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.603+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faea805aa80 con 0x7faeac1037a0 2024-03-14T13:20:21.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.877+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faeac109da0 con 0x7faeac1037a0 2024-03-14T13:20:21.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.877+0000 7faea27fc700 1 -- 172.21.15.111:0/2335893637 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7faea805a610 con 0x7faeac1037a0 2024-03-14T13:20:21.877 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:20:21.877 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:20:21.877 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:20:21.877 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:20:21.877 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:20:21.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 msgr2=0x7fae9806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 0x7fae9806ec70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009fd0 tx=0x7fae9c009ee0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 msgr2=0x7faeac113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7faea80093b0 tx=0x7faea8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 shutdown_connections 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fae9806c7c0 0x7fae9806ec70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faeac102e00 0x7faeac1138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 --2- 172.21.15.111:0/2335893637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeac1037a0 0x7faeac113e00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 >> 172.21.15.111:0/2335893637 conn(0x7faeac0f89b0 msgr2=0x7faeac0f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 shutdown_connections 2024-03-14T13:20:21.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:21.879+0000 7faeb4047700 1 -- 172.21.15.111:0/2335893637 wait complete. 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: pgmap v487: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: from='client.34940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2651757915' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:20:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:21 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2329347478' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:20:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: from='client.34932 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: from='client.34936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: pgmap v487: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: from='client.34940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2651757915' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:20:22.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:21 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2329347478' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:20:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:22 smithi111 ceph-mon[145314]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:22.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:22 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2335893637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:20:23.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:22 smithi137 ceph-mon[108145]: from='client.34952 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:23.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:22 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2335893637' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:20:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:23 smithi111 ceph-mon[145314]: pgmap v488: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:23 smithi137 ceph-mon[108145]: pgmap v488: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:25 smithi111 ceph-mon[145314]: pgmap v489: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:25 smithi137 ceph-mon[108145]: pgmap v489: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:27 smithi111 ceph-mon[145314]: pgmap v490: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:27 smithi137 ceph-mon[108145]: pgmap v490: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:29 smithi111 ceph-mon[145314]: pgmap v491: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:29 smithi137 ceph-mon[108145]: pgmap v491: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:30.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:30 smithi111 ceph-mon[145314]: pgmap v492: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:31.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:30 smithi137 ceph-mon[108145]: pgmap v492: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:33.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:33 smithi137 ceph-mon[108145]: pgmap v493: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:33 smithi111 ceph-mon[145314]: pgmap v493: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:35 smithi111 ceph-mon[145314]: pgmap v494: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:35 smithi137 ceph-mon[108145]: pgmap v494: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:37 smithi111 ceph-mon[145314]: pgmap v495: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:37 smithi137 ceph-mon[108145]: pgmap v495: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:39 smithi111 ceph-mon[145314]: pgmap v496: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:39 smithi137 ceph-mon[108145]: pgmap v496: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:40.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:41.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:41 smithi111 ceph-mon[145314]: pgmap v497: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:41 smithi137 ceph-mon[108145]: pgmap v497: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:43.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:43 smithi111 ceph-mon[145314]: pgmap v498: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:43 smithi137 ceph-mon[108145]: pgmap v498: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:45.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:45 smithi111 ceph-mon[145314]: pgmap v499: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:45 smithi137 ceph-mon[108145]: pgmap v499: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:47.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:47 smithi111 ceph-mon[145314]: pgmap v500: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:47 smithi137 ceph-mon[108145]: pgmap v500: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:49.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:49 smithi111 ceph-mon[145314]: pgmap v501: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:49 smithi137 ceph-mon[108145]: pgmap v501: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:51.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:51 smithi111 ceph-mon[145314]: pgmap v502: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 -- 172.21.15.111:0/3014238705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038071f90 msgr2=0x7f1038070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 --2- 172.21.15.111:0/3014238705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038071f90 0x7f1038070360 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1028004970 tx=0x7f1028009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 -- 172.21.15.111:0/3014238705 shutdown_connections 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 --2- 172.21.15.111:0/3014238705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10380708a0 0x7f1038070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 --2- 172.21.15.111:0/3014238705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1038071f90 0x7f1038070360 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 -- 172.21.15.111:0/3014238705 >> 172.21.15.111:0/3014238705 conn(0x7f10380f89b0 msgr2=0x7f10380fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 -- 172.21.15.111:0/3014238705 shutdown_connections 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 -- 172.21.15.111:0/3014238705 wait complete. 2024-03-14T13:20:51.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.993+0000 7f103f834700 1 Processor -- start 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103f834700 1 -- start start 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103f834700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103f834700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038071f90 0x7f103811a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103f834700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103811aca0 con 0x7f10380708a0 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103f834700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1038114090 con 0x7f1038071f90 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56572/0 (socket says 172.21.15.111:56572) 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 -- 172.21.15.111:0/3907179296 learned_addr learned my addr 172.21.15.111:0/3907179296 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103cdcf700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038071f90 0x7f103811a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 -- 172.21.15.111:0/3907179296 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038071f90 msgr2=0x7f103811a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:51.995 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038071f90 0x7f103811a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.994+0000 7f103d5d0700 1 -- 172.21.15.111:0/3907179296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028005040 con 0x7f10380708a0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f103d5d0700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1028009380 tx=0x7f1028009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102800e040 con 0x7f10380708a0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1028013420 con 0x7f10380708a0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1038114310 con 0x7f10380708a0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102801c630 con 0x7f10380708a0 2024-03-14T13:20:51.996 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.995+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1038114850 con 0x7f10380708a0 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.996+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1038071aa0 con 0x7f10380708a0 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.997+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f10280139e0 con 0x7f10380708a0 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.997+0000 7f102dffb700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 0x7f102406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.997+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f102808cd20 con 0x7f10380708a0 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.997+0000 7f103cdcf700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 0x7f102406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:51.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:51.998+0000 7f103cdcf700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 0x7f102406ec70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.001 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.002+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1028056fa0 con 0x7f10380708a0 2024-03-14T13:20:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:51 smithi137 ceph-mon[108145]: pgmap v502: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:52.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.211+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1038110460 con 0x7f102406c7c0 2024-03-14T13:20:52.211 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.212+0000 7f102dffb700 1 -- 172.21.15.111:0/3907179296 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1038110460 con 0x7f102406c7c0 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 msgr2=0x7f102406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 0x7f102406ec70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034008040 comp rx=0 tx=0).stop 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 msgr2=0x7f103811a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1028009380 tx=0x7f1028009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 shutdown_connections 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.214+0000 7f103f834700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f102406c7c0 0x7f102406ec70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.215+0000 7f103f834700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10380708a0 0x7f103811a080 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.215+0000 7f103f834700 1 --2- 172.21.15.111:0/3907179296 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1038071f90 0x7f103811a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.215+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 >> 172.21.15.111:0/3907179296 conn(0x7f10380f89b0 msgr2=0x7f10380f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.214 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.215+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 shutdown_connections 2024-03-14T13:20:52.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.215+0000 7f103f834700 1 -- 172.21.15.111:0/3907179296 wait complete. 2024-03-14T13:20:52.224 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/997019087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc18102f50 msgr2=0x7fcc18103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/997019087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc18102f50 0x7fcc18103320 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fcc08004940 tx=0x7fcc08009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/997019087 shutdown_connections 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/997019087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc180fb4d0 0x7fcc180fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/997019087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc18102f50 0x7fcc18103320 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/997019087 >> 172.21.15.111:0/997019087 conn(0x7fcc180714c0 msgr2=0x7fcc180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/997019087 shutdown_connections 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/997019087 wait complete. 2024-03-14T13:20:52.316 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 Processor -- start 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.316+0000 7fcc1d8ea700 1 -- start start 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc18102f50 0x7fcc18113e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181144d0 con 0x7fcc180fb4d0 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1810dad0 con 0x7fcc18102f50 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56592/0 (socket says 172.21.15.111:56592) 2024-03-14T13:20:52.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 -- 172.21.15.111:0/1480059169 learned_addr learned my addr 172.21.15.111:0/1480059169 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:52.318 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 -- 172.21.15.111:0/1480059169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc18102f50 msgr2=0x7fcc18113e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc167fc700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc18102f50 0x7fcc18113e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc18102f50 0x7fcc18113e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 -- 172.21.15.111:0/1480059169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc08005040 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc16ffd700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009380 tx=0x7fcc08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc0800e040 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc08013420 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1810dd50 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc0801c630 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.317+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1810e290 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.319+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18100200 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.319+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcc080139e0 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.319+0000 7fcbff7fe700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 0x7fcc0006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.319+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcc0808ccd0 con 0x7fcc180fb4d0 2024-03-14T13:20:52.319 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.319+0000 7fcc167fc700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 0x7fcc0006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.320 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.320+0000 7fcc167fc700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 0x7fcc0006ecc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcc1810ec90 tx=0x7fcc0c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.323 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.324+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcc0805a960 con 0x7fcc180fb4d0 2024-03-14T13:20:52.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.545+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc1810ef90 con 0x7fcc0006c810 2024-03-14T13:20:52.545 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.546+0000 7fcbff7fe700 1 -- 172.21.15.111:0/1480059169 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fcc1810ef90 con 0x7fcc0006c810 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 msgr2=0x7fcc0006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 0x7fcc0006ecc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcc1810ec90 tx=0x7fcc0c009040 comp rx=0 tx=0).stop 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 msgr2=0x7fcc18113940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009380 tx=0x7fcc08009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 shutdown_connections 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcc0006c810 0x7fcc0006ecc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc180fb4d0 0x7fcc18113940 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 --2- 172.21.15.111:0/1480059169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc18102f50 0x7fcc18113e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.548+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 >> 172.21.15.111:0/1480059169 conn(0x7fcc180714c0 msgr2=0x7fcc180fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.549+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 shutdown_connections 2024-03-14T13:20:52.548 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.549+0000 7fcc1d8ea700 1 -- 172.21.15.111:0/1480059169 wait complete. 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 -- 172.21.15.111:0/3584194943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 msgr2=0x7f39c41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/3584194943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c41031e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f39b8004940 tx=0x7f39b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 -- 172.21.15.111:0/3584194943 shutdown_connections 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/3584194943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/3584194943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c41031e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.648 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 -- 172.21.15.111:0/3584194943 >> 172.21.15.111:0/3584194943 conn(0x7f39c40f89a0 msgr2=0x7f39c40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 -- 172.21.15.111:0/3584194943 shutdown_connections 2024-03-14T13:20:52.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.648+0000 7f39c9bd4700 1 -- 172.21.15.111:0/3584194943 wait complete. 2024-03-14T13:20:52.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 Processor -- start 2024-03-14T13:20:52.649 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 -- start start 2024-03-14T13:20:52.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c4113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c41146d0 con 0x7f39c4102e10 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c9bd4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c410da30 con 0x7f39c41037b0 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56608/0 (socket says 172.21.15.111:56608) 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.649+0000 7f39c37fe700 1 -- 172.21.15.111:0/294782432 learned_addr learned my addr 172.21.15.111:0/294782432 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c37fe700 1 -- 172.21.15.111:0/294782432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 msgr2=0x7f39c4113ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c2ffd700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c4113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c37fe700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c4113ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c37fe700 1 -- 172.21.15.111:0/294782432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b8005040 con 0x7f39c4102e10 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c2ffd700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c4113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c37fe700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009180 tx=0x7f39b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.651 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b800e040 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39b8004290 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c410dcb0 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39b801c3f0 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.650+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c410e270 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.651+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c40fe560 con 0x7f39c4102e10 2024-03-14T13:20:52.652 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.651+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f39b8013790 con 0x7f39c4102e10 2024-03-14T13:20:52.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.652+0000 7f39abfff700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 0x7f39ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.652+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f39b808cd30 con 0x7f39c4102e10 2024-03-14T13:20:52.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.652+0000 7f39c2ffd700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 0x7f39ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.653 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.652+0000 7f39c2ffd700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 0x7f39ac06ec70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f39b4006fd0 tx=0x7f39b4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.656+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f39b805aa40 con 0x7f39c4102e10 2024-03-14T13:20:52.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.857+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f39c40fe560 con 0x7f39ac06c7c0 2024-03-14T13:20:52.865 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.866+0000 7f39abfff700 1 -- 172.21.15.111:0/294782432 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f39c40fe560 con 0x7f39ac06c7c0 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (17m) 10m ago 28m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (14m) 10m ago 28m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (14m) 10m ago 27m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (16m) 10m ago 28m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (10m) 10m ago 22m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (10m) 10m ago 22m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (10m) 10m ago 22m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (10m) 10m ago 22m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (19m) 10m ago 30m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (18m) 10m ago 27m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (15m) 10m ago 30m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (15m) 10m ago 27m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:20:52.866 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (18m) 10m ago 28m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (17m) 10m ago 27m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (14m) 10m ago 26m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (13m) 10m ago 25m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (12m) 10m ago 25m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (12m) 10m ago 25m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (11m) 10m ago 24m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (11m) 10m ago 24m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:20:52.867 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (17m) 10m ago 27m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 msgr2=0x7f39ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 0x7f39ac06ec70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f39b4006fd0 tx=0x7f39b4008040 comp rx=0 tx=0).stop 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 msgr2=0x7f39c4113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009180 tx=0x7f39b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 shutdown_connections 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f39ac06c7c0 0x7f39ac06ec70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39c4102e10 0x7f39c4113ab0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 --2- 172.21.15.111:0/294782432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39c41037b0 0x7f39c4113ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 >> 172.21.15.111:0/294782432 conn(0x7f39c40f89a0 msgr2=0x7f39c40f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 shutdown_connections 2024-03-14T13:20:52.868 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.868+0000 7f39c9bd4700 1 -- 172.21.15.111:0/294782432 wait complete. 2024-03-14T13:20:52.967 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.966+0000 7f20b8fd1700 1 -- 172.21.15.111:0/435847564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fe830 msgr2=0x7f20b4106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.966+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/435847564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fe830 0x7f20b4106f00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f20a8004940 tx=0x7f20a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- 172.21.15.111:0/435847564 shutdown_connections 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/435847564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fe830 0x7f20b4106f00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/435847564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fde90 0x7f20b40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- 172.21.15.111:0/435847564 >> 172.21.15.111:0/435847564 conn(0x7f20b4072bc0 msgr2=0x7f20b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- 172.21.15.111:0/435847564 shutdown_connections 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- 172.21.15.111:0/435847564 wait complete. 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 Processor -- start 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- start start 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fe830 0x7f20b4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b4114670 con 0x7f20b40fde90 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b8fd1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b410da60 con 0x7f20b40fe830 2024-03-14T13:20:52.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56626/0 (socket says 172.21.15.111:56626) 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.967+0000 7f20b259c700 1 -- 172.21.15.111:0/957060145 learned_addr learned my addr 172.21.15.111:0/957060145 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b1d9b700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fe830 0x7f20b4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b259c700 1 -- 172.21.15.111:0/957060145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fe830 msgr2=0x7f20b4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b259c700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fe830 0x7f20b4114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b259c700 1 -- 172.21.15.111:0/957060145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8005040 con 0x7f20b40fde90 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b259c700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20a4002a50 tx=0x7f20a40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20a4007cc0 con 0x7f20b40fde90 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f20a400f040 con 0x7f20b40fde90 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b410dd40 con 0x7f20b40fde90 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20a4009540 con 0x7f20b40fde90 2024-03-14T13:20:52.969 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.968+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b410e280 con 0x7f20b40fde90 2024-03-14T13:20:52.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.969+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20b4065e30 con 0x7f20b40fde90 2024-03-14T13:20:52.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.969+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f20a40096a0 con 0x7f20b40fde90 2024-03-14T13:20:52.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.970+0000 7f20a2ffd700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 0x7f209c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:52.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.970+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20a408b8a0 con 0x7f20b40fde90 2024-03-14T13:20:52.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.970+0000 7f20b1d9b700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 0x7f209c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:52.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.970+0000 7f20b1d9b700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 0x7f209c06ec70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f20a800b3b0 tx=0x7f20a8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:52.973 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:52.974+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f20a4059530 con 0x7f20b40fde90 2024-03-14T13:20:53.251 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.252+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f20b4109dd0 con 0x7f20b40fde90 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.253+0000 7f20a2ffd700 1 -- 172.21.15.111:0/957060145 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f20a40590c0 con 0x7f20b40fde90 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:53.252 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:20:53.253 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 msgr2=0x7f209c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 0x7f209c06ec70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f20a800b3b0 tx=0x7f20a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 msgr2=0x7f20b4113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20a4002a50 tx=0x7f20a40046b0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 shutdown_connections 2024-03-14T13:20:53.254 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f209c06c7c0 0x7f209c06ec70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20b40fde90 0x7f20b4113ae0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 --2- 172.21.15.111:0/957060145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b40fe830 0x7f20b4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 >> 172.21.15.111:0/957060145 conn(0x7f20b4072bc0 msgr2=0x7f20b4103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 shutdown_connections 2024-03-14T13:20:53.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.255+0000 7f20b8fd1700 1 -- 172.21.15.111:0/957060145 wait complete. 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- 172.21.15.111:0/2614426568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 msgr2=0x7fb41c0fe180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 --2- 172.21.15.111:0/2614426568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c0fe180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb40c004940 tx=0x7fb40c009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- 172.21.15.111:0/2614426568 shutdown_connections 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 --2- 172.21.15.111:0/2614426568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 --2- 172.21.15.111:0/2614426568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c0fe180 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- 172.21.15.111:0/2614426568 >> 172.21.15.111:0/2614426568 conn(0x7fb41c0f7860 msgr2=0x7fb41c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- 172.21.15.111:0/2614426568 shutdown_connections 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- 172.21.15.111:0/2614426568 wait complete. 2024-03-14T13:20:53.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 Processor -- start 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.355+0000 7fb423bbb700 1 -- start start 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c111dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c112410 con 0x7fb41c0fddb0 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c10b800 con 0x7fb41c0fe750 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56642/0 (socket says 172.21.15.111:56642) 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 -- 172.21.15.111:0/1482666602 learned_addr learned my addr 172.21.15.111:0/1482666602 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421156700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c111dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 -- 172.21.15.111:0/1482666602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 msgr2=0x7fb41c111dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c111dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 -- 172.21.15.111:0/1482666602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c005040 con 0x7fb41c0fddb0 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421156700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb421957700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb40c008f80 tx=0x7fb40c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:53.357 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40c01e070 con 0x7fb41c0fddb0 2024-03-14T13:20:53.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c10ba80 con 0x7fb41c0fddb0 2024-03-14T13:20:53.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb40c004290 con 0x7fb41c0fddb0 2024-03-14T13:20:53.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb40c019420 con 0x7fb41c0fddb0 2024-03-14T13:20:53.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.356+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c10bf60 con 0x7fb41c0fddb0 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.357+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c066770 con 0x7fb41c0fddb0 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.358+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb40c016020 con 0x7fb41c0fddb0 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.358+0000 7fb4127fc700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 0x7fb40806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.358+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb40c090be0 con 0x7fb41c0fddb0 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.358+0000 7fb421156700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 0x7fb40806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.359+0000 7fb421156700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 0x7fb40806ec70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb418004600 tx=0x7fb41800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:53.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.363+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb40c05e8f0 con 0x7fb41c0fddb0 2024-03-14T13:20:53.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.601+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb41c107af0 con 0x7fb41c0fddb0 2024-03-14T13:20:53.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.601+0000 7fb4127fc700 1 -- 172.21.15.111:0/1482666602 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fb40c004490 con 0x7fb41c0fddb0 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:20:53.601 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:20:53.602 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:53.603 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 msgr2=0x7fb40806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 0x7fb40806ec70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb418004600 tx=0x7fb41800b040 comp rx=0 tx=0).stop 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 msgr2=0x7fb41c111880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb40c008f80 tx=0x7fb40c009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 shutdown_connections 2024-03-14T13:20:53.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb40806c7c0 0x7fb40806ec70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb41c0fddb0 0x7fb41c111880 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 --2- 172.21.15.111:0/1482666602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb41c0fe750 0x7fb41c111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 >> 172.21.15.111:0/1482666602 conn(0x7fb41c0f7860 msgr2=0x7fb41c0f8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 shutdown_connections 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.603+0000 7fb423bbb700 1 -- 172.21.15.111:0/1482666602 wait complete. 2024-03-14T13:20:53.605 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:20:53.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- 172.21.15.111:0/535424885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c0fb4a0 msgr2=0x7f646c0fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 --2- 172.21.15.111:0/535424885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c0fb4a0 0x7f646c0fb910 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6468004940 tx=0x7f6468009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- 172.21.15.111:0/535424885 shutdown_connections 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 --2- 172.21.15.111:0/535424885 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c0fb4a0 0x7f646c0fb910 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 --2- 172.21.15.111:0/535424885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c102e70 0x7f646c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- 172.21.15.111:0/535424885 >> 172.21.15.111:0/535424885 conn(0x7f646c0714c0 msgr2=0x7f646c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- 172.21.15.111:0/535424885 shutdown_connections 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- 172.21.15.111:0/535424885 wait complete. 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 Processor -- start 2024-03-14T13:20:53.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.704+0000 7f6473807700 1 -- start start 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6473807700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c0fb4a0 0x7f646c113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6473807700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6473807700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c114600 con 0x7f646c102e70 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6473807700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646c10d9f0 con 0x7f646c0fb4a0 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56650/0 (socket says 172.21.15.111:56650) 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 -- 172.21.15.111:0/2497929591 learned_addr learned my addr 172.21.15.111:0/2497929591 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f64715a3700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c0fb4a0 0x7f646c113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 -- 172.21.15.111:0/2497929591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c0fb4a0 msgr2=0x7f646c113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c0fb4a0 0x7f646c113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 -- 172.21.15.111:0/2497929591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6468005040 con 0x7f646c102e70 2024-03-14T13:20:53.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6470da2700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f646800b3b0 tx=0x7f6468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646800e040 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6468013420 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f646c10dc70 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.705+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f646801c610 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.706+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646c10e230 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.707+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f646c100110 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.707+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6468013a00 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.707+0000 7f6461ffb700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 0x7f645806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.707+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f646808cac0 con 0x7f646c102e70 2024-03-14T13:20:53.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.707+0000 7f64715a3700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 0x7f645806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:53.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.708+0000 7f64715a3700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 0x7f645806ec70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f645c007fd0 tx=0x7f645c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:53.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.712+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f646805a890 con 0x7f646c102e70 2024-03-14T13:20:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:53 smithi111 ceph-mon[145314]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:53 smithi111 ceph-mon[145314]: pgmap v503: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:53 smithi111 ceph-mon[145314]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:53.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:53 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/957060145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:20:53.922 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.923+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f646c10f0c0 con 0x7f645806c7c0 2024-03-14T13:20:53.923 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.924+0000 7f6461ffb700 1 -- 172.21.15.111:0/2497929591 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f646c10f0c0 con 0x7f645806c7c0 2024-03-14T13:20:53.923 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:20:53.923 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:20:53.923 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:20:53.923 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:20:53.924 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:20:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 msgr2=0x7f645806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 0x7f645806ec70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f645c007fd0 tx=0x7f645c009040 comp rx=0 tx=0).stop 2024-03-14T13:20:53.925 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 msgr2=0x7f646c113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f646800b3b0 tx=0x7f6468009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 shutdown_connections 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f645806c7c0 0x7f645806ec70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f646c0fb4a0 0x7f646c113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 --2- 172.21.15.111:0/2497929591 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f646c102e70 0x7f646c113fb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 >> 172.21.15.111:0/2497929591 conn(0x7f646c0714c0 msgr2=0x7f646c0fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 shutdown_connections 2024-03-14T13:20:53.926 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:53.926+0000 7f6473807700 1 -- 172.21.15.111:0/2497929591 wait complete. 2024-03-14T13:20:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:53 smithi137 ceph-mon[108145]: from='client.34960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:53 smithi137 ceph-mon[108145]: pgmap v503: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:53 smithi137 ceph-mon[108145]: from='client.34964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:54.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:53 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/957060145' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:20:54.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- 172.21.15.111:0/603382565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858102710 msgr2=0x7fc8580fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 --2- 172.21.15.111:0/603382565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858102710 0x7fc8580fbd80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc844004940 tx=0x7fc844009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- 172.21.15.111:0/603382565 shutdown_connections 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 --2- 172.21.15.111:0/603382565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858102710 0x7fc8580fbd80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 --2- 172.21.15.111:0/603382565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858101d70 0x7fc858102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- 172.21.15.111:0/603382565 >> 172.21.15.111:0/603382565 conn(0x7fc858072bc0 msgr2=0x7fc858072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- 172.21.15.111:0/603382565 shutdown_connections 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- 172.21.15.111:0/603382565 wait complete. 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 Processor -- start 2024-03-14T13:20:54.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.027+0000 7fc85f220700 1 -- start start 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858102710 0x7fc858103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8581036a0 con 0x7fc858101d70 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8581037e0 con 0x7fc858102710 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56672/0 (socket says 172.21.15.111:56672) 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 -- 172.21.15.111:0/1836562528 learned_addr learned my addr 172.21.15.111:0/1836562528 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc857fff700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858102710 0x7fc858103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 -- 172.21.15.111:0/1836562528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858102710 msgr2=0x7fc858103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858102710 0x7fc858103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 -- 172.21.15.111:0/1836562528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844005040 con 0x7fc858101d70 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85cfbc700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc84c002a50 tx=0x7fc84c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:54.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84c007cc0 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc858119dc0 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc84c00f040 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc84c00d9c0 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.028+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85811a240 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.029+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8580fd4c0 con 0x7fc858101d70 2024-03-14T13:20:54.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.030+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc84c00db20 con 0x7fc858101d70 2024-03-14T13:20:54.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.030+0000 7fc8557fa700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 0x7fc84806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:20:54.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.030+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc84c08b7e0 con 0x7fc858101d70 2024-03-14T13:20:54.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.030+0000 7fc857fff700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 0x7fc84806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:20:54.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.031+0000 7fc857fff700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 0x7fc84806ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc844004890 tx=0x7fc844009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:20:54.034 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.035+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc84c0559e0 con 0x7fc858101d70 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.309+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc85805f5f0 con 0x7fc858101d70 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.310+0000 7fc8557fa700 1 -- 172.21.15.111:0/1836562528 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fc84c059000 con 0x7fc858101d70 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:20:54.309 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:20:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 msgr2=0x7fc84806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 0x7fc84806ec70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc844004890 tx=0x7fc844009e30 comp rx=0 tx=0).stop 2024-03-14T13:20:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 msgr2=0x7fc858102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:20:54.311 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc84c002a50 tx=0x7fc84c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 shutdown_connections 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc84806c7c0 0x7fc84806ec70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc858101d70 0x7fc858102bd0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 --2- 172.21.15.111:0/1836562528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc858102710 0x7fc858103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 >> 172.21.15.111:0/1836562528 conn(0x7fc858072bc0 msgr2=0x7fc8580659e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 shutdown_connections 2024-03-14T13:20:54.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:20:54.312+0000 7fc85f220700 1 -- 172.21.15.111:0/1836562528 wait complete. 2024-03-14T13:20:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:54 smithi111 ceph-mon[145314]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:54 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1482666602' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:20:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:54.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:54 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1836562528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:20:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:54 smithi137 ceph-mon[108145]: from='client.34968 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:55.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:54 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1482666602' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:20:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:20:55.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:54 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1836562528' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:20:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:55 smithi111 ceph-mon[145314]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:55.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:55 smithi111 ceph-mon[145314]: pgmap v504: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:55 smithi137 ceph-mon[108145]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:20:56.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:55 smithi137 ceph-mon[108145]: pgmap v504: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:57.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:57 smithi111 ceph-mon[145314]: pgmap v505: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:58.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:57 smithi137 ceph-mon[108145]: pgmap v505: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:20:59.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:20:59 smithi111 ceph-mon[145314]: pgmap v506: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:20:59 smithi137 ceph-mon[108145]: pgmap v506: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:01.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:01 smithi111 ceph-mon[145314]: pgmap v507: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:02.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:01 smithi137 ceph-mon[108145]: pgmap v507: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:03 smithi111 ceph-mon[145314]: pgmap v508: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:04.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:03 smithi137 ceph-mon[108145]: pgmap v508: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:05.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:05 smithi111 ceph-mon[145314]: pgmap v509: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:06.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:05 smithi137 ceph-mon[108145]: pgmap v509: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:07.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:07 smithi111 ceph-mon[145314]: pgmap v510: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:08.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:07 smithi137 ceph-mon[108145]: pgmap v510: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:09 smithi111 ceph-mon[145314]: pgmap v511: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:09.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:09 smithi137 ceph-mon[108145]: pgmap v511: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:10.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:11.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:11 smithi111 ceph-mon[145314]: pgmap v512: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:12.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:11 smithi137 ceph-mon[108145]: pgmap v512: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:13.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:13 smithi111 ceph-mon[145314]: pgmap v513: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:14.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:13 smithi137 ceph-mon[108145]: pgmap v513: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:15.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:15 smithi111 ceph-mon[145314]: pgmap v514: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:16.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:15 smithi137 ceph-mon[108145]: pgmap v514: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:17.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:16 smithi137 ceph-mon[108145]: pgmap v515: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:17.140 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:16 smithi111 ceph-mon[145314]: pgmap v515: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:19.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:19 smithi137 ceph-mon[108145]: pgmap v516: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:19.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:19 smithi111 ceph-mon[145314]: pgmap v516: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:21 smithi111 ceph-mon[145314]: pgmap v517: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:22.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:21 smithi137 ceph-mon[108145]: pgmap v517: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:23.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:23 smithi111 ceph-mon[145314]: pgmap v518: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:24.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:23 smithi137 ceph-mon[108145]: pgmap v518: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:24.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 -- 172.21.15.111:0/181860619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c1037b0 msgr2=0x7fe11c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 --2- 172.21.15.111:0/181860619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c1037b0 0x7fe11c0fce20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe108004940 tx=0x7fe108009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:24.424 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 -- 172.21.15.111:0/181860619 shutdown_connections 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 --2- 172.21.15.111:0/181860619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c1037b0 0x7fe11c0fce20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 --2- 172.21.15.111:0/181860619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c102e10 0x7fe11c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 -- 172.21.15.111:0/181860619 >> 172.21.15.111:0/181860619 conn(0x7fe11c0f89a0 msgr2=0x7fe11c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 -- 172.21.15.111:0/181860619 shutdown_connections 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.423+0000 7fe122e74700 1 -- 172.21.15.111:0/181860619 wait complete. 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 Processor -- start 2024-03-14T13:21:24.425 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 -- start start 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c1037b0 0x7fe11c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c114660 con 0x7fe11c102e10 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.424+0000 7fe122e74700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c10da50 con 0x7fe11c1037b0 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49986/0 (socket says 172.21.15.111:49986) 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 -- 172.21.15.111:0/2320251393 learned_addr learned my addr 172.21.15.111:0/2320251393 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe11bfff700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c1037b0 0x7fe11c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 -- 172.21.15.111:0/2320251393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c1037b0 msgr2=0x7fe11c114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c1037b0 0x7fe11c114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 -- 172.21.15.111:0/2320251393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe108005040 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe120c10700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c810 tx=0x7fe110004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe110012810 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe110012e20 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c10dd30 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe11000a370 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.425+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c10e2f0 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.426+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c0fe560 con 0x7fe11c102e10 2024-03-14T13:21:24.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.427+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe110011780 con 0x7fe11c102e10 2024-03-14T13:21:24.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.427+0000 7fe1197fa700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 0x7fe10c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.427+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe11008bf00 con 0x7fe11c102e10 2024-03-14T13:21:24.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.427+0000 7fe11bfff700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 0x7fe10c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.428+0000 7fe11bfff700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 0x7fe10c06ec70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe11c10ec10 tx=0x7fe108009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:24.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.432+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe110059b90 con 0x7fe11c102e10 2024-03-14T13:21:24.646 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.646+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe11c109e40 con 0x7fe10c06c7c0 2024-03-14T13:21:24.647 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.647+0000 7fe1197fa700 1 -- 172.21.15.111:0/2320251393 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe11c109e40 con 0x7fe10c06c7c0 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 msgr2=0x7fe10c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 0x7fe10c06ec70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe11c10ec10 tx=0x7fe108009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 msgr2=0x7fe11c113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c810 tx=0x7fe110004470 comp rx=0 tx=0).stop 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 shutdown_connections 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe10c06c7c0 0x7fe10c06ec70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe11c102e10 0x7fe11c113ad0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 --2- 172.21.15.111:0/2320251393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c1037b0 0x7fe11c114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 >> 172.21.15.111:0/2320251393 conn(0x7fe11c0f89a0 msgr2=0x7fe11c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 shutdown_connections 2024-03-14T13:21:24.650 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.649+0000 7fe122e74700 1 -- 172.21.15.111:0/2320251393 wait complete. 2024-03-14T13:21:24.660 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- 172.21.15.111:0/1608561918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 msgr2=0x7f6528102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 --2- 172.21.15.111:0/1608561918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528102140 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6524004940 tx=0x7f6524009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- 172.21.15.111:0/1608561918 shutdown_connections 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 --2- 172.21.15.111:0/1608561918 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f65280fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 --2- 172.21.15.111:0/1608561918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528102140 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- 172.21.15.111:0/1608561918 >> 172.21.15.111:0/1608561918 conn(0x7f6528072bc0 msgr2=0x7f6528072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- 172.21.15.111:0/1608561918 shutdown_connections 2024-03-14T13:21:24.753 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- 172.21.15.111:0/1608561918 wait complete. 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 Processor -- start 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.751+0000 7f652f265700 1 -- start start 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652f265700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652f265700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f6528113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652f265700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65281145d0 con 0x7f6528101d70 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652f265700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652810d9c0 con 0x7f6528102710 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50006/0 (socket says 172.21.15.111:50006) 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 -- 172.21.15.111:0/3790303061 learned_addr learned my addr 172.21.15.111:0/3790303061 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652c800700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f6528113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 -- 172.21.15.111:0/3790303061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 msgr2=0x7f6528113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f6528113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 -- 172.21.15.111:0/3790303061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6524005040 con 0x7f6528101d70 2024-03-14T13:21:24.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652c800700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f6528113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652d001700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6524000c00 tx=0x7f6524009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6524021070 con 0x7f6528101d70 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6524013de0 con 0x7f6528101d70 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652810dc40 con 0x7f6528101d70 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.752+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f652400e070 con 0x7f6528101d70 2024-03-14T13:21:24.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.753+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652810e1a0 con 0x7f6528101d70 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.753+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65280fd4c0 con 0x7f6528101d70 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.754+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f652401c600 con 0x7f6528101d70 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.755+0000 7f651dffb700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 0x7f651406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.755+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f652408d320 con 0x7f6528101d70 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.755+0000 7f652c800700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 0x7f651406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:24.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.755+0000 7f652c800700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 0x7f651406ec70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6518004600 tx=0x7f651800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:24.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.759+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f652405b0f0 con 0x7f6528101d70 2024-03-14T13:21:24.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:24.967 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.967+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f652804bed0 con 0x7f651406c7c0 2024-03-14T13:21:24.968 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.968+0000 7f651dffb700 1 -- 172.21.15.111:0/3790303061 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f652804bed0 con 0x7f651406c7c0 2024-03-14T13:21:24.970 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 msgr2=0x7f651406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 0x7f651406ec70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6518004600 tx=0x7f651800b040 comp rx=0 tx=0).stop 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 msgr2=0x7f6528113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6524000c00 tx=0x7f6524009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 shutdown_connections 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f651406c7c0 0x7f651406ec70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6528101d70 0x7f6528113a40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 --2- 172.21.15.111:0/3790303061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6528102710 0x7f6528113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 >> 172.21.15.111:0/3790303061 conn(0x7f6528072bc0 msgr2=0x7f6528065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 shutdown_connections 2024-03-14T13:21:24.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:24.970+0000 7f652f265700 1 -- 172.21.15.111:0/3790303061 wait complete. 2024-03-14T13:21:25.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 -- 172.21.15.111:0/3031867035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 msgr2=0x7fdfe0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 --2- 172.21.15.111:0/3031867035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0102140 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0004940 tx=0x7fdfd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 -- 172.21.15.111:0/3031867035 shutdown_connections 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 --2- 172.21.15.111:0/3031867035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 0x7fdfe00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 --2- 172.21.15.111:0/3031867035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0102140 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 -- 172.21.15.111:0/3031867035 >> 172.21.15.111:0/3031867035 conn(0x7fdfe0072bc0 msgr2=0x7fdfe0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 -- 172.21.15.111:0/3031867035 shutdown_connections 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 -- 172.21.15.111:0/3031867035 wait complete. 2024-03-14T13:21:25.071 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.070+0000 7fdfe7291700 1 Processor -- start 2024-03-14T13:21:25.072 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe7291700 1 -- start start 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe7291700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe7291700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 0x7fdfe0112e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe7291700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe01134b0 con 0x7fdfe0101d70 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe7291700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe010c8a0 con 0x7fdfe0102710 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50024/0 (socket says 172.21.15.111:50024) 2024-03-14T13:21:25.073 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 -- 172.21.15.111:0/1240246889 learned_addr learned my addr 172.21.15.111:0/1240246889 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe482c700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 0x7fdfe0112e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 -- 172.21.15.111:0/1240246889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 msgr2=0x7fdfe0112e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 0x7fdfe0112e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 -- 172.21.15.111:0/1240246889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd0005040 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.071+0000 7fdfe502d700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009380 tx=0x7fdfd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.072+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd000e040 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.072+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe010cb20 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.072+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfd0004290 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.072+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfd0013660 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.072+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe010d080 con 0x7fdfe0101d70 2024-03-14T13:21:25.074 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.073+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe00fd4c0 con 0x7fdfe0101d70 2024-03-14T13:21:25.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.073+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fdfd0013920 con 0x7fdfe0101d70 2024-03-14T13:21:25.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.074+0000 7fdfd5ffb700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 0x7fdfcc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.074+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdfd008f870 con 0x7fdfe0101d70 2024-03-14T13:21:25.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.074+0000 7fdfe482c700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 0x7fdfcc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.075 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.074+0000 7fdfe482c700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 0x7fdfcc06ec70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdfe010da60 tx=0x7fdfdc00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.078+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdfd005d500 con 0x7fdfe0101d70 2024-03-14T13:21:25.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.278+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe0108dd0 con 0x7fdfcc06c7c0 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.287+0000 7fdfd5ffb700 1 -- 172.21.15.111:0/1240246889 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fdfdc00f040 con 0x7fdfcc06c7c0 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (17m) 11m ago 29m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (15m) 11m ago 29m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (15m) 10m ago 27m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (17m) 11m ago 28m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (11m) 11m ago 23m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (11m) 11m ago 23m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (10m) 10m ago 22m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (10m) 10m ago 23m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (20m) 11m ago 30m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (18m) 10m ago 27m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (16m) 11m ago 30m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (15m) 10m ago 27m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (18m) 11m ago 29m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (18m) 10m ago 27m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (14m) 11m ago 26m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (14m) 11m ago 26m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (13m) 11m ago 25m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (12m) 10m ago 25m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (12m) 10m ago 25m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (11m) 10m ago 24m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (18m) 11m ago 28m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 msgr2=0x7fdfcc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 0x7fdfcc06ec70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdfe010da60 tx=0x7fdfdc00b040 comp rx=0 tx=0).stop 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 msgr2=0x7fdfe0112920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009380 tx=0x7fdfd0009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 shutdown_connections 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdfcc06c7c0 0x7fdfcc06ec70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfe0101d70 0x7fdfe0112920 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 --2- 172.21.15.111:0/1240246889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfe0102710 0x7fdfe0112e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 >> 172.21.15.111:0/1240246889 conn(0x7fdfe0072bc0 msgr2=0x7fdfe00faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 shutdown_connections 2024-03-14T13:21:25.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.289+0000 7fdfe7291700 1 -- 172.21.15.111:0/1240246889 wait complete. 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 -- 172.21.15.111:0/3252472125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894101e60 msgr2=0x7f0894102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 --2- 172.21.15.111:0/3252472125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894101e60 0x7f0894102230 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0888004940 tx=0x7f0888009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 -- 172.21.15.111:0/3252472125 shutdown_connections 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 --2- 172.21.15.111:0/3252472125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894102800 0x7f08940fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 --2- 172.21.15.111:0/3252472125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894101e60 0x7f0894102230 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.389+0000 7f0899278700 1 -- 172.21.15.111:0/3252472125 >> 172.21.15.111:0/3252472125 conn(0x7f0894071530 msgr2=0x7f0894071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 -- 172.21.15.111:0/3252472125 shutdown_connections 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 -- 172.21.15.111:0/3252472125 wait complete. 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 Processor -- start 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 -- start start 2024-03-14T13:21:25.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894101e60 0x7f08941138e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0894114470 con 0x7f0894102800 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.390+0000 7f0899278700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089410da70 con 0x7f0894101e60 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50050/0 (socket says 172.21.15.111:50050) 2024-03-14T13:21:25.392 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 -- 172.21.15.111:0/1447776398 learned_addr learned my addr 172.21.15.111:0/1447776398 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:25.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f0892ffd700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894101e60 0x7f08941138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 -- 172.21.15.111:0/1447776398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894101e60 msgr2=0x7f08941138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894101e60 0x7f08941138e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 -- 172.21.15.111:0/1447776398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0888005040 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f08927fc700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0884002a50 tx=0x7f0884004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0884007cc0 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089410dd50 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f088400f040 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088400d990 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.391+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089410e310 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.392+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08941001d0 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.393+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0884009710 con 0x7f0894102800 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.393+0000 7f087b7fe700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 0x7f087c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.393+0000 7f0892ffd700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 0x7f087c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.394 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.393+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f088408b890 con 0x7f0894102800 2024-03-14T13:21:25.395 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.394+0000 7f0892ffd700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 0x7f087c06ec70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0888004560 tx=0x7f0888009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.398 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.398+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0884059520 con 0x7f0894102800 2024-03-14T13:21:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.669+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f089410ef30 con 0x7f0894102800 2024-03-14T13:21:25.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.670+0000 7f087b7fe700 1 -- 172.21.15.111:0/1447776398 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f08840590b0 con 0x7f0894102800 2024-03-14T13:21:25.670 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:21:25.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 msgr2=0x7f087c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 0x7f087c06ec70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0888004560 tx=0x7f0888009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 msgr2=0x7f0894113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.672 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0884002a50 tx=0x7f0884004600 comp rx=0 tx=0).stop 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 shutdown_connections 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f087c06c7c0 0x7f087c06ec70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0894101e60 0x7f08941138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 --2- 172.21.15.111:0/1447776398 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0894102800 0x7f0894113e20 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 >> 172.21.15.111:0/1447776398 conn(0x7f0894071530 msgr2=0x7f0894105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 shutdown_connections 2024-03-14T13:21:25.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.672+0000 7f0899278700 1 -- 172.21.15.111:0/1447776398 wait complete. 2024-03-14T13:21:25.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- 172.21.15.111:0/3090696047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 msgr2=0x7fa8901031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 --2- 172.21.15.111:0/3090696047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901031e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa880004940 tx=0x7fa880009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:25.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- 172.21.15.111:0/3090696047 shutdown_connections 2024-03-14T13:21:25.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 --2- 172.21.15.111:0/3090696047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 0x7fa8900fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 --2- 172.21.15.111:0/3090696047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901031e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- 172.21.15.111:0/3090696047 >> 172.21.15.111:0/3090696047 conn(0x7fa8900f89a0 msgr2=0x7fa8900fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:25.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- 172.21.15.111:0/3090696047 shutdown_connections 2024-03-14T13:21:25.773 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- 172.21.15.111:0/3090696047 wait complete. 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 Processor -- start 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.772+0000 7fa898507700 1 -- start start 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa898507700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa898507700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 0x7fa890113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa898507700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa890114580 con 0x7fa890102e10 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa898507700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89010d9b0 con 0x7fa8901037b0 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50060/0 (socket says 172.21.15.111:50060) 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 -- 172.21.15.111:0/954123764 learned_addr learned my addr 172.21.15.111:0/954123764 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa895aa2700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 0x7fa890113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 -- 172.21.15.111:0/954123764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 msgr2=0x7fa890113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:25.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 0x7fa890113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 -- 172.21.15.111:0/954123764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa880005040 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa8962a3700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa880000c00 tx=0x7fa880009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa880021070 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8800093b0 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa88000e070 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.773+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa89010dc30 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.774+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89010e1f0 con 0x7fa890102e10 2024-03-14T13:21:25.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.774+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8900fe560 con 0x7fa890102e10 2024-03-14T13:21:25.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.775+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa88001c490 con 0x7fa890102e10 2024-03-14T13:21:25.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.776+0000 7fa886ffd700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 0x7fa87c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:25.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.776+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa88008d930 con 0x7fa890102e10 2024-03-14T13:21:25.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.776+0000 7fa895aa2700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 0x7fa87c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:25.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.776+0000 7fa895aa2700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 0x7fa87c06ec70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa890107080 tx=0x7fa88c00b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:25.780 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:25.779+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa88005b680 con 0x7fa890102e10 2024-03-14T13:21:25.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:25 smithi111 ceph-mon[145314]: pgmap v519: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:26.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:25 smithi137 ceph-mon[108145]: pgmap v519: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:26.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.015+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa89004bf00 con 0x7fa890102e10 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.015+0000 7fa886ffd700 1 -- 172.21.15.111:0/954123764 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fa88005b210 con 0x7fa890102e10 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:21:26.016 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:21:26.017 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:26.018 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:26.019 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 msgr2=0x7fa87c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 0x7fa87c06ec70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa890107080 tx=0x7fa88c00b040 comp rx=0 tx=0).stop 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 msgr2=0x7fa8901139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa880000c00 tx=0x7fa880009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 shutdown_connections 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa87c06c7c0 0x7fa87c06ec70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa890102e10 0x7fa8901139f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 --2- 172.21.15.111:0/954123764 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8901037b0 0x7fa890113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.017+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 >> 172.21.15.111:0/954123764 conn(0x7fa8900f89a0 msgr2=0x7fa8900f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.018+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 shutdown_connections 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.018+0000 7fa898507700 1 -- 172.21.15.111:0/954123764 wait complete. 2024-03-14T13:21:26.020 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 -- 172.21.15.111:0/3028768277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c1037b0 msgr2=0x7f718c0fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3028768277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c1037b0 0x7f718c0fce20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f717c004940 tx=0x7f717c009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 -- 172.21.15.111:0/3028768277 shutdown_connections 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3028768277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c1037b0 0x7f718c0fce20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3028768277 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c102e10 0x7f718c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 -- 172.21.15.111:0/3028768277 >> 172.21.15.111:0/3028768277 conn(0x7f718c0f89a0 msgr2=0x7f718c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 -- 172.21.15.111:0/3028768277 shutdown_connections 2024-03-14T13:21:26.119 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.118+0000 7f7191ad1700 1 -- 172.21.15.111:0/3028768277 wait complete. 2024-03-14T13:21:26.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 Processor -- start 2024-03-14T13:21:26.120 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 -- start start 2024-03-14T13:21:26.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c1037b0 0x7f718c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c114660 con 0x7f718c102e10 2024-03-14T13:21:26.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f7191ad1700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718c10da50 con 0x7f718c1037b0 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50076/0 (socket says 172.21.15.111:50076) 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 -- 172.21.15.111:0/3666607461 learned_addr learned my addr 172.21.15.111:0/3666607461 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718affd700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c1037b0 0x7f718c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 -- 172.21.15.111:0/3666607461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c1037b0 msgr2=0x7f718c114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c1037b0 0x7f718c114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.119+0000 7f718b7fe700 1 -- 172.21.15.111:0/3666607461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c005040 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f718b7fe700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7174002a50 tx=0x7f71740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7174007cc0 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f717400f040 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718c10dd30 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7174009540 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.120+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f718c10e2f0 con 0x7f718c102e10 2024-03-14T13:21:26.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.121+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f718c0fe560 con 0x7f718c102e10 2024-03-14T13:21:26.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.121+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f71740096a0 con 0x7f718c102e10 2024-03-14T13:21:26.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.122+0000 7f7183fff700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 0x7f717806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.122+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f717408b820 con 0x7f718c102e10 2024-03-14T13:21:26.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.122+0000 7f718affd700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 0x7f717806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.123 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.122+0000 7f718affd700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 0x7f717806ed40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f718c10ec10 tx=0x7f717c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:26.126 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.126+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f71740594b0 con 0x7f718c102e10 2024-03-14T13:21:26.336 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.336+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f718c109e50 con 0x7f717806c890 2024-03-14T13:21:26.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.337+0000 7f7183fff700 1 -- 172.21.15.111:0/3666607461 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f717c007cc0 con 0x7f717806c890 2024-03-14T13:21:26.337 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:21:26.337 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:21:26.337 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:21:26.337 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:21:26.338 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:21:26.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 msgr2=0x7f717806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 0x7f717806ed40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f718c10ec10 tx=0x7f717c009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 msgr2=0x7f718c113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7174002a50 tx=0x7f71740046b0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 shutdown_connections 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f717806c890 0x7f717806ed40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f718c102e10 0x7f718c113ad0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 --2- 172.21.15.111:0/3666607461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f718c1037b0 0x7f718c114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 >> 172.21.15.111:0/3666607461 conn(0x7f718c0f89a0 msgr2=0x7f718c0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 shutdown_connections 2024-03-14T13:21:26.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.339+0000 7f7191ad1700 1 -- 172.21.15.111:0/3666607461 wait complete. 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- 172.21.15.111:0/3841863550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 msgr2=0x7fa0f81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 --2- 172.21.15.111:0/3841863550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f81031e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4004940 tx=0x7fa0f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- 172.21.15.111:0/3841863550 shutdown_connections 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 --2- 172.21.15.111:0/3841863550 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 0x7fa0f80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 --2- 172.21.15.111:0/3841863550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f81031e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.439 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- 172.21.15.111:0/3841863550 >> 172.21.15.111:0/3841863550 conn(0x7fa0f80f8980 msgr2=0x7fa0f80fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:26.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- 172.21.15.111:0/3841863550 shutdown_connections 2024-03-14T13:21:26.440 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- 172.21.15.111:0/3841863550 wait complete. 2024-03-14T13:21:26.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 Processor -- start 2024-03-14T13:21:26.441 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.439+0000 7fa0ff699700 1 -- start start 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0ff699700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0ff699700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 0x7fa0f8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0ff699700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8114660 con 0x7fa0f8102e10 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0ff699700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f810da50 con 0x7fa0f81037b0 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:50102/0 (socket says 172.21.15.111:50102) 2024-03-14T13:21:26.442 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 -- 172.21.15.111:0/2037166639 learned_addr learned my addr 172.21.15.111:0/2037166639 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fcc34700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 0x7fa0f8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 -- 172.21.15.111:0/2037166639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 msgr2=0x7fa0f8114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 0x7fa0f8114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 -- 172.21.15.111:0/2037166639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4005040 con 0x7fa0f8102e10 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0fd435700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009180 tx=0x7fa0f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:26.443 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f400e040 con 0x7fa0f8102e10 2024-03-14T13:21:26.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0f4004290 con 0x7fa0f8102e10 2024-03-14T13:21:26.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.440+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0f4013e60 con 0x7fa0f8102e10 2024-03-14T13:21:26.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.441+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f810dcd0 con 0x7fa0f8102e10 2024-03-14T13:21:26.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.441+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f810e290 con 0x7fa0f8102e10 2024-03-14T13:21:26.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.442+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f80fe560 con 0x7fa0f8102e10 2024-03-14T13:21:26.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.442+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa0f40136b0 con 0x7fa0f8102e10 2024-03-14T13:21:26.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.442+0000 7fa0edffb700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 0x7fa0e406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:26.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.443+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa0f408cbd0 con 0x7fa0f8102e10 2024-03-14T13:21:26.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.443+0000 7fa0fcc34700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 0x7fa0e406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:26.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.443+0000 7fa0fcc34700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 0x7fa0e406ec70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa0f810ec10 tx=0x7fa0e800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:26.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.447+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa0f405a9a0 con 0x7fa0f8102e10 2024-03-14T13:21:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.719+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa0f8109dd0 con 0x7fa0f8102e10 2024-03-14T13:21:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.720+0000 7fa0edffb700 1 -- 172.21.15.111:0/2037166639 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7fa0f405a530 con 0x7fa0f8102e10 2024-03-14T13:21:26.720 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:21:26.720 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:21:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:21:26.721 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:21:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:21:26.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 msgr2=0x7fa0e406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 0x7fa0e406ec70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa0f810ec10 tx=0x7fa0e800b040 comp rx=0 tx=0).stop 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 msgr2=0x7fa0f8113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009180 tx=0x7fa0f4009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 shutdown_connections 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa0e406c7c0 0x7fa0e406ec70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0f8102e10 0x7fa0f8113ad0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 --2- 172.21.15.111:0/2037166639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0f81037b0 0x7fa0f8114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 >> 172.21.15.111:0/2037166639 conn(0x7fa0f80f8980 msgr2=0x7fa0f80f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 shutdown_connections 2024-03-14T13:21:26.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:26.722+0000 7fa0ff699700 1 -- 172.21.15.111:0/2037166639 wait complete. 2024-03-14T13:21:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:26 smithi111 ceph-mon[145314]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:26 smithi111 ceph-mon[145314]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:26 smithi111 ceph-mon[145314]: from='client.34996 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:26 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1447776398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:21:26.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:26 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/954123764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:21:27.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:26 smithi137 ceph-mon[108145]: from='client.34988 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:26 smithi137 ceph-mon[108145]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:26 smithi137 ceph-mon[108145]: from='client.34996 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:26 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1447776398' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:21:27.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:26 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/954123764' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:21:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:27 smithi111 ceph-mon[145314]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:27 smithi111 ceph-mon[145314]: pgmap v520: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:27.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:27 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2037166639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:21:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:27 smithi137 ceph-mon[108145]: from='client.35008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:28.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:27 smithi137 ceph-mon[108145]: pgmap v520: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:28.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:27 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2037166639' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:21:29.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:29 smithi111 ceph-mon[145314]: pgmap v521: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:30.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:29 smithi137 ceph-mon[108145]: pgmap v521: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:31.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:31 smithi111 ceph-mon[145314]: pgmap v522: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:32.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:31 smithi137 ceph-mon[108145]: pgmap v522: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:33.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:33 smithi111 ceph-mon[145314]: pgmap v523: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:34.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:33 smithi137 ceph-mon[108145]: pgmap v523: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:35.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:35 smithi111 ceph-mon[145314]: pgmap v524: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:36.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:35 smithi137 ceph-mon[108145]: pgmap v524: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:37 smithi111 ceph-mon[145314]: pgmap v525: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:38.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:37 smithi137 ceph-mon[108145]: pgmap v525: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:39 smithi111 ceph-mon[145314]: pgmap v526: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:39.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:39 smithi137 ceph-mon[108145]: pgmap v526: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:40.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:41.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:41 smithi111 ceph-mon[145314]: pgmap v527: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:42.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:41 smithi137 ceph-mon[108145]: pgmap v527: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:43.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:43 smithi111 ceph-mon[145314]: pgmap v528: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:44.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:43 smithi137 ceph-mon[108145]: pgmap v528: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:45.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:45 smithi111 ceph-mon[145314]: pgmap v529: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:46.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:45 smithi137 ceph-mon[108145]: pgmap v529: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:47.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:47 smithi111 ceph-mon[145314]: pgmap v530: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:48.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:47 smithi137 ceph-mon[108145]: pgmap v530: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:49.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:49 smithi111 ceph-mon[145314]: pgmap v531: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:50.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:49 smithi137 ceph-mon[108145]: pgmap v531: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:51.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:51 smithi111 ceph-mon[145314]: pgmap v532: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:52.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:51 smithi137 ceph-mon[108145]: pgmap v532: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:53.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:53 smithi111 ceph-mon[145314]: pgmap v533: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:54.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:53 smithi137 ceph-mon[108145]: pgmap v533: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:54.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:55.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:21:55.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:55 smithi111 ceph-mon[145314]: pgmap v534: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:56.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:55 smithi137 ceph-mon[108145]: pgmap v534: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:56.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 -- 172.21.15.111:0/1587210905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070300 msgr2=0x7f65540706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:56.834 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 --2- 172.21.15.111:0/1587210905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070300 0x7f65540706d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6548004970 tx=0x7f6548009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 -- 172.21.15.111:0/1587210905 shutdown_connections 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 --2- 172.21.15.111:0/1587210905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070c10 0x7f6554109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 --2- 172.21.15.111:0/1587210905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070300 0x7f65540706d0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 -- 172.21.15.111:0/1587210905 >> 172.21.15.111:0/1587210905 conn(0x7f65540f8950 msgr2=0x7f65540fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 -- 172.21.15.111:0/1587210905 shutdown_connections 2024-03-14T13:21:56.835 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.836+0000 7f65599c4700 1 -- 172.21.15.111:0/1587210905 wait complete. 2024-03-14T13:21:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 Processor -- start 2024-03-14T13:21:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 -- start start 2024-03-14T13:21:56.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070300 0x7f6554115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65541165e0 con 0x7f6554070c10 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65599c4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655410fbc0 con 0x7f6554070300 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52928/0 (socket says 172.21.15.111:52928) 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 -- 172.21.15.111:0/3667777632 learned_addr learned my addr 172.21.15.111:0/3667777632 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f6552ffd700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070300 0x7f6554115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 -- 172.21.15.111:0/3667777632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070300 msgr2=0x7f6554115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070300 0x7f6554115a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 -- 172.21.15.111:0/3667777632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6548005040 con 0x7f6554070c10 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.837+0000 7f65527fc700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f654400eb70 tx=0x7f654400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.838+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6544007cc0 con 0x7f6554070c10 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.838+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655410fe40 con 0x7f6554070c10 2024-03-14T13:21:56.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.838+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f654400f040 con 0x7f6554070c10 2024-03-14T13:21:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.838+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f654400c9f0 con 0x7f6554070c10 2024-03-14T13:21:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.838+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6554110380 con 0x7f6554070c10 2024-03-14T13:21:56.838 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.839+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6554071a40 con 0x7f6554070c10 2024-03-14T13:21:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.839+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f654400cb50 con 0x7f6554070c10 2024-03-14T13:21:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.840+0000 7f653b7fe700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 0x7f653c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.840+0000 7f6552ffd700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 0x7f653c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.840+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f654408b660 con 0x7f6554070c10 2024-03-14T13:21:56.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.840+0000 7f6552ffd700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 0x7f653c06ec70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6548009fd0 tx=0x7f6548009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:56.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:56.845+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f65440592f0 con 0x7f6554070c10 2024-03-14T13:21:56.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:56 smithi111 ceph-mon[145314]: pgmap v535: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:57.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:56 smithi137 ceph-mon[108145]: pgmap v535: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:57.054 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.055+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f655410c020 con 0x7f653c06c7c0 2024-03-14T13:21:57.055 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.057+0000 7f653b7fe700 1 -- 172.21.15.111:0/3667777632 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f655410c020 con 0x7f653c06c7c0 2024-03-14T13:21:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 msgr2=0x7f653c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 0x7f653c06ec70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6548009fd0 tx=0x7f6548009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 msgr2=0x7f6554115f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f654400eb70 tx=0x7f654400b040 comp rx=0 tx=0).stop 2024-03-14T13:21:57.057 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 shutdown_connections 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f653c06c7c0 0x7f653c06ec70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6554070300 0x7f6554115a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 --2- 172.21.15.111:0/3667777632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6554070c10 0x7f6554115f90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 >> 172.21.15.111:0/3667777632 conn(0x7f65540f8950 msgr2=0x7f6554105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 shutdown_connections 2024-03-14T13:21:57.058 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.059+0000 7f65599c4700 1 -- 172.21.15.111:0/3667777632 wait complete. 2024-03-14T13:21:57.067 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 -- 172.21.15.111:0/1246726067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c102e10 msgr2=0x7fb24c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 --2- 172.21.15.111:0/1246726067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c102e10 0x7fb24c1031e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb240004940 tx=0x7fb240009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 -- 172.21.15.111:0/1246726067 shutdown_connections 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 --2- 172.21.15.111:0/1246726067 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c1037b0 0x7fb24c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 --2- 172.21.15.111:0/1246726067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c102e10 0x7fb24c1031e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 -- 172.21.15.111:0/1246726067 >> 172.21.15.111:0/1246726067 conn(0x7fb24c0f89a0 msgr2=0x7fb24c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 -- 172.21.15.111:0/1246726067 shutdown_connections 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.159+0000 7fb250b81700 1 -- 172.21.15.111:0/1246726067 wait complete. 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 Processor -- start 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 -- start start 2024-03-14T13:21:57.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 0x7fb24c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c114680 con 0x7fb24c1037b0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb250b81700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c10da70 con 0x7fb24c102e10 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52936/0 (socket says 172.21.15.111:52936) 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 -- 172.21.15.111:0/2571948009 learned_addr learned my addr 172.21.15.111:0/2571948009 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb24a59c700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 0x7fb24c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 -- 172.21.15.111:0/2571948009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 msgr2=0x7fb24c113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 0x7fb24c113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb249d9b700 1 -- 172.21.15.111:0/2571948009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb240005040 con 0x7fb24c1037b0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.160+0000 7fb24a59c700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 0x7fb24c113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb249d9b700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb23c002a50 tx=0x7fb23c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb23c007cc0 con 0x7fb24c1037b0 2024-03-14T13:21:57.161 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb23c00f040 con 0x7fb24c1037b0 2024-03-14T13:21:57.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c10dd50 con 0x7fb24c1037b0 2024-03-14T13:21:57.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb23c00d990 con 0x7fb24c1037b0 2024-03-14T13:21:57.162 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.161+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c10e310 con 0x7fb24c1037b0 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.162+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c0fe560 con 0x7fb24c1037b0 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.162+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb23c009710 con 0x7fb24c1037b0 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.163+0000 7fb23affd700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 0x7fb23406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.163+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb23c08b890 con 0x7fb24c1037b0 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.163+0000 7fb24a59c700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 0x7fb23406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.163+0000 7fb24a59c700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 0x7fb23406ec70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb240004560 tx=0x7fb240009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.167+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb23c059520 con 0x7fb24c1037b0 2024-03-14T13:21:57.378 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.380+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb24c1137c0 con 0x7fb23406c7c0 2024-03-14T13:21:57.379 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.381+0000 7fb23affd700 1 -- 172.21.15.111:0/2571948009 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb24c1137c0 con 0x7fb23406c7c0 2024-03-14T13:21:57.381 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 msgr2=0x7fb23406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 0x7fb23406ec70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb240004560 tx=0x7fb240009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 msgr2=0x7fb24c114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb23c002a50 tx=0x7fb23c004600 comp rx=0 tx=0).stop 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 shutdown_connections 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb23406c7c0 0x7fb23406ec70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb24c102e10 0x7fb24c113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 --2- 172.21.15.111:0/2571948009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb24c1037b0 0x7fb24c114030 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 >> 172.21.15.111:0/2571948009 conn(0x7fb24c0f89a0 msgr2=0x7fb24c0f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 shutdown_connections 2024-03-14T13:21:57.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.383+0000 7fb250b81700 1 -- 172.21.15.111:0/2571948009 wait complete. 2024-03-14T13:21:57.484 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 -- 172.21.15.111:0/2529513097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50101d70 msgr2=0x7fbe50102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2529513097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50101d70 0x7fbe50102140 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbe44004940 tx=0x7fbe44009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 -- 172.21.15.111:0/2529513097 shutdown_connections 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2529513097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50102710 0x7fbe500fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2529513097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50101d70 0x7fbe50102140 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 -- 172.21.15.111:0/2529513097 >> 172.21.15.111:0/2529513097 conn(0x7fbe50072bc0 msgr2=0x7fbe50072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 -- 172.21.15.111:0/2529513097 shutdown_connections 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.484+0000 7fbe582e4700 1 -- 172.21.15.111:0/2529513097 wait complete. 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 Processor -- start 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 -- start start 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50101d70 0x7fbe50112940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501134d0 con 0x7fbe50102710 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe582e4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5010c8c0 con 0x7fbe50101d70 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52948/0 (socket says 172.21.15.111:52948) 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 -- 172.21.15.111:0/2312339160 learned_addr learned my addr 172.21.15.111:0/2312339160 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe56080700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50101d70 0x7fbe50112940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.486 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 -- 172.21.15.111:0/2312339160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50101d70 msgr2=0x7fbe50112940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50101d70 0x7fbe50112940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.485+0000 7fbe5587f700 1 -- 172.21.15.111:0/2312339160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe44005040 con 0x7fbe50102710 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe5587f700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00eb40 tx=0x7fbe4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4c007cc0 con 0x7fbe50102710 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5010cb40 con 0x7fbe50102710 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe4c00f040 con 0x7fbe50102710 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe4c009510 con 0x7fbe50102710 2024-03-14T13:21:57.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.486+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5010d100 con 0x7fbe50102710 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.487+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe500ff080 con 0x7fbe50102710 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.487+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fbe4c009670 con 0x7fbe50102710 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.488+0000 7fbe42ffd700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 0x7fbe3c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.488+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbe4c08b6f0 con 0x7fbe50102710 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.488+0000 7fbe56080700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 0x7fbe3c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.488+0000 7fbe56080700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 0x7fbe3c06ec70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400b0d0 tx=0x7fbe44009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.492+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fbe4c059380 con 0x7fbe50102710 2024-03-14T13:21:57.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.695+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe5010e030 con 0x7fbe3c06c7c0 2024-03-14T13:21:57.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.705+0000 7fbe42ffd700 1 -- 172.21.15.111:0/2312339160 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fbe5010e030 con 0x7fbe3c06c7c0 2024-03-14T13:21:57.703 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:21:57.703 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (18m) 11m ago 29m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:21:57.703 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (15m) 11m ago 29m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:21:57.703 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (15m) 11m ago 28m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (17m) 11m ago 29m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (11m) 11m ago 23m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (12m) 11m ago 23m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (11m) 11m ago 23m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (11m) 11m ago 23m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (20m) 11m ago 31m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (19m) 11m ago 28m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (16m) 11m ago 31m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (16m) 11m ago 28m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (19m) 11m ago 29m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (18m) 11m ago 28m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (15m) 11m ago 27m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (14m) 11m ago 27m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:21:57.704 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (14m) 11m ago 26m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (13m) 11m ago 26m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (12m) 11m ago 25m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (12m) 11m ago 25m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (18m) 11m ago 28m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 msgr2=0x7fbe3c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 0x7fbe3c06ec70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400b0d0 tx=0x7fbe44009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 msgr2=0x7fbe50112e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c00eb40 tx=0x7fbe4c00b040 comp rx=0 tx=0).stop 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 shutdown_connections 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbe3c06c7c0 0x7fbe3c06ec70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe50101d70 0x7fbe50112940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 --2- 172.21.15.111:0/2312339160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe50102710 0x7fbe50112e80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 >> 172.21.15.111:0/2312339160 conn(0x7fbe50072bc0 msgr2=0x7fbe500faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 shutdown_connections 2024-03-14T13:21:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.707+0000 7fbe582e4700 1 -- 172.21.15.111:0/2312339160 wait complete. 2024-03-14T13:21:57.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4007357635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 msgr2=0x7fb058102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4007357635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102140 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb048004940 tx=0x7fb048009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:57.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4007357635 shutdown_connections 2024-03-14T13:21:57.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4007357635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb0580fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4007357635 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102140 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4007357635 >> 172.21.15.111:0/4007357635 conn(0x7fb058072bc0 msgr2=0x7fb058072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4007357635 shutdown_connections 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.806+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4007357635 wait complete. 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 Processor -- start 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 -- start start 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb058103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581036a0 con 0x7fb058101d70 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb05e8ae700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0581037e0 con 0x7fb058102710 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52978/0 (socket says 172.21.15.111:52978) 2024-03-14T13:21:57.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 -- 172.21.15.111:0/4032195824 learned_addr learned my addr 172.21.15.111:0/4032195824 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb04ffff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb058103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 -- 172.21.15.111:0/4032195824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 msgr2=0x7fb058103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb058103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 -- 172.21.15.111:0/4032195824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048005040 con 0x7fb058101d70 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb04ffff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb058103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.807+0000 7fb057fff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb04800b3b0 tx=0x7fb048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04800e040 con 0x7fb058101d70 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb048013420 con 0x7fb058101d70 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb058103a30 con 0x7fb058101d70 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04801c630 con 0x7fb058101d70 2024-03-14T13:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05811a1f0 con 0x7fb058101d70 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.808+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0580fd4c0 con 0x7fb058101d70 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.809+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb048013970 con 0x7fb058101d70 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.809+0000 7fb0557fa700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 0x7fb04406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.810+0000 7fb04ffff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 0x7fb04406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.810+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb04808ccc0 con 0x7fb058101d70 2024-03-14T13:21:57.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.810+0000 7fb04ffff700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 0x7fb04406ec70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb040009fd0 tx=0x7fb040009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:57.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:57.814+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb048056f40 con 0x7fb058101d70 2024-03-14T13:21:57.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:57 smithi111 ceph-mon[145314]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:57.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:57 smithi111 ceph-mon[145314]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:58.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:57 smithi137 ceph-mon[108145]: from='client.35016 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:58.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:57 smithi137 ceph-mon[108145]: from='client.35020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:58.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.083+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb0581042c0 con 0x7fb058101d70 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.083+0000 7fb0557fa700 1 -- 172.21.15.111:0/4032195824 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fb04805a560 con 0x7fb058101d70 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:21:58.082 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:21:58.083 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 msgr2=0x7fb04406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 0x7fb04406ec70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb040009fd0 tx=0x7fb040009ee0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 msgr2=0x7fb058102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb04800b3b0 tx=0x7fb048009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 shutdown_connections 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb04406c7c0 0x7fb04406ec70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb058101d70 0x7fb058102bd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 --2- 172.21.15.111:0/4032195824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb058102710 0x7fb058103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 >> 172.21.15.111:0/4032195824 conn(0x7fb058072bc0 msgr2=0x7fb0580659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 shutdown_connections 2024-03-14T13:21:58.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.085+0000 7fb05e8ae700 1 -- 172.21.15.111:0/4032195824 wait complete. 2024-03-14T13:21:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 -- 172.21.15.111:0/1308364953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28102710 msgr2=0x7f5a280fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/1308364953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28102710 0x7f5a280fbd80 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5a14004940 tx=0x7f5a14009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 -- 172.21.15.111:0/1308364953 shutdown_connections 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/1308364953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28102710 0x7f5a280fbd80 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/1308364953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28101d70 0x7f5a28102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 -- 172.21.15.111:0/1308364953 >> 172.21.15.111:0/1308364953 conn(0x7f5a28072bc0 msgr2=0x7f5a28072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 -- 172.21.15.111:0/1308364953 shutdown_connections 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.185+0000 7f5a2ed84700 1 -- 172.21.15.111:0/1308364953 wait complete. 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 Processor -- start 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 -- start start 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28102710 0x7f5a2810ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2810f5b0 con 0x7f5a28101d70 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2ed84700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2810f6f0 con 0x7f5a28102710 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2cb20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2cb20700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52996/0 (socket says 172.21.15.111:52996) 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a2cb20700 1 -- 172.21.15.111:0/2324066715 learned_addr learned my addr 172.21.15.111:0/2324066715 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.186+0000 7f5a27fff700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28102710 0x7f5a2810ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2cb20700 1 -- 172.21.15.111:0/2324066715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28102710 msgr2=0x7f5a2810ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2cb20700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28102710 0x7f5a2810ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2cb20700 1 -- 172.21.15.111:0/2324066715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14005040 con 0x7f5a28101d70 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2cb20700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00eb70 tx=0x7f5a1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1c007cc0 con 0x7f5a28101d70 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a1c00f040 con 0x7f5a28101d70 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a1c0095f0 con 0x7f5a28101d70 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a28117a90 con 0x7f5a28101d70 2024-03-14T13:21:58.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.187+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a28117fa0 con 0x7f5a28101d70 2024-03-14T13:21:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.188+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a28104240 con 0x7f5a28101d70 2024-03-14T13:21:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.188+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5a1c009750 con 0x7f5a28101d70 2024-03-14T13:21:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.189+0000 7f5a257fa700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 0x7f5a1806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.189+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5a1c08b6f0 con 0x7f5a28101d70 2024-03-14T13:21:58.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.189+0000 7f5a27fff700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 0x7f5a1806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.189+0000 7f5a27fff700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 0x7f5a1806ec70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5a28066000 tx=0x7f5a14009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.193+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5a1c0558f0 con 0x7f5a28101d70 2024-03-14T13:21:58.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.430+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f5a28108dc0 con 0x7f5a28101d70 2024-03-14T13:21:58.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.430+0000 7f5a257fa700 1 -- 172.21.15.111:0/2324066715 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f5a1c058f10 con 0x7f5a28101d70 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:21:58.429 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:21:58.430 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:58.431 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:21:58.432 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:21:58.432 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:58.432 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:58.432 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:58.432 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 msgr2=0x7f5a1806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 0x7f5a1806ec70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5a28066000 tx=0x7f5a14009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 msgr2=0x7f5a2810ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00eb70 tx=0x7f5a1c00b040 comp rx=0 tx=0).stop 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 shutdown_connections 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5a1806c7c0 0x7f5a1806ec70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a28101d70 0x7f5a2810ea20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 --2- 172.21.15.111:0/2324066715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a28102710 0x7f5a2810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 >> 172.21.15.111:0/2324066715 conn(0x7f5a28072bc0 msgr2=0x7f5a280fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 shutdown_connections 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.432+0000 7f5a2ed84700 1 -- 172.21.15.111:0/2324066715 wait complete. 2024-03-14T13:21:58.433 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- 172.21.15.111:0/2256136049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 msgr2=0x7f8b9c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 --2- 172.21.15.111:0/2256136049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c0fbd80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8b98004940 tx=0x7f8b98009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- 172.21.15.111:0/2256136049 shutdown_connections 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 --2- 172.21.15.111:0/2256136049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c0fbd80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 --2- 172.21.15.111:0/2256136049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 0x7f8b9c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- 172.21.15.111:0/2256136049 >> 172.21.15.111:0/2256136049 conn(0x7f8b9c072bc0 msgr2=0x7f8b9c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- 172.21.15.111:0/2256136049 shutdown_connections 2024-03-14T13:21:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- 172.21.15.111:0/2256136049 wait complete. 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 Processor -- start 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- start start 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 0x7f8b9c113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c114650 con 0x7f8b9c102710 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.533+0000 7f8ba356e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9c10da40 con 0x7f8b9c101d70 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53022/0 (socket says 172.21.15.111:53022) 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 -- 172.21.15.111:0/3526167651 learned_addr learned my addr 172.21.15.111:0/3526167651 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba130a700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 0x7f8b9c113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 -- 172.21.15.111:0/3526167651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 msgr2=0x7f8b9c113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 0x7f8b9c113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 -- 172.21.15.111:0/3526167651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98005040 con 0x7f8b9c102710 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba0b09700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009380 tx=0x7f8b98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9800e040 con 0x7f8b9c102710 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9c10dcc0 con 0x7f8b9c102710 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b98013420 con 0x7f8b9c102710 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b9801c630 con 0x7f8b9c102710 2024-03-14T13:21:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.534+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9c10e280 con 0x7f8b9c102710 2024-03-14T13:21:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.535+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9c0fd4c0 con 0x7f8b9c102710 2024-03-14T13:21:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.536+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8b980139e0 con 0x7f8b9c102710 2024-03-14T13:21:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.536+0000 7f8b91ffb700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.536+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f8b9808cc60 con 0x7f8b9c102710 2024-03-14T13:21:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.536+0000 7f8ba130a700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.537+0000 7f8ba130a700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c007fd0 tx=0x7f8b8c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.541+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8b9805a970 con 0x7f8b9c102710 2024-03-14T13:21:58.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.753+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b9c108d10 con 0x7f8b8806c7c0 2024-03-14T13:21:58.752 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.754+0000 7f8b91ffb700 1 -- 172.21.15.111:0/3526167651 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f8b9c108d10 con 0x7f8b8806c7c0 2024-03-14T13:21:58.752 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:21:58.752 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:21:58.752 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:21:58.752 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:21:58.753 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:21:58.754 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 msgr2=0x7f8b8806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c007fd0 tx=0x7f8b8c009040 comp rx=0 tx=0).stop 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 msgr2=0x7f8b9c114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009380 tx=0x7f8b98009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 shutdown_connections 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f8b8806c7c0 0x7f8b8806ec70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b9c101d70 0x7f8b9c113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 --2- 172.21.15.111:0/3526167651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b9c102710 0x7f8b9c114000 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 >> 172.21.15.111:0/3526167651 conn(0x7f8b9c072bc0 msgr2=0x7f8b9c065a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 shutdown_connections 2024-03-14T13:21:58.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.756+0000 7f8ba356e700 1 -- 172.21.15.111:0/3526167651 wait complete. 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 -- 172.21.15.111:0/3934282386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000fb4d0 msgr2=0x7f64000fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 --2- 172.21.15.111:0/3934282386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000fb4d0 0x7f64000fb940 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f63ec004940 tx=0x7f63ec009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 -- 172.21.15.111:0/3934282386 shutdown_connections 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 --2- 172.21.15.111:0/3934282386 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000fb4d0 0x7f64000fb940 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 --2- 172.21.15.111:0/3934282386 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6400102f50 0x7f6400103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 -- 172.21.15.111:0/3934282386 >> 172.21.15.111:0/3934282386 conn(0x7f64000714c0 msgr2=0x7f64000718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 -- 172.21.15.111:0/3934282386 shutdown_connections 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.855+0000 7f6406af0700 1 -- 172.21.15.111:0/3934282386 wait complete. 2024-03-14T13:21:58.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 Processor -- start 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 -- start start 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 0x7f6400113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64001145a0 con 0x7f6400102f50 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f6406af0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640010db30 con 0x7f64000fb4d0 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f63fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f63fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53036/0 (socket says 172.21.15.111:53036) 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.856+0000 7f63fffff700 1 -- 172.21.15.111:0/3892336065 learned_addr learned my addr 172.21.15.111:0/3892336065 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fffff700 1 -- 172.21.15.111:0/3892336065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 msgr2=0x7f6400113980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f640488c700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 0x7f6400113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fffff700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 0x7f6400113980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fffff700 1 -- 172.21.15.111:0/3892336065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec005040 con 0x7f6400102f50 2024-03-14T13:21:58.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f640488c700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 0x7f6400113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fffff700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009180 tx=0x7f63ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63ec00e040 con 0x7f6400102f50 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640010ddb0 con 0x7f6400102f50 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63ec004290 con 0x7f6400102f50 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63ec01c3f0 con 0x7f6400102f50 2024-03-14T13:21:58.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.857+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640010e2f0 con 0x7f6400102f50 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.858+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6400100200 con 0x7f6400102f50 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.859+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f63ec013790 con 0x7f6400102f50 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.859+0000 7f63fd7fa700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 0x7f63f006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.859+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f63ec08cd70 con 0x7f6400102f50 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.859+0000 7f640488c700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 0x7f63f006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:21:58.859 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.860+0000 7f640488c700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 0x7f63f006ec70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006fd0 tx=0x7f63f4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:21:58.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:58.864+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f63ec056fb0 con 0x7f6400102f50 2024-03-14T13:21:59.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:58 smithi137 ceph-mon[108145]: from='client.35024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:58 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4032195824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:21:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:58 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2324066715' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:21:59.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:58 smithi137 ceph-mon[108145]: pgmap v536: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:21:59.129 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.131+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6400100200 con 0x7f6400102f50 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.131+0000 7f63fd7fa700 1 -- 172.21.15.111:0/3892336065 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f63ec013420 con 0x7f6400102f50 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:21:59.130 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:21:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 msgr2=0x7f63f006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:59.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 0x7f63f006ec70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006fd0 tx=0x7f63f4008040 comp rx=0 tx=0).stop 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 msgr2=0x7f6400113ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009180 tx=0x7f63ec009e30 comp rx=0 tx=0).stop 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 shutdown_connections 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f006c7c0 0x7f63f006ec70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64000fb4d0 0x7f6400113980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 --2- 172.21.15.111:0/3892336065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6400102f50 0x7f6400113ec0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 >> 172.21.15.111:0/3892336065 conn(0x7f64000714c0 msgr2=0x7f64000fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 shutdown_connections 2024-03-14T13:21:59.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:21:59.133+0000 7f6406af0700 1 -- 172.21.15.111:0/3892336065 wait complete. 2024-03-14T13:21:59.138 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:58 smithi111 ceph-mon[145314]: from='client.35024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:21:59.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:58 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4032195824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:21:59.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:58 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2324066715' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:21:59.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:58 smithi111 ceph-mon[145314]: pgmap v536: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:00.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:59 smithi137 ceph-mon[108145]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:00.005 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:21:59 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3892336065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:22:00.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:59 smithi111 ceph-mon[145314]: from='client.35036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:00.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:21:59 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3892336065' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:22:01.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:00 smithi137 ceph-mon[108145]: pgmap v537: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:01.139 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:00 smithi111 ceph-mon[145314]: pgmap v537: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:03.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:03 smithi111 ceph-mon[145314]: pgmap v538: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:04.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:03 smithi137 ceph-mon[108145]: pgmap v538: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:05.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:05 smithi111 ceph-mon[145314]: pgmap v539: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:06.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:05 smithi137 ceph-mon[108145]: pgmap v539: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:07.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:07 smithi111 ceph-mon[145314]: pgmap v540: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:08.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:07 smithi137 ceph-mon[108145]: pgmap v540: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:09.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:09 smithi111 ceph-mon[145314]: pgmap v541: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:09.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:10.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:09 smithi137 ceph-mon[108145]: pgmap v541: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:10.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:11.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:11 smithi111 ceph-mon[145314]: pgmap v542: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:12.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:11 smithi137 ceph-mon[108145]: pgmap v542: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:13.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:13 smithi111 ceph-mon[145314]: pgmap v543: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:14.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:13 smithi137 ceph-mon[108145]: pgmap v543: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:15.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:15 smithi111 ceph-mon[145314]: pgmap v544: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:16.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:15 smithi137 ceph-mon[108145]: pgmap v544: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:17.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:17 smithi111 ceph-mon[145314]: pgmap v545: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:18.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:17 smithi137 ceph-mon[108145]: pgmap v545: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:19.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:19 smithi111 ceph-mon[145314]: pgmap v546: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:20.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:19 smithi137 ceph-mon[108145]: pgmap v546: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:21.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:21 smithi111 ceph-mon[145314]: pgmap v547: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:22.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:21 smithi137 ceph-mon[108145]: pgmap v547: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:23.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:23 smithi111 ceph-mon[145314]: pgmap v548: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:24.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:23 smithi137 ceph-mon[108145]: pgmap v548: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:24.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:25.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:25.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:25 smithi111 ceph-mon[145314]: pgmap v549: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:26.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:25 smithi137 ceph-mon[108145]: pgmap v549: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:27.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:27 smithi111 ceph-mon[145314]: pgmap v550: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:28.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:27 smithi137 ceph-mon[108145]: pgmap v550: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 -- 172.21.15.111:0/3583391936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fe820 msgr2=0x7fe4b0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 --2- 172.21.15.111:0/3583391936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fe820 0x7fe4b0106e60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4004940 tx=0x7fe4a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 -- 172.21.15.111:0/3583391936 shutdown_connections 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 --2- 172.21.15.111:0/3583391936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fe820 0x7fe4b0106e60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 --2- 172.21.15.111:0/3583391936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fde80 0x7fe4b00fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 -- 172.21.15.111:0/3583391936 >> 172.21.15.111:0/3583391936 conn(0x7fe4b00f78d0 msgr2=0x7fe4b00f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:29.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 -- 172.21.15.111:0/3583391936 shutdown_connections 2024-03-14T13:22:29.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.245+0000 7fe4b682b700 1 -- 172.21.15.111:0/3583391936 wait complete. 2024-03-14T13:22:29.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 Processor -- start 2024-03-14T13:22:29.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 -- start start 2024-03-14T13:22:29.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fe820 0x7fe4b0113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.246 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b01146a0 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4b682b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b010da00 con 0x7fe4b00fe820 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33442/0 (socket says 172.21.15.111:33442) 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.246+0000 7fe4affff700 1 -- 172.21.15.111:0/2234897713 learned_addr learned my addr 172.21.15.111:0/2234897713 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4af7fe700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fe820 0x7fe4b0113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4affff700 1 -- 172.21.15.111:0/2234897713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fe820 msgr2=0x7fe4b0113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4affff700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fe820 0x7fe4b0113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4affff700 1 -- 172.21.15.111:0/2234897713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4005040 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4affff700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000b790 tx=0x7fe4a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a000f770 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4a000fd80 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b010dce0 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4a000d310 con 0x7fe4b00fde80 2024-03-14T13:22:29.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.247+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b010e220 con 0x7fe4b00fde80 2024-03-14T13:22:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.248+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b0066710 con 0x7fe4b00fde80 2024-03-14T13:22:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.249+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe4a0010420 con 0x7fe4b00fde80 2024-03-14T13:22:29.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.249+0000 7fe4acff9700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 0x7fe49806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.249+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4a008ba80 con 0x7fe4b00fde80 2024-03-14T13:22:29.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.249+0000 7fe4af7fe700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 0x7fe49806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.249 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.249+0000 7fe4af7fe700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 0x7fe49806ec80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe4b010ebc0 tx=0x7fe4a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.252 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.254+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe4a0059790 con 0x7fe4b00fde80 2024-03-14T13:22:29.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.467+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4b0109df0 con 0x7fe49806c7d0 2024-03-14T13:22:29.467 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.468+0000 7fe4acff9700 1 -- 172.21.15.111:0/2234897713 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe4b0109df0 con 0x7fe49806c7d0 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 msgr2=0x7fe49806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 0x7fe49806ec80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe4b010ebc0 tx=0x7fe4a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 msgr2=0x7fe4b0113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000b790 tx=0x7fe4a0009250 comp rx=0 tx=0).stop 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 shutdown_connections 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe49806c7d0 0x7fe49806ec80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4b00fde80 0x7fe4b0113a80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 --2- 172.21.15.111:0/2234897713 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4b00fe820 0x7fe4b0113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.469 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 >> 172.21.15.111:0/2234897713 conn(0x7fe4b00f78d0 msgr2=0x7fe4b00f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:29.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 shutdown_connections 2024-03-14T13:22:29.470 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.470+0000 7fe4b682b700 1 -- 172.21.15.111:0/2234897713 wait complete. 2024-03-14T13:22:29.480 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:22:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 -- 172.21.15.111:0/2723715527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8101d70 msgr2=0x7f1dc8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2723715527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8101d70 0x7f1dc8102140 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1db8004940 tx=0x7f1db8009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 -- 172.21.15.111:0/2723715527 shutdown_connections 2024-03-14T13:22:29.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2723715527 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8102710 0x7f1dc80fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2723715527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8101d70 0x7f1dc8102140 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 -- 172.21.15.111:0/2723715527 >> 172.21.15.111:0/2723715527 conn(0x7f1dc8072bc0 msgr2=0x7f1dc8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 -- 172.21.15.111:0/2723715527 shutdown_connections 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 -- 172.21.15.111:0/2723715527 wait complete. 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.572+0000 7f1dcd021700 1 Processor -- start 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 -- start start 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8101d70 0x7f1dc810c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc810d2d0 con 0x7f1dc8102710 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc810d440 con 0x7f1dc8101d70 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33464/0 (socket says 172.21.15.111:33464) 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 -- 172.21.15.111:0/2222197143 learned_addr learned my addr 172.21.15.111:0/2222197143 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:29.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc6d9d700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8101d70 0x7f1dc810c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 -- 172.21.15.111:0/2222197143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8101d70 msgr2=0x7f1dc810c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8101d70 0x7f1dc810c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 -- 172.21.15.111:0/2222197143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8005040 con 0x7f1dc8102710 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dc659c700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc002a80 tx=0x7f1dbc004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc007cc0 con 0x7f1dc8102710 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dbc00f040 con 0x7f1dc8102710 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc8119ec0 con 0x7f1dc8102710 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc00da40 con 0x7f1dc8102710 2024-03-14T13:22:29.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.573+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc811a290 con 0x7f1dc8102710 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.574+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc80ff080 con 0x7f1dc8102710 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.575+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1dbc00dbe0 con 0x7f1dc8102710 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.575+0000 7f1daf7fe700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 0x7f1db006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.575+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1dbc08b0a0 con 0x7f1dc8102710 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.575+0000 7f1dc6d9d700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 0x7f1db006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.576+0000 7f1dc6d9d700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 0x7f1db006ec80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1db800b020 tx=0x7f1db8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.580+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1dbc058db0 con 0x7f1dc8102710 2024-03-14T13:22:29.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.790+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dc8108d50 con 0x7f1db006c7d0 2024-03-14T13:22:29.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.792+0000 7f1daf7fe700 1 -- 172.21.15.111:0/2222197143 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f1dc8108d50 con 0x7f1db006c7d0 2024-03-14T13:22:29.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.793+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 msgr2=0x7f1db006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 0x7f1db006ec80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1db800b020 tx=0x7f1db8009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:29.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 msgr2=0x7f1dc810cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc002a80 tx=0x7f1dbc004630 comp rx=0 tx=0).stop 2024-03-14T13:22:29.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 shutdown_connections 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1db006c7d0 0x7f1db006ec80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1dc8101d70 0x7f1dc810c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 --2- 172.21.15.111:0/2222197143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1dc8102710 0x7f1dc810cd90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 >> 172.21.15.111:0/2222197143 conn(0x7f1dc8072bc0 msgr2=0x7f1dc80fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 shutdown_connections 2024-03-14T13:22:29.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.794+0000 7f1dcd021700 1 -- 172.21.15.111:0/2222197143 wait complete. 2024-03-14T13:22:29.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:29 smithi111 ceph-mon[145314]: pgmap v551: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:29.892 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/3598829899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b41037b0 msgr2=0x7fb1b40fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.892 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/3598829899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b41037b0 0x7fb1b40fce20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4004940 tx=0x7fb1a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/3598829899 shutdown_connections 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/3598829899 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b41037b0 0x7fb1b40fce20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/3598829899 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b4102e10 0x7fb1b41031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/3598829899 >> 172.21.15.111:0/3598829899 conn(0x7fb1b40f8980 msgr2=0x7fb1b40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/3598829899 shutdown_connections 2024-03-14T13:22:29.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.894+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/3598829899 wait complete. 2024-03-14T13:22:29.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 Processor -- start 2024-03-14T13:22:29.894 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 -- start start 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b41037b0 0x7fb1b4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b4114670 con 0x7fb1b4102e10 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1bbc4b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b410da60 con 0x7fb1b41037b0 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33484/0 (socket says 172.21.15.111:33484) 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 -- 172.21.15.111:0/4175584562 learned_addr learned my addr 172.21.15.111:0/4175584562 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b91e6700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b41037b0 0x7fb1b4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 -- 172.21.15.111:0/4175584562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b41037b0 msgr2=0x7fb1b4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b41037b0 0x7fb1b4114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 -- 172.21.15.111:0/4175584562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a4005040 con 0x7fb1b4102e10 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1b99e7700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0002a50 tx=0x7fb1b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.895+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1b0007cc0 con 0x7fb1b4102e10 2024-03-14T13:22:29.895 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.896+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b410dd40 con 0x7fb1b4102e10 2024-03-14T13:22:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.896+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1b000f040 con 0x7fb1b4102e10 2024-03-14T13:22:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.896+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1b000d9c0 con 0x7fb1b4102e10 2024-03-14T13:22:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.896+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b410e2d0 con 0x7fb1b4102e10 2024-03-14T13:22:29.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.896+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b40fe560 con 0x7fb1b4102e10 2024-03-14T13:22:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.897+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb1b000db20 con 0x7fb1b4102e10 2024-03-14T13:22:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.897+0000 7fb1aa7fc700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 0x7fb1a006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.897+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb1b008b7a0 con 0x7fb1b4102e10 2024-03-14T13:22:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.897+0000 7fb1b91e6700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 0x7fb1a006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:29.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.898+0000 7fb1b91e6700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 0x7fb1a006ec70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4004890 tx=0x7fb1a4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:29.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:29.902+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb1b0059430 con 0x7fb1b4102e10 2024-03-14T13:22:30.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:29 smithi137 ceph-mon[108145]: pgmap v551: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:30.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.104+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1b4109e00 con 0x7fb1a006c7c0 2024-03-14T13:22:30.112 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.114+0000 7fb1aa7fc700 1 -- 172.21.15.111:0/4175584562 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fb1b4109e00 con 0x7fb1a006c7c0 2024-03-14T13:22:30.112 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:22:30.112 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (18m) 12m ago 30m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (16m) 12m ago 30m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (16m) 11m ago 29m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (18m) 12m ago 29m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (12m) 12m ago 24m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (12m) 12m ago 24m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (11m) 11m ago 24m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (12m) 11m ago 24m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (21m) 12m ago 31m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (19m) 11m ago 28m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (17m) 12m ago 31m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (16m) 11m ago 28m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (19m) 12m ago 30m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (19m) 11m ago 28m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:22:30.113 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (15m) 12m ago 28m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (15m) 12m ago 27m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (14m) 12m ago 27m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (13m) 11m ago 26m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (13m) 11m ago 26m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (12m) 11m ago 25m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:22:30.114 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (19m) 12m ago 29m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.115+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 msgr2=0x7fb1a006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.115+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 0x7fb1a006ec70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4004890 tx=0x7fb1a4009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.115+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 msgr2=0x7fb1b4113ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.115+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0002a50 tx=0x7fb1b00046b0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 shutdown_connections 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fb1a006c7c0 0x7fb1a006ec70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1b4102e10 0x7fb1b4113ae0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 --2- 172.21.15.111:0/4175584562 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1b41037b0 0x7fb1b4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 >> 172.21.15.111:0/4175584562 conn(0x7fb1b40f8980 msgr2=0x7fb1b40f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 shutdown_connections 2024-03-14T13:22:30.115 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.116+0000 7fb1bbc4b700 1 -- 172.21.15.111:0/4175584562 wait complete. 2024-03-14T13:22:30.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 -- 172.21.15.111:0/1668377090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 msgr2=0x7f30540fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 --2- 172.21.15.111:0/1668377090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f30540fe260 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3048004940 tx=0x7f3048009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 -- 172.21.15.111:0/1668377090 shutdown_connections 2024-03-14T13:22:30.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 --2- 172.21.15.111:0/1668377090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30540fe830 0x7f3054106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.215 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 --2- 172.21.15.111:0/1668377090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f30540fe260 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 -- 172.21.15.111:0/1668377090 >> 172.21.15.111:0/1668377090 conn(0x7f3054072bc0 msgr2=0x7f3054072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 -- 172.21.15.111:0/1668377090 shutdown_connections 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.214+0000 7f3058ea5700 1 -- 172.21.15.111:0/1668377090 wait complete. 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 Processor -- start 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 -- start start 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30540fe830 0x7f3054114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3054114710 con 0x7f30540fde90 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f3058ea5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305410da70 con 0x7f30540fe830 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:33502/0 (socket says 172.21.15.111:33502) 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 -- 172.21.15.111:0/163656994 learned_addr learned my addr 172.21.15.111:0/163656994 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 -- 172.21.15.111:0/163656994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30540fe830 msgr2=0x7f3054114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.216 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30540fe830 0x7f3054114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 -- 172.21.15.111:0/163656994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048005040 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.215+0000 7f305259c700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f304800b3b0 tx=0x7f3048009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.216+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f304800e040 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.216+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3048013420 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.216+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305410dcf0 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.216+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f304801c5d0 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.216+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305410e230 con 0x7f30540fde90 2024-03-14T13:22:30.217 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.217+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3054065e30 con 0x7f30540fde90 2024-03-14T13:22:30.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.217+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3048013590 con 0x7f30540fde90 2024-03-14T13:22:30.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.217+0000 7f3042ffd700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 0x7f303c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.218+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f304808cc10 con 0x7f30540fde90 2024-03-14T13:22:30.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.218+0000 7f3051d9b700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 0x7f303c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.218+0000 7f3051d9b700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 0x7f303c06ec70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3044006fd0 tx=0x7f3044006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.222+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f304805a920 con 0x7f30540fde90 2024-03-14T13:22:30.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.494+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3054109e40 con 0x7f30540fde90 2024-03-14T13:22:30.493 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.495+0000 7f3042ffd700 1 -- 172.21.15.111:0/163656994 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f304805a4b0 con 0x7f30540fde90 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:22:30.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:22:30.495 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:22:30.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 msgr2=0x7f303c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 0x7f303c06ec70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3044006fd0 tx=0x7f3044006db0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 msgr2=0x7f3054113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.496 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f304800b3b0 tx=0x7f3048009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 shutdown_connections 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f303c06c7c0 0x7f303c06ec70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30540fde90 0x7f3054113af0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 --2- 172.21.15.111:0/163656994 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30540fe830 0x7f3054114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 >> 172.21.15.111:0/163656994 conn(0x7f3054072bc0 msgr2=0x7f30541037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 shutdown_connections 2024-03-14T13:22:30.497 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.497+0000 7f3058ea5700 1 -- 172.21.15.111:0/163656994 wait complete. 2024-03-14T13:22:30.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 -- 172.21.15.111:0/1554532711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86501026b0 msgr2=0x7f86500fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 --2- 172.21.15.111:0/1554532711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86501026b0 0x7f86500fbd20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8644004940 tx=0x7f8644009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 -- 172.21.15.111:0/1554532711 shutdown_connections 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 --2- 172.21.15.111:0/1554532711 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f86501026b0 0x7f86500fbd20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 --2- 172.21.15.111:0/1554532711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8650101d10 0x7f86501020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 -- 172.21.15.111:0/1554532711 >> 172.21.15.111:0/1554532711 conn(0x7f86500f7880 msgr2=0x7f86500f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 -- 172.21.15.111:0/1554532711 shutdown_connections 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.596+0000 7f8655970700 1 -- 172.21.15.111:0/1554532711 wait complete. 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 Processor -- start 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 -- start start 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 0x7f865010ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865010f4d0 con 0x7f8650101d10 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f8655970700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865010f610 con 0x7f86501026b0 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f864effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f864effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43962/0 (socket says 172.21.15.111:43962) 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f864effd700 1 -- 172.21.15.111:0/1387736004 learned_addr learned my addr 172.21.15.111:0/1387736004 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.597+0000 7f864e7fc700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 0x7f865010ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f864effd700 1 -- 172.21.15.111:0/1387736004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 msgr2=0x7f865010ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f864effd700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 0x7f865010ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f864effd700 1 -- 172.21.15.111:0/1387736004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8644005040 con 0x7f8650101d10 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f864e7fc700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 0x7f865010ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f864effd700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8640002a50 tx=0x7f86400046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8640007cc0 con 0x7f8650101d10 2024-03-14T13:22:30.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f864000f040 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8640009540 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8650117ac0 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.598+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8650117e90 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.599+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86500fd460 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.599+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f86400096a0 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.600+0000 7f86377fe700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 0x7f863806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.600+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f864008b7a0 con 0x7f8650101d10 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.600+0000 7f864e7fc700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 0x7f863806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.600+0000 7f864e7fc700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 0x7f863806ec70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8650066ac0 tx=0x7f8644009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.605+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8640055a20 con 0x7f8650101d10 2024-03-14T13:22:30.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.838+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f86500668a0 con 0x7f8650101d10 2024-03-14T13:22:30.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.838+0000 7f86377fe700 1 -- 172.21.15.111:0/1387736004 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f8640059040 con 0x7f8650101d10 2024-03-14T13:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:22:30.838 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:22:30.839 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:22:30.840 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.840+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 msgr2=0x7f863806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.840+0000 7f8655970700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 0x7f863806ec70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8650066ac0 tx=0x7f8644009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.840+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 msgr2=0x7f865010ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.840+0000 7f8655970700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8640002a50 tx=0x7f86400046b0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 shutdown_connections 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f863806c7c0 0x7f863806ec70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8650101d10 0x7f865010ea50 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 --2- 172.21.15.111:0/1387736004 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86501026b0 0x7f865010ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 >> 172.21.15.111:0/1387736004 conn(0x7f86500f7880 msgr2=0x7f86500f8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 shutdown_connections 2024-03-14T13:22:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.841+0000 7f8655970700 1 -- 172.21.15.111:0/1387736004 wait complete. 2024-03-14T13:22:30.842 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:22:30.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:30 smithi111 ceph-mon[145314]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:30.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:30 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/163656994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 -- 172.21.15.111:0/2465836761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 msgr2=0x7f68ec1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 --2- 172.21.15.111:0/2465836761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec1031b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f68dc004940 tx=0x7f68dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 -- 172.21.15.111:0/2465836761 shutdown_connections 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 --2- 172.21.15.111:0/2465836761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 0x7f68ec0fcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 --2- 172.21.15.111:0/2465836761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec1031b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 -- 172.21.15.111:0/2465836761 >> 172.21.15.111:0/2465836761 conn(0x7f68ec0f8950 msgr2=0x7f68ec0fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 -- 172.21.15.111:0/2465836761 shutdown_connections 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.941+0000 7f68f172a700 1 -- 172.21.15.111:0/2465836761 wait complete. 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 Processor -- start 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 -- start start 2024-03-14T13:22:30.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 0x7f68ec070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec072210 con 0x7f68ec102de0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68f172a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec070d40 con 0x7f68ec103780 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43970/0 (socket says 172.21.15.111:43970) 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 -- 172.21.15.111:0/4217880375 learned_addr learned my addr 172.21.15.111:0/4217880375 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68ea7fc700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 0x7f68ec070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 -- 172.21.15.111:0/4217880375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 msgr2=0x7f68ec070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 0x7f68ec070800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.942+0000 7f68eaffd700 1 -- 172.21.15.111:0/4217880375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68dc005040 con 0x7f68ec102de0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68eaffd700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00b3b0 tx=0x7f68dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68dc00e040 con 0x7f68ec102de0 2024-03-14T13:22:30.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68dc013420 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec070fc0 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68dc01c610 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.943+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec11a000 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.944+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec0fe530 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.944+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f68dc013a00 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.945+0000 7f68e37fe700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 0x7f68d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.945+0000 7f68ea7fc700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 0x7f68d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.945+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f68dc08cc10 con 0x7f68ec102de0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.945+0000 7f68ea7fc700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 0x7f68d806ec70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0719f0 tx=0x7f68d400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:30.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:30.949+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f68dc056e90 con 0x7f68ec102de0 2024-03-14T13:22:31.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:30 smithi137 ceph-mon[108145]: from='client.35044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:31.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:30 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/163656994' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:22:31.156 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.158+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68ec0fa0b0 con 0x7f68d806c7c0 2024-03-14T13:22:31.157 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.159+0000 7f68e37fe700 1 -- 172.21.15.111:0/4217880375 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f68ec0fa0b0 con 0x7f68d806c7c0 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:22:31.158 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:22:31.159 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:22:31.159 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:22:31.159 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:22:31.159 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 msgr2=0x7f68d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 0x7f68d806ec70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0719f0 tx=0x7f68d400b040 comp rx=0 tx=0).stop 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 msgr2=0x7f68ec0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00b3b0 tx=0x7f68dc009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 shutdown_connections 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68d806c7c0 0x7f68d806ec70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68ec102de0 0x7f68ec0702c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 --2- 172.21.15.111:0/4217880375 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ec103780 0x7f68ec070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 >> 172.21.15.111:0/4217880375 conn(0x7f68ec0f8950 msgr2=0x7f68ec0f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 shutdown_connections 2024-03-14T13:22:31.160 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.161+0000 7f68f172a700 1 -- 172.21.15.111:0/4217880375 wait complete. 2024-03-14T13:22:31.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 -- 172.21.15.111:0/3301411561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a4102f50 msgr2=0x7f51a4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/3301411561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a4102f50 0x7f51a4103320 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5194004940 tx=0x7f5194009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:31.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 -- 172.21.15.111:0/3301411561 shutdown_connections 2024-03-14T13:22:31.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/3301411561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a40fb4d0 0x7f51a40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/3301411561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a4102f50 0x7f51a4103320 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.261+0000 7f51a8b4d700 1 -- 172.21.15.111:0/3301411561 >> 172.21.15.111:0/3301411561 conn(0x7f51a40714c0 msgr2=0x7f51a40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 -- 172.21.15.111:0/3301411561 shutdown_connections 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 -- 172.21.15.111:0/3301411561 wait complete. 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 Processor -- start 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 -- start start 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 0x7f51a4113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a41145c0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a8b4d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a410db50 con 0x7f51a4102f50 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:44002/0 (socket says 172.21.15.111:44002) 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a259c700 1 -- 172.21.15.111:0/484024141 learned_addr learned my addr 172.21.15.111:0/484024141 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:22:31.262 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.262+0000 7f51a1d9b700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 0x7f51a4113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a259c700 1 -- 172.21.15.111:0/484024141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 msgr2=0x7f51a4113ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a259c700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 0x7f51a4113ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a259c700 1 -- 172.21.15.111:0/484024141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5194005040 con 0x7f51a40fb4d0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a1d9b700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 0x7f51a4113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a259c700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5194009390 tx=0x7f5194009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519400e040 con 0x7f51a40fb4d0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a410ddd0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5194013420 con 0x7f51a40fb4d0 2024-03-14T13:22:31.263 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f519401c630 con 0x7f51a40fb4d0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.263+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a410e310 con 0x7f51a40fb4d0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.264+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a4100200 con 0x7f51a40fb4d0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.264+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f51940139e0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.265+0000 7f5192ffd700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 0x7f518c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.265+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f519408cd20 con 0x7f51a40fb4d0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.265+0000 7f51a1d9b700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 0x7f518c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:22:31.264 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.265+0000 7f51a1d9b700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 0x7f518c06ec70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51a410f240 tx=0x7f5198008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:22:31.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.270+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5194056fa0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.552+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f51a4109fe0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.552+0000 7f5192ffd700 1 -- 172.21.15.111:0/484024141 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f519405a5c0 con 0x7f51a40fb4d0 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:22:31.551 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 msgr2=0x7f518c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 0x7f518c06ec70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f51a410f240 tx=0x7f5198008040 comp rx=0 tx=0).stop 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 msgr2=0x7f51a41139a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5194009390 tx=0x7f5194009e30 comp rx=0 tx=0).stop 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 shutdown_connections 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f518c06c7c0 0x7f518c06ec70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51a40fb4d0 0x7f51a41139a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 --2- 172.21.15.111:0/484024141 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51a4102f50 0x7f51a4113ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 >> 172.21.15.111:0/484024141 conn(0x7f51a40714c0 msgr2=0x7f51a40fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 shutdown_connections 2024-03-14T13:22:31.553 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:22:31.554+0000 7f51a8b4d700 1 -- 172.21.15.111:0/484024141 wait complete. 2024-03-14T13:22:31.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:31 smithi111 ceph-mon[145314]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:31.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:31 smithi111 ceph-mon[145314]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:31.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:31 smithi111 ceph-mon[145314]: pgmap v552: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:31.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:31 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1387736004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:22:31.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:31 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/484024141' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:22:32.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:31 smithi137 ceph-mon[108145]: from='client.35048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:32.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:31 smithi137 ceph-mon[108145]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:32.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:31 smithi137 ceph-mon[108145]: pgmap v552: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:32.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:31 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1387736004' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:22:32.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:31 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/484024141' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:22:32.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:32 smithi111 ceph-mon[145314]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:33.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:32 smithi137 ceph-mon[108145]: from='client.35064 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:22:33.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:33 smithi111 ceph-mon[145314]: pgmap v553: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:34.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:33 smithi137 ceph-mon[108145]: pgmap v553: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:35.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:35 smithi111 ceph-mon[145314]: pgmap v554: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:36.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:35 smithi137 ceph-mon[108145]: pgmap v554: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:37.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:37 smithi111 ceph-mon[145314]: pgmap v555: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:38.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:37 smithi137 ceph-mon[108145]: pgmap v555: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:39.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:39 smithi111 ceph-mon[145314]: pgmap v556: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:39.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:40.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:39 smithi137 ceph-mon[108145]: pgmap v556: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:40.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:41.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:41 smithi111 ceph-mon[145314]: pgmap v557: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:42.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:41 smithi137 ceph-mon[108145]: pgmap v557: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:43.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:43 smithi111 ceph-mon[145314]: pgmap v558: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:44.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:43 smithi137 ceph-mon[108145]: pgmap v558: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:45.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:45 smithi111 ceph-mon[145314]: pgmap v559: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:46.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:45 smithi137 ceph-mon[108145]: pgmap v559: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:47.888 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:47 smithi111 ceph-mon[145314]: pgmap v560: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:48.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:47 smithi137 ceph-mon[108145]: pgmap v560: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:49.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:49 smithi111 ceph-mon[145314]: pgmap v561: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:50.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:49 smithi137 ceph-mon[108145]: pgmap v561: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:51.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:51 smithi111 ceph-mon[145314]: pgmap v562: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:52.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:51 smithi137 ceph-mon[108145]: pgmap v562: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:53.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:53 smithi111 ceph-mon[145314]: pgmap v563: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:54.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:53 smithi137 ceph-mon[108145]: pgmap v563: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:54.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:55.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:22:55.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:55 smithi111 ceph-mon[145314]: pgmap v564: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:56.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:55 smithi137 ceph-mon[108145]: pgmap v564: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:57.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:57 smithi111 ceph-mon[145314]: pgmap v565: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:58.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:57 smithi137 ceph-mon[108145]: pgmap v565: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:22:59.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:22:59 smithi111 ceph-mon[145314]: pgmap v566: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:00.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:22:59 smithi137 ceph-mon[108145]: pgmap v566: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.665+0000 7f9c968ea700 1 -- 172.21.15.111:0/2008009177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c901026b0 msgr2=0x7f9c900fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 --2- 172.21.15.111:0/2008009177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c901026b0 0x7f9c900fbd20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9c84004940 tx=0x7f9c84009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 -- 172.21.15.111:0/2008009177 shutdown_connections 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 --2- 172.21.15.111:0/2008009177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c901026b0 0x7f9c900fbd20 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 --2- 172.21.15.111:0/2008009177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c90101d10 0x7f9c901020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 -- 172.21.15.111:0/2008009177 >> 172.21.15.111:0/2008009177 conn(0x7f9c900f7880 msgr2=0x7f9c900f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 -- 172.21.15.111:0/2008009177 shutdown_connections 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 -- 172.21.15.111:0/2008009177 wait complete. 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 Processor -- start 2024-03-14T13:23:01.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.666+0000 7f9c968ea700 1 -- start start 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c901026b0 0x7f9c90113220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c90113870 con 0x7f9c90101d10 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9010cc40 con 0x7f9c901026b0 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38556/0 (socket says 172.21.15.111:38556) 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 -- 172.21.15.111:0/3010907761 learned_addr learned my addr 172.21.15.111:0/3010907761 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8f7fe700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c901026b0 0x7f9c90113220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:01.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 -- 172.21.15.111:0/3010907761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c901026b0 msgr2=0x7f9c90113220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c901026b0 0x7f9c90113220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 -- 172.21.15.111:0/3010907761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c84005040 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8ffff700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c830 tx=0x7f9c80004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c800106f0 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9010cf20 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c80010d00 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c80012e90 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.667+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9010d430 con 0x7f9c90101d10 2024-03-14T13:23:01.666 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.668+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c900fd460 con 0x7f9c90101d10 2024-03-14T13:23:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.669+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9c80010850 con 0x7f9c90101d10 2024-03-14T13:23:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.669+0000 7f9c8cff9700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 0x7f9c78070e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.669+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9c8008ba30 con 0x7f9c90101d10 2024-03-14T13:23:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.669+0000 7f9c8f7fe700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 0x7f9c78070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:01.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.670+0000 7f9c8f7fe700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 0x7f9c78070e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9c84004890 tx=0x7f9c84009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:01.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.674+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9c8008de60 con 0x7f9c90101d10 2024-03-14T13:23:01.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.888+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9004bf40 con 0x7f9c7806e9d0 2024-03-14T13:23:01.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.889+0000 7f9c8cff9700 1 -- 172.21.15.111:0/3010907761 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f9c9004bf40 con 0x7f9c7806e9d0 2024-03-14T13:23:01.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:01 smithi111 ceph-mon[145314]: pgmap v567: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:01.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 msgr2=0x7f9c78070e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 0x7f9c78070e80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9c84004890 tx=0x7f9c84009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:01.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 msgr2=0x7f9c90112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c830 tx=0x7f9c80004470 comp rx=0 tx=0).stop 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 shutdown_connections 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9c7806e9d0 0x7f9c78070e80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c90101d10 0x7f9c90112ce0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 --2- 172.21.15.111:0/3010907761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c901026b0 0x7f9c90113220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.891+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 >> 172.21.15.111:0/3010907761 conn(0x7f9c900f7880 msgr2=0x7f9c900f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.892+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 shutdown_connections 2024-03-14T13:23:01.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.892+0000 7f9c968ea700 1 -- 172.21.15.111:0/3010907761 wait complete. 2024-03-14T13:23:01.898 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:23:01.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/2510567838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fe7e0 msgr2=0x7fe5d4106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/2510567838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fe7e0 0x7fe5d4106eb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4004940 tx=0x7fe5c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/2510567838 shutdown_connections 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/2510567838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fe7e0 0x7fe5d4106eb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/2510567838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d40fde40 0x7fe5d40fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/2510567838 >> 172.21.15.111:0/2510567838 conn(0x7fe5d40f7880 msgr2=0x7fe5d40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/2510567838 shutdown_connections 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.992+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/2510567838 wait complete. 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 Processor -- start 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 -- start start 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d40fe7e0 0x7fe5d4114350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d41149a0 con 0x7fe5d40fde40 2024-03-14T13:23:01.991 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5dc6a9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d410ddc0 con 0x7fe5d40fe7e0 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5da445700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5da445700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38574/0 (socket says 172.21.15.111:38574) 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5da445700 1 -- 172.21.15.111:0/3893000103 learned_addr learned my addr 172.21.15.111:0/3893000103 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5da445700 1 -- 172.21.15.111:0/3893000103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d40fe7e0 msgr2=0x7fe5d4114350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.993+0000 7fe5da445700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d40fe7e0 0x7fe5d4114350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5da445700 1 -- 172.21.15.111:0/3893000103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c4005040 con 0x7fe5d40fde40 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5da445700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000c8c0 tx=0x7fe5d00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5d000a980 con 0x7fe5d40fde40 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5d000ce60 con 0x7fe5d40fde40 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d410e0a0 con 0x7fe5d40fde40 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5d00113a0 con 0x7fe5d40fde40 2024-03-14T13:23:01.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.994+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d410e5e0 con 0x7fe5d40fde40 2024-03-14T13:23:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.995+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d40666b0 con 0x7fe5d40fde40 2024-03-14T13:23:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.995+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe5d0012400 con 0x7fe5d40fde40 2024-03-14T13:23:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.995+0000 7fe5caffd700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 0x7fe5c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:01.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.996+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe5d008bb90 con 0x7fe5d40fde40 2024-03-14T13:23:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.996+0000 7fe5d9c44700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 0x7fe5c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:01.994 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:01.996+0000 7fe5d9c44700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 0x7fe5c006ec70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4000c00 tx=0x7fe5c4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:01.997 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.000+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe5d00598a0 con 0x7fe5d40fde40 2024-03-14T13:23:02.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:01 smithi137 ceph-mon[108145]: pgmap v567: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:02.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.209+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5d4109d80 con 0x7fe5c006c7c0 2024-03-14T13:23:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.210+0000 7fe5caffd700 1 -- 172.21.15.111:0/3893000103 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fe5d4109d80 con 0x7fe5c006c7c0 2024-03-14T13:23:02.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.212+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 msgr2=0x7fe5c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.209 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.212+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 0x7fe5c006ec70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4000c00 tx=0x7fe5c4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.212+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 msgr2=0x7fe5d4113e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.212+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000c8c0 tx=0x7fe5d00044c0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 shutdown_connections 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe5c006c7c0 0x7fe5c006ec70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5d40fde40 0x7fe5d4113e10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 --2- 172.21.15.111:0/3893000103 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d40fe7e0 0x7fe5d4114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 >> 172.21.15.111:0/3893000103 conn(0x7fe5d40f7880 msgr2=0x7fe5d40f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 shutdown_connections 2024-03-14T13:23:02.210 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.213+0000 7fe5dc6a9700 1 -- 172.21.15.111:0/3893000103 wait complete. 2024-03-14T13:23:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 -- 172.21.15.111:0/2351101341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fde60 msgr2=0x7f077c0fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 --2- 172.21.15.111:0/2351101341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fde60 0x7f077c0fe230 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f076c004970 tx=0x7f076c009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 -- 172.21.15.111:0/2351101341 shutdown_connections 2024-03-14T13:23:02.309 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 --2- 172.21.15.111:0/2351101341 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fe800 0x7f077c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 --2- 172.21.15.111:0/2351101341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fde60 0x7f077c0fe230 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 -- 172.21.15.111:0/2351101341 >> 172.21.15.111:0/2351101341 conn(0x7f077c0f7870 msgr2=0x7f077c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 -- 172.21.15.111:0/2351101341 shutdown_connections 2024-03-14T13:23:02.310 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.312+0000 7f0781373700 1 -- 172.21.15.111:0/2351101341 wait complete. 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 Processor -- start 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 -- start start 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fde60 0x7f077c113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c114420 con 0x7f077c0fe800 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f0781373700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c10da20 con 0x7f077c0fde60 2024-03-14T13:23:02.312 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f077a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f077a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38586/0 (socket says 172.21.15.111:38586) 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.313+0000 7f077a7fc700 1 -- 172.21.15.111:0/3652880809 learned_addr learned my addr 172.21.15.111:0/3652880809 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f077affd700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fde60 0x7f077c113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f077a7fc700 1 -- 172.21.15.111:0/3652880809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fde60 msgr2=0x7f077c113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f077a7fc700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fde60 0x7f077c113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f077a7fc700 1 -- 172.21.15.111:0/3652880809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c005040 con 0x7f077c0fe800 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f077a7fc700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0770002a80 tx=0x7f07700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0770007cc0 con 0x7f077c0fe800 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f077000f040 con 0x7f077c0fe800 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c10dd00 con 0x7f077c0fe800 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0770009640 con 0x7f077c0fe800 2024-03-14T13:23:02.313 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.314+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c10e240 con 0x7f077c0fe800 2024-03-14T13:23:02.314 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.315+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c066b30 con 0x7f077c0fe800 2024-03-14T13:23:02.314 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.315+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f07700097a0 con 0x7f077c0fe800 2024-03-14T13:23:02.314 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.316+0000 7f07637fe700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 0x7f076406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.316+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f077008b2b0 con 0x7f077c0fe800 2024-03-14T13:23:02.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.316+0000 7f077affd700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 0x7f076406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.315 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.316+0000 7f077affd700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 0x7f076406ec70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f076c004540 tx=0x7f076c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:02.317 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.320+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0770058f40 con 0x7f077c0fe800 2024-03-14T13:23:02.519 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.522+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f077c113560 con 0x7f076406c7c0 2024-03-14T13:23:02.528 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.532+0000 7f07637fe700 1 -- 172.21.15.111:0/3652880809 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f077c113560 con 0x7f076406c7c0 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (19m) 12m ago 30m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (16m) 12m ago 31m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (16m) 12m ago 29m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (18m) 12m ago 30m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (12m) 12m ago 24m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (13m) 12m ago 24m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:23:02.529 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (12m) 12m ago 24m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (12m) 12m ago 24m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (21m) 12m ago 32m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (20m) 12m ago 29m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (17m) 12m ago 32m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (17m) 12m ago 29m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (20m) 12m ago 30m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (20m) 12m ago 29m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (16m) 12m ago 28m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (15m) 12m ago 28m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (15m) 12m ago 27m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (14m) 12m ago 27m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:23:02.530 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (13m) 12m ago 26m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:23:02.531 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (13m) 12m ago 26m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:23:02.531 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (19m) 12m ago 29m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 msgr2=0x7f076406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 0x7f076406ec70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f076c004540 tx=0x7f076c009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 msgr2=0x7f077c113dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0770002a80 tx=0x7f07700046b0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 shutdown_connections 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f076406c7c0 0x7f076406ec70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f077c0fde60 0x7f077c113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 --2- 172.21.15.111:0/3652880809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f077c0fe800 0x7f077c113dd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 >> 172.21.15.111:0/3652880809 conn(0x7f077c0f7870 msgr2=0x7f077c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:02.532 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 shutdown_connections 2024-03-14T13:23:02.533 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.534+0000 7f0781373700 1 -- 172.21.15.111:0/3652880809 wait complete. 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1778225748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 msgr2=0x7fcdb8106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1778225748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb8106e70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4004940 tx=0x7fcdb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1778225748 shutdown_connections 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1778225748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb8106e70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1778225748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 0x7fcdb80fe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1778225748 >> 172.21.15.111:0/1778225748 conn(0x7fcdb80f7880 msgr2=0x7fcdb80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:02.631 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1778225748 shutdown_connections 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1778225748 wait complete. 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 Processor -- start 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 -- start start 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.633+0000 7fcdbf6ac700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 0x7fcdb8113d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbf6ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbf6ac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb81148f0 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbf6ac700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb810dcc0 con 0x7fcdb80fde00 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38606/0 (socket says 172.21.15.111:38606) 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 -- 172.21.15.111:0/1973460451 learned_addr learned my addr 172.21.15.111:0/1973460451 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbd448700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 0x7fcdb8113d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 -- 172.21.15.111:0/1973460451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 msgr2=0x7fcdb8113d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 0x7fcdb8113d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.632 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 -- 172.21.15.111:0/1973460451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4005040 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbcc47700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4000c00 tx=0x7fcdb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb4021070 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb810df40 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdb4013e00 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.633 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb400e070 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.634+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb810e480 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.635+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb80666f0 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.636+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcdb4013950 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.636+0000 7fcdadffb700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 0x7fcda406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.636+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcdb408cb60 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.636+0000 7fcdbd448700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 0x7fcda406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:02.634 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.637+0000 7fcdbd448700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 0x7fcda406ec80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcda8004600 tx=0x7fcda800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:02.637 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.641+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcdb405a870 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.912+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcdb8109d80 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.912+0000 7fcdadffb700 1 -- 172.21.15.111:0/1973460451 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fcdb405a400 con 0x7fcdb80fe7a0 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:23:02.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:23:02.910 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:23:02.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 msgr2=0x7fcda406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 0x7fcda406ec80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcda8004600 tx=0x7fcda800b040 comp rx=0 tx=0).stop 2024-03-14T13:23:02.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 msgr2=0x7fcdb81142a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:02.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4000c00 tx=0x7fcdb4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:02.911 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 shutdown_connections 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcda406c7d0 0x7fcda406ec80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdb80fde00 0x7fcdb8113d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 --2- 172.21.15.111:0/1973460451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcdb80fe7a0 0x7fcdb81142a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 >> 172.21.15.111:0/1973460451 conn(0x7fcdb80f7880 msgr2=0x7fcdb80f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 shutdown_connections 2024-03-14T13:23:02.912 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:02.914+0000 7fcdbf6ac700 1 -- 172.21.15.111:0/1973460451 wait complete. 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- 172.21.15.111:0/1310392399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 msgr2=0x7ff638102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/1310392399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff638102140 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ff628004940 tx=0x7ff628009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- 172.21.15.111:0/1310392399 shutdown_connections 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/1310392399 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 0x7ff6380fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/1310392399 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff638102140 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- 172.21.15.111:0/1310392399 >> 172.21.15.111:0/1310392399 conn(0x7ff6380714c0 msgr2=0x7ff6380718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- 172.21.15.111:0/1310392399 shutdown_connections 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- 172.21.15.111:0/1310392399 wait complete. 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 Processor -- start 2024-03-14T13:23:03.013 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.015+0000 7ff63dbc8700 1 -- start start 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff63dbc8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 0x7ff638113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff63dbc8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff63dbc8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff638114710 con 0x7ff638101d70 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff63dbc8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810db00 con 0x7ff6380fc5b0 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38620/0 (socket says 172.21.15.111:38620) 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 -- 172.21.15.111:0/2775116486 learned_addr learned my addr 172.21.15.111:0/2775116486 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff6377fe700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 0x7ff638113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 -- 172.21.15.111:0/2775116486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 msgr2=0x7ff638113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 0x7ff638113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 -- 172.21.15.111:0/2775116486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628005040 con 0x7ff638101d70 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff636ffd700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff62c002a50 tx=0x7ff62c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.014 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62c007cc0 con 0x7ff638101d70 2024-03-14T13:23:03.015 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63810dde0 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff62c00f040 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.016+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62c00d9c0 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.017+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63810e320 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.018+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6380fd5c0 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.018+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff62c00db20 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.018+0000 7ff61ffff700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 0x7ff62006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.018+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff62c08b820 con 0x7ff638101d70 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.018+0000 7ff6377fe700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 0x7ff62006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.016 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.019+0000 7ff6377fe700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 0x7ff62006ec70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff628004890 tx=0x7ff628009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.019 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.023+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff62c059520 con 0x7ff638101d70 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.258+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff638108cb0 con 0x7ff638101d70 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.258+0000 7ff61ffff700 1 -- 172.21.15.111:0/2775116486 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ff62c0590b0 con 0x7ff638101d70 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:23:03.255 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:23:03.256 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:03.257 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:03.258 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:03.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 msgr2=0x7ff62006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 0x7ff62006ec70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff628004890 tx=0x7ff628009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 msgr2=0x7ff6381140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff62c002a50 tx=0x7ff62c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 shutdown_connections 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62006c7c0 0x7ff62006ec70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff6380fc5b0 0x7ff638113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 --2- 172.21.15.111:0/2775116486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff638101d70 0x7ff6381140c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 >> 172.21.15.111:0/2775116486 conn(0x7ff6380714c0 msgr2=0x7ff638072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 shutdown_connections 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.260+0000 7ff63dbc8700 1 -- 172.21.15.111:0/2775116486 wait complete. 2024-03-14T13:23:03.259 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:23:03.358 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 -- 172.21.15.111:0/958666932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 msgr2=0x7f4ae41020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 --2- 172.21.15.111:0/958666932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae41020e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4004940 tx=0x7f4ad4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 -- 172.21.15.111:0/958666932 shutdown_connections 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 --2- 172.21.15.111:0/958666932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ae41026b0 0x7f4ae40fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 --2- 172.21.15.111:0/958666932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae41020e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 -- 172.21.15.111:0/958666932 >> 172.21.15.111:0/958666932 conn(0x7f4ae40f7880 msgr2=0x7f4ae40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.361+0000 7f4aeb628700 1 -- 172.21.15.111:0/958666932 shutdown_connections 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 -- 172.21.15.111:0/958666932 wait complete. 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 Processor -- start 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 -- start start 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ae41026b0 0x7f4ae4116490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae4116ae0 con 0x7f4ae4101d10 2024-03-14T13:23:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4aeb628700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae410feb0 con 0x7f4ae41026b0 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4ae93c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4ae93c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38644/0 (socket says 172.21.15.111:38644) 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.362+0000 7f4ae93c4700 1 -- 172.21.15.111:0/3264976942 learned_addr learned my addr 172.21.15.111:0/3264976942 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ae93c4700 1 -- 172.21.15.111:0/3264976942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ae41026b0 msgr2=0x7f4ae4116490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ae93c4700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ae41026b0 0x7f4ae4116490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ae93c4700 1 -- 172.21.15.111:0/3264976942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4005040 con 0x7f4ae4101d10 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ae93c4700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4008d80 tx=0x7f4ad4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad400e030 con 0x7f4ae4101d10 2024-03-14T13:23:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae4110130 con 0x7f4ae4101d10 2024-03-14T13:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ad4013de0 con 0x7f4ae4101d10 2024-03-14T13:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ad401cd60 con 0x7f4ae4101d10 2024-03-14T13:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.363+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae41106f0 con 0x7f4ae4101d10 2024-03-14T13:23:03.362 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.365+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae40fd460 con 0x7f4ae4101d10 2024-03-14T13:23:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.365+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4ad401c590 con 0x7f4ae4101d10 2024-03-14T13:23:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.365+0000 7f4ad9ffb700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.365+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4ad408d360 con 0x7f4ae4101d10 2024-03-14T13:23:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.365+0000 7f4ae8bc3700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.366+0000 7f4ae8bc3700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 0x7f4ad006ec80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ae41117f0 tx=0x7f4ae000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.366 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.370+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4ad405b070 con 0x7f4ae4101d10 2024-03-14T13:23:03.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.582+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ae404bed0 con 0x7f4ad006c7d0 2024-03-14T13:23:03.579 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.583+0000 7f4ad9ffb700 1 -- 172.21.15.111:0/3264976942 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f4ae404bed0 con 0x7f4ad006c7d0 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:23:03.580 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:23:03.581 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 msgr2=0x7f4ad006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 0x7f4ad006ec80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4ae41117f0 tx=0x7f4ae000b040 comp rx=0 tx=0).stop 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 msgr2=0x7f4ae4115f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4008d80 tx=0x7f4ad4009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 shutdown_connections 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4ad006c7d0 0x7f4ad006ec80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ae4101d10 0x7f4ae4115f50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 --2- 172.21.15.111:0/3264976942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ae41026b0 0x7f4ae4116490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 >> 172.21.15.111:0/3264976942 conn(0x7f4ae40f7880 msgr2=0x7f4ae4065ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 shutdown_connections 2024-03-14T13:23:03.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.585+0000 7f4aeb628700 1 -- 172.21.15.111:0/3264976942 wait complete. 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 -- 172.21.15.111:0/2269971147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460102710 msgr2=0x7f24600fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 --2- 172.21.15.111:0/2269971147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460102710 0x7f24600fbd80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2454004940 tx=0x7f2454009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 -- 172.21.15.111:0/2269971147 shutdown_connections 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 --2- 172.21.15.111:0/2269971147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460102710 0x7f24600fbd80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 --2- 172.21.15.111:0/2269971147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460101d70 0x7f2460102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 -- 172.21.15.111:0/2269971147 >> 172.21.15.111:0/2269971147 conn(0x7f2460072bc0 msgr2=0x7f2460072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 -- 172.21.15.111:0/2269971147 shutdown_connections 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.685+0000 7f2465c94700 1 -- 172.21.15.111:0/2269971147 wait complete. 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 Processor -- start 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 -- start start 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.684 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460102710 0x7f2460103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460103650 con 0x7f2460101d70 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f2465c94700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24601037c0 con 0x7f2460102710 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.686+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38654/0 (socket says 172.21.15.111:38654) 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 -- 172.21.15.111:0/3464163596 learned_addr learned my addr 172.21.15.111:0/3464163596 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245effd700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460102710 0x7f2460103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 -- 172.21.15.111:0/3464163596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460102710 msgr2=0x7f2460103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460102710 0x7f2460103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 -- 172.21.15.111:0/3464163596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454005040 con 0x7f2460101d70 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f245f7fe700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f245000eb40 tx=0x7f245000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2450007cc0 con 0x7f2460101d70 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2460103a40 con 0x7f2460101d70 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f245000f040 con 0x7f2460101d70 2024-03-14T13:23:03.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2450009570 con 0x7f2460101d70 2024-03-14T13:23:03.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.687+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246011a0e0 con 0x7f2460101d70 2024-03-14T13:23:03.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.688+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24600fd4c0 con 0x7f2460101d70 2024-03-14T13:23:03.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.689+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f24500096d0 con 0x7f2460101d70 2024-03-14T13:23:03.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.689+0000 7f2447fff700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 0x7f244806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:03.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.689+0000 7f245effd700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 0x7f244806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:03.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.689+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f245008b5a0 con 0x7f2460101d70 2024-03-14T13:23:03.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.690+0000 7f245effd700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 0x7f244806ec80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24600fa690 tx=0x7f2454009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:03.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.694+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2450055850 con 0x7f2460101d70 2024-03-14T13:23:03.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: pgmap v568: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: from='client.35080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1973460451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:23:03.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:03 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2775116486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.965+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f24601042c0 con 0x7f2460101d70 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.965+0000 7f2447fff700 1 -- 172.21.15.111:0/3464163596 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f24601042c0 con 0x7f2460101d70 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:23:03.962 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:23:03.963 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 msgr2=0x7f244806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 0x7f244806ec80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24600fa690 tx=0x7f2454009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 msgr2=0x7f2460102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f245000eb40 tx=0x7f245000b040 comp rx=0 tx=0).stop 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 shutdown_connections 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f244806c7d0 0x7f244806ec80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2460101d70 0x7f2460102bd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 --2- 172.21.15.111:0/3464163596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2460102710 0x7f2460103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:03.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 >> 172.21.15.111:0/3464163596 conn(0x7f2460072bc0 msgr2=0x7f2460065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:03.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.967+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 shutdown_connections 2024-03-14T13:23:03.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:03.968+0000 7f2465c94700 1 -- 172.21.15.111:0/3464163596 wait complete. 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: from='client.35072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: from='client.35076 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: pgmap v568: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: from='client.35080 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1973460451' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:23:04.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:03 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2775116486' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:23:04.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:04 smithi111 ceph-mon[145314]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:04.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:04 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3464163596' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:23:05.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:04 smithi137 ceph-mon[108145]: from='client.35092 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:05.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:04 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3464163596' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:23:05.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:05 smithi111 ceph-mon[145314]: pgmap v569: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:06.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:05 smithi137 ceph-mon[108145]: pgmap v569: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:07.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:06 smithi137 ceph-mon[108145]: pgmap v570: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:07.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:06 smithi111 ceph-mon[145314]: pgmap v570: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:09.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:09 smithi111 ceph-mon[145314]: pgmap v571: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:09.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:10.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:09 smithi137 ceph-mon[108145]: pgmap v571: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:10.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:11.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:11 smithi111 ceph-mon[145314]: pgmap v572: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:12.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:11 smithi137 ceph-mon[108145]: pgmap v572: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:13.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:13 smithi111 ceph-mon[145314]: pgmap v573: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:14.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:13 smithi137 ceph-mon[108145]: pgmap v573: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:15.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:15 smithi111 ceph-mon[145314]: pgmap v574: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:16.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:15 smithi137 ceph-mon[108145]: pgmap v574: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:17.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:17 smithi111 ceph-mon[145314]: pgmap v575: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:18.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:17 smithi137 ceph-mon[108145]: pgmap v575: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:19.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:19 smithi111 ceph-mon[145314]: pgmap v576: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:20.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:19 smithi137 ceph-mon[108145]: pgmap v576: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:21.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:21 smithi111 ceph-mon[145314]: pgmap v577: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:22.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:21 smithi137 ceph-mon[108145]: pgmap v577: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:23.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:23 smithi111 ceph-mon[145314]: pgmap v578: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:24.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:23 smithi137 ceph-mon[108145]: pgmap v578: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:24.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:25.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:25.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:25 smithi111 ceph-mon[145314]: pgmap v579: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:26.334 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:25 smithi137 ceph-mon[108145]: pgmap v579: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:27.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:27 smithi111 ceph-mon[145314]: pgmap v580: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:28.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:27 smithi137 ceph-mon[108145]: pgmap v580: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:29.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:29 smithi111 ceph-mon[145314]: pgmap v581: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:30.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:29 smithi137 ceph-mon[108145]: pgmap v581: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:31.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:31 smithi111 ceph-mon[145314]: pgmap v582: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:32.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:31 smithi137 ceph-mon[108145]: pgmap v582: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:33.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:33 smithi111 ceph-mon[145314]: pgmap v583: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:34.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:33 smithi137 ceph-mon[108145]: pgmap v583: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:34.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.080+0000 7fbd33561700 1 -- 172.21.15.111:0/3210104676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 msgr2=0x7fbd2c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.078 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.080+0000 7fbd33561700 1 --2- 172.21.15.111:0/3210104676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c102140 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c004940 tx=0x7fbd1c009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- 172.21.15.111:0/3210104676 shutdown_connections 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 --2- 172.21.15.111:0/3210104676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 0x7fbd2c0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 --2- 172.21.15.111:0/3210104676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c102140 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- 172.21.15.111:0/3210104676 >> 172.21.15.111:0/3210104676 conn(0x7fbd2c072bc0 msgr2=0x7fbd2c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- 172.21.15.111:0/3210104676 shutdown_connections 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- 172.21.15.111:0/3210104676 wait complete. 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 Processor -- start 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- start start 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 0x7fbd2c10f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c10f660 con 0x7fbd2c101d70 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.081+0000 7fbd33561700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c10f7a0 con 0x7fbd2c102710 2024-03-14T13:23:34.079 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38348/0 (socket says 172.21.15.111:38348) 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 -- 172.21.15.111:0/3649703207 learned_addr learned my addr 172.21.15.111:0/3649703207 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd30afc700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 0x7fbd2c10f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 -- 172.21.15.111:0/3649703207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 msgr2=0x7fbd2c10f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 0x7fbd2c10f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 -- 172.21.15.111:0/3649703207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c005040 con 0x7fbd2c101d70 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd312fd700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009180 tx=0x7fbd1c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1c00e040 con 0x7fbd2c101d70 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd1c004290 con 0x7fbd2c101d70 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd1c013e60 con 0x7fbd2c101d70 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.082+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c117a90 con 0x7fbd2c101d70 2024-03-14T13:23:34.080 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.083+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c118020 con 0x7fbd2c101d70 2024-03-14T13:23:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.084+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c0ff080 con 0x7fbd2c101d70 2024-03-14T13:23:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.084+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fbd1c0136b0 con 0x7fbd2c101d70 2024-03-14T13:23:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.084+0000 7fbd21ffb700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 0x7fbd1806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.085+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbd1c08ccd0 con 0x7fbd2c101d70 2024-03-14T13:23:34.081 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.085+0000 7fbd30afc700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 0x7fbd1806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.082 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.085+0000 7fbd30afc700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 0x7fbd1806ec00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c066020 tx=0x7fbd2800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.089+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fbd1c057000 con 0x7fbd2c101d70 2024-03-14T13:23:34.295 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.299+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd2c04bed0 con 0x7fbd1806c750 2024-03-14T13:23:34.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.300+0000 7fbd21ffb700 1 -- 172.21.15.111:0/3649703207 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fbd2c04bed0 con 0x7fbd1806c750 2024-03-14T13:23:34.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 msgr2=0x7fbd1806ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 0x7fbd1806ec00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c066020 tx=0x7fbd2800b040 comp rx=0 tx=0).stop 2024-03-14T13:23:34.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 msgr2=0x7fbd2c10ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009180 tx=0x7fbd1c009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 shutdown_connections 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fbd1806c750 0x7fbd1806ec00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd2c101d70 0x7fbd2c10ebe0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 --2- 172.21.15.111:0/3649703207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd2c102710 0x7fbd2c10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 >> 172.21.15.111:0/3649703207 conn(0x7fbd2c072bc0 msgr2=0x7fbd2c0fb070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 shutdown_connections 2024-03-14T13:23:34.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.302+0000 7fbd33561700 1 -- 172.21.15.111:0/3649703207 wait complete. 2024-03-14T13:23:34.308 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:23:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- 172.21.15.111:0/1640329561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654102710 msgr2=0x7f26540fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 --2- 172.21.15.111:0/1640329561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654102710 0x7f26540fbd80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2640004940 tx=0x7f2640009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- 172.21.15.111:0/1640329561 shutdown_connections 2024-03-14T13:23:34.401 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 --2- 172.21.15.111:0/1640329561 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654102710 0x7f26540fbd80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 --2- 172.21.15.111:0/1640329561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2654101d70 0x7f2654102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- 172.21.15.111:0/1640329561 >> 172.21.15.111:0/1640329561 conn(0x7f2654072bc0 msgr2=0x7f2654072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- 172.21.15.111:0/1640329561 shutdown_connections 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- 172.21.15.111:0/1640329561 wait complete. 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 Processor -- start 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- start start 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2654102710 0x7f265410efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265410f5f0 con 0x7f2654101d70 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.404+0000 7f265ad52700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265410f730 con 0x7f2654102710 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38358/0 (socket says 172.21.15.111:38358) 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 -- 172.21.15.111:0/1433264527 learned_addr learned my addr 172.21.15.111:0/1433264527 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:34.402 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 -- 172.21.15.111:0/1433264527 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2654102710 msgr2=0x7f265410efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2654102710 0x7f265410efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 -- 172.21.15.111:0/1433264527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640005040 con 0x7f2654101d70 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f2658aee700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f264800c830 tx=0x7f2648004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f264800a760 con 0x7f2654101d70 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f264800ad70 con 0x7f2654101d70 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26480113a0 con 0x7f2654101d70 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654117ad0 con 0x7f2654101d70 2024-03-14T13:23:34.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.405+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2654118030 con 0x7f2654101d70 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.406+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26540ff080 con 0x7f2654101d70 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.407+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2648012400 con 0x7f2654101d70 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.407+0000 7f26517fa700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.407+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f264808b9e0 con 0x7f2654101d70 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.407+0000 7f2653fff700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.407+0000 7f2653fff700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 0x7f264406ec70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2640004f90 tx=0x7f2640009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.412+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2648055d20 con 0x7f2654101d70 2024-03-14T13:23:34.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.623+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2654108d10 con 0x7f264406c7c0 2024-03-14T13:23:34.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.624+0000 7f26517fa700 1 -- 172.21.15.111:0/1433264527 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2654108d10 con 0x7f264406c7c0 2024-03-14T13:23:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 msgr2=0x7f264406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 0x7f264406ec70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2640004f90 tx=0x7f2640009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 msgr2=0x7f265410ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f264800c830 tx=0x7f2648004470 comp rx=0 tx=0).stop 2024-03-14T13:23:34.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 shutdown_connections 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f264406c7c0 0x7f264406ec70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2654101d70 0x7f265410ea60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 --2- 172.21.15.111:0/1433264527 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2654102710 0x7f265410efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 >> 172.21.15.111:0/1433264527 conn(0x7f2654072bc0 msgr2=0x7f26540fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 shutdown_connections 2024-03-14T13:23:34.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.626+0000 7f265ad52700 1 -- 172.21.15.111:0/1433264527 wait complete. 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- 172.21.15.111:0/339189998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba90102e00 msgr2=0x7fba901031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 --2- 172.21.15.111:0/339189998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba90102e00 0x7fba901031d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba80004940 tx=0x7fba80009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- 172.21.15.111:0/339189998 shutdown_connections 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 --2- 172.21.15.111:0/339189998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba901037a0 0x7fba900fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 --2- 172.21.15.111:0/339189998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba90102e00 0x7fba901031d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- 172.21.15.111:0/339189998 >> 172.21.15.111:0/339189998 conn(0x7fba900f89b0 msgr2=0x7fba900fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- 172.21.15.111:0/339189998 shutdown_connections 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- 172.21.15.111:0/339189998 wait complete. 2024-03-14T13:23:34.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 Processor -- start 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- start start 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba90102e00 0x7fba900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90070d90 con 0x7fba901037a0 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.727+0000 7fba97fb6700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90070ed0 con 0x7fba90102e00 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38374/0 (socket says 172.21.15.111:38374) 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 -- 172.21.15.111:0/2546084716 learned_addr learned my addr 172.21.15.111:0/2546084716 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95d52700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba90102e00 0x7fba900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 -- 172.21.15.111:0/2546084716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba90102e00 msgr2=0x7fba900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.726 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba90102e00 0x7fba900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 -- 172.21.15.111:0/2546084716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80005040 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba95551700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba8c002a50 tx=0x7fba8c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c007cc0 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba90119e20 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba8c00f040 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba8c00d920 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.728+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9011a2a0 con 0x7fba901037a0 2024-03-14T13:23:34.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.729+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba900fe550 con 0x7fba901037a0 2024-03-14T13:23:34.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.730+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fba8c009680 con 0x7fba901037a0 2024-03-14T13:23:34.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.730+0000 7fba867fc700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 0x7fba7c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:34.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.731+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fba8c08be60 con 0x7fba901037a0 2024-03-14T13:23:34.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.731+0000 7fba95d52700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 0x7fba7c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:34.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.731+0000 7fba95d52700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 0x7fba7c06ec70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba80004f90 tx=0x7fba80009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:34.730 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.734+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fba8c056110 con 0x7fba901037a0 2024-03-14T13:23:34.930 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.934+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fba9004bed0 con 0x7fba7c06c7c0 2024-03-14T13:23:34.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.943+0000 7fba867fc700 1 -- 172.21.15.111:0/2546084716 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fba9004bed0 con 0x7fba7c06c7c0 2024-03-14T13:23:34.939 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (19m) 13m ago 31m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (17m) 13m ago 31m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (17m) 12m ago 30m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (19m) 13m ago 30m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (13m) 13m ago 25m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (13m) 13m ago 25m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (12m) 12m ago 25m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (13m) 12m ago 25m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (22m) 13m ago 32m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:23:34.940 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (20m) 12m ago 29m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (18m) 13m ago 32m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (17m) 12m ago 29m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (20m) 13m ago 31m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (20m) 12m ago 30m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (16m) 13m ago 29m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (16m) 13m ago 28m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (15m) 13m ago 28m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (15m) 12m ago 27m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (14m) 12m ago 27m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (14m) 12m ago 26m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:23:34.941 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (20m) 13m ago 30m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:23:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.945+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 msgr2=0x7fba7c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 0x7fba7c06ec70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fba80004f90 tx=0x7fba80009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:34.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 msgr2=0x7fba90070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba8c002a50 tx=0x7fba8c004600 comp rx=0 tx=0).stop 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 shutdown_connections 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fba7c06c7c0 0x7fba7c06ec70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba90102e00 0x7fba900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 --2- 172.21.15.111:0/2546084716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba901037a0 0x7fba90070800 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 >> 172.21.15.111:0/2546084716 conn(0x7fba900f89b0 msgr2=0x7fba900f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 shutdown_connections 2024-03-14T13:23:34.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:34.946+0000 7fba97fb6700 1 -- 172.21.15.111:0/2546084716 wait complete. 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3824507665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e38102e10 msgr2=0x7f1e381031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3824507665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e38102e10 0x7f1e381031e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1e28004940 tx=0x7f1e28009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3824507665 shutdown_connections 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3824507665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e381037b0 0x7f1e380fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3824507665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e38102e10 0x7f1e381031e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3824507665 >> 172.21.15.111:0/3824507665 conn(0x7f1e380f8980 msgr2=0x7f1e380fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3824507665 shutdown_connections 2024-03-14T13:23:35.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3824507665 wait complete. 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.044+0000 7f1e3fca8700 1 Processor -- start 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 -- start start 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 0x7f1e38113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e381146a0 con 0x7f1e381037b0 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3810da90 con 0x7f1e38102e10 2024-03-14T13:23:35.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38404/0 (socket says 172.21.15.111:38404) 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 -- 172.21.15.111:0/3016906638 learned_addr learned my addr 172.21.15.111:0/3016906638 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3da44700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 0x7f1e38113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 -- 172.21.15.111:0/3016906638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 msgr2=0x7f1e38113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 0x7f1e38113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 -- 172.21.15.111:0/3016906638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28005040 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3da44700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 0x7f1e38113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3d243700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400c800 tx=0x7f1e34004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e3400a760 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e3400ad70 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3810dd70 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e340113a0 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.045+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3810e330 con 0x7f1e381037b0 2024-03-14T13:23:35.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.046+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e380fe560 con 0x7f1e381037b0 2024-03-14T13:23:35.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.047+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1e34012400 con 0x7f1e381037b0 2024-03-14T13:23:35.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.047+0000 7f1e2e7fc700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 0x7f1e2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.047+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1e3408ba90 con 0x7f1e381037b0 2024-03-14T13:23:35.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.047+0000 7f1e3da44700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 0x7f1e2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.048+0000 7f1e3da44700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 0x7f1e2406ec70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e28004f90 tx=0x7f1e28009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.052+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1e340597a0 con 0x7f1e381037b0 2024-03-14T13:23:35.321 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.326+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1e38109dd0 con 0x7f1e381037b0 2024-03-14T13:23:35.322 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.326+0000 7f1e2e7fc700 1 -- 172.21.15.111:0/3016906638 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f1e34059330 con 0x7f1e381037b0 2024-03-14T13:23:35.322 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:23:35.322 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:23:35.322 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:23:35.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:23:35.323 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:23:35.324 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:23:35.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 msgr2=0x7f1e2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 0x7f1e2406ec70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1e28004f90 tx=0x7f1e28009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 msgr2=0x7f1e38114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1e3400c800 tx=0x7f1e34004470 comp rx=0 tx=0).stop 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 shutdown_connections 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1e2406c7c0 0x7f1e2406ec70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e38102e10 0x7f1e38113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 --2- 172.21.15.111:0/3016906638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e381037b0 0x7f1e38114050 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 >> 172.21.15.111:0/3016906638 conn(0x7f1e380f8980 msgr2=0x7f1e380f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 shutdown_connections 2024-03-14T13:23:35.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.328+0000 7f1e3fca8700 1 -- 172.21.15.111:0/3016906638 wait complete. 2024-03-14T13:23:35.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 -- 172.21.15.111:0/1517346174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3278102e10 msgr2=0x7f32781031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 --2- 172.21.15.111:0/1517346174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3278102e10 0x7f32781031e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3268004940 tx=0x7f3268009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 -- 172.21.15.111:0/1517346174 shutdown_connections 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 --2- 172.21.15.111:0/1517346174 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32781037b0 0x7f32780fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 --2- 172.21.15.111:0/1517346174 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3278102e10 0x7f32781031e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 -- 172.21.15.111:0/1517346174 >> 172.21.15.111:0/1517346174 conn(0x7f32780f89a0 msgr2=0x7f32780fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 -- 172.21.15.111:0/1517346174 shutdown_connections 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.430+0000 7f327fad0700 1 -- 172.21.15.111:0/1517346174 wait complete. 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 Processor -- start 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 -- start start 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3278102e10 0x7f3278113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3278114670 con 0x7f32781037b0 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327fad0700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327810da10 con 0x7f3278102e10 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38424/0 (socket says 172.21.15.111:38424) 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 -- 172.21.15.111:0/376798149 learned_addr learned my addr 172.21.15.111:0/376798149 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d86c700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3278102e10 0x7f3278113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 -- 172.21.15.111:0/376798149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3278102e10 msgr2=0x7f3278113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3278102e10 0x7f3278113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 -- 172.21.15.111:0/376798149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3268005040 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.431+0000 7f327d06b700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f327400b790 tx=0x7f3274009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f327400f6f0 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f327400fd00 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327810dcf0 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32740103a0 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327810e2b0 con 0x7f32781037b0 2024-03-14T13:23:35.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.432+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32780fe560 con 0x7f32781037b0 2024-03-14T13:23:35.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.433+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f327400f850 con 0x7f32781037b0 2024-03-14T13:23:35.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.434+0000 7f326e7fc700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 0x7f326406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.434+0000 7f327d86c700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 0x7f326406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.434+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f327408b770 con 0x7f32781037b0 2024-03-14T13:23:35.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.434+0000 7f327d86c700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 0x7f326406ec70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3268004f90 tx=0x7f3268009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.438+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3274059540 con 0x7f32781037b0 2024-03-14T13:23:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.672+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3278109e70 con 0x7f32781037b0 2024-03-14T13:23:35.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.672+0000 7f326e7fc700 1 -- 172.21.15.111:0/376798149 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f32740590d0 con 0x7f32781037b0 2024-03-14T13:23:35.668 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:23:35.669 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:23:35.670 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:23:35.671 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:35.672 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:23:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 msgr2=0x7f326406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 0x7f326406ec70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3268004f90 tx=0x7f3268009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 msgr2=0x7f3278113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f327400b790 tx=0x7f3274009220 comp rx=0 tx=0).stop 2024-03-14T13:23:35.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 shutdown_connections 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f326406c7c0 0x7f326406ec70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3278102e10 0x7f3278113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 --2- 172.21.15.111:0/376798149 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32781037b0 0x7f3278113f90 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 >> 172.21.15.111:0/376798149 conn(0x7f32780f89a0 msgr2=0x7f32780f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 shutdown_connections 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.674+0000 7f327fad0700 1 -- 172.21.15.111:0/376798149 wait complete. 2024-03-14T13:23:35.674 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 -- 172.21.15.111:0/2191405368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac8102e10 msgr2=0x7f7ac81031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 --2- 172.21.15.111:0/2191405368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac8102e10 0x7f7ac81031e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7abc004940 tx=0x7f7abc009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 -- 172.21.15.111:0/2191405368 shutdown_connections 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 --2- 172.21.15.111:0/2191405368 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac81037b0 0x7f7ac80fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 --2- 172.21.15.111:0/2191405368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac8102e10 0x7f7ac81031e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.775+0000 7f7acf075700 1 -- 172.21.15.111:0/2191405368 >> 172.21.15.111:0/2191405368 conn(0x7f7ac80f89a0 msgr2=0x7f7ac80fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 -- 172.21.15.111:0/2191405368 shutdown_connections 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 -- 172.21.15.111:0/2191405368 wait complete. 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 Processor -- start 2024-03-14T13:23:35.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 -- start start 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac8102e10 0x7f7ac8113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.776+0000 7f7acf075700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac81146a0 con 0x7f7ac81037b0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7acf075700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac810da90 con 0x7f7ac8102e10 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38452/0 (socket says 172.21.15.111:38452) 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 -- 172.21.15.111:0/1746946862 learned_addr learned my addr 172.21.15.111:0/1746946862 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7acce11700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac8102e10 0x7f7ac8113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 -- 172.21.15.111:0/1746946862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac8102e10 msgr2=0x7f7ac8113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac8102e10 0x7f7ac8113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 -- 172.21.15.111:0/1746946862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc005040 con 0x7f7ac81037b0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac7fff700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400c7e0 tx=0x7f7ab4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab4012810 con 0x7f7ac81037b0 2024-03-14T13:23:35.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac810dd70 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ab4012e20 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ab400a310 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.777+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac810e330 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.778+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80fe560 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.779+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7ab4011380 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.779+0000 7f7ac57fa700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 0x7f7ab806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.779+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7ab408bf30 con 0x7f7ac81037b0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.779+0000 7f7acce11700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 0x7f7ab806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:35.778 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.780+0000 7f7acce11700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 0x7f7ab806ec70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7abc004f90 tx=0x7f7abc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:35.779 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.784+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7ab4059b40 con 0x7f7ac81037b0 2024-03-14T13:23:35.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:35 smithi111 ceph-mon[145314]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:35.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:35 smithi111 ceph-mon[145314]: pgmap v584: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:35.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:35 smithi111 ceph-mon[145314]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:35.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:35 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3016906638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:23:35.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.993+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ac8109e60 con 0x7f7ab806c7c0 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.994+0000 7f7ac57fa700 1 -- 172.21.15.111:0/1746946862 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f7ac8109e60 con 0x7f7ab806c7c0 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:23:35.990 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:23:35.991 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:23:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 msgr2=0x7f7ab806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 0x7f7ab806ec70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7abc004f90 tx=0x7f7abc009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 msgr2=0x7f7ac8114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:35.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7ab400c7e0 tx=0x7f7ab4007f90 comp rx=0 tx=0).stop 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 shutdown_connections 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7ab806c7c0 0x7f7ab806ec70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ac8102e10 0x7f7ac8113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 --2- 172.21.15.111:0/1746946862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac81037b0 0x7f7ac8114050 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.996+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 >> 172.21.15.111:0/1746946862 conn(0x7f7ac80f89a0 msgr2=0x7f7ac80f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.997+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 shutdown_connections 2024-03-14T13:23:35.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:35.997+0000 7f7acf075700 1 -- 172.21.15.111:0/1746946862 wait complete. 2024-03-14T13:23:36.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:35 smithi137 ceph-mon[108145]: from='client.35100 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:36.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:35 smithi137 ceph-mon[108145]: pgmap v584: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:36.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:35 smithi137 ceph-mon[108145]: from='client.35104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:36.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:35 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3016906638' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:23:36.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 -- 172.21.15.111:0/2234002674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 msgr2=0x7f51080fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:36.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 --2- 172.21.15.111:0/2234002674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f51080fce10 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f50f0004940 tx=0x7f50f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:36.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 -- 172.21.15.111:0/2234002674 shutdown_connections 2024-03-14T13:23:36.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 --2- 172.21.15.111:0/2234002674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f51080fce10 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.092 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 --2- 172.21.15.111:0/2234002674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 0x7f51081031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 -- 172.21.15.111:0/2234002674 >> 172.21.15.111:0/2234002674 conn(0x7f51080f89b0 msgr2=0x7f51080fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:36.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 -- 172.21.15.111:0/2234002674 shutdown_connections 2024-03-14T13:23:36.093 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.096+0000 7f510d910700 1 -- 172.21.15.111:0/2234002674 wait complete. 2024-03-14T13:23:36.094 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 Processor -- start 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 -- start start 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 0x7f51081138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5108114450 con 0x7f51081037a0 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f510d910700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510810da50 con 0x7f5108102e00 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f51067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f51067fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:38472/0 (socket says 172.21.15.111:38472) 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.097+0000 7f51067fc700 1 -- 172.21.15.111:0/3006642835 learned_addr learned my addr 172.21.15.111:0/3006642835 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f5106ffd700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 0x7f51081138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f51067fc700 1 -- 172.21.15.111:0/3006642835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 msgr2=0x7f51081138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:36.095 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f51067fc700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 0x7f51081138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f51067fc700 1 -- 172.21.15.111:0/3006642835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f0005040 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f51067fc700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009380 tx=0x7f50f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50f000e040 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510810dcd0 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50f0013420 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50f001c630 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.098+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f510810e290 con 0x7f51081037a0 2024-03-14T13:23:36.096 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.099+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51080fe550 con 0x7f51081037a0 2024-03-14T13:23:36.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.100+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f50f00139b0 con 0x7f51081037a0 2024-03-14T13:23:36.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.100+0000 7f50ff7fe700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 0x7f50f406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:23:36.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.100+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f50f008cbe0 con 0x7f51081037a0 2024-03-14T13:23:36.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.100+0000 7f5106ffd700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 0x7f50f406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:23:36.097 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.100+0000 7f5106ffd700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 0x7f50f406ec70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f50f8004f40 tx=0x7f50f800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:23:36.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.105+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f50f005a8f0 con 0x7f51081037a0 2024-03-14T13:23:36.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.374+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f51080fe550 con 0x7f51081037a0 2024-03-14T13:23:36.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.375+0000 7f50ff7fe700 1 -- 172.21.15.111:0/3006642835 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f50f005a480 con 0x7f51081037a0 2024-03-14T13:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:23:36.372 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 msgr2=0x7f50f406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 0x7f50f406ec70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f50f8004f40 tx=0x7f50f800b040 comp rx=0 tx=0).stop 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 msgr2=0x7f5108113e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009380 tx=0x7f50f0009e30 comp rx=0 tx=0).stop 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 shutdown_connections 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f50f406c7c0 0x7f50f406ec70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5108102e00 0x7f51081138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 --2- 172.21.15.111:0/3006642835 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51081037a0 0x7f5108113e00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 >> 172.21.15.111:0/3006642835 conn(0x7f51080f89b0 msgr2=0x7f51080f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 shutdown_connections 2024-03-14T13:23:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:23:36.377+0000 7f510d910700 1 -- 172.21.15.111:0/3006642835 wait complete. 2024-03-14T13:23:37.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:36 smithi137 ceph-mon[108145]: from='client.35108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:37.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:36 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/376798149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:23:37.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:36 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3006642835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:23:37.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:36 smithi111 ceph-mon[145314]: from='client.35108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:37.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:36 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/376798149' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:23:37.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:36 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3006642835' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:23:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:37 smithi111 ceph-mon[145314]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:37.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:37 smithi111 ceph-mon[145314]: pgmap v585: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:38.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:37 smithi137 ceph-mon[108145]: from='client.35120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:23:38.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:37 smithi137 ceph-mon[108145]: pgmap v585: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:39.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:38 smithi137 ceph-mon[108145]: pgmap v586: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:39.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:38 smithi111 ceph-mon[145314]: pgmap v586: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:40.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:40.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:41.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:40 smithi137 ceph-mon[108145]: pgmap v587: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:41.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:40 smithi111 ceph-mon[145314]: pgmap v587: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:43.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:43 smithi111 ceph-mon[145314]: pgmap v588: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:44.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:43 smithi137 ceph-mon[108145]: pgmap v588: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:45.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:45 smithi111 ceph-mon[145314]: pgmap v589: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:46.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:45 smithi137 ceph-mon[108145]: pgmap v589: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:47.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:47 smithi111 ceph-mon[145314]: pgmap v590: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:48.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:47 smithi137 ceph-mon[108145]: pgmap v590: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:49.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:49 smithi111 ceph-mon[145314]: pgmap v591: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:50.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:49 smithi137 ceph-mon[108145]: pgmap v591: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:51.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:51 smithi111 ceph-mon[145314]: pgmap v592: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:52.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:51 smithi137 ceph-mon[108145]: pgmap v592: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:53.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:53 smithi111 ceph-mon[145314]: pgmap v593: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:54.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:53 smithi137 ceph-mon[108145]: pgmap v593: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:54.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:55.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:23:55.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:55 smithi111 ceph-mon[145314]: pgmap v594: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:56.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:55 smithi137 ceph-mon[108145]: pgmap v594: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:57.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:57 smithi111 ceph-mon[145314]: pgmap v595: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:58.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:57 smithi137 ceph-mon[108145]: pgmap v595: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:23:59.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:23:59 smithi111 ceph-mon[145314]: pgmap v596: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:00.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:23:59 smithi137 ceph-mon[108145]: pgmap v596: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:01.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:01 smithi111 ceph-mon[145314]: pgmap v597: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:02.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:01 smithi137 ceph-mon[108145]: pgmap v597: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:03.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:03 smithi111 ceph-mon[145314]: pgmap v598: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:04.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:03 smithi137 ceph-mon[108145]: pgmap v598: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:06.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:05 smithi137 ceph-mon[108145]: pgmap v599: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:06.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:05 smithi111 ceph-mon[145314]: pgmap v599: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:06.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- 172.21.15.111:0/4192080401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fe830 msgr2=0x7f37e8106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 --2- 172.21.15.111:0/4192080401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fe830 0x7f37e8106f00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f37e4004940 tx=0x7f37e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- 172.21.15.111:0/4192080401 shutdown_connections 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 --2- 172.21.15.111:0/4192080401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fe830 0x7f37e8106f00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 --2- 172.21.15.111:0/4192080401 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fde90 0x7f37e80fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- 172.21.15.111:0/4192080401 >> 172.21.15.111:0/4192080401 conn(0x7f37e8072bc0 msgr2=0x7f37e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- 172.21.15.111:0/4192080401 shutdown_connections 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- 172.21.15.111:0/4192080401 wait complete. 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 Processor -- start 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 -- start start 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.490+0000 7f37efaee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37efaee700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fe830 0x7f37e80fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37efaee700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80fc820 con 0x7f37e80fde90 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37efaee700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e80fc960 con 0x7f37e80fe830 2024-03-14T13:24:06.488 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55722/0 (socket says 172.21.15.111:55722) 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 -- 172.21.15.111:0/2190962707 learned_addr learned my addr 172.21.15.111:0/2190962707 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed089700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fe830 0x7f37e80fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 -- 172.21.15.111:0/2190962707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fe830 msgr2=0x7f37e80fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fe830 0x7f37e80fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 -- 172.21.15.111:0/2190962707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e4005040 con 0x7f37e80fde90 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37ed88a700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00eb40 tx=0x7f37dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37dc007cc0 con 0x7f37e80fde90 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc00f040 con 0x7f37e80fde90 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e80fcbb0 con 0x7f37e80fde90 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f37dc009570 con 0x7f37e80fde90 2024-03-14T13:24:06.489 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.491+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e811a130 con 0x7f37e80fde90 2024-03-14T13:24:06.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.492+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e8065e30 con 0x7f37e80fde90 2024-03-14T13:24:06.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.493+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f37dc0096d0 con 0x7f37e80fde90 2024-03-14T13:24:06.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.493+0000 7f37da7fc700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 0x7f37d406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.493+0000 7f37ed089700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 0x7f37d406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.493+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f37dc08b740 con 0x7f37e80fde90 2024-03-14T13:24:06.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.494+0000 7f37ed089700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 0x7f37d406ec70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b0d0 tx=0x7f37e4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:06.494 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.498+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f37dc0559f0 con 0x7f37e80fde90 2024-03-14T13:24:06.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.708+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37e80fd440 con 0x7f37d406c7c0 2024-03-14T13:24:06.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.709+0000 7f37da7fc700 1 -- 172.21.15.111:0/2190962707 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f37e80fd440 con 0x7f37d406c7c0 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.711+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 msgr2=0x7f37d406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.711+0000 7f37efaee700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 0x7f37d406ec70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b0d0 tx=0x7f37e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.711+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 msgr2=0x7f37e80fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.711+0000 7f37efaee700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00eb40 tx=0x7f37dc00b040 comp rx=0 tx=0).stop 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 shutdown_connections 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f37d406c7c0 0x7f37d406ec70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37e80fde90 0x7f37e80fbd50 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 --2- 172.21.15.111:0/2190962707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37e80fe830 0x7f37e80fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 >> 172.21.15.111:0/2190962707 conn(0x7f37e8072bc0 msgr2=0x7f37e80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 shutdown_connections 2024-03-14T13:24:06.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.712+0000 7f37efaee700 1 -- 172.21.15.111:0/2190962707 wait complete. 2024-03-14T13:24:06.718 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- 172.21.15.111:0/117919072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 msgr2=0x7fb000106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 --2- 172.21.15.111:0/117919072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000106f00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7faffc004940 tx=0x7faffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- 172.21.15.111:0/117919072 shutdown_connections 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 --2- 172.21.15.111:0/117919072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000106f00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 --2- 172.21.15.111:0/117919072 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb0000fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- 172.21.15.111:0/117919072 >> 172.21.15.111:0/117919072 conn(0x7fb000072bc0 msgr2=0x7fb000072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- 172.21.15.111:0/117919072 shutdown_connections 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- 172.21.15.111:0/117919072 wait complete. 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 Processor -- start 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- start start 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb000113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.811 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb000114690 con 0x7fb0000fe830 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.813+0000 7fb008304700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00010da80 con 0x7fb0000fde90 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55748/0 (socket says 172.21.15.111:55748) 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 -- 172.21.15.111:0/4228647815 learned_addr learned my addr 172.21.15.111:0/4228647815 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb0060a0700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb000113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 -- 172.21.15.111:0/4228647815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 msgr2=0x7fb000113b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb000113b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 -- 172.21.15.111:0/4228647815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc005040 con 0x7fb0000fe830 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb0060a0700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb000113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb00589f700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faffc009360 tx=0x7faffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc00e040 con 0x7fb0000fe830 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faffc013420 con 0x7fb0000fe830 2024-03-14T13:24:06.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00010dd00 con 0x7fb0000fe830 2024-03-14T13:24:06.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faffc01c630 con 0x7fb0000fe830 2024-03-14T13:24:06.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.814+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00010e240 con 0x7fb0000fe830 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.815+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb000065e30 con 0x7fb0000fe830 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.815+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faffc0139e0 con 0x7fb0000fe830 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.816+0000 7faff6ffd700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 0x7fafec06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.816+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faffc08cbc0 con 0x7fb0000fe830 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.816+0000 7fb0060a0700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 0x7fafec06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:06.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.816+0000 7fb0060a0700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 0x7fafec06ec70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faff0009fd0 tx=0x7faff000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:06.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:06.820+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faffc05a990 con 0x7fb0000fe830 2024-03-14T13:24:07.027 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.031+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb000109df0 con 0x7fafec06c7c0 2024-03-14T13:24:07.028 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.032+0000 7faff6ffd700 1 -- 172.21.15.111:0/4228647815 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7fb000109df0 con 0x7fafec06c7c0 2024-03-14T13:24:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 msgr2=0x7fafec06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 0x7fafec06ec70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faff0009fd0 tx=0x7faff000b040 comp rx=0 tx=0).stop 2024-03-14T13:24:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 msgr2=0x7fb000114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7faffc009360 tx=0x7faffc009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 shutdown_connections 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fafec06c7c0 0x7fafec06ec70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0000fde90 0x7fb000113b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 --2- 172.21.15.111:0/4228647815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0000fe830 0x7fb000114040 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 >> 172.21.15.111:0/4228647815 conn(0x7fb000072bc0 msgr2=0x7fb000103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 shutdown_connections 2024-03-14T13:24:07.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.034+0000 7fb008304700 1 -- 172.21.15.111:0/4228647815 wait complete. 2024-03-14T13:24:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 -- 172.21.15.111:0/2796558572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fde90 msgr2=0x7fe4bc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/2796558572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc0fe260 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0004940 tx=0x7fe4b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 -- 172.21.15.111:0/2796558572 shutdown_connections 2024-03-14T13:24:07.130 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/2796558572 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/2796558572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc0fe260 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 -- 172.21.15.111:0/2796558572 >> 172.21.15.111:0/2796558572 conn(0x7fe4bc072bc0 msgr2=0x7fe4bc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 -- 172.21.15.111:0/2796558572 shutdown_connections 2024-03-14T13:24:07.131 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 -- 172.21.15.111:0/2796558572 wait complete. 2024-03-14T13:24:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.134+0000 7fe4c1e55700 1 Processor -- start 2024-03-14T13:24:07.132 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4c1e55700 1 -- start start 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4c1e55700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4c1e55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4c1e55700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc114680 con 0x7fe4bc0fe830 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4c1e55700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10da70 con 0x7fe4bc0fde90 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55770/0 (socket says 172.21.15.111:55770) 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 -- 172.21.15.111:0/3840773329 learned_addr learned my addr 172.21.15.111:0/3840773329 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4bb7fe700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 -- 172.21.15.111:0/3840773329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fde90 msgr2=0x7fe4bc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 -- 172.21.15.111:0/3840773329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b0005040 con 0x7fe4bc0fe830 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.135+0000 7fe4baffd700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00c8c0 tx=0x7fe4ac0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4ac00a980 con 0x7fe4bc0fe830 2024-03-14T13:24:07.133 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4ac00ce60 con 0x7fe4bc0fe830 2024-03-14T13:24:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4bc10dd50 con 0x7fe4bc0fe830 2024-03-14T13:24:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4ac0113a0 con 0x7fe4bc0fe830 2024-03-14T13:24:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc10e290 con 0x7fe4bc0fe830 2024-03-14T13:24:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.136+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4bc1019b0 con 0x7fe4bc0fe830 2024-03-14T13:24:07.134 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.137+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe4ac012400 con 0x7fe4bc0fe830 2024-03-14T13:24:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.138+0000 7fe4a3fff700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 0x7fe4a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.138+0000 7fe4bb7fe700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 0x7fe4a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.138+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe4ac08c680 con 0x7fe4bc0fe830 2024-03-14T13:24:07.135 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.138+0000 7fe4bb7fe700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 0x7fe4a406ec70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b3b0 tx=0x7fe4b0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.142+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe4ac05a390 con 0x7fe4bc0fe830 2024-03-14T13:24:07.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.346+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4bc1137c0 con 0x7fe4a406c7c0 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.357+0000 7fe4a3fff700 1 -- 172.21.15.111:0/3840773329 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fe4bc1137c0 con 0x7fe4a406c7c0 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (20m) 13m ago 31m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (17m) 13m ago 32m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (17m) 13m ago 30m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (19m) 13m ago 31m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (13m) 13m ago 25m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:24:07.353 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (14m) 13m ago 25m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (13m) 13m ago 25m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (13m) 13m ago 25m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (22m) 13m ago 33m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (21m) 13m ago 30m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (18m) 13m ago 33m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (18m) 13m ago 30m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (21m) 13m ago 32m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (21m) 13m ago 30m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (17m) 13m ago 29m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (16m) 13m ago 29m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (16m) 13m ago 28m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (15m) 13m ago 28m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (15m) 13m ago 27m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:24:07.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (14m) 13m ago 27m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:24:07.355 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (20m) 13m ago 31m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:24:07.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 msgr2=0x7fe4a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 0x7fe4a406ec70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b3b0 tx=0x7fe4b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 msgr2=0x7fe4bc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00c8c0 tx=0x7fe4ac0044c0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 shutdown_connections 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe4a406c7c0 0x7fe4a406ec70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4bc0fde90 0x7fe4bc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 --2- 172.21.15.111:0/3840773329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc0fe830 0x7fe4bc114030 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 >> 172.21.15.111:0/3840773329 conn(0x7fe4bc072bc0 msgr2=0x7fe4bc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 shutdown_connections 2024-03-14T13:24:07.356 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.359+0000 7fe4c1e55700 1 -- 172.21.15.111:0/3840773329 wait complete. 2024-03-14T13:24:07.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 -- 172.21.15.111:0/2565220465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa64101520 msgr2=0x7ffa641018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 --2- 172.21.15.111:0/2565220465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa64101520 0x7ffa641018f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ffa54004940 tx=0x7ffa54009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 -- 172.21.15.111:0/2565220465 shutdown_connections 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 --2- 172.21.15.111:0/2565220465 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa640fb4c0 0x7ffa640fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 --2- 172.21.15.111:0/2565220465 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa64101520 0x7ffa641018f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 -- 172.21.15.111:0/2565220465 >> 172.21.15.111:0/2565220465 conn(0x7ffa64072310 msgr2=0x7ffa64072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 -- 172.21.15.111:0/2565220465 shutdown_connections 2024-03-14T13:24:07.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.459+0000 7ffa6b436700 1 -- 172.21.15.111:0/2565220465 wait complete. 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 Processor -- start 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 -- start start 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa64101520 0x7ffa64113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64114440 con 0x7ffa640fb4c0 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa6b436700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6410da40 con 0x7ffa64101520 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa691d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa691d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55790/0 (socket says 172.21.15.111:55790) 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa691d2700 1 -- 172.21.15.111:0/3486521580 learned_addr learned my addr 172.21.15.111:0/3486521580 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.460+0000 7ffa689d1700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa64101520 0x7ffa64113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa691d2700 1 -- 172.21.15.111:0/3486521580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa64101520 msgr2=0x7ffa64113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa691d2700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa64101520 0x7ffa64113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa691d2700 1 -- 172.21.15.111:0/3486521580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa54005040 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa691d2700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400b3b0 tx=0x7ffa54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5400e040 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa54013420 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6410dcc0 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa5401c610 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.461+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6410e280 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.462+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa640fcc00 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.463+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ffa54013a00 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.463+0000 7ffa59ffb700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 0x7ffa5006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.463+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa5408cbf0 con 0x7ffa640fb4c0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.463+0000 7ffa689d1700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 0x7ffa5006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.463+0000 7ffa689d1700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 0x7ffa5006ec70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffa6410ec00 tx=0x7ffa6000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.467+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffa54093080 con 0x7ffa640fb4c0 2024-03-14T13:24:07.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.743+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ffa64108450 con 0x7ffa640fb4c0 2024-03-14T13:24:07.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.743+0000 7ffa59ffb700 1 -- 172.21.15.111:0/3486521580 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7ffa5405a900 con 0x7ffa640fb4c0 2024-03-14T13:24:07.739 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:24:07.740 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:24:07.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:24:07.741 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:24:07.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 msgr2=0x7ffa5006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 0x7ffa5006ec70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffa6410ec00 tx=0x7ffa6000b040 comp rx=0 tx=0).stop 2024-03-14T13:24:07.741 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 msgr2=0x7ffa641138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400b3b0 tx=0x7ffa54009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 shutdown_connections 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5006c7c0 0x7ffa5006ec70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa640fb4c0 0x7ffa641138b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 --2- 172.21.15.111:0/3486521580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa64101520 0x7ffa64113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 >> 172.21.15.111:0/3486521580 conn(0x7ffa64072310 msgr2=0x7ffa640ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 shutdown_connections 2024-03-14T13:24:07.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.745+0000 7ffa6b436700 1 -- 172.21.15.111:0/3486521580 wait complete. 2024-03-14T13:24:07.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 -- 172.21.15.111:0/1098437663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b74102e10 msgr2=0x7f4b741031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.841 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 --2- 172.21.15.111:0/1098437663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b74102e10 0x7f4b741031e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4b64004940 tx=0x7f4b64009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 -- 172.21.15.111:0/1098437663 shutdown_connections 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 --2- 172.21.15.111:0/1098437663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b741037b0 0x7f4b740fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 --2- 172.21.15.111:0/1098437663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b74102e10 0x7f4b741031e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 -- 172.21.15.111:0/1098437663 >> 172.21.15.111:0/1098437663 conn(0x7f4b740f8980 msgr2=0x7f4b740fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 -- 172.21.15.111:0/1098437663 shutdown_connections 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 -- 172.21.15.111:0/1098437663 wait complete. 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.845+0000 7f4b7a179700 1 Processor -- start 2024-03-14T13:24:07.842 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b7a179700 1 -- start start 2024-03-14T13:24:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b7a179700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74102e10 0x7f4b74113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b7a179700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b7a179700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b741146a0 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b7a179700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7410da90 con 0x7f4b74102e10 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55794/0 (socket says 172.21.15.111:55794) 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 -- 172.21.15.111:0/2124690060 learned_addr learned my addr 172.21.15.111:0/2124690060 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b737fe700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74102e10 0x7f4b74113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 -- 172.21.15.111:0/2124690060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74102e10 msgr2=0x7f4b74113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74102e10 0x7f4b74113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 -- 172.21.15.111:0/2124690060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64005040 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.846+0000 7f4b72ffd700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4b68002a50 tx=0x7f4b68004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.847+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b68007cc0 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.847+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b6800f040 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.847+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7410dd70 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.847+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b6800d920 con 0x7f4b741037b0 2024-03-14T13:24:07.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.847+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7410e330 con 0x7f4b741037b0 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.848+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b740fe560 con 0x7f4b741037b0 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.848+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4b68009680 con 0x7f4b741037b0 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.848+0000 7f4b5bfff700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 0x7f4b5c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.849+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4b6808b810 con 0x7f4b741037b0 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.849+0000 7f4b737fe700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 0x7f4b5c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:07.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.849+0000 7f4b737fe700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 0x7f4b5c06ec70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4b64004f90 tx=0x7f4b64009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:07.849 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:07.853+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4b680594a0 con 0x7f4b741037b0 2024-03-14T13:24:07.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:07 smithi111 ceph-mon[145314]: pgmap v600: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:08.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:07 smithi137 ceph-mon[108145]: pgmap v600: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:08.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.088+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4b74109dd0 con 0x7f4b741037b0 2024-03-14T13:24:08.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.088+0000 7f4b5bfff700 1 -- 172.21.15.111:0/2124690060 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f4b68059030 con 0x7f4b741037b0 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:24:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:24:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:24:08.087 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:08.088 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:08.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.090+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 msgr2=0x7f4b5c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.090+0000 7f4b7a179700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 0x7f4b5c06ec70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4b64004f90 tx=0x7f4b64009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:08.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.090+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 msgr2=0x7f4b74114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.090+0000 7f4b7a179700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4b68002a50 tx=0x7f4b68004600 comp rx=0 tx=0).stop 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 shutdown_connections 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4b5c06c7c0 0x7f4b5c06ec70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b74102e10 0x7f4b74113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 --2- 172.21.15.111:0/2124690060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4b741037b0 0x7f4b74114050 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 >> 172.21.15.111:0/2124690060 conn(0x7f4b740f8980 msgr2=0x7f4b740f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 shutdown_connections 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.091+0000 7f4b7a179700 1 -- 172.21.15.111:0/2124690060 wait complete. 2024-03-14T13:24:08.090 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- 172.21.15.111:0/325501445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 msgr2=0x7febf00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 --2- 172.21.15.111:0/325501445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf00fbd80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7febe0004940 tx=0x7febe0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- 172.21.15.111:0/325501445 shutdown_connections 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 --2- 172.21.15.111:0/325501445 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf00fbd80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 --2- 172.21.15.111:0/325501445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 0x7febf0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- 172.21.15.111:0/325501445 >> 172.21.15.111:0/325501445 conn(0x7febf0072bc0 msgr2=0x7febf0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- 172.21.15.111:0/325501445 shutdown_connections 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- 172.21.15.111:0/325501445 wait complete. 2024-03-14T13:24:08.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 Processor -- start 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- start start 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 0x7febf0115ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf0116870 con 0x7febf0102710 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.192+0000 7febf4a9c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf010fc90 con 0x7febf0101d70 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55804/0 (socket says 172.21.15.111:55804) 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 -- 172.21.15.111:0/3105577637 learned_addr learned my addr 172.21.15.111:0/3105577637 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febee59c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 0x7febf0115ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 -- 172.21.15.111:0/3105577637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 msgr2=0x7febf0115ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 0x7febf0115ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 -- 172.21.15.111:0/3105577637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0005040 con 0x7febf0102710 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febedd9b700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7febe00093b0 tx=0x7febe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:08.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febe000e040 con 0x7febf0102710 2024-03-14T13:24:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf010ff10 con 0x7febf0102710 2024-03-14T13:24:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7febe0013420 con 0x7febf0102710 2024-03-14T13:24:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7febe001c630 con 0x7febf0102710 2024-03-14T13:24:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.193+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf01104d0 con 0x7febf0102710 2024-03-14T13:24:08.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.194+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf00fd4c0 con 0x7febf0102710 2024-03-14T13:24:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.195+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7febe00139e0 con 0x7febf0102710 2024-03-14T13:24:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.195+0000 7febe6ffd700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 0x7febdc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.195+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7febe008cbc0 con 0x7febf0102710 2024-03-14T13:24:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.195+0000 7febee59c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 0x7febdc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.193 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.195+0000 7febee59c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 0x7febdc06ec70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7febd8004f60 tx=0x7febd800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:08.195 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.199+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7febe005a980 con 0x7febf0102710 2024-03-14T13:24:08.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.407+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7febf0108d80 con 0x7febdc06c7c0 2024-03-14T13:24:08.404 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.409+0000 7febe6ffd700 1 -- 172.21.15.111:0/3105577637 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7febf0108d80 con 0x7febdc06c7c0 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:24:08.405 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:24:08.406 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:24:08.406 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.410+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 msgr2=0x7febdc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.410+0000 7febf4a9c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 0x7febdc06ec70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7febd8004f60 tx=0x7febd800b040 comp rx=0 tx=0).stop 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 msgr2=0x7febf0116220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7febe00093b0 tx=0x7febe0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 shutdown_connections 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7febdc06c7c0 0x7febdc06ec70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7febf0101d70 0x7febf0115ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 --2- 172.21.15.111:0/3105577637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7febf0102710 0x7febf0116220 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 >> 172.21.15.111:0/3105577637 conn(0x7febf0072bc0 msgr2=0x7febf0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 shutdown_connections 2024-03-14T13:24:08.407 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.411+0000 7febf4a9c700 1 -- 172.21.15.111:0/3105577637 wait complete. 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 -- 172.21.15.111:0/2784868952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fde20 msgr2=0x7f62800fe1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 --2- 172.21.15.111:0/2784868952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fde20 0x7f62800fe1f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6270004970 tx=0x7f6270009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 -- 172.21.15.111:0/2784868952 shutdown_connections 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 --2- 172.21.15.111:0/2784868952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fe7c0 0x7f6280106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 --2- 172.21.15.111:0/2784868952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fde20 0x7f62800fe1f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 -- 172.21.15.111:0/2784868952 >> 172.21.15.111:0/2784868952 conn(0x7f62800f7880 msgr2=0x7f62800f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 -- 172.21.15.111:0/2784868952 shutdown_connections 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.510+0000 7f628699b700 1 -- 172.21.15.111:0/2784868952 wait complete. 2024-03-14T13:24:08.507 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 Processor -- start 2024-03-14T13:24:08.508 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 -- start start 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fde20 0x7f6280113d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801148d0 con 0x7f62800fe7c0 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f628699b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628010dce0 con 0x7f62800fde20 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f627f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f627f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:55818/0 (socket says 172.21.15.111:55818) 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.511+0000 7f627f7fe700 1 -- 172.21.15.111:0/2605483056 learned_addr learned my addr 172.21.15.111:0/2605483056 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627ffff700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fde20 0x7f6280113d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.509 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627f7fe700 1 -- 172.21.15.111:0/2605483056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fde20 msgr2=0x7f6280113d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627f7fe700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fde20 0x7f6280113d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627f7fe700 1 -- 172.21.15.111:0/2605483056 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6270005040 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627f7fe700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6274002a50 tx=0x7f62740046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6274007cc0 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628010dfc0 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f627400f040 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f627400d9c0 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.512+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628010e500 con 0x7f62800fe7c0 2024-03-14T13:24:08.510 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.513+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62800666d0 con 0x7f62800fe7c0 2024-03-14T13:24:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.513+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f627400db20 con 0x7f62800fe7c0 2024-03-14T13:24:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.514+0000 7f627cff9700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 0x7f626806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.514+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f627408b7a0 con 0x7f62800fe7c0 2024-03-14T13:24:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.514+0000 7f627ffff700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 0x7f626806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:08.511 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.514+0000 7f627ffff700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 0x7f626806ec70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6270004f50 tx=0x7f6270009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:08.514 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.518+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f62740559a0 con 0x7f62800fe7c0 2024-03-14T13:24:08.789 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.793+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f6280109dd0 con 0x7f62800fe7c0 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.794+0000 7f627cff9700 1 -- 172.21.15.111:0/2605483056 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7f6274058fc0 con 0x7f62800fe7c0 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:24:08.790 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 msgr2=0x7f626806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 0x7f626806ec70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6270004f50 tx=0x7f6270009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 msgr2=0x7f6280114280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6274002a50 tx=0x7f62740046b0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 shutdown_connections 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f626806c7c0 0x7f626806ec70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62800fde20 0x7f6280113d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.792 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 --2- 172.21.15.111:0/2605483056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800fe7c0 0x7f6280114280 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:08.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 >> 172.21.15.111:0/2605483056 conn(0x7f62800f7880 msgr2=0x7f62800f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:08.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 shutdown_connections 2024-03-14T13:24:08.793 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:08.796+0000 7f628699b700 1 -- 172.21.15.111:0/2605483056 wait complete. 2024-03-14T13:24:09.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:08 smithi137 ceph-mon[108145]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:08 smithi137 ceph-mon[108145]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:08 smithi137 ceph-mon[108145]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:08 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3486521580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:24:09.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:08 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2124690060' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:24:09.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:08 smithi111 ceph-mon[145314]: from='client.35128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:08 smithi111 ceph-mon[145314]: from='client.35132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:08 smithi111 ceph-mon[145314]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:09.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:08 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3486521580' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:24:09.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:08 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2124690060' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:24:10.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:09 smithi137 ceph-mon[108145]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:10.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:09 smithi137 ceph-mon[108145]: pgmap v601: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:10.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:09 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2605483056' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:24:10.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:10.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:09 smithi111 ceph-mon[145314]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:10.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:09 smithi111 ceph-mon[145314]: pgmap v601: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:10.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:09 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2605483056' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:24:10.136 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:11.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:10 smithi137 ceph-mon[108145]: pgmap v602: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:11.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:10 smithi111 ceph-mon[145314]: pgmap v602: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:13.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:13 smithi111 ceph-mon[145314]: pgmap v603: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:14.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:13 smithi137 ceph-mon[108145]: pgmap v603: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:15.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:15 smithi111 ceph-mon[145314]: pgmap v604: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:16.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:15 smithi137 ceph-mon[108145]: pgmap v604: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:17.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:17 smithi111 ceph-mon[145314]: pgmap v605: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:18.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:17 smithi137 ceph-mon[108145]: pgmap v605: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:19.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:19 smithi111 ceph-mon[145314]: pgmap v606: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:20.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:19 smithi137 ceph-mon[108145]: pgmap v606: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:21.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:21 smithi111 ceph-mon[145314]: pgmap v607: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:22.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:21 smithi137 ceph-mon[108145]: pgmap v607: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:23.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:23 smithi111 ceph-mon[145314]: pgmap v608: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:24.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:23 smithi137 ceph-mon[108145]: pgmap v608: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:24.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:25.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:25.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:25 smithi111 ceph-mon[145314]: pgmap v609: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:26.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:25 smithi137 ceph-mon[108145]: pgmap v609: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:27.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:27 smithi111 ceph-mon[145314]: pgmap v610: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:28.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:27 smithi137 ceph-mon[108145]: pgmap v610: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:29.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:29 smithi111 ceph-mon[145314]: pgmap v611: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:30.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:29 smithi137 ceph-mon[108145]: pgmap v611: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:31.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:31 smithi111 ceph-mon[145314]: pgmap v612: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:32.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:31 smithi137 ceph-mon[108145]: pgmap v612: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:33.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:33 smithi111 ceph-mon[145314]: pgmap v613: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:34.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:33 smithi137 ceph-mon[108145]: pgmap v613: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:35.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:35 smithi111 ceph-mon[145314]: pgmap v614: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:36.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:35 smithi137 ceph-mon[108145]: pgmap v614: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:37.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:37 smithi111 ceph-mon[145314]: pgmap v615: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:38.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:37 smithi137 ceph-mon[108145]: pgmap v615: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:38.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 -- 172.21.15.111:0/2911946413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 msgr2=0x7f2048102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 --2- 172.21.15.111:0/2911946413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f2048102140 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2030004940 tx=0x7f2030009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 -- 172.21.15.111:0/2911946413 shutdown_connections 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 --2- 172.21.15.111:0/2911946413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 0x7f20480fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 --2- 172.21.15.111:0/2911946413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f2048102140 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 -- 172.21.15.111:0/2911946413 >> 172.21.15.111:0/2911946413 conn(0x7f2048072bc0 msgr2=0x7f2048072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 -- 172.21.15.111:0/2911946413 shutdown_connections 2024-03-14T13:24:38.905 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.910+0000 7f204d012700 1 -- 172.21.15.111:0/2911946413 wait complete. 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 Processor -- start 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 -- start start 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 0x7f204810ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204810f670 con 0x7f2048101d70 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f204d012700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204810f7b0 con 0x7f2048102710 2024-03-14T13:24:38.907 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f2046d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f2046d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46926/0 (socket says 172.21.15.111:46926) 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.911+0000 7f2046d9d700 1 -- 172.21.15.111:0/2739021904 learned_addr learned my addr 172.21.15.111:0/2739021904 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f204659c700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 0x7f204810ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f2046d9d700 1 -- 172.21.15.111:0/2739021904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 msgr2=0x7f204810ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f2046d9d700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 0x7f204810ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f2046d9d700 1 -- 172.21.15.111:0/2739021904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2030005040 con 0x7f2048101d70 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f2046d9d700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2030009180 tx=0x7f2030009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203000e040 con 0x7f2048101d70 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2048117ac0 con 0x7f2048101d70 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2030004290 con 0x7f2048101d70 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203001c3f0 con 0x7f2048101d70 2024-03-14T13:24:38.908 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.912+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2048118050 con 0x7f2048101d70 2024-03-14T13:24:38.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.913+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20480ff080 con 0x7f2048101d70 2024-03-14T13:24:38.909 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.914+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2030013790 con 0x7f2048101d70 2024-03-14T13:24:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.914+0000 7f203f7fe700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 0x7f203406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.914+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f203008cd30 con 0x7f2048101d70 2024-03-14T13:24:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.914+0000 7f204659c700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 0x7f203406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:38.910 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.915+0000 7f204659c700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 0x7f203406ec70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2038006fd0 tx=0x7f2038008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:38.913 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:38.919+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2030056fb0 con 0x7f2048101d70 2024-03-14T13:24:39.121 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.126+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f204804bed0 con 0x7f203406c7c0 2024-03-14T13:24:39.122 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.128+0000 7f203f7fe700 1 -- 172.21.15.111:0/2739021904 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f204804bed0 con 0x7f203406c7c0 2024-03-14T13:24:39.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 msgr2=0x7f203406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.124 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 0x7f203406ec70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2038006fd0 tx=0x7f2038008040 comp rx=0 tx=0).stop 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 msgr2=0x7f204810ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2030009180 tx=0x7f2030009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 shutdown_connections 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f203406c7c0 0x7f203406ec70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2048101d70 0x7f204810ea50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 --2- 172.21.15.111:0/2739021904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2048102710 0x7f204810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 >> 172.21.15.111:0/2739021904 conn(0x7f2048072bc0 msgr2=0x7f20480faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 shutdown_connections 2024-03-14T13:24:39.125 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.130+0000 7f204d012700 1 -- 172.21.15.111:0/2739021904 wait complete. 2024-03-14T13:24:39.134 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:24:39.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 -- 172.21.15.111:0/3458591578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fde90 msgr2=0x7f73dc0fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 --2- 172.21.15.111:0/3458591578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fde90 0x7f73dc0fe260 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f73d8004940 tx=0x7f73d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 -- 172.21.15.111:0/3458591578 shutdown_connections 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 --2- 172.21.15.111:0/3458591578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fe830 0x7f73dc106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 --2- 172.21.15.111:0/3458591578 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fde90 0x7f73dc0fe260 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.230+0000 7f73e449d700 1 -- 172.21.15.111:0/3458591578 >> 172.21.15.111:0/3458591578 conn(0x7f73dc072bc0 msgr2=0x7f73dc072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.231+0000 7f73e449d700 1 -- 172.21.15.111:0/3458591578 shutdown_connections 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.231+0000 7f73e449d700 1 -- 172.21.15.111:0/3458591578 wait complete. 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.231+0000 7f73e449d700 1 Processor -- start 2024-03-14T13:24:39.226 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.231+0000 7f73e449d700 1 -- start start 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.231+0000 7f73e449d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fde90 0x7f73dc113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e449d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e449d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc114680 con 0x7f73dc0fe830 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e449d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc10da70 con 0x7f73dc0fde90 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46944/0 (socket says 172.21.15.111:46944) 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 -- 172.21.15.111:0/3916535981 learned_addr learned my addr 172.21.15.111:0/3916535981 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:39.228 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e2239700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fde90 0x7f73dc113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 -- 172.21.15.111:0/3916535981 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fde90 msgr2=0x7f73dc113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fde90 0x7f73dc113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 -- 172.21.15.111:0/3916535981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8005040 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e1a38700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f73cc002a50 tx=0x7f73cc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73cc007cc0 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73cc00f040 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc10dd50 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73cc009540 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.232+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73dc10e290 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.233+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73dc1019b0 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.234+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f73cc0096a0 con 0x7f73dc0fe830 2024-03-14T13:24:39.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.234+0000 7f73d2ffd700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 0x7f73c806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.234+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f73cc08b8a0 con 0x7f73dc0fe830 2024-03-14T13:24:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.234+0000 7f73e2239700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 0x7f73c806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.234+0000 7f73e2239700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 0x7f73c806ec70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f73d800b3b0 tx=0x7f73d8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.232 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.238+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f73cc059530 con 0x7f73dc0fe830 2024-03-14T13:24:39.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.464+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f73dc10f140 con 0x7f73c806c7c0 2024-03-14T13:24:39.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.466+0000 7f73d2ffd700 1 -- 172.21.15.111:0/3916535981 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f73dc10f140 con 0x7f73c806c7c0 2024-03-14T13:24:39.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 msgr2=0x7f73c806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 0x7f73c806ec70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f73d800b3b0 tx=0x7f73d8009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 msgr2=0x7f73dc114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f73cc002a50 tx=0x7f73cc0046b0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 shutdown_connections 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f73c806c7c0 0x7f73c806ec70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc0fde90 0x7f73dc113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 --2- 172.21.15.111:0/3916535981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73dc0fe830 0x7f73dc114030 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 >> 172.21.15.111:0/3916535981 conn(0x7f73dc072bc0 msgr2=0x7f73dc0ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 shutdown_connections 2024-03-14T13:24:39.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.468+0000 7f73e449d700 1 -- 172.21.15.111:0/3916535981 wait complete. 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 -- 172.21.15.111:0/988017904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 msgr2=0x7fe1b81020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 --2- 172.21.15.111:0/988017904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b81020e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac004940 tx=0x7fe1ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 -- 172.21.15.111:0/988017904 shutdown_connections 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 --2- 172.21.15.111:0/988017904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b80fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 --2- 172.21.15.111:0/988017904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b81020e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 -- 172.21.15.111:0/988017904 >> 172.21.15.111:0/988017904 conn(0x7fe1b80f7880 msgr2=0x7fe1b80f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 -- 172.21.15.111:0/988017904 shutdown_connections 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.568+0000 7fe1befb8700 1 -- 172.21.15.111:0/988017904 wait complete. 2024-03-14T13:24:39.563 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 Processor -- start 2024-03-14T13:24:39.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 -- start start 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b8113230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b8113910 con 0x7fe1b8101d10 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1befb8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b810cc50 con 0x7fe1b81026b0 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46956/0 (socket says 172.21.15.111:46956) 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 -- 172.21.15.111:0/279877654 learned_addr learned my addr 172.21.15.111:0/279877654 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1b7fff700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b8113230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 -- 172.21.15.111:0/279877654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 msgr2=0x7fe1b8113230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.566 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b8113230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1bcd54700 1 -- 172.21.15.111:0/279877654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac005040 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.569+0000 7fe1b7fff700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b8113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1bcd54700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009180 tx=0x7fe1ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac00e040 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1ac004290 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b810ce70 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac01c3f0 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.570+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b810d430 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.571+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b80fd460 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.571+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe1ac013790 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.572+0000 7fe1b57fa700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 0x7fe1a806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.572+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe1ac08cd30 con 0x7fe1b8101d10 2024-03-14T13:24:39.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.572+0000 7fe1b7fff700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 0x7fe1a806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.572+0000 7fe1b7fff700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 0x7fe1a806ec70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1b810e590 tx=0x7fe1a4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.576+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe1ac05aa40 con 0x7fe1b8101d10 2024-03-14T13:24:39.775 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.781+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1b80fd460 con 0x7fe1a806c7c0 2024-03-14T13:24:39.784 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.790+0000 7fe1b57fa700 1 -- 172.21.15.111:0/279877654 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fe1b80fd460 con 0x7fe1a806c7c0 2024-03-14T13:24:39.784 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (20m) 14m ago 32m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (18m) 14m ago 32m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (18m) 13m ago 31m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (20m) 14m ago 31m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (14m) 14m ago 26m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (14m) 14m ago 26m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (13m) 13m ago 26m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (14m) 13m ago 26m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (23m) 14m ago 33m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (22m) 13m ago 31m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (19m) 14m ago 33m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (19m) 13m ago 30m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (21m) 14m ago 32m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:24:39.785 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (21m) 13m ago 31m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (17m) 14m ago 30m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (17m) 14m ago 29m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (16m) 14m ago 29m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (16m) 13m ago 28m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (15m) 13m ago 28m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (15m) 13m ago 27m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:24:39.786 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (21m) 14m ago 31m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 msgr2=0x7fe1a806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 0x7fe1a806ec70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1b810e590 tx=0x7fe1a4008040 comp rx=0 tx=0).stop 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 msgr2=0x7fe1b8112cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009180 tx=0x7fe1ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 shutdown_connections 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1a806c7c0 0x7fe1a806ec70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1b8101d10 0x7fe1b8112cf0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 --2- 172.21.15.111:0/279877654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1b81026b0 0x7fe1b8113230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 >> 172.21.15.111:0/279877654 conn(0x7fe1b80f7880 msgr2=0x7fe1b80f8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 shutdown_connections 2024-03-14T13:24:39.787 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.792+0000 7fe1befb8700 1 -- 172.21.15.111:0/279877654 wait complete. 2024-03-14T13:24:39.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- 172.21.15.111:0/2530594054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 msgr2=0x7fe9200fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 --2- 172.21.15.111:0/2530594054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe9200fbd30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe91c004940 tx=0x7fe91c009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:39.885 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- 172.21.15.111:0/2530594054 shutdown_connections 2024-03-14T13:24:39.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 --2- 172.21.15.111:0/2530594054 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe9200fbd30 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 --2- 172.21.15.111:0/2530594054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 0x7fe9201020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- 172.21.15.111:0/2530594054 >> 172.21.15.111:0/2530594054 conn(0x7fe9200f78d0 msgr2=0x7fe9200f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:39.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- 172.21.15.111:0/2530594054 shutdown_connections 2024-03-14T13:24:39.886 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- 172.21.15.111:0/2530594054 wait complete. 2024-03-14T13:24:39.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 Processor -- start 2024-03-14T13:24:39.887 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.891+0000 7fe928637700 1 -- start start 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe928637700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 0x7fe920112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe928637700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe928637700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe920113590 con 0x7fe9201026c0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe928637700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92010c980 con 0x7fe920101d20 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46978/0 (socket says 172.21.15.111:46978) 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 -- 172.21.15.111:0/7627834 learned_addr learned my addr 172.21.15.111:0/7627834 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe9263d3700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 0x7fe920112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 -- 172.21.15.111:0/7627834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 msgr2=0x7fe920112a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 0x7fe920112a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 -- 172.21.15.111:0/7627834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c005040 con 0x7fe9201026c0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe925bd2700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0093b0 tx=0x7fe91c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91c00e040 con 0x7fe9201026c0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe91c013420 con 0x7fe9201026c0 2024-03-14T13:24:39.888 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92010cc00 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.892+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe91c01c630 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.893+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92010d1c0 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.893+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9200fd470 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.894+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe91c0139e0 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.894+0000 7fe916ffd700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 0x7fe90c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.894+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe91c08cbc0 con 0x7fe9201026c0 2024-03-14T13:24:39.889 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.894+0000 7fe9263d3700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 0x7fe90c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:39.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.895+0000 7fe9263d3700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 0x7fe90c06ec70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe910004f40 tx=0x7fe91000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:39.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:39.899+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe91c05a980 con 0x7fe9201026c0 2024-03-14T13:24:40.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:39 smithi137 ceph-mon[108145]: pgmap v616: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:40.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:40.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:39 smithi111 ceph-mon[145314]: pgmap v616: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:40.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:40.163 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.169+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe92005f5f0 con 0x7fe9201026c0 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.169+0000 7fe916ffd700 1 -- 172.21.15.111:0/7627834 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fe91c05a510 con 0x7fe9201026c0 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:40.164 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:24:40.165 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.171+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 msgr2=0x7fe90c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.171+0000 7fe928637700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 0x7fe90c06ec70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe910004f40 tx=0x7fe91000b040 comp rx=0 tx=0).stop 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.171+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 msgr2=0x7fe920112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.171+0000 7fe928637700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe91c0093b0 tx=0x7fe91c009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 shutdown_connections 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe90c06c7c0 0x7fe90c06ec70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe920101d20 0x7fe920112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 --2- 172.21.15.111:0/7627834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9201026c0 0x7fe920112f40 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 >> 172.21.15.111:0/7627834 conn(0x7fe9200f78d0 msgr2=0x7fe9200f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 shutdown_connections 2024-03-14T13:24:40.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.172+0000 7fe928637700 1 -- 172.21.15.111:0/7627834 wait complete. 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.271+0000 7f34e825b700 1 -- 172.21.15.111:0/3423543404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 msgr2=0x7f34e00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.271+0000 7f34e825b700 1 --2- 172.21.15.111:0/3423543404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e00fce10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f34d0004940 tx=0x7f34d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- 172.21.15.111:0/3423543404 shutdown_connections 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 --2- 172.21.15.111:0/3423543404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e00fce10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 --2- 172.21.15.111:0/3423543404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- 172.21.15.111:0/3423543404 >> 172.21.15.111:0/3423543404 conn(0x7f34e00f89b0 msgr2=0x7f34e00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- 172.21.15.111:0/3423543404 shutdown_connections 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- 172.21.15.111:0/3423543404 wait complete. 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 Processor -- start 2024-03-14T13:24:40.268 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- start start 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e0113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0114650 con 0x7f34e01037a0 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.272+0000 7f34e825b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e010da40 con 0x7f34e0102e00 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46990/0 (socket says 172.21.15.111:46990) 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 -- 172.21.15.111:0/3208385621 learned_addr learned my addr 172.21.15.111:0/3208385621 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e5ff7700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e0113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 -- 172.21.15.111:0/3208385621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 msgr2=0x7f34e0113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e0113ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 -- 172.21.15.111:0/3208385621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0005040 con 0x7f34e01037a0 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e5ff7700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e0113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e57f6700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009380 tx=0x7f34d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.269 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d000e040 con 0x7f34e01037a0 2024-03-14T13:24:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34d0013420 con 0x7f34e01037a0 2024-03-14T13:24:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34d001c630 con 0x7f34e01037a0 2024-03-14T13:24:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e010dcc0 con 0x7f34e01037a0 2024-03-14T13:24:40.270 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.273+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e010e230 con 0x7f34e01037a0 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.274+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e00fe550 con 0x7f34e01037a0 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.275+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f34d00139e0 con 0x7f34e01037a0 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.275+0000 7f34d67fc700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 0x7f34cc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.275+0000 7f34e5ff7700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 0x7f34cc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.275+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f34d008cc60 con 0x7f34e01037a0 2024-03-14T13:24:40.271 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.276+0000 7f34e5ff7700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 0x7f34cc06ec70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0041d0 tx=0x7f34dc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.274 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.280+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f34d005a970 con 0x7f34e01037a0 2024-03-14T13:24:40.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.519+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f34e0109da0 con 0x7f34e01037a0 2024-03-14T13:24:40.513 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.519+0000 7f34d67fc700 1 -- 172.21.15.111:0/3208385621 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7f34d005a500 con 0x7f34e01037a0 2024-03-14T13:24:40.513 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:24:40.513 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:24:40.514 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:24:40.515 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:40.516 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 msgr2=0x7f34cc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 0x7f34cc06ec70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0041d0 tx=0x7f34dc009040 comp rx=0 tx=0).stop 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 msgr2=0x7f34e0114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009380 tx=0x7f34d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 shutdown_connections 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f34cc06c7c0 0x7f34cc06ec70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.517 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34e0102e00 0x7f34e0113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 --2- 172.21.15.111:0/3208385621 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e01037a0 0x7f34e0114000 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 >> 172.21.15.111:0/3208385621 conn(0x7f34e00f89b0 msgr2=0x7f34e00f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 shutdown_connections 2024-03-14T13:24:40.518 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.521+0000 7f34e825b700 1 -- 172.21.15.111:0/3208385621 wait complete. 2024-03-14T13:24:40.518 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:24:40.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1266827407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 msgr2=0x7ffbb01020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.617 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1266827407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb01020e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ffba0004940 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1266827407 shutdown_connections 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1266827407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 0x7ffbb00fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1266827407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb01020e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1266827407 >> 172.21.15.111:0/1266827407 conn(0x7ffbb00f7880 msgr2=0x7ffbb00f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1266827407 shutdown_connections 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.621+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1266827407 wait complete. 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 Processor -- start 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 -- start start 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 0x7ffbb0114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb01148e0 con 0x7ffbb0101d10 2024-03-14T13:24:40.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbb4eeb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb010dcb0 con 0x7ffbb01026b0 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42946/0 (socket says 172.21.15.111:42946) 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbae59c700 1 -- 172.21.15.111:0/1069664961 learned_addr learned my addr 172.21.15.111:0/1069664961 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.622+0000 7ffbadd9b700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 0x7ffbb0114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbae59c700 1 -- 172.21.15.111:0/1069664961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 msgr2=0x7ffbb0114290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbae59c700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 0x7ffbb0114290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbae59c700 1 -- 172.21.15.111:0/1069664961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0005040 con 0x7ffbb0101d10 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbae59c700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000c00 tx=0x7ffba0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba0021070 con 0x7ffbb0101d10 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba0013de0 con 0x7ffbb0101d10 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb010df30 con 0x7ffbb0101d10 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba000e070 con 0x7ffbb0101d10 2024-03-14T13:24:40.619 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.623+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb010e490 con 0x7ffbb0101d10 2024-03-14T13:24:40.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.624+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb00fd460 con 0x7ffbb0101d10 2024-03-14T13:24:40.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.624+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ffba001c600 con 0x7ffbb0101d10 2024-03-14T13:24:40.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.625+0000 7ffb9effd700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 0x7ffb9806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.625+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffba008cb60 con 0x7ffbb0101d10 2024-03-14T13:24:40.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.625+0000 7ffbadd9b700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 0x7ffb9806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.625+0000 7ffbadd9b700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 0x7ffb9806ec70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffba4004600 tx=0x7ffba400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.629+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffba005a870 con 0x7ffbb0101d10 2024-03-14T13:24:40.836 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.842+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ffbb004bed0 con 0x7ffb9806c7c0 2024-03-14T13:24:40.837 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.843+0000 7ffb9effd700 1 -- 172.21.15.111:0/1069664961 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ffbb004bed0 con 0x7ffb9806c7c0 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:24:40.838 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:24:40.839 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:24:40.839 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:24:40.839 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:24:40.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 msgr2=0x7ffb9806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.839 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 0x7ffb9806ec70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffba4004600 tx=0x7ffba400b040 comp rx=0 tx=0).stop 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 msgr2=0x7ffbb0113d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000c00 tx=0x7ffba0009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 shutdown_connections 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffb9806c7c0 0x7ffb9806ec70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbb0101d10 0x7ffbb0113d50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 --2- 172.21.15.111:0/1069664961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffbb01026b0 0x7ffbb0114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 >> 172.21.15.111:0/1069664961 conn(0x7ffbb00f7880 msgr2=0x7ffbb00f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 shutdown_connections 2024-03-14T13:24:40.840 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.845+0000 7ffbb4eeb700 1 -- 172.21.15.111:0/1069664961 wait complete. 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/2358250884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac101d70 msgr2=0x7ff1ac102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/2358250884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac101d70 0x7ff1ac102140 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff19c004940 tx=0x7ff19c009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/2358250884 shutdown_connections 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/2358250884 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac102710 0x7ff1ac0fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/2358250884 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac101d70 0x7ff1ac102140 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/2358250884 >> 172.21.15.111:0/2358250884 conn(0x7ff1ac072bc0 msgr2=0x7ff1ac072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/2358250884 shutdown_connections 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/2358250884 wait complete. 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 Processor -- start 2024-03-14T13:24:40.941 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.945+0000 7ff1b3c6c700 1 -- start start 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac101d70 0x7ff1ac1129b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac1135d0 con 0x7ff1ac102710 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac10c930 con 0x7ff1ac101d70 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42972/0 (socket says 172.21.15.111:42972) 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 -- 172.21.15.111:0/1476313617 learned_addr learned my addr 172.21.15.111:0/1476313617 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1a08700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac101d70 0x7ff1ac1129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 -- 172.21.15.111:0/1476313617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac101d70 msgr2=0x7ff1ac1129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac101d70 0x7ff1ac1129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 -- 172.21.15.111:0/1476313617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c005040 con 0x7ff1ac102710 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b1207700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800b790 tx=0x7ff1a8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a800f6f0 con 0x7ff1ac102710 2024-03-14T13:24:40.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1a800fd00 con 0x7ff1ac102710 2024-03-14T13:24:40.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac10cc10 con 0x7ff1ac102710 2024-03-14T13:24:40.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1a80103a0 con 0x7ff1ac102710 2024-03-14T13:24:40.943 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.946+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac10d1d0 con 0x7ff1ac102710 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.947+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac0ff080 con 0x7ff1ac102710 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.948+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff1a800f850 con 0x7ff1ac102710 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.948+0000 7ff1a27fc700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 0x7ff19806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.948+0000 7ff1b1a08700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 0x7ff19806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.948+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff1a808b920 con 0x7ff1ac102710 2024-03-14T13:24:40.944 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.949+0000 7ff1b1a08700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 0x7ff19806ec70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff19c004f90 tx=0x7ff19c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:24:40.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:40.953+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff1a80595b0 con 0x7ff1ac102710 2024-03-14T13:24:41.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:40 smithi137 ceph-mon[108145]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:41.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:40 smithi137 ceph-mon[108145]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:41.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:40 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/7627834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:24:41.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:40 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3208385621' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:24:41.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:40 smithi111 ceph-mon[145314]: from='client.35156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:41.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:40 smithi111 ceph-mon[145314]: from='client.35160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:41.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:40 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/7627834' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:24:41.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:40 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3208385621' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.223+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff1ac10e000 con 0x7ff1ac102710 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.224+0000 7ff1a27fc700 1 -- 172.21.15.111:0/1476313617 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff1a8059140 con 0x7ff1ac102710 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:24:41.218 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:24:41.219 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:24:41.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 msgr2=0x7ff19806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:41.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 0x7ff19806ec70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff19c004f90 tx=0x7ff19c009e30 comp rx=0 tx=0).stop 2024-03-14T13:24:41.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 msgr2=0x7ff1ac112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800b790 tx=0x7ff1a8009220 comp rx=0 tx=0).stop 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 shutdown_connections 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff19806c7c0 0x7ff19806ec70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1ac101d70 0x7ff1ac1129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 --2- 172.21.15.111:0/1476313617 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff1ac102710 0x7ff1ac112ef0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 >> 172.21.15.111:0/1476313617 conn(0x7ff1ac072bc0 msgr2=0x7ff1ac0faf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 shutdown_connections 2024-03-14T13:24:41.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:24:41.226+0000 7ff1b3c6c700 1 -- 172.21.15.111:0/1476313617 wait complete. 2024-03-14T13:24:42.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:41 smithi137 ceph-mon[108145]: from='client.35164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:42.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:41 smithi137 ceph-mon[108145]: pgmap v617: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:42.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:41 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1476313617' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:24:42.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:41 smithi111 ceph-mon[145314]: from='client.35164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:42.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:41 smithi111 ceph-mon[145314]: pgmap v617: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:42.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:41 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1476313617' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:24:43.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:42 smithi137 ceph-mon[108145]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:43.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:42 smithi137 ceph-mon[108145]: pgmap v618: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:43.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:42 smithi111 ceph-mon[145314]: from='client.35176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:24:43.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:42 smithi111 ceph-mon[145314]: pgmap v618: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:45.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:45 smithi111 ceph-mon[145314]: pgmap v619: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:45 smithi137 ceph-mon[108145]: pgmap v619: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:47.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:47 smithi111 ceph-mon[145314]: pgmap v620: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:48.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:47 smithi137 ceph-mon[108145]: pgmap v620: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:49.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:49 smithi111 ceph-mon[145314]: pgmap v621: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:50.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:49 smithi137 ceph-mon[108145]: pgmap v621: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:51.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:51 smithi111 ceph-mon[145314]: pgmap v622: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:52.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:51 smithi137 ceph-mon[108145]: pgmap v622: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:53.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:53 smithi111 ceph-mon[145314]: pgmap v623: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:54.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:53 smithi137 ceph-mon[108145]: pgmap v623: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:54.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:55.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:24:55.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:55 smithi111 ceph-mon[145314]: pgmap v624: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:56.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:55 smithi137 ceph-mon[108145]: pgmap v624: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:57.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:57 smithi111 ceph-mon[145314]: pgmap v625: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:58.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:57 smithi137 ceph-mon[108145]: pgmap v625: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:24:59.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:24:59 smithi111 ceph-mon[145314]: pgmap v626: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:00.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:24:59 smithi137 ceph-mon[108145]: pgmap v626: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:01.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:01 smithi111 ceph-mon[145314]: pgmap v627: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:02.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:01 smithi137 ceph-mon[108145]: pgmap v627: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:04.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:03 smithi137 ceph-mon[108145]: pgmap v628: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:04.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:03 smithi111 ceph-mon[145314]: pgmap v628: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:06.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:05 smithi137 ceph-mon[108145]: pgmap v629: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:06.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:05 smithi111 ceph-mon[145314]: pgmap v629: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:07.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:07 smithi111 ceph-mon[145314]: pgmap v630: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:08.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:07 smithi137 ceph-mon[108145]: pgmap v630: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:10.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:09 smithi137 ceph-mon[108145]: pgmap v631: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:10.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:10.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:09 smithi111 ceph-mon[145314]: pgmap v631: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:10.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.340+0000 7f468f1db700 1 -- 172.21.15.111:0/1924156799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4688102de0 msgr2=0x7f46881031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.340+0000 7f468f1db700 1 --2- 172.21.15.111:0/1924156799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4688102de0 0x7f46881031b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f467c004940 tx=0x7f467c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 -- 172.21.15.111:0/1924156799 shutdown_connections 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 --2- 172.21.15.111:0/1924156799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46880fb430 0x7f46880fb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 --2- 172.21.15.111:0/1924156799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4688102de0 0x7f46881031b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 -- 172.21.15.111:0/1924156799 >> 172.21.15.111:0/1924156799 conn(0x7f4688071450 msgr2=0x7f4688071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 -- 172.21.15.111:0/1924156799 shutdown_connections 2024-03-14T13:25:11.337 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 -- 172.21.15.111:0/1924156799 wait complete. 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 Processor -- start 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.341+0000 7f468f1db700 1 -- start start 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4688102de0 0x7f46880fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46880fd0e0 con 0x7f46880fb430 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46880fd220 con 0x7f4688102de0 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52684/0 (socket says 172.21.15.111:52684) 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 -- 172.21.15.111:0/3186187737 learned_addr learned my addr 172.21.15.111:0/3186187737 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f4687fff700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4688102de0 0x7f46880fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 -- 172.21.15.111:0/3186187737 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4688102de0 msgr2=0x7f46880fcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4688102de0 0x7f46880fcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 -- 172.21.15.111:0/3186187737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c005040 con 0x7f46880fb430 2024-03-14T13:25:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468cf77700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f467c00b3b0 tx=0x7f467c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c00e040 con 0x7f46880fb430 2024-03-14T13:25:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f467c013420 con 0x7f46880fb430 2024-03-14T13:25:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46880fd470 con 0x7f46880fb430 2024-03-14T13:25:11.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f467c01c610 con 0x7f46880fb430 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.342+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468811a140 con 0x7f46880fb430 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.343+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46881000a0 con 0x7f46880fb430 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.344+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f467c013a00 con 0x7f46880fb430 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.345+0000 7f46857fa700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 0x7f467806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.345+0000 7f4687fff700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 0x7f467806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.345+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f467c08d750 con 0x7f46880fb430 2024-03-14T13:25:11.340 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.345+0000 7f4687fff700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 0x7f467806ec70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f46880fdf50 tx=0x7f467400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.349+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f467c0579d0 con 0x7f46880fb430 2024-03-14T13:25:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.560+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f468804bed0 con 0x7f467806c7c0 2024-03-14T13:25:11.555 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.561+0000 7f46857fa700 1 -- 172.21.15.111:0/3186187737 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f468804bed0 con 0x7f467806c7c0 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 msgr2=0x7f467806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 0x7f467806ec70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f46880fdf50 tx=0x7f467400b040 comp rx=0 tx=0).stop 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 msgr2=0x7f46880fc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f467c00b3b0 tx=0x7f467c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 shutdown_connections 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f467806c7c0 0x7f467806ec70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46880fb430 0x7f46880fc610 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.558 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 --2- 172.21.15.111:0/3186187737 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4688102de0 0x7f46880fcb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 >> 172.21.15.111:0/3186187737 conn(0x7f4688071450 msgr2=0x7f46880fa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 shutdown_connections 2024-03-14T13:25:11.559 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.563+0000 7f468f1db700 1 -- 172.21.15.111:0/3186187737 wait complete. 2024-03-14T13:25:11.568 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.664+0000 7f6e0286b700 1 -- 172.21.15.111:0/4293257886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 msgr2=0x7f6dfc0fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.664+0000 7f6e0286b700 1 --2- 172.21.15.111:0/4293257886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc0fbd20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6dec004940 tx=0x7f6dec009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.664+0000 7f6e0286b700 1 -- 172.21.15.111:0/4293257886 shutdown_connections 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.664+0000 7f6e0286b700 1 --2- 172.21.15.111:0/4293257886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc0fbd20 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.664+0000 7f6e0286b700 1 --2- 172.21.15.111:0/4293257886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6dfc101d10 0x7f6dfc1020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- 172.21.15.111:0/4293257886 >> 172.21.15.111:0/4293257886 conn(0x7f6dfc0f7880 msgr2=0x7f6dfc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- 172.21.15.111:0/4293257886 shutdown_connections 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- 172.21.15.111:0/4293257886 wait complete. 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 Processor -- start 2024-03-14T13:25:11.661 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- start start 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6dfc101d10 0x7f6dfc115ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc116b80 con 0x7f6dfc1026b0 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.665+0000 7f6e0286b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc10ff50 con 0x7f6dfc101d10 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52694/0 (socket says 172.21.15.111:52694) 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 -- 172.21.15.111:0/274259604 learned_addr learned my addr 172.21.15.111:0/274259604 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 -- 172.21.15.111:0/274259604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6dfc101d10 msgr2=0x7f6dfc115ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6dfc101d10 0x7f6dfc115ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 -- 172.21.15.111:0/274259604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec005040 con 0x7f6dfc1026b0 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6dfb7fe700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009380 tx=0x7f6dec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.662 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dec00e040 con 0x7f6dfc1026b0 2024-03-14T13:25:11.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dec013420 con 0x7f6dfc1026b0 2024-03-14T13:25:11.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc1101d0 con 0x7f6dfc1026b0 2024-03-14T13:25:11.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dec01c630 con 0x7f6dfc1026b0 2024-03-14T13:25:11.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.666+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc110790 con 0x7f6dfc1026b0 2024-03-14T13:25:11.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.667+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc0fd460 con 0x7f6dfc1026b0 2024-03-14T13:25:11.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.667+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6dec0139e0 con 0x7f6dfc1026b0 2024-03-14T13:25:11.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.668+0000 7f6df8ff9700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 0x7f6de806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.668+0000 7f6dfbfff700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 0x7f6de806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.668+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6dec08cb40 con 0x7f6dfc1026b0 2024-03-14T13:25:11.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.668+0000 7f6dfbfff700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 0x7f6de806ec70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004d00 tx=0x7f6de4009ce0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.672+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6dec05a910 con 0x7f6dfc1026b0 2024-03-14T13:25:11.875 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.880+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dfc108d20 con 0x7f6de806c7c0 2024-03-14T13:25:11.876 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.881+0000 7f6df8ff9700 1 -- 172.21.15.111:0/274259604 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f6dfc108d20 con 0x7f6de806c7c0 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 msgr2=0x7f6de806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 0x7f6de806ec70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6de4004d00 tx=0x7f6de4009ce0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 msgr2=0x7f6dfc116530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6dec009380 tx=0x7f6dec009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 shutdown_connections 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6de806c7c0 0x7f6de806ec70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.878 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6dfc101d10 0x7f6dfc115ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 --2- 172.21.15.111:0/274259604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6dfc1026b0 0x7f6dfc116530 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.883+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 >> 172.21.15.111:0/274259604 conn(0x7f6dfc0f7880 msgr2=0x7f6dfc0f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:11.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.884+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 shutdown_connections 2024-03-14T13:25:11.879 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.884+0000 7f6e0286b700 1 -- 172.21.15.111:0/274259604 wait complete. 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 -- 172.21.15.111:0/738462003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 msgr2=0x7fcb1c1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 --2- 172.21.15.111:0/738462003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c1031f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c004940 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 -- 172.21.15.111:0/738462003 shutdown_connections 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 --2- 172.21.15.111:0/738462003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 0x7fcb1c0fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 --2- 172.21.15.111:0/738462003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c1031f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 -- 172.21.15.111:0/738462003 >> 172.21.15.111:0/738462003 conn(0x7fcb1c0f89d0 msgr2=0x7fcb1c0fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 -- 172.21.15.111:0/738462003 shutdown_connections 2024-03-14T13:25:11.980 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.983+0000 7fcb22789700 1 -- 172.21.15.111:0/738462003 wait complete. 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 Processor -- start 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 -- start start 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 0x7fcb1c114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c114710 con 0x7fcb1c102e20 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb22789700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c10da70 con 0x7fcb1c1037c0 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb1bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb1bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52716/0 (socket says 172.21.15.111:52716) 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.984+0000 7fcb1bfff700 1 -- 172.21.15.111:0/321159134 learned_addr learned my addr 172.21.15.111:0/321159134 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb1b7fe700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 0x7fcb1c114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb1bfff700 1 -- 172.21.15.111:0/321159134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 msgr2=0x7fcb1c114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb1bfff700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 0x7fcb1c114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:11.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb1bfff700 1 -- 172.21.15.111:0/321159134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c005040 con 0x7fcb1c102e20 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb1bfff700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00b3b0 tx=0x7fcb0c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c00e040 con 0x7fcb1c102e20 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c10dcf0 con 0x7fcb1c102e20 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb0c013420 con 0x7fcb1c102e20 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c01c630 con 0x7fcb1c102e20 2024-03-14T13:25:11.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.985+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c10e2b0 con 0x7fcb1c102e20 2024-03-14T13:25:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.986+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c0fe570 con 0x7fcb1c102e20 2024-03-14T13:25:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.987+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcb0c013970 con 0x7fcb1c102e20 2024-03-14T13:25:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.987+0000 7fcb18ff9700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 0x7fcb0406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.987+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcb0c08d750 con 0x7fcb1c102e20 2024-03-14T13:25:11.983 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.987+0000 7fcb1b7fe700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 0x7fcb0406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:11.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.988+0000 7fcb1b7fe700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 0x7fcb0406ec70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcb10006fd0 tx=0x7fcb10008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:11.986 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:11.991+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcb0c05b520 con 0x7fcb1c102e20 2024-03-14T13:25:11.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:11 smithi137 ceph-mon[108145]: pgmap v632: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:12.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:11 smithi111 ceph-mon[145314]: pgmap v632: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:12.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.190+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb1c04bed0 con 0x7fcb0406c7c0 2024-03-14T13:25:12.194 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.199+0000 7fcb18ff9700 1 -- 172.21.15.111:0/321159134 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7fcb1c04bed0 con 0x7fcb0406c7c0 2024-03-14T13:25:12.194 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:25:12.194 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (21m) 14m ago 32m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:25:12.194 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (18m) 14m ago 33m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:25:12.194 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (18m) 14m ago 31m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (21m) 14m ago 32m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (14m) 14m ago 26m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (15m) 14m ago 26m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (14m) 14m ago 26m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (14m) 14m ago 26m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (24m) 14m ago 34m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (22m) 14m ago 31m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (19m) 14m ago 34m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (19m) 14m ago 31m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (22m) 14m ago 33m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (22m) 14m ago 31m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (18m) 14m ago 30m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:25:12.195 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (17m) 14m ago 30m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:25:12.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (17m) 14m ago 29m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:25:12.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (16m) 14m ago 29m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:25:12.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (16m) 14m ago 28m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:25:12.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (15m) 14m ago 28m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:25:12.196 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (21m) 14m ago 32m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.201+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 msgr2=0x7fcb0406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.201+0000 7fcb22789700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 0x7fcb0406ec70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcb10006fd0 tx=0x7fcb10008040 comp rx=0 tx=0).stop 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 msgr2=0x7fcb1c113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00b3b0 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 shutdown_connections 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcb0406c7c0 0x7fcb0406ec70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb1c102e20 0x7fcb1c113af0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 --2- 172.21.15.111:0/321159134 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb1c1037c0 0x7fcb1c114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 >> 172.21.15.111:0/321159134 conn(0x7fcb1c0f89d0 msgr2=0x7fcb1c0f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 shutdown_connections 2024-03-14T13:25:12.197 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.202+0000 7fcb22789700 1 -- 172.21.15.111:0/321159134 wait complete. 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- 172.21.15.111:0/1814315863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05241026b0 msgr2=0x7f05240fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 --2- 172.21.15.111:0/1814315863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05241026b0 0x7f05240fbd20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0514004940 tx=0x7f0514009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- 172.21.15.111:0/1814315863 shutdown_connections 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 --2- 172.21.15.111:0/1814315863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05241026b0 0x7f05240fbd20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 --2- 172.21.15.111:0/1814315863 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0524101d10 0x7f05241020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- 172.21.15.111:0/1814315863 >> 172.21.15.111:0/1814315863 conn(0x7f05240f7880 msgr2=0x7f05240f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- 172.21.15.111:0/1814315863 shutdown_connections 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- 172.21.15.111:0/1814315863 wait complete. 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 Processor -- start 2024-03-14T13:25:12.296 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 -- start start 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.300+0000 7f0528bf9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0528bf9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05241026b0 0x7f05241030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0528bf9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0524103640 con 0x7f0524101d10 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0528bf9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0524103780 con 0x7f05241026b0 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52728/0 (socket says 172.21.15.111:52728) 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 -- 172.21.15.111:0/2994053969 learned_addr learned my addr 172.21.15.111:0/2994053969 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 -- 172.21.15.111:0/2994053969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05241026b0 msgr2=0x7f05241030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05241026b0 0x7f05241030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 -- 172.21.15.111:0/2994053969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514005040 con 0x7f0524101d10 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f052259c700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0518002a50 tx=0x7f05180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0518007cc0 con 0x7f0524101d10 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f051800f040 con 0x7f0524101d10 2024-03-14T13:25:12.297 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052411a120 con 0x7f0524101d10 2024-03-14T13:25:12.298 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0518009540 con 0x7f0524101d10 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.301+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052411a5a0 con 0x7f0524101d10 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.302+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05240fd460 con 0x7f0524101d10 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.303+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f05180096a0 con 0x7f0524101d10 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.303+0000 7f0512ffd700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 0x7f050c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.303+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f051808b7a0 con 0x7f0524101d10 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.303+0000 7f0521d9b700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 0x7f050c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:12.299 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.303+0000 7f0521d9b700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 0x7f050c06ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0514004f90 tx=0x7f0514009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:12.302 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.307+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0518055a20 con 0x7f0524101d10 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.583+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f052404bed0 con 0x7f0524101d10 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.583+0000 7f0512ffd700 1 -- 172.21.15.111:0/2994053969 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f0518059040 con 0x7f0524101d10 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:25:12.578 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:25:12.579 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 msgr2=0x7f050c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 0x7f050c06ec70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0514004f90 tx=0x7f0514009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 msgr2=0x7f0524102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0518002a50 tx=0x7f05180046b0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 shutdown_connections 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f050c06c7c0 0x7f050c06ec70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0524101d10 0x7f0524102b70 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 --2- 172.21.15.111:0/2994053969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05241026b0 0x7f05241030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 >> 172.21.15.111:0/2994053969 conn(0x7f05240f7880 msgr2=0x7f05240f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:12.580 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 shutdown_connections 2024-03-14T13:25:12.581 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.585+0000 7f0528bf9700 1 -- 172.21.15.111:0/2994053969 wait complete. 2024-03-14T13:25:12.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.688+0000 7fd294d6f700 1 -- 172.21.15.111:0/1105631993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2901026c0 msgr2=0x7fd2900fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.685 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.688+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1105631993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2901026c0 0x7fd2900fbd30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd284004940 tx=0x7fd284009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- 172.21.15.111:0/1105631993 shutdown_connections 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1105631993 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2901026c0 0x7fd2900fbd30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1105631993 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd290101d20 0x7fd2901020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- 172.21.15.111:0/1105631993 >> 172.21.15.111:0/1105631993 conn(0x7fd2900f7870 msgr2=0x7fd2900f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- 172.21.15.111:0/1105631993 shutdown_connections 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- 172.21.15.111:0/1105631993 wait complete. 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 Processor -- start 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- start start 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2901026c0 0x7fd29010efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.686 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29010f500 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.689+0000 7fd294d6f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29010f640 con 0x7fd2901026c0 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52746/0 (socket says 172.21.15.111:52746) 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 -- 172.21.15.111:0/1645551854 learned_addr learned my addr 172.21.15.111:0/1645551854 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28dd9b700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2901026c0 0x7fd29010efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 -- 172.21.15.111:0/1645551854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2901026c0 msgr2=0x7fd29010efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2901026c0 0x7fd29010efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 -- 172.21.15.111:0/1645551854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284005040 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd28e59c700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd280002a50 tx=0x7fd2800046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd280007cc0 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd290117af0 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd28000f040 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28000d9c0 con 0x7fd290101d20 2024-03-14T13:25:12.687 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.690+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd290117e40 con 0x7fd290101d20 2024-03-14T13:25:12.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.691+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2900fd470 con 0x7fd290101d20 2024-03-14T13:25:12.688 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.691+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd280009070 con 0x7fd290101d20 2024-03-14T13:25:12.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.692+0000 7fd27effd700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 0x7fd27806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:12.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.692+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd28008b740 con 0x7fd290101d20 2024-03-14T13:25:12.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.692+0000 7fd28dd9b700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 0x7fd27806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:12.689 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.692+0000 7fd28dd9b700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 0x7fd27806ec70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd284004890 tx=0x7fd284009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:12.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.696+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd2800559c0 con 0x7fd290101d20 2024-03-14T13:25:12.924 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.929+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd290066810 con 0x7fd290101d20 2024-03-14T13:25:12.924 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.930+0000 7fd27effd700 1 -- 172.21.15.111:0/1645551854 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7fd280058fe0 con 0x7fd290101d20 2024-03-14T13:25:12.924 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:25:12.925 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:25:12.926 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 msgr2=0x7fd27806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 0x7fd27806ec70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd284004890 tx=0x7fd284009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:12.928 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 msgr2=0x7fd29010ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd280002a50 tx=0x7fd2800046b0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 shutdown_connections 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd27806c7c0 0x7fd27806ec70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd290101d20 0x7fd29010ea80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 --2- 172.21.15.111:0/1645551854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2901026c0 0x7fd29010efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 >> 172.21.15.111:0/1645551854 conn(0x7fd2900f7870 msgr2=0x7fd2900f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 shutdown_connections 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:12.932+0000 7fd294d6f700 1 -- 172.21.15.111:0/1645551854 wait complete. 2024-03-14T13:25:12.929 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:25:12.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:12 smithi137 ceph-mon[108145]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:13.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:12 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2994053969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.032+0000 7f5aecc5d700 1 -- 172.21.15.111:0/4019921860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae8101d20 msgr2=0x7f5ae81020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.032+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/4019921860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae8101d20 0x7f5ae81020f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8004940 tx=0x7f5ad8009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- 172.21.15.111:0/4019921860 shutdown_connections 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/4019921860 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae81026c0 0x7f5ae80fbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/4019921860 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae8101d20 0x7f5ae81020f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.029 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- 172.21.15.111:0/4019921860 >> 172.21.15.111:0/4019921860 conn(0x7f5ae80f7870 msgr2=0x7f5ae80f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- 172.21.15.111:0/4019921860 shutdown_connections 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- 172.21.15.111:0/4019921860 wait complete. 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 Processor -- start 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- start start 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 0x7f5ae8113900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8114490 con 0x7f5ae81026c0 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5aecc5d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae810da90 con 0x7f5ae8101d20 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5ae5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5ae5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52760/0 (socket says 172.21.15.111:52760) 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.033+0000 7f5ae5d9b700 1 -- 172.21.15.111:0/336925985 learned_addr learned my addr 172.21.15.111:0/336925985 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae659c700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 0x7f5ae8113900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.030 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae5d9b700 1 -- 172.21.15.111:0/336925985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 msgr2=0x7f5ae8113900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae5d9b700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 0x7f5ae8113900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae5d9b700 1 -- 172.21.15.111:0/336925985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8005040 con 0x7f5ae81026c0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae659c700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 0x7f5ae8113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ae5d9b700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5adc002a50 tx=0x7f5adc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc007cc0 con 0x7f5ae81026c0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5adc00f040 con 0x7f5ae81026c0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae810dd70 con 0x7f5ae81026c0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5adc00d9a0 con 0x7f5ae81026c0 2024-03-14T13:25:13.031 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.034+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae810e330 con 0x7f5ae81026c0 2024-03-14T13:25:13.032 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.035+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae80fd470 con 0x7f5ae81026c0 2024-03-14T13:25:13.032 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.035+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5adc00dc60 con 0x7f5ae81026c0 2024-03-14T13:25:13.032 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.036+0000 7f5ad6ffd700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 0x7f5ad006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.032 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.036+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5adc08b8b0 con 0x7f5ae81026c0 2024-03-14T13:25:13.032 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.036+0000 7f5ae659c700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 0x7f5ad006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.033 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.036+0000 7f5ae659c700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 0x7f5ad006ec70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8004560 tx=0x7f5ad8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:13.035 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.040+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5adc059540 con 0x7f5ae81026c0 2024-03-14T13:25:13.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:12 smithi111 ceph-mon[145314]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:13.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:12 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2994053969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:13.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.249+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ae8108ce0 con 0x7f5ad006c7c0 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.250+0000 7f5ad6ffd700 1 -- 172.21.15.111:0/336925985 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f5ae8108ce0 con 0x7f5ad006c7c0 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:25:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:25:13.246 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 msgr2=0x7f5ad006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 0x7f5ad006ec70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8004560 tx=0x7f5ad8009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 msgr2=0x7f5ae8113e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5adc002a50 tx=0x7f5adc004600 comp rx=0 tx=0).stop 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 shutdown_connections 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ad006c7c0 0x7f5ad006ec70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ae8101d20 0x7f5ae8113900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.247 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 --2- 172.21.15.111:0/336925985 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae81026c0 0x7f5ae8113e40 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.252+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 >> 172.21.15.111:0/336925985 conn(0x7f5ae80f7870 msgr2=0x7f5ae80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.253+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 shutdown_connections 2024-03-14T13:25:13.248 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.253+0000 7f5aecc5d700 1 -- 172.21.15.111:0/336925985 wait complete. 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 -- 172.21.15.111:0/1391894432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b0102e10 msgr2=0x7ff5b01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 --2- 172.21.15.111:0/1391894432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b0102e10 0x7ff5b01031e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac004940 tx=0x7ff5ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 -- 172.21.15.111:0/1391894432 shutdown_connections 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 --2- 172.21.15.111:0/1391894432 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b01037b0 0x7ff5b00fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 --2- 172.21.15.111:0/1391894432 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b0102e10 0x7ff5b01031e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.352+0000 7ff5b744e700 1 -- 172.21.15.111:0/1391894432 >> 172.21.15.111:0/1391894432 conn(0x7ff5b00f89a0 msgr2=0x7ff5b00fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 -- 172.21.15.111:0/1391894432 shutdown_connections 2024-03-14T13:25:13.349 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 -- 172.21.15.111:0/1391894432 wait complete. 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 Processor -- start 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 -- start start 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 0x7ff5b0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b0114680 con 0x7ff5b01037b0 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b744e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b010da70 con 0x7ff5b0102e10 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.353+0000 7ff5b49e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:52764/0 (socket says 172.21.15.111:52764) 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 -- 172.21.15.111:0/2521915158 learned_addr learned my addr 172.21.15.111:0/2521915158 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b51ea700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 0x7ff5b0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 -- 172.21.15.111:0/2521915158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 msgr2=0x7ff5b0113af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.350 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 0x7ff5b0113af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 -- 172.21.15.111:0/2521915158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac005040 con 0x7ff5b01037b0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b51ea700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 0x7ff5b0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b49e9700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c890 tx=0x7ff5a00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a000a7d0 con 0x7ff5b01037b0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b010dd50 con 0x7ff5b01037b0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5a000ade0 con 0x7ff5b01037b0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a00113a0 con 0x7ff5b01037b0 2024-03-14T13:25:13.351 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.354+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b010e310 con 0x7ff5b01037b0 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.355+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b00fe560 con 0x7ff5b01037b0 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.356+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff5a0012400 con 0x7ff5b01037b0 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.356+0000 7ff5a5ffb700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 0x7ff59c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.356+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff5a008bb70 con 0x7ff5b01037b0 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.356+0000 7ff5b51ea700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 0x7ff59c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:13.352 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.357+0000 7ff5b51ea700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 0x7ff59c06ec70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac004f90 tx=0x7ff5ac009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:13.355 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.361+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff5a0059880 con 0x7ff5b01037b0 2024-03-14T13:25:13.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.629+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff5b0109dd0 con 0x7ff5b01037b0 2024-03-14T13:25:13.624 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.629+0000 7ff5a5ffb700 1 -- 172.21.15.111:0/2521915158 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+233 (secure 0 0 0) 0x7ff5a0059410 con 0x7ff5b01037b0 2024-03-14T13:25:13.624 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:25:13.624 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:25:13.625 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:25:13.625 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:25:13.625 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 msgr2=0x7ff59c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 0x7ff59c06ec70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac004f90 tx=0x7ff5ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 msgr2=0x7ff5b0114030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c890 tx=0x7ff5a00044c0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 shutdown_connections 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff59c06c7c0 0x7ff59c06ec70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff5b0102e10 0x7ff5b0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 --2- 172.21.15.111:0/2521915158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5b01037b0 0x7ff5b0114030 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:13.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 >> 172.21.15.111:0/2521915158 conn(0x7ff5b00f89a0 msgr2=0x7ff5b00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:13.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 shutdown_connections 2024-03-14T13:25:13.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:13.631+0000 7ff5b744e700 1 -- 172.21.15.111:0/2521915158 wait complete. 2024-03-14T13:25:14.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:13 smithi137 ceph-mon[108145]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:14.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:13 smithi137 ceph-mon[108145]: from='client.35192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:14.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:13 smithi137 ceph-mon[108145]: pgmap v633: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:14.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:13 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1645551854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:25:14.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:13 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2521915158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:25:14.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:13 smithi111 ceph-mon[145314]: from='client.35188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:14.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:13 smithi111 ceph-mon[145314]: from='client.35192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:14.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:13 smithi111 ceph-mon[145314]: pgmap v633: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:14.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:13 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1645551854' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:25:14.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:13 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2521915158' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:25:14.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:14 smithi137 ceph-mon[108145]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:15.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:14 smithi137 ceph-mon[108145]: pgmap v634: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:15.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:14 smithi111 ceph-mon[145314]: from='client.35204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:15.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:14 smithi111 ceph-mon[145314]: pgmap v634: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:17.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:17 smithi111 ceph-mon[145314]: pgmap v635: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:17.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:17 smithi137 ceph-mon[108145]: pgmap v635: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:19.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:19 smithi111 ceph-mon[145314]: pgmap v636: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:19.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:19 smithi137 ceph-mon[108145]: pgmap v636: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:21.868 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:21 smithi137 ceph-mon[108145]: pgmap v637: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:21.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:21 smithi111 ceph-mon[145314]: pgmap v637: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:23.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:23 smithi111 ceph-mon[145314]: pgmap v638: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:23.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:23 smithi137 ceph-mon[108145]: pgmap v638: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:24.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:24.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:25.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:25 smithi111 ceph-mon[145314]: pgmap v639: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:25.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:25 smithi137 ceph-mon[108145]: pgmap v639: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:27.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:27 smithi111 ceph-mon[145314]: pgmap v640: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:27.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:27 smithi137 ceph-mon[108145]: pgmap v640: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:29.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:29 smithi137 ceph-mon[108145]: pgmap v641: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:30.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:29 smithi111 ceph-mon[145314]: pgmap v641: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:31.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:31 smithi137 ceph-mon[108145]: pgmap v642: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:32.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:31 smithi111 ceph-mon[145314]: pgmap v642: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:33.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:33 smithi137 ceph-mon[108145]: pgmap v643: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:34.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:33 smithi111 ceph-mon[145314]: pgmap v643: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:35.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:35 smithi137 ceph-mon[108145]: pgmap v644: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:36.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:35 smithi111 ceph-mon[145314]: pgmap v644: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:37.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:37 smithi111 ceph-mon[145314]: pgmap v645: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:37.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:37 smithi137 ceph-mon[108145]: pgmap v645: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:39.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:39 smithi137 ceph-mon[108145]: pgmap v646: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:39.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:40.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:39 smithi111 ceph-mon[145314]: pgmap v646: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:40.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:41.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:41 smithi137 ceph-mon[108145]: pgmap v647: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:42.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:41 smithi111 ceph-mon[145314]: pgmap v647: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:43.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 -- 172.21.15.111:0/3931057330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102710 msgr2=0x7f524c0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:43.739 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 --2- 172.21.15.111:0/3931057330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102710 0x7f524c0fbd80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f5234004940 tx=0x7f5234009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 -- 172.21.15.111:0/3931057330 shutdown_connections 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 --2- 172.21.15.111:0/3931057330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c102710 0x7f524c0fbd80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 --2- 172.21.15.111:0/3931057330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c101d70 0x7f524c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 -- 172.21.15.111:0/3931057330 >> 172.21.15.111:0/3931057330 conn(0x7f524c072bc0 msgr2=0x7f524c072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 -- 172.21.15.111:0/3931057330 shutdown_connections 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 -- 172.21.15.111:0/3931057330 wait complete. 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.746+0000 7f525286e700 1 Processor -- start 2024-03-14T13:25:43.740 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f525286e700 1 -- start start 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f525286e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f525286e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c102710 0x7f524c103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f525286e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c103650 con 0x7f524c101d70 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f525286e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c1037c0 con 0x7f524c102710 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51558/0 (socket says 172.21.15.111:51558) 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 -- 172.21.15.111:0/2914037558 learned_addr learned my addr 172.21.15.111:0/2914037558 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524b7fe700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c102710 0x7f524c103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:43.742 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 -- 172.21.15.111:0/2914037558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c102710 msgr2=0x7f524c103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c102710 0x7f524c103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.747+0000 7f524bfff700 1 -- 172.21.15.111:0/2914037558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5234005040 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f524bfff700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f523c002a50 tx=0x7f523c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f523c007cc0 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c119dc0 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f523c00f040 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f523c009540 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.748+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524c11a1c0 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.749+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c0fd4c0 con 0x7f524c101d70 2024-03-14T13:25:43.743 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.749+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f523c0096a0 con 0x7f524c101d70 2024-03-14T13:25:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.750+0000 7f5248ff9700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 0x7f523806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.750+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f523c08b870 con 0x7f524c101d70 2024-03-14T13:25:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.750+0000 7f524b7fe700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 0x7f523806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:43.744 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.750+0000 7f524b7fe700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 0x7f523806ec70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f523400b3b0 tx=0x7f5234009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:43.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.755+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f523c055a70 con 0x7f524c101d70 2024-03-14T13:25:43.955 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.962+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f524c04bf40 con 0x7f523806c7c0 2024-03-14T13:25:43.956 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.964+0000 7f5248ff9700 1 -- 172.21.15.111:0/2914037558 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f524c04bf40 con 0x7f523806c7c0 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 msgr2=0x7f523806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 0x7f523806ec70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f523400b3b0 tx=0x7f5234009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 msgr2=0x7f524c102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f523c002a50 tx=0x7f523c0046b0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 shutdown_connections 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f523806c7c0 0x7f523806ec70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f524c101d70 0x7f524c102bd0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 --2- 172.21.15.111:0/2914037558 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f524c102710 0x7f524c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 >> 172.21.15.111:0/2914037558 conn(0x7f524c072bc0 msgr2=0x7f524c065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 shutdown_connections 2024-03-14T13:25:43.959 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:43.966+0000 7f525286e700 1 -- 172.21.15.111:0/2914037558 wait complete. 2024-03-14T13:25:43.969 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:25:43.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:43 smithi137 ceph-mon[108145]: pgmap v648: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 -- 172.21.15.111:0/869843253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 msgr2=0x7f2fb00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 --2- 172.21.15.111:0/869843253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb00fbd80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2fac004940 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 -- 172.21.15.111:0/869843253 shutdown_connections 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 --2- 172.21.15.111:0/869843253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb00fbd80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 --2- 172.21.15.111:0/869843253 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 0x7f2fb0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 -- 172.21.15.111:0/869843253 >> 172.21.15.111:0/869843253 conn(0x7f2fb0072bc0 msgr2=0x7f2fb0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 -- 172.21.15.111:0/869843253 shutdown_connections 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.067+0000 7f2fb8470700 1 -- 172.21.15.111:0/869843253 wait complete. 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 Processor -- start 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 -- start start 2024-03-14T13:25:44.062 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 0x7f2fb01129c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0113550 con 0x7f2fb0102710 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb8470700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb010c940 con 0x7f2fb0101d70 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51574/0 (socket says 172.21.15.111:51574) 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 -- 172.21.15.111:0/4146855691 learned_addr learned my addr 172.21.15.111:0/4146855691 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb620c700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 0x7f2fb01129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 -- 172.21.15.111:0/4146855691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 msgr2=0x7f2fb01129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 0x7f2fb01129c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 -- 172.21.15.111:0/4146855691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fb0102710 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.068+0000 7f2fb5a0b700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009380 tx=0x7f2fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac00e040 con 0x7f2fb0102710 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb010cbc0 con 0x7f2fb0102710 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fac013420 con 0x7f2fb0102710 2024-03-14T13:25:44.063 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fac01c630 con 0x7f2fb0102710 2024-03-14T13:25:44.064 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb010d180 con 0x7f2fb0102710 2024-03-14T13:25:44.064 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.069+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb00ff080 con 0x7f2fb0102710 2024-03-14T13:25:44.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.070+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2fac0139e0 con 0x7f2fb0102710 2024-03-14T13:25:44.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.070+0000 7f2fa6ffd700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 0x7f2f9c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.070+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f2fac08cc50 con 0x7f2fb0102710 2024-03-14T13:25:44.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.070+0000 7f2fb620c700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 0x7f2f9c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.065 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.071+0000 7f2fb620c700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 0x7f2f9c06ec70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0004990 tx=0x7f2fa0009ee0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.067 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.075+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2fac05a960 con 0x7f2fb0102710 2024-03-14T13:25:44.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:43 smithi111 ceph-mon[145314]: pgmap v648: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:44.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.284+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fb010e010 con 0x7f2f9c06c7c0 2024-03-14T13:25:44.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.285+0000 7f2fa6ffd700 1 -- 172.21.15.111:0/4146855691 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7f2fb010e010 con 0x7f2f9c06c7c0 2024-03-14T13:25:44.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 msgr2=0x7f2f9c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 0x7f2f9c06ec70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0004990 tx=0x7f2fa0009ee0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 msgr2=0x7f2fb0112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009380 tx=0x7f2fac009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 shutdown_connections 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f2f9c06c7c0 0x7f2f9c06ec70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fb0101d70 0x7f2fb01129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 --2- 172.21.15.111:0/4146855691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fb0102710 0x7f2fb0112f00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 >> 172.21.15.111:0/4146855691 conn(0x7f2fb0072bc0 msgr2=0x7f2fb00fb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 shutdown_connections 2024-03-14T13:25:44.281 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.287+0000 7f2fb8470700 1 -- 172.21.15.111:0/4146855691 wait complete. 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- 172.21.15.111:0/3893951206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0101d70 msgr2=0x7f4bc0102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/3893951206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102140 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0004940 tx=0x7f4bb0009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- 172.21.15.111:0/3893951206 shutdown_connections 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/3893951206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0102710 0x7f4bc00fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/3893951206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102140 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- 172.21.15.111:0/3893951206 >> 172.21.15.111:0/3893951206 conn(0x7f4bc0072bc0 msgr2=0x7f4bc0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- 172.21.15.111:0/3893951206 shutdown_connections 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- 172.21.15.111:0/3893951206 wait complete. 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 Processor -- start 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- start start 2024-03-14T13:25:44.382 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0103650 con 0x7f4bc0102710 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.387+0000 7f4bc86c4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0103790 con 0x7f4bc0101d70 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51592/0 (socket says 172.21.15.111:51592) 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 -- 172.21.15.111:0/245089868 learned_addr learned my addr 172.21.15.111:0/245089868 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc6460700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 -- 172.21.15.111:0/245089868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0101d70 msgr2=0x7f4bc0102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 -- 172.21.15.111:0/245089868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb0005040 con 0x7f4bc0102710 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc5c5f700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c7e0 tx=0x7f4bbc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc012810 con 0x7f4bc0102710 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bbc012e20 con 0x7f4bc0102710 2024-03-14T13:25:44.383 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc00a310 con 0x7f4bc0102710 2024-03-14T13:25:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc0103a40 con 0x7f4bc0102710 2024-03-14T13:25:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.388+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc011a230 con 0x7f4bc0102710 2024-03-14T13:25:44.384 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.389+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc00fd4c0 con 0x7f4bc0102710 2024-03-14T13:25:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.390+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4bbc011380 con 0x7f4bc0102710 2024-03-14T13:25:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.390+0000 7f4bb6ffd700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 0x7f4bac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.390+0000 7f4bc6460700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 0x7f4bac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.390+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4bbc08bc50 con 0x7f4bc0102710 2024-03-14T13:25:44.385 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.391+0000 7f4bc6460700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 0x7f4bac06ec70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0004560 tx=0x7f4bb0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.387 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.395+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4bbc055ed0 con 0x7f4bc0102710 2024-03-14T13:25:44.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.596+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bc0103a40 con 0x7f4bac06c7c0 2024-03-14T13:25:44.599 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.606+0000 7f4bb6ffd700 1 -- 172.21.15.111:0/245089868 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f4bc0103a40 con 0x7f4bac06c7c0 2024-03-14T13:25:44.599 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:25:44.599 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (21m) 15m ago 33m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:25:44.599 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (19m) 15m ago 33m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:25:44.599 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (19m) 14m ago 32m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (21m) 15m ago 32m 88.9M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (15m) 15m ago 27m 12.4M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (15m) 15m ago 27m 13.9M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (14m) 14m ago 27m 11.2M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (15m) 14m ago 27m 11.6M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (24m) 15m ago 34m 505M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (23m) 14m ago 32m 452M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (20m) 15m ago 35m 56.2M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (20m) 14m ago 32m 44.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (22m) 15m ago 33m 19.5M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (22m) 14m ago 32m 18.7M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (19m) 15m ago 31m 88.7M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (18m) 15m ago 30m 95.1M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (17m) 15m ago 30m 85.0M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:25:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (17m) 14m ago 29m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (16m) 14m ago 29m 94.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (16m) 14m ago 29m 86.8M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (22m) 15m ago 32m 62.3M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 msgr2=0x7f4bac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 0x7f4bac06ec70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0004560 tx=0x7f4bb0009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 msgr2=0x7f4bc0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c7e0 tx=0x7f4bbc007f90 comp rx=0 tx=0).stop 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 shutdown_connections 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f4bac06c7c0 0x7f4bac06ec70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4bc0101d70 0x7f4bc0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 --2- 172.21.15.111:0/245089868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc0102710 0x7f4bc0103110 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 >> 172.21.15.111:0/245089868 conn(0x7f4bc0072bc0 msgr2=0x7f4bc0065a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 shutdown_connections 2024-03-14T13:25:44.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.608+0000 7f4bc86c4700 1 -- 172.21.15.111:0/245089868 wait complete. 2024-03-14T13:25:44.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 -- 172.21.15.111:0/2044148164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70040fb4a0 msgr2=0x7f70040fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 --2- 172.21.15.111:0/2044148164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70040fb4a0 0x7f70040fb910 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8004940 tx=0x7f6ff8009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 -- 172.21.15.111:0/2044148164 shutdown_connections 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 --2- 172.21.15.111:0/2044148164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70040fb4a0 0x7f70040fb910 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 --2- 172.21.15.111:0/2044148164 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7004102e70 0x7f7004103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 -- 172.21.15.111:0/2044148164 >> 172.21.15.111:0/2044148164 conn(0x7f70040714c0 msgr2=0x7f70040718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 -- 172.21.15.111:0/2044148164 shutdown_connections 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.708+0000 7f7009f0b700 1 -- 172.21.15.111:0/2044148164 wait complete. 2024-03-14T13:25:44.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 Processor -- start 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 -- start start 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70040fb4a0 0x7f70040fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040fd150 con 0x7f7004102e70 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7009f0b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040fd290 con 0x7f70040fb4a0 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7002ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7002ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51610/0 (socket says 172.21.15.111:51610) 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f7002ffd700 1 -- 172.21.15.111:0/3735603194 learned_addr learned my addr 172.21.15.111:0/3735603194 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.709+0000 7f70037fe700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70040fb4a0 0x7f70040fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7002ffd700 1 -- 172.21.15.111:0/3735603194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70040fb4a0 msgr2=0x7f70040fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7002ffd700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70040fb4a0 0x7f70040fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7002ffd700 1 -- 172.21.15.111:0/3735603194 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff8005040 con 0x7f7004102e70 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7002ffd700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009180 tx=0x7f6ff8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.704 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff800e040 con 0x7f7004102e70 2024-03-14T13:25:44.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ff8004290 con 0x7f7004102e70 2024-03-14T13:25:44.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70040fd4e0 con 0x7f7004102e70 2024-03-14T13:25:44.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ff8013e60 con 0x7f7004102e70 2024-03-14T13:25:44.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.710+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700411a200 con 0x7f7004102e70 2024-03-14T13:25:44.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.711+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004100170 con 0x7f7004102e70 2024-03-14T13:25:44.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.711+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6ff80136b0 con 0x7f7004102e70 2024-03-14T13:25:44.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.712+0000 7f6febfff700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 0x7f6fec06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:44.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.712+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ff808ccd0 con 0x7f7004102e70 2024-03-14T13:25:44.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.712+0000 7f70037fe700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 0x7f6fec06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:44.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.712+0000 7f70037fe700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 0x7f6fec06ec00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007fd0 tx=0x7f6ff4009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:44.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.716+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ff8056f50 con 0x7f7004102e70 2024-03-14T13:25:44.981 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.988+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f7004100170 con 0x7f7004102e70 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.989+0000 7f6febfff700 1 -- 172.21.15.111:0/3735603194 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f6ff805a570 con 0x7f7004102e70 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:25:44.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:25:44.983 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 msgr2=0x7f6fec06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 0x7f6fec06ec00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007fd0 tx=0x7f6ff4009040 comp rx=0 tx=0).stop 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 msgr2=0x7f70040fcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009180 tx=0x7f6ff8009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 shutdown_connections 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6fec06c750 0x7f6fec06ec00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70040fb4a0 0x7f70040fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 --2- 172.21.15.111:0/3735603194 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7004102e70 0x7f70040fcbc0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 >> 172.21.15.111:0/3735603194 conn(0x7f70040714c0 msgr2=0x7f70040fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 shutdown_connections 2024-03-14T13:25:44.985 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:44.991+0000 7f7009f0b700 1 -- 172.21.15.111:0/3735603194 wait complete. 2024-03-14T13:25:45.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 -- 172.21.15.111:0/880034303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 msgr2=0x7ffa701092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 --2- 172.21.15.111:0/880034303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa701092f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffa60004940 tx=0x7ffa60009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:45.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 -- 172.21.15.111:0/880034303 shutdown_connections 2024-03-14T13:25:45.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 --2- 172.21.15.111:0/880034303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa701092f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.084 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 --2- 172.21.15.111:0/880034303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 0x7ffa700fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 -- 172.21.15.111:0/880034303 >> 172.21.15.111:0/880034303 conn(0x7ffa70072bc0 msgr2=0x7ffa70072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:45.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 -- 172.21.15.111:0/880034303 shutdown_connections 2024-03-14T13:25:45.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 -- 172.21.15.111:0/880034303 wait complete. 2024-03-14T13:25:45.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.091+0000 7ffa75e74700 1 Processor -- start 2024-03-14T13:25:45.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 -- start start 2024-03-14T13:25:45.086 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 0x7ffa70115cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa70116860 con 0x7ffa700fc590 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7010fc30 con 0x7ffa700fbc80 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51620/0 (socket says 172.21.15.111:51620) 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 -- 172.21.15.111:0/1132506788 learned_addr learned my addr 172.21.15.111:0/1132506788 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6f7fe700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 0x7ffa70115cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 -- 172.21.15.111:0/1132506788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 msgr2=0x7ffa70115cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 0x7ffa70115cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 -- 172.21.15.111:0/1132506788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60005040 con 0x7ffa700fc590 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa6effd700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009180 tx=0x7ffa60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.087 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa6000e040 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa60004290 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa6001c3f0 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.092+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7010feb0 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.093+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70110470 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.093+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa70103d70 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.094+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ffa60013790 con 0x7ffa700fc590 2024-03-14T13:25:45.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.094+0000 7ffa57fff700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 0x7ffa5806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.094+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ffa6008cc30 con 0x7ffa700fc590 2024-03-14T13:25:45.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.094+0000 7ffa6f7fe700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 0x7ffa5806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.095+0000 7ffa6f7fe700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 0x7ffa5806ec70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa64007fd0 tx=0x7ffa64009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.099+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffa6005aa00 con 0x7ffa700fc590 2024-03-14T13:25:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.331+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ffa70111320 con 0x7ffa700fc590 2024-03-14T13:25:45.324 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.331+0000 7ffa57fff700 1 -- 172.21.15.111:0/1132506788 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 33 v33) v1 ==== 76+0+1845 (secure 0 0 0) 0x7ffa60013420 con 0x7ffa700fc590 2024-03-14T13:25:45.324 INFO:teuthology.orchestra.run.smithi111.stdout:e33 2024-03-14T13:25:45.324 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 24 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:flags 13 allow_snaps allow_multimds_snaps 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:09:44.383471+0000 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:25:45.325 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:up {} 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:failed 0,1 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:25:45.326 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:Standby daemons: 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{-1:34388} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{-1:34420} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:45.329 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{-1:44305} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:45.330 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{-1:44335} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:25:45.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 msgr2=0x7ffa5806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 0x7ffa5806ec70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffa64007fd0 tx=0x7ffa64009040 comp rx=0 tx=0).stop 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 msgr2=0x7ffa70116210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009180 tx=0x7ffa60009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 shutdown_connections 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ffa5806c7c0 0x7ffa5806ec70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa700fbc80 0x7ffa70115cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 --2- 172.21.15.111:0/1132506788 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa700fc590 0x7ffa70116210 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.333+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 >> 172.21.15.111:0/1132506788 conn(0x7ffa70072bc0 msgr2=0x7ffa700fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.334+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 shutdown_connections 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.334+0000 7ffa75e74700 1 -- 172.21.15.111:0/1132506788 wait complete. 2024-03-14T13:25:45.331 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:25:45.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 -- 172.21.15.111:0/2301600156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d81037a0 msgr2=0x7ff4d80fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 --2- 172.21.15.111:0/2301600156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d81037a0 0x7ff4d80fce10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc004940 tx=0x7ff4cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:45.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 -- 172.21.15.111:0/2301600156 shutdown_connections 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 --2- 172.21.15.111:0/2301600156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d81037a0 0x7ff4d80fce10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 --2- 172.21.15.111:0/2301600156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d8102e00 0x7ff4d81031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 -- 172.21.15.111:0/2301600156 >> 172.21.15.111:0/2301600156 conn(0x7ff4d80f89b0 msgr2=0x7ff4d80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 -- 172.21.15.111:0/2301600156 shutdown_connections 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.434+0000 7ff4de392700 1 -- 172.21.15.111:0/2301600156 wait complete. 2024-03-14T13:25:45.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 Processor -- start 2024-03-14T13:25:45.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 -- start start 2024-03-14T13:25:45.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d81037a0 0x7ff4d8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8114690 con 0x7ff4d8102e00 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4de392700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d810da80 con 0x7ff4d81037a0 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51640/0 (socket says 172.21.15.111:51640) 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4d7fff700 1 -- 172.21.15.111:0/368728900 learned_addr learned my addr 172.21.15.111:0/368728900 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.435+0000 7ff4d77fe700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d81037a0 0x7ff4d8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d7fff700 1 -- 172.21.15.111:0/368728900 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d81037a0 msgr2=0x7ff4d8114040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d7fff700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d81037a0 0x7ff4d8114040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d7fff700 1 -- 172.21.15.111:0/368728900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc005040 con 0x7ff4d8102e00 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d7fff700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8002a50 tx=0x7ff4c80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4c8007cc0 con 0x7ff4d8102e00 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d810dd60 con 0x7ff4d8102e00 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4c800f040 con 0x7ff4d8102e00 2024-03-14T13:25:45.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4c800d9c0 con 0x7ff4d8102e00 2024-03-14T13:25:45.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.436+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d810e2f0 con 0x7ff4d8102e00 2024-03-14T13:25:45.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.437+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d80fe550 con 0x7ff4d8102e00 2024-03-14T13:25:45.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.437+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff4c800db20 con 0x7ff4d8102e00 2024-03-14T13:25:45.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.438+0000 7ff4d4ff9700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 0x7ff4c006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.438+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4c808b7a0 con 0x7ff4d8102e00 2024-03-14T13:25:45.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.438+0000 7ff4d77fe700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 0x7ff4c006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.438+0000 7ff4d77fe700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 0x7ff4c006ec70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc004890 tx=0x7ff4cc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.442+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff4c8059430 con 0x7ff4d8102e00 2024-03-14T13:25:45.663 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.671+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d8109da0 con 0x7ff4c006c7c0 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.672+0000 7ff4d4ff9700 1 -- 172.21.15.111:0/368728900 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+431 (secure 0 0 0) 0x7ff4d8109da0 con 0x7ff4c006c7c0 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [ 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "mon", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "mds", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "crash", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "osd", 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr" 2024-03-14T13:25:45.665 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-03-14T13:25:45.666 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "16/21 daemons upgraded", 2024-03-14T13:25:45.666 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mds daemons", 2024-03-14T13:25:45.666 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:25:45.666 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:25:45.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 msgr2=0x7ff4c006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 0x7ff4c006ec70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc004890 tx=0x7ff4cc009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:45.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 msgr2=0x7ff4d8113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8002a50 tx=0x7ff4c80046b0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 shutdown_connections 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4c006c7c0 0x7ff4c006ec70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4d8102e00 0x7ff4d8113b00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 --2- 172.21.15.111:0/368728900 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4d81037a0 0x7ff4d8114040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.674+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 >> 172.21.15.111:0/368728900 conn(0x7ff4d80f89b0 msgr2=0x7ff4d80f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.675+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 shutdown_connections 2024-03-14T13:25:45.668 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.675+0000 7ff4de392700 1 -- 172.21.15.111:0/368728900 wait complete. 2024-03-14T13:25:45.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 -- 172.21.15.111:0/486951427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73601026c0 msgr2=0x7f73600fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 --2- 172.21.15.111:0/486951427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73601026c0 0x7f73600fbd30 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f735c004940 tx=0x7f735c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:45.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 -- 172.21.15.111:0/486951427 shutdown_connections 2024-03-14T13:25:45.768 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 --2- 172.21.15.111:0/486951427 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73601026c0 0x7f73600fbd30 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 --2- 172.21.15.111:0/486951427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7360101d20 0x7f73601020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 -- 172.21.15.111:0/486951427 >> 172.21.15.111:0/486951427 conn(0x7f73600f78d0 msgr2=0x7f73600f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 -- 172.21.15.111:0/486951427 shutdown_connections 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.775+0000 7f73678db700 1 -- 172.21.15.111:0/486951427 wait complete. 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 Processor -- start 2024-03-14T13:25:45.769 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 -- start start 2024-03-14T13:25:45.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73601026c0 0x7f736010f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736010f630 con 0x7f7360101d20 2024-03-14T13:25:45.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f73678db700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736010f770 con 0x7f73601026c0 2024-03-14T13:25:45.770 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:51668/0 (socket says 172.21.15.111:51668) 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 -- 172.21.15.111:0/968878309 learned_addr learned my addr 172.21.15.111:0/968878309 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7364e76700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73601026c0 0x7f736010f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 -- 172.21.15.111:0/968878309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73601026c0 msgr2=0x7f736010f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73601026c0 0x7f736010f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 -- 172.21.15.111:0/968878309 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c005040 con 0x7f7360101d20 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.776+0000 7f7365677700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7350002a80 tx=0x7f73500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.777+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7350007cc0 con 0x7f7360101d20 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.777+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7360117a60 con 0x7f7360101d20 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.777+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f735000f040 con 0x7f7360101d20 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.777+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f735000da60 con 0x7f7360101d20 2024-03-14T13:25:45.771 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.777+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7360117ff0 con 0x7f7360101d20 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.778+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73600fd470 con 0x7f7360101d20 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.778+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f735000dc00 con 0x7f7360101d20 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.779+0000 7f7355ffb700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 0x7f734c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.779+0000 7f7364e76700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 0x7f734c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.779+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f735008b710 con 0x7f7360101d20 2024-03-14T13:25:45.772 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.779+0000 7f7364e76700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 0x7f734c06ec70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f73600668b0 tx=0x7f735c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:25:45.776 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:45.783+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7350055910 con 0x7f7360101d20 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: pgmap v649: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3735603194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:45.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1132506788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:46.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:25:46.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.yzwmcx"}]': finished 2024-03-14T13:25:46.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:25:46.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.zmslha"}]': finished 2024-03-14T13:25:46.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:45 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:25:46.047 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.054+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f736004bf00 con 0x7f7360101d20 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.055+0000 7f7355ffb700 1 -- 172.21.15.111:0/968878309 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+3583 (secure 0 0 0) 0x7f7350058f30 con 0x7f7360101d20 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_ERR failed to probe daemons or devices; 1 filesystem is degraded; 1 filesystem is offline 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: host smithi137 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi137/config 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout:/usr/bin/podman: timeout after 895 seconds 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout:Non-zero exit code 124 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de -e NODE_NAME=smithi137 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/591fecbc-e201-11ee-95c9-87774f69a715:/var/run/ceph:z -v /var/log/ceph/591fecbc-e201-11ee-95c9-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7ifk0fif:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de inventory --format=json-pretty --filter-for-batch 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout:Traceback (most recent call last): 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "__main__", mod_spec) 2024-03-14T13:25:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: exec(code, run_globals) 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 10863, in 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 10851, in main 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 2568, in _infer_config 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 2484, in _infer_fsid 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 2596, in _infer_image 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 2471, in _validate_fsid 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 7129, in command_ceph_volume 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: File "/tmp/tmp_6bye4f9.cephadm.build/__main__.py", line 2259, in call_throws 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout:RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de -e NODE_NAME=smithi137 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/591fecbc-e201-11ee-95c9-87774f69a715:/var/run/ceph:z -v /var/log/ceph/591fecbc-e201-11ee-95c9-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp7ifk0fif:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph@sha256:309491eeecd7000f600be6c48906f2b9b8131c5d6115ffe991865a8b787279de inventory --format=json-pretty --filter-for-batch 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout:[WRN] FS_DEGRADED: 1 filesystem is degraded 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is degraded 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout:[ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-14T13:25:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: fs cephfs is offline because no MDS is active for it. 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 msgr2=0x7f734c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 0x7f734c06ec70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f73600668b0 tx=0x7f735c009e30 comp rx=0 tx=0).stop 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 msgr2=0x7f736010ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7350002a80 tx=0x7f73500046b0 comp rx=0 tx=0).stop 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 shutdown_connections 2024-03-14T13:25:46.050 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f734c06c7c0 0x7f734c06ec70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:46.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7360101d20 0x7f736010ebb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:46.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 --2- 172.21.15.111:0/968878309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73601026c0 0x7f736010f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:25:46.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 >> 172.21.15.111:0/968878309 conn(0x7f73600f78d0 msgr2=0x7f73600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:25:46.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 shutdown_connections 2024-03-14T13:25:46.051 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:25:46.057+0000 7f73678db700 1 -- 172.21.15.111:0/968878309 wait complete. 2024-03-14T13:25:46.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='client.35212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='client.35216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: pgmap v649: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3735603194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1132506788' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.yzwmcx"}]: dispatch 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.yzwmcx"}]': finished 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.zmslha"}]: dispatch 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi111.zmslha"}]': finished 2024-03-14T13:25:46.135 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:45 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.iiqzcp"}]: dispatch 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: pgmap v650: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all mds 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.iiqzcp"}]': finished 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.ryucox"}]': finished 2024-03-14T13:25:46.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: Upgrade: Setting filesystem cephfs Joinable 2024-03-14T13:25:47.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-14T13:25:47.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:46 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/968878309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:25:47.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: pgmap v650: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all mds 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED) 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.iiqzcp"}]': finished 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.ryucox"}]: dispatch 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='client.35232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi137.ryucox"}]': finished 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: Upgrade: Setting filesystem cephfs Joinable 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]: dispatch 2024-03-14T13:25:47.134 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:46 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/968878309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: Standby daemon mds.cephfs.smithi111.zmslha assigned to filesystem cephfs as rank 0 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: Standby daemon mds.cephfs.smithi111.yzwmcx assigned to filesystem cephfs as rank 1 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:replay} 2 up:standby 2024-03-14T13:25:47.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:47 smithi111 ceph-mon[145314]: pgmap v651: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s rd, 1 op/s 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]': finished 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: Standby daemon mds.cephfs.smithi111.zmslha assigned to filesystem cephfs as rank 0 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: Standby daemon mds.cephfs.smithi111.yzwmcx assigned to filesystem cephfs as rank 1 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: fsmap cephfs:0/2 4 up:standby, 2 failed 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: Upgrade: Enabling allow_standby_replay on filesystem cephfs 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]: dispatch 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:replay} 2 up:standby 2024-03-14T13:25:47.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:47 smithi137 ceph-mon[108145]: pgmap v651: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 4.0 MiB/s rd, 1 op/s 2024-03-14T13:25:48.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:replay} 2 up:standby 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all rgw 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all rbd-mirror 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all ceph-exporter 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.000 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all iscsi 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all nfs 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Setting container_image for all nvmeof 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Finalizing container_image settings 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-14T13:25:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: Upgrade: Complete! 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.004 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:48 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "1"}]': finished 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:replay} 2 up:standby 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all rgw 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all rbd-mirror 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.005 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all ceph-exporter 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all iscsi 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all nfs 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Setting container_image for all nvmeof 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.006 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Finalizing container_image settings 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-14T13:25:49.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.008 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]: dispatch 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix": "config rm", "name": "mon_mds_skip_sanity", "who": "mon"}]': finished 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: Upgrade: Complete! 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:49.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:48 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.685 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.685 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:50 smithi111 ceph-mon[145314]: pgmap v652: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 5.5 MiB/s rd, 1 op/s 2024-03-14T13:25:50.685 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.685 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:50 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:50 smithi137 ceph-mon[108145]: pgmap v652: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 5.5 MiB/s rd, 1 op/s 2024-03-14T13:25:50.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:50.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:50 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:52.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:52 smithi111 ceph-mon[145314]: pgmap v653: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 5 op/s 2024-03-14T13:25:52.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:52 smithi137 ceph-mon[108145]: pgmap v653: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 15 MiB/s rd, 5 op/s 2024-03-14T13:25:54.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: pgmap v654: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:resolve 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:resolve} 2 up:standby 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:54.635 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: pgmap v654: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 23 MiB/s rd, 7 op/s 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:resolve 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:replay,1=cephfs.smithi111.yzwmcx=up:resolve} 2 up:standby 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:54.750 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:25:54.751 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:25:54.751 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:25:54.751 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:55.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: pgmap v655: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-14T13:25:55.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-14T13:25:55.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:55.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:55.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:resolve 2024-03-14T13:25:55.634 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:55 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:resolve,1=cephfs.smithi111.yzwmcx=up:resolve} 2 up:standby 2024-03-14T13:25:55.748 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: pgmap v655: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 27 MiB/s rd, 9 op/s 2024-03-14T13:25:55.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: Health check cleared: CEPHADM_REFRESH_FAILED (was: failed to probe daemons or devices) 2024-03-14T13:25:55.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:25:55.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:25:55.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:resolve 2024-03-14T13:25:55.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:55 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:resolve,1=cephfs.smithi111.yzwmcx=up:resolve} 2 up:standby 2024-03-14T13:25:56.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: reconnect by client.14546 192.168.0.1:0/2694268800 after 0.000999981 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: reconnect by client.14546 192.168.0.1:0/2694268800 after 0.000999981 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: reconnect by client.24309 192.168.0.1:0/584861229 after 0.000999981 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: reconnect by client.24309 192.168.0.1:0/584861229 after 0.000999981 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:reconnect 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:reconnect 2024-03-14T13:25:56.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:56 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:reconnect,1=cephfs.smithi111.yzwmcx=up:reconnect} 2 up:standby 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: reconnect by client.14546 192.168.0.1:0/2694268800 after 0.000999981 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: reconnect by client.14546 192.168.0.1:0/2694268800 after 0.000999981 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: reconnect by client.24309 192.168.0.1:0/584861229 after 0.000999981 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: reconnect by client.24309 192.168.0.1:0/584861229 after 0.000999981 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:reconnect 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:reconnect 2024-03-14T13:25:56.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:56 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:reconnect,1=cephfs.smithi111.yzwmcx=up:reconnect} 2 up:standby 2024-03-14T13:25:57.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: pgmap v656: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 8 op/s 2024-03-14T13:25:57.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:rejoin 2024-03-14T13:25:57.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:rejoin 2024-03-14T13:25:57.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:rejoin,1=cephfs.smithi111.yzwmcx=up:rejoin} 2 up:standby 2024-03-14T13:25:57.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: daemon mds.cephfs.smithi111.zmslha is now active in filesystem cephfs as rank 0 2024-03-14T13:25:57.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:57 smithi111 ceph-mon[145314]: daemon mds.cephfs.smithi111.yzwmcx is now active in filesystem cephfs as rank 1 2024-03-14T13:25:57.939 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: pgmap v656: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 25 MiB/s rd, 8 op/s 2024-03-14T13:25:57.939 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:rejoin 2024-03-14T13:25:57.939 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:rejoin 2024-03-14T13:25:57.939 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: fsmap cephfs:2/2 {0=cephfs.smithi111.zmslha=up:rejoin,1=cephfs.smithi111.yzwmcx=up:rejoin} 2 up:standby 2024-03-14T13:25:57.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: daemon mds.cephfs.smithi111.zmslha is now active in filesystem cephfs as rank 0 2024-03-14T13:25:57.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:57 smithi137 ceph-mon[108145]: daemon mds.cephfs.smithi111.yzwmcx is now active in filesystem cephfs as rank 1 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: Cluster is now healthy 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:active 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:active 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi111.yzwmcx=up:active} 2 up:standby 2024-03-14T13:25:58.749 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:58 smithi137 ceph-mon[108145]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi111.yzwmcx=up:active} 2 up:standby-replay 2024-03-14T13:25:58.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-14T13:25:58.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: Cluster is now healthy 2024-03-14T13:25:58.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] up:active 2024-03-14T13:25:58.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: mds.? [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] up:active 2024-03-14T13:25:58.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi111.yzwmcx=up:active} 2 up:standby 2024-03-14T13:25:58.884 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:58 smithi111 ceph-mon[145314]: fsmap cephfs:2 {0=cephfs.smithi111.zmslha=up:active,1=cephfs.smithi111.yzwmcx=up:active} 2 up:standby-replay 2024-03-14T13:25:59.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:25:59 smithi111 ceph-mon[145314]: pgmap v657: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 100 B/s wr, 7 op/s 2024-03-14T13:25:59.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:25:59 smithi137 ceph-mon[108145]: pgmap v657: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 20 MiB/s rd, 100 B/s wr, 7 op/s 2024-03-14T13:26:01.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:01 smithi111 ceph-mon[145314]: pgmap v658: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 100 B/s wr, 8 op/s 2024-03-14T13:26:01.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:01 smithi137 ceph-mon[108145]: pgmap v658: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 100 B/s wr, 8 op/s 2024-03-14T13:26:03.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:03 smithi111 ceph-mon[145314]: pgmap v659: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 10 KiB/s wr, 17 op/s 2024-03-14T13:26:03.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:03 smithi137 ceph-mon[108145]: pgmap v659: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 22 MiB/s rd, 10 KiB/s wr, 17 op/s 2024-03-14T13:26:05.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:05 smithi111 ceph-mon[145314]: pgmap v660: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-03-14T13:26:05.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:05 smithi137 ceph-mon[108145]: pgmap v660: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 24 MiB/s rd, 10 KiB/s wr, 19 op/s 2024-03-14T13:26:07.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:07 smithi111 ceph-mon[145314]: pgmap v661: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.9 KiB/s wr, 17 op/s 2024-03-14T13:26:07.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:07 smithi137 ceph-mon[108145]: pgmap v661: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.9 KiB/s wr, 17 op/s 2024-03-14T13:26:09.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:09 smithi137 ceph-mon[108145]: pgmap v662: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.9 KiB/s wr, 18 op/s 2024-03-14T13:26:09.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:26:09.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:10.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:09 smithi111 ceph-mon[145314]: pgmap v662: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.9 KiB/s wr, 18 op/s 2024-03-14T13:26:10.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:26:10.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:11.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:11 smithi137 ceph-mon[108145]: pgmap v663: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-03-14T13:26:12.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:11 smithi111 ceph-mon[145314]: pgmap v663: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 21 MiB/s rd, 8.8 KiB/s wr, 18 op/s 2024-03-14T13:26:13.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:13 smithi137 ceph-mon[108145]: pgmap v664: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 8.8 KiB/s wr, 17 op/s 2024-03-14T13:26:14.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:13 smithi111 ceph-mon[145314]: pgmap v664: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 19 MiB/s rd, 8.8 KiB/s wr, 17 op/s 2024-03-14T13:26:15.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:15 smithi137 ceph-mon[108145]: pgmap v665: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 9.3 MiB/s rd, 7 op/s 2024-03-14T13:26:16.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:15 smithi111 ceph-mon[145314]: pgmap v665: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 9.3 MiB/s rd, 7 op/s 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 -- 172.21.15.111:0/1192535612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 msgr2=0x7f51c40fb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/1192535612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c40fb910 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f51b4004940 tx=0x7f51b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 -- 172.21.15.111:0/1192535612 shutdown_connections 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/1192535612 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c40fb910 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/1192535612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 -- 172.21.15.111:0/1192535612 >> 172.21.15.111:0/1192535612 conn(0x7f51c40714c0 msgr2=0x7f51c40718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 -- 172.21.15.111:0/1192535612 shutdown_connections 2024-03-14T13:26:16.165 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.172+0000 7f51c9aaf700 1 -- 172.21.15.111:0/1192535612 wait complete. 2024-03-14T13:26:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 Processor -- start 2024-03-14T13:26:16.166 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 -- start start 2024-03-14T13:26:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:16.167 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c4114600 con 0x7f51c40fb4a0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c9aaf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c410d9f0 con 0x7f51c4102e70 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40136/0 (socket says 172.21.15.111:40136) 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c37fe700 1 -- 172.21.15.111:0/3390244928 learned_addr learned my addr 172.21.15.111:0/3390244928 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.173+0000 7f51c2ffd700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c37fe700 1 -- 172.21.15.111:0/3390244928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 msgr2=0x7f51c4113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c37fe700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c37fe700 1 -- 172.21.15.111:0/3390244928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b4005040 con 0x7f51c40fb4a0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c2ffd700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c37fe700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c8c0 tx=0x7f51b80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b800a980 con 0x7f51c40fb4a0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51b800ce60 con 0x7f51c40fb4a0 2024-03-14T13:26:16.168 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51b80113a0 con 0x7f51c40fb4a0 2024-03-14T13:26:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c410dcd0 con 0x7f51c40fb4a0 2024-03-14T13:26:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.174+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c410e290 con 0x7f51c40fb4a0 2024-03-14T13:26:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.175+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4100110 con 0x7f51c40fb4a0 2024-03-14T13:26:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.175+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f51b800aae0 con 0x7f51c40fb4a0 2024-03-14T13:26:16.169 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.176+0000 7f51abfff700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 0x7f51ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.176+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f51b808bb50 con 0x7f51c40fb4a0 2024-03-14T13:26:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.176+0000 7f51c2ffd700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 0x7f51ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:16.170 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.176+0000 7f51c2ffd700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 0x7f51ac06ec70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009fd0 tx=0x7f51b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:16.172 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.180+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f51b8059860 con 0x7f51c40fb4a0 2024-03-14T13:26:16.387 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.395+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f51c410f0c0 con 0x7f51ac06c7c0 2024-03-14T13:26:16.388 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.396+0000 7f51abfff700 1 -- 172.21.15.111:0/3390244928 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f51c410f0c0 con 0x7f51ac06c7c0 2024-03-14T13:26:16.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 msgr2=0x7f51ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:16.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 0x7f51ac06ec70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009fd0 tx=0x7f51b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:16.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 msgr2=0x7f51c4113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f51b800c8c0 tx=0x7f51b80044c0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 shutdown_connections 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f51ac06c7c0 0x7f51ac06ec70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51c40fb4a0 0x7f51c4113a70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 --2- 172.21.15.111:0/3390244928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51c4102e70 0x7f51c4113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 >> 172.21.15.111:0/3390244928 conn(0x7f51c40714c0 msgr2=0x7f51c40fa910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 shutdown_connections 2024-03-14T13:26:16.391 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:16.398+0000 7f51c9aaf700 1 -- 172.21.15.111:0/3390244928 wait complete. 2024-03-14T13:26:17.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch ps' 2024-03-14T13:26:17.451 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:17.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:17 smithi111 ceph-mon[145314]: pgmap v666: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 690 KiB/s rd, 2 op/s 2024-03-14T13:26:17.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:17 smithi111 ceph-mon[145314]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:17.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:17 smithi137 ceph-mon[108145]: pgmap v666: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 690 KiB/s rd, 2 op/s 2024-03-14T13:26:17.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:17 smithi137 ceph-mon[108145]: from='client.35240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 -- 172.21.15.111:0/1319902378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 msgr2=0x7f6abc101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/1319902378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc101900 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4004940 tx=0x7f6aa4009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 -- 172.21.15.111:0/1319902378 shutdown_connections 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/1319902378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/1319902378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc101900 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.413+0000 7f6ac0caf700 1 -- 172.21.15.111:0/1319902378 >> 172.21.15.111:0/1319902378 conn(0x7f6abc0714c0 msgr2=0x7f6abc0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 -- 172.21.15.111:0/1319902378 shutdown_connections 2024-03-14T13:26:19.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 -- 172.21.15.111:0/1319902378 wait complete. 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 Processor -- start 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 -- start start 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0ff0a0 con 0x7f6abc101530 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ac0caf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0fdbd0 con 0x7f6abc0fb570 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ab9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ab9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40144/0 (socket says 172.21.15.111:40144) 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.414+0000 7f6ab9d9b700 1 -- 172.21.15.111:0/2755268757 learned_addr learned my addr 172.21.15.111:0/2755268757 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6aba59c700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab9d9b700 1 -- 172.21.15.111:0/2755268757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 msgr2=0x7f6abc0fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:19.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab9d9b700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab9d9b700 1 -- 172.21.15.111:0/2755268757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4005040 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6aba59c700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab9d9b700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6aac002a50 tx=0x7f6aac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aac007cc0 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6aac00f040 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc0fdeb0 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aac009540 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.415+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc11a090 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.416+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0fcc90 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.416+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6aac0096a0 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.417+0000 7f6ab2ffd700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 0x7f6aa806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.417+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6aac08b800 con 0x7f6abc101530 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.417+0000 7f6aba59c700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 0x7f6aa806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.417+0000 7f6aba59c700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 0x7f6aa806ec70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4004560 tx=0x7f6aa4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:19.435 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.421+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6aac055a40 con 0x7f6abc101530 2024-03-14T13:26:19.618 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.625+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6abc0fa980 con 0x7f6aa806c7c0 2024-03-14T13:26:19.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.635+0000 7f6ab2ffd700 1 -- 172.21.15.111:0/2755268757 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f6abc0fa980 con 0x7f6aa806c7c0 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (22m) 26s ago 34m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (20m) 26s ago 34m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (19m) 29s ago 32m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (22m) 26s ago 33m 94.3M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (16m) 26s ago 27m 122M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (16m) 26s ago 28m 97.7M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (15m) 29s ago 27m 18.0M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (15m) 29s ago 27m 17.2M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (25m) 26s ago 35m 509M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (23m) 29s ago 32m 458M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:26:19.628 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (21m) 26s ago 35m 72.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (20m) 29s ago 32m 51.6M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (23m) 26s ago 34m 19.3M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (23m) 29s ago 32m 18.9M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (19m) 26s ago 31m 93.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (19m) 26s ago 31m 106M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (18m) 26s ago 30m 92.2M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (17m) 29s ago 30m 112M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (17m) 29s ago 30m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (16m) 29s ago 29m 97.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:26:19.629 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (22m) 26s ago 33m 83.8M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 msgr2=0x7f6aa806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 0x7f6aa806ec70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4004560 tx=0x7f6aa4009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 msgr2=0x7f6abc0fd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6aac002a50 tx=0x7f6aac0046b0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 shutdown_connections 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6aa806c7c0 0x7f6aa806ec70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6abc0fb570 0x7f6abc0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 --2- 172.21.15.111:0/2755268757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6abc101530 0x7f6abc0fd690 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.637+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 >> 172.21.15.111:0/2755268757 conn(0x7f6abc0714c0 msgr2=0x7f6abc0fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.638+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 shutdown_connections 2024-03-14T13:26:19.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:19.638+0000 7f6ac0caf700 1 -- 172.21.15.111:0/2755268757 wait complete. 2024-03-14T13:26:19.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:19 smithi137 ceph-mon[108145]: pgmap v667: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 8.3 KiB/s rd, 4 op/s 2024-03-14T13:26:20.011 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:19 smithi111 ceph-mon[145314]: pgmap v667: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 8.3 KiB/s rd, 4 op/s 2024-03-14T13:26:20.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch upgrade status' 2024-03-14T13:26:20.605 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:20.935 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:20 smithi111 ceph-mon[145314]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:20.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:20 smithi137 ceph-mon[108145]: from='client.35244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:21.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:21 smithi111 ceph-mon[145314]: pgmap v668: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:21.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:21 smithi137 ceph-mon[108145]: pgmap v668: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 -- 172.21.15.111:0/328561806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228103780 msgr2=0x7f12280fcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 --2- 172.21.15.111:0/328561806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228103780 0x7f12280fcdf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1224004940 tx=0x7f1224009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 -- 172.21.15.111:0/328561806 shutdown_connections 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 --2- 172.21.15.111:0/328561806 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228103780 0x7f12280fcdf0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 --2- 172.21.15.111:0/328561806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228102de0 0x7f12281031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 -- 172.21.15.111:0/328561806 >> 172.21.15.111:0/328561806 conn(0x7f12280f8950 msgr2=0x7f12280fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 -- 172.21.15.111:0/328561806 shutdown_connections 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.542+0000 7f122f5ac700 1 -- 172.21.15.111:0/328561806 wait complete. 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 Processor -- start 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 -- start start 2024-03-14T13:26:22.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228103780 0x7f1228113df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12281144d0 con 0x7f1228102de0 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122f5ac700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122810da40 con 0x7f1228103780 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53666/0 (socket says 172.21.15.111:53666) 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 -- 172.21.15.111:0/1514271240 learned_addr learned my addr 172.21.15.111:0/1514271240 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122cb47700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228103780 0x7f1228113df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 -- 172.21.15.111:0/1514271240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228103780 msgr2=0x7f1228113df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228103780 0x7f1228113df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.543+0000 7f122d348700 1 -- 172.21.15.111:0/1514271240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1224005040 con 0x7f1228102de0 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f122d348700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f121800b790 tx=0x7f1218009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:22.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121800f770 con 0x7f1228102de0 2024-03-14T13:26:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f121800fd80 con 0x7f1228102de0 2024-03-14T13:26:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122810dd20 con 0x7f1228102de0 2024-03-14T13:26:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f121800d310 con 0x7f1228102de0 2024-03-14T13:26:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.544+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122810e2e0 con 0x7f1228102de0 2024-03-14T13:26:22.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.545+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12280fe530 con 0x7f1228102de0 2024-03-14T13:26:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.545+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1218010420 con 0x7f1228102de0 2024-03-14T13:26:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.546+0000 7f121dffb700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 0x7f121406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.546+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f121808ba80 con 0x7f1228102de0 2024-03-14T13:26:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.546+0000 7f122cb47700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 0x7f121406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:22.540 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.546+0000 7f122cb47700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 0x7f121406ec70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f122810ec00 tx=0x7f1224009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:22.542 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.550+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1218059790 con 0x7f1228102de0 2024-03-14T13:26:22.755 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.763+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1228109e60 con 0x7f121406c7c0 2024-03-14T13:26:22.756 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.764+0000 7f121dffb700 1 -- 172.21.15.111:0/1514271240 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f1228109e60 con 0x7f121406c7c0 2024-03-14T13:26:22.756 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:26:22.756 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": null, 2024-03-14T13:26:22.756 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": false, 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout: "which": "", 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": null, 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "", 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout: "is_paused": false 2024-03-14T13:26:22.757 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 msgr2=0x7f121406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 0x7f121406ec70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f122810ec00 tx=0x7f1224009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 msgr2=0x7f12281138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f121800b790 tx=0x7f1218009250 comp rx=0 tx=0).stop 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 shutdown_connections 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f121406c7c0 0x7f121406ec70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1228102de0 0x7f12281138b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 --2- 172.21.15.111:0/1514271240 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1228103780 0x7f1228113df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 >> 172.21.15.111:0/1514271240 conn(0x7f12280f8950 msgr2=0x7f12280f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 shutdown_connections 2024-03-14T13:26:22.759 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:22.766+0000 7f122f5ac700 1 -- 172.21.15.111:0/1514271240 wait complete. 2024-03-14T13:26:23.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph health detail' 2024-03-14T13:26:23.697 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:23.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:23 smithi137 ceph-mon[108145]: pgmap v669: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:24.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:23 smithi111 ceph-mon[145314]: pgmap v669: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:24 smithi137 ceph-mon[108145]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:25.013 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:24 smithi111 ceph-mon[145314]: from='client.35248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:26:25.013 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- 172.21.15.111:0/2987624569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 msgr2=0x7f59f80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 --2- 172.21.15.111:0/2987624569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f80fbd80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f59e80049c0 tx=0x7f59e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- 172.21.15.111:0/2987624569 shutdown_connections 2024-03-14T13:26:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 --2- 172.21.15.111:0/2987624569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f80fbd80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.620 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 --2- 172.21.15.111:0/2987624569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 0x7f59f8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- 172.21.15.111:0/2987624569 >> 172.21.15.111:0/2987624569 conn(0x7f59f8072bc0 msgr2=0x7f59f8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- 172.21.15.111:0/2987624569 shutdown_connections 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- 172.21.15.111:0/2987624569 wait complete. 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 Processor -- start 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- start start 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 0x7f59f8111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f8112310 con 0x7f59f8102710 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.625+0000 7f59ff44c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f810b700 con 0x7f59f8101d70 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53688/0 (socket says 172.21.15.111:53688) 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 -- 172.21.15.111:0/3119329588 learned_addr learned my addr 172.21.15.111:0/3119329588 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:26:25.621 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fd1e8700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 0x7f59f8111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 -- 172.21.15.111:0/3119329588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 msgr2=0x7f59f8111780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 0x7f59f8111780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 -- 172.21.15.111:0/3119329588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e8005040 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59fc9e7700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009150 tx=0x7f59e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e800ba90 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59e8009aa0 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f59e8011e50 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f810b980 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.626+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f810bec0 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.627+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f80ff080 con 0x7f59f8102710 2024-03-14T13:26:25.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.628+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f59e8021070 con 0x7f59f8102710 2024-03-14T13:26:25.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.628+0000 7f59edffb700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 0x7f59e406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:25.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.628+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f59e808f430 con 0x7f59f8102710 2024-03-14T13:26:25.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.628+0000 7f59fd1e8700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 0x7f59e406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:25.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.629+0000 7f59fd1e8700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 0x7f59e406ec70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009fd0 tx=0x7f59f4009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:25.625 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.633+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f59e805d140 con 0x7f59f8102710 2024-03-14T13:26:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.905+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f59f810cdd0 con 0x7f59f8102710 2024-03-14T13:26:25.898 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.906+0000 7f59edffb700 1 -- 172.21.15.111:0/3119329588 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f59e805ccd0 con 0x7f59f8102710 2024-03-14T13:26:25.898 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 msgr2=0x7f59e406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 0x7f59e406ec70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009fd0 tx=0x7f59f4009f60 comp rx=0 tx=0).stop 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 msgr2=0x7f59f8111cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009150 tx=0x7f59e8009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 shutdown_connections 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f59e406c7c0 0x7f59e406ec70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59f8101d70 0x7f59f8111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 --2- 172.21.15.111:0/3119329588 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f8102710 0x7f59f8111cc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 >> 172.21.15.111:0/3119329588 conn(0x7f59f8072bc0 msgr2=0x7f59f80faf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 shutdown_connections 2024-03-14T13:26:25.901 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:25.908+0000 7f59ff44c700 1 -- 172.21.15.111:0/3119329588 wait complete. 2024-03-14T13:26:25.919 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:25 smithi111 ceph-mon[145314]: pgmap v670: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:25.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:25 smithi137 ceph-mon[108145]: pgmap v670: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:26.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions' 2024-03-14T13:26:26.819 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:26.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:26 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3119329588' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:26:26.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:26 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3119329588' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-14T13:26:27.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:27 smithi111 ceph-mon[145314]: pgmap v671: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:27.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:27 smithi137 ceph-mon[108145]: pgmap v671: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 -- 172.21.15.111:0/909477666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fde50 msgr2=0x7f5adc0fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/909477666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fde50 0x7f5adc0fe220 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5acc004940 tx=0x7f5acc009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 -- 172.21.15.111:0/909477666 shutdown_connections 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/909477666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/909477666 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fde50 0x7f5adc0fe220 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 -- 172.21.15.111:0/909477666 >> 172.21.15.111:0/909477666 conn(0x7f5adc0f78b0 msgr2=0x7f5adc0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 -- 172.21.15.111:0/909477666 shutdown_connections 2024-03-14T13:26:28.690 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.696+0000 7f5ae15ed700 1 -- 172.21.15.111:0/909477666 wait complete. 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 Processor -- start 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 -- start start 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fde50 0x7f5adc113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc1143e0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ae15ed700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc10d9e0 con 0x7f5adc0fde50 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ada7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ada7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:53716/0 (socket says 172.21.15.111:53716) 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5ada7fc700 1 -- 172.21.15.111:0/3601144241 learned_addr learned my addr 172.21.15.111:0/3601144241 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.697+0000 7f5adaffd700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fde50 0x7f5adc113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ada7fc700 1 -- 172.21.15.111:0/3601144241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fde50 msgr2=0x7f5adc113850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ada7fc700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fde50 0x7f5adc113850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ada7fc700 1 -- 172.21.15.111:0/3601144241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc005040 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ada7fc700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0002a80 tx=0x7f5ad00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:28.691 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ad0007cc0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc10dcc0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ad000f040 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.692 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ad0009640 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.698+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc10e200 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.699+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc066a90 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.699+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5ad00097a0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.700+0000 7f5ac37fe700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 0x7f5ac406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.700+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5ad008b2f0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.700+0000 7f5adaffd700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 0x7f5ac406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:26:28.693 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.700+0000 7f5adaffd700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 0x7f5ac406ec70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00b3b0 tx=0x7f5acc009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:26:28.696 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.704+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5ad0058f40 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.971 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.979+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5adc109d80 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.974 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.979+0000 7f5ac37fe700 1 -- 172.21.15.111:0/3601144241 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f5ad0058ad0 con 0x7f5adc0fe7f0 2024-03-14T13:26:28.974 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:26:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:26:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:26:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:26:28.975 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 msgr2=0x7f5ac406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 0x7f5ac406ec70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5acc00b3b0 tx=0x7f5acc009e30 comp rx=0 tx=0).stop 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 msgr2=0x7f5adc113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5ad0002a80 tx=0x7f5ad00046b0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 shutdown_connections 2024-03-14T13:26:28.976 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5ac406c7c0 0x7f5ac406ec70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5adc0fde50 0x7f5adc113850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 --2- 172.21.15.111:0/3601144241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5adc0fe7f0 0x7f5adc113d90 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:26:28.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 >> 172.21.15.111:0/3601144241 conn(0x7f5adc0f78b0 msgr2=0x7f5adc0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:26:28.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 shutdown_connections 2024-03-14T13:26:28.977 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:26:28.982+0000 7f5ae15ed700 1 -- 172.21.15.111:0/3601144241 wait complete. 2024-03-14T13:26:28.994 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:28 smithi111 ceph-mon[145314]: pgmap v672: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:28.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:28 smithi137 ceph-mon[108145]: pgmap v672: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:29.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-14T13:26:29.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:29 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3601144241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:26:29.925 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:29.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:29 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3601144241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:26:30.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:30 smithi137 ceph-mon[108145]: pgmap v673: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:31.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:30 smithi111 ceph-mon[145314]: pgmap v673: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:31.848 INFO:teuthology.orchestra.run.smithi111.stdout:wait for servicemap items w/ changing names to refresh 2024-03-14T13:26:32.613 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'sleep 60' 2024-03-14T13:26:32.883 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:32 smithi111 ceph-mon[145314]: pgmap v674: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:32.884 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:26:33.248 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:32 smithi137 ceph-mon[108145]: pgmap v674: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:35.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:35 smithi111 ceph-mon[145314]: pgmap v675: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:35.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:35 smithi137 ceph-mon[108145]: pgmap v675: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:37.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:37 smithi111 ceph-mon[145314]: pgmap v676: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:37.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:37 smithi137 ceph-mon[108145]: pgmap v676: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:39.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:39 smithi137 ceph-mon[108145]: pgmap v677: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:39.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:40.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:39 smithi111 ceph-mon[145314]: pgmap v677: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:40.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:41.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:41 smithi137 ceph-mon[108145]: pgmap v678: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:41 smithi111 ceph-mon[145314]: pgmap v678: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:43.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:43 smithi137 ceph-mon[108145]: pgmap v679: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:44.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:43 smithi111 ceph-mon[145314]: pgmap v679: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:45.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:45 smithi137 ceph-mon[108145]: pgmap v680: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:46.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:45 smithi111 ceph-mon[145314]: pgmap v680: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:47.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:47 smithi111 ceph-mon[145314]: pgmap v681: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:47.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:47 smithi137 ceph-mon[108145]: pgmap v681: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:49.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:49 smithi137 ceph-mon[108145]: pgmap v682: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:50.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:49 smithi111 ceph-mon[145314]: pgmap v682: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:26:51.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:51 smithi137 ceph-mon[108145]: pgmap v683: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:52.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:51 smithi111 ceph-mon[145314]: pgmap v683: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:53.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:53 smithi137 ceph-mon[108145]: pgmap v684: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:54.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:53 smithi111 ceph-mon[145314]: pgmap v684: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:26:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:26:54.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:54.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:26:54.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:26:54.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:26:55.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:26:55.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:26:55.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:26:55.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:26:55.133 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:26:55.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:55 smithi137 ceph-mon[108145]: pgmap v685: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:56.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:55 smithi111 ceph-mon[145314]: pgmap v685: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:26:57.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:57 smithi111 ceph-mon[145314]: pgmap v686: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:57.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:57 smithi137 ceph-mon[108145]: pgmap v686: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:26:59.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:26:59 smithi137 ceph-mon[108145]: pgmap v687: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:00.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:26:59 smithi111 ceph-mon[145314]: pgmap v687: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:00.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:00 smithi137 ceph-mon[108145]: pgmap v688: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:01.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:00 smithi111 ceph-mon[145314]: pgmap v688: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:02.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:02 smithi137 ceph-mon[108145]: pgmap v689: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:03.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:02 smithi111 ceph-mon[145314]: pgmap v689: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:05.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:05 smithi137 ceph-mon[108145]: pgmap v690: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:06.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:05 smithi111 ceph-mon[145314]: pgmap v690: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:07.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:07 smithi111 ceph-mon[145314]: pgmap v691: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:07.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:07 smithi137 ceph-mon[108145]: pgmap v691: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:09.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:09 smithi137 ceph-mon[108145]: pgmap v692: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:09.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:10.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:09 smithi111 ceph-mon[145314]: pgmap v692: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:10.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:11.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:11 smithi137 ceph-mon[108145]: pgmap v693: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:12.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:11 smithi111 ceph-mon[145314]: pgmap v693: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:13.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:13 smithi137 ceph-mon[108145]: pgmap v694: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:14.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:13 smithi111 ceph-mon[145314]: pgmap v694: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:15.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:15 smithi137 ceph-mon[108145]: pgmap v695: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:16.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:15 smithi111 ceph-mon[145314]: pgmap v695: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:17.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:17 smithi111 ceph-mon[145314]: pgmap v696: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:17.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:17 smithi137 ceph-mon[108145]: pgmap v696: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:19.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:19 smithi137 ceph-mon[108145]: pgmap v697: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:20.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:19 smithi111 ceph-mon[145314]: pgmap v697: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:21.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:21 smithi137 ceph-mon[108145]: pgmap v698: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:22.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:21 smithi111 ceph-mon[145314]: pgmap v698: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:23.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:23 smithi137 ceph-mon[108145]: pgmap v699: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:24.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:23 smithi111 ceph-mon[145314]: pgmap v699: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:24.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:25.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:25.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:25 smithi137 ceph-mon[108145]: pgmap v700: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:26.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:25 smithi111 ceph-mon[145314]: pgmap v700: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:27.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:27 smithi111 ceph-mon[145314]: pgmap v701: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:27.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:27 smithi137 ceph-mon[108145]: pgmap v701: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:29.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:29 smithi137 ceph-mon[108145]: pgmap v702: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:30.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:29 smithi111 ceph-mon[145314]: pgmap v702: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:30.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:30 smithi137 ceph-mon[108145]: pgmap v703: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:31.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:30 smithi111 ceph-mon[145314]: pgmap v703: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:32.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:32 smithi137 ceph-mon[108145]: pgmap v704: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:33.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:32 smithi111 ceph-mon[145314]: pgmap v704: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:35.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph orch ps' 2024-03-14T13:27:35.669 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:27:35.901 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:35 smithi111 ceph-mon[145314]: pgmap v705: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:35.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:35 smithi137 ceph-mon[108145]: pgmap v705: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 -- 172.21.15.111:0/4214526269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c4102e10 msgr2=0x7f24c41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/4214526269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c4102e10 0x7f24c41031e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f24b8004940 tx=0x7f24b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 -- 172.21.15.111:0/4214526269 shutdown_connections 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/4214526269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c41037b0 0x7f24c40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/4214526269 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c4102e10 0x7f24c41031e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 -- 172.21.15.111:0/4214526269 >> 172.21.15.111:0/4214526269 conn(0x7f24c40f8980 msgr2=0x7f24c40fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:37.444 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 -- 172.21.15.111:0/4214526269 shutdown_connections 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.451+0000 7f24cb1ca700 1 -- 172.21.15.111:0/4214526269 wait complete. 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 Processor -- start 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 -- start start 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c4102e10 0x7f24c40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c4070d90 con 0x7f24c41037b0 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24cb1ca700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c4070ed0 con 0x7f24c4102e10 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:59174/0 (socket says 172.21.15.111:59174) 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.452+0000 7f24c3fff700 1 -- 172.21.15.111:0/258471810 learned_addr learned my addr 172.21.15.111:0/258471810 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c8f66700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c4102e10 0x7f24c40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c3fff700 1 -- 172.21.15.111:0/258471810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c4102e10 msgr2=0x7f24c40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:37.445 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c3fff700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c4102e10 0x7f24c40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c3fff700 1 -- 172.21.15.111:0/258471810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b8005040 con 0x7f24c41037b0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c3fff700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f24b0002a50 tx=0x7f24b0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24b0007cc0 con 0x7f24c41037b0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24b000f040 con 0x7f24c41037b0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c4119e10 con 0x7f24c41037b0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24b000d990 con 0x7f24c41037b0 2024-03-14T13:27:37.446 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.453+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c411a290 con 0x7f24c41037b0 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.454+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c40fe560 con 0x7f24c41037b0 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.454+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f24b0009710 con 0x7f24c41037b0 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.455+0000 7f24c17fa700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 0x7f24b406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.455+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f24b008b910 con 0x7f24c41037b0 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.455+0000 7f24c8f66700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 0x7f24b406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:37.447 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.455+0000 7f24c8f66700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 0x7f24b406ecc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f24b8004f90 tx=0x7f24b8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:37.450 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.459+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f24b0055a90 con 0x7f24c41037b0 2024-03-14T13:27:37.655 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.664+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 --> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f24c404bed0 con 0x7f24b406c810 2024-03-14T13:27:37.664 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.673+0000 7f24c17fa700 1 -- 172.21.15.111:0/258471810 <== mgr.34104 v2:172.21.15.111:6800/977525747 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3476 (secure 0 0 0) 0x7f24c404bed0 con 0x7f24b406c810 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (23m) 104s ago 35m 27.8M - 0.25.0 c8568f914cd2 9d8fc65254ef 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (21m) 104s ago 35m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 6a36d0b1cf0b 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi137 smithi137 running (21m) 107s ago 34m 6656k - 18.2.2-825-g2f465319 a9e65ad305ae 9360d481baa0 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (23m) 104s ago 34m 94.3M - 9.4.7 954c08fa6188 1ee018486ddf 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.yzwmcx smithi111 running (17m) 104s ago 29m 122M - 18.2.2-825-g2f465319 a9e65ad305ae 593e1f307e5e 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi111.zmslha smithi111 running (17m) 104s ago 29m 97.7M - 18.2.2-825-g2f465319 a9e65ad305ae 403961ac9553 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.iiqzcp smithi137 running (16m) 107s ago 29m 18.0M - 18.2.2-825-g2f465319 a9e65ad305ae 5ab4d7fb9235 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mds.cephfs.smithi137.ryucox smithi137 running (17m) 107s ago 29m 17.2M - 18.2.2-825-g2f465319 a9e65ad305ae d507a5ac9224 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.atysez smithi111 *:8443,9283,8765 running (26m) 104s ago 36m 509M - 18.2.2-825-g2f465319 a9e65ad305ae 5262c02256f2 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi137.xbyzbp smithi137 *:8443,9283,8765 running (25m) 107s ago 34m 458M - 18.2.2-825-g2f465319 a9e65ad305ae 4e33aa7706c2 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (22m) 104s ago 36m 72.9M 2048M 18.2.2-825-g2f465319 a9e65ad305ae ba28dff3939f 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi137 smithi137 running (21m) 107s ago 33m 51.6M 2048M 18.2.2-825-g2f465319 a9e65ad305ae 512b2064ceae 2024-03-14T13:27:37.665 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (24m) 104s ago 35m 19.3M - 1.5.0 0da6a335fe13 ab19ab7ebfce 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi137 smithi137 *:9100 running (24m) 107s ago 34m 18.9M - 1.5.0 0da6a335fe13 e08c7441a652 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (20m) 104s ago 33m 93.9M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 02bbe6f32bcc 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (20m) 104s ago 32m 106M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 09dcfc43c596 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (19m) 104s ago 32m 92.2M 1568M 18.2.2-825-g2f465319 a9e65ad305ae 206403cb01ba 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi137 running (19m) 107s ago 31m 112M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 9df61411961e 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi137 running (18m) 107s ago 31m 101M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 5382a85f9809 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi137 running (18m) 107s ago 30m 97.2M 2592M 18.2.2-825-g2f465319 a9e65ad305ae 98cbc02db785 2024-03-14T13:27:37.666 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (24m) 104s ago 34m 83.8M - 2.43.0 a07b618ecd1d 9d94fa9399a1 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 msgr2=0x7f24b406ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 0x7f24b406ecc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f24b8004f90 tx=0x7f24b8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 msgr2=0x7f24c4070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f24b0002a50 tx=0x7f24b0004600 comp rx=0 tx=0).stop 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 shutdown_connections 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f24b406c810 0x7f24b406ecc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c4102e10 0x7f24c40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 --2- 172.21.15.111:0/258471810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24c41037b0 0x7f24c4070800 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 >> 172.21.15.111:0/258471810 conn(0x7f24c40f8980 msgr2=0x7f24c40f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 shutdown_connections 2024-03-14T13:27:37.667 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:37.676+0000 7f24cb1ca700 1 -- 172.21.15.111:0/258471810 wait complete. 2024-03-14T13:27:37.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:37 smithi111 ceph-mon[145314]: pgmap v706: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:37.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:37 smithi137 ceph-mon[108145]: pgmap v706: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:38.392 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions' 2024-03-14T13:27:38.664 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:27:38.919 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:38 smithi111 ceph-mon[145314]: from='client.35260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:27:38.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:38 smithi137 ceph-mon[108145]: from='client.35260 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-14T13:27:39.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:39 smithi111 ceph-mon[145314]: pgmap v707: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:39.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:39.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:39 smithi137 ceph-mon[108145]: pgmap v707: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:39.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:40.452 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/424902583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 msgr2=0x7fa1f41031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/424902583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f41031e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4004940 tx=0x7fa1e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/424902583 shutdown_connections 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/424902583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 0x7fa1f40fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/424902583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f41031e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/424902583 >> 172.21.15.111:0/424902583 conn(0x7fa1f40f89a0 msgr2=0x7fa1f40fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/424902583 shutdown_connections 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.459+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/424902583 wait complete. 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 Processor -- start 2024-03-14T13:27:40.453 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 -- start start 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 0x7fa1f4114030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f4114680 con 0x7fa1f4102e10 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fc4bd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f410da70 con 0x7fa1f41037b0 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34684/0 (socket says 172.21.15.111:34684) 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 -- 172.21.15.111:0/1396656266 learned_addr learned my addr 172.21.15.111:0/1396656266 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1f9a58700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 0x7fa1f4114030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 -- 172.21.15.111:0/1396656266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 msgr2=0x7fa1f4114030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 0x7fa1f4114030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.454 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 -- 172.21.15.111:0/1396656266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4005040 con 0x7fa1f4102e10 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1fa259700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400b3b0 tx=0x7fa1e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1e400e040 con 0x7fa1f4102e10 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.460+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1e4013420 con 0x7fa1f4102e10 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.461+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f410dcf0 con 0x7fa1f4102e10 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.461+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1e401c610 con 0x7fa1f4102e10 2024-03-14T13:27:40.455 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.461+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f410e2b0 con 0x7fa1f4102e10 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.461+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f40fe560 con 0x7fa1f4102e10 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.462+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa1e4013a00 con 0x7fa1f4102e10 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.462+0000 7fa1eaffd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 0x7fa1e006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.462+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa1e408cb80 con 0x7fa1f4102e10 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.462+0000 7fa1f9a58700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 0x7fa1e006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:40.456 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.463+0000 7fa1f9a58700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 0x7fa1e006ec70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1f410ec30 tx=0x7fa1f000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:40.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.467+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa1e405a890 con 0x7fa1f4102e10 2024-03-14T13:27:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.741+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa1f404bed0 con 0x7fa1f4102e10 2024-03-14T13:27:40.733 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.742+0000 7fa1eaffd700 1 -- 172.21.15.111:0/1396656266 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fa1e405a420 con 0x7fa1f4102e10 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 2 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-03-14T13:27:40.734 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 6 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 4 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)": 14 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-03-14T13:27:40.735 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-03-14T13:27:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 msgr2=0x7fa1e006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 0x7fa1e006ec70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1f410ec30 tx=0x7fa1f000b040 comp rx=0 tx=0).stop 2024-03-14T13:27:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 msgr2=0x7fa1f4113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa1e400b3b0 tx=0x7fa1e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:40.736 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 shutdown_connections 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fa1e006c7c0 0x7fa1e006ec70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1f4102e10 0x7fa1f4113af0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 --2- 172.21.15.111:0/1396656266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa1f41037b0 0x7fa1f4114030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 >> 172.21.15.111:0/1396656266 conn(0x7fa1f40f89a0 msgr2=0x7fa1f40f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 shutdown_connections 2024-03-14T13:27:40.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:40.744+0000 7fa1fc4bd700 1 -- 172.21.15.111:0/1396656266 wait complete. 2024-03-14T13:27:41.421 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-14T13:27:41.692 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:27:41.957 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:41 smithi111 ceph-mon[145314]: pgmap v708: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:41.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:41 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1396656266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:41.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:41 smithi137 ceph-mon[108145]: pgmap v708: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:41.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:41 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1396656266' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- 172.21.15.111:0/2181031660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 msgr2=0x7f1a580fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/2181031660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fe260 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1a54004940 tx=0x7f1a54009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- 172.21.15.111:0/2181031660 shutdown_connections 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/2181031660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 0x7f1a58106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/2181031660 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fe260 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- 172.21.15.111:0/2181031660 >> 172.21.15.111:0/2181031660 conn(0x7f1a58072bc0 msgr2=0x7f1a58072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- 172.21.15.111:0/2181031660 shutdown_connections 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- 172.21.15.111:0/2181031660 wait complete. 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 Processor -- start 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.460+0000 7f1a5fd46700 1 -- start start 2024-03-14T13:27:43.459 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 0x7f1a580fc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580fc820 con 0x7f1a580fde90 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a580fc960 con 0x7f1a580fe830 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34704/0 (socket says 172.21.15.111:34704) 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 -- 172.21.15.111:0/1418751224 learned_addr learned my addr 172.21.15.111:0/1418751224 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5d2e1700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 0x7f1a580fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 -- 172.21.15.111:0/1418751224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 msgr2=0x7f1a580fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 0x7f1a580fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 -- 172.21.15.111:0/1418751224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54005040 con 0x7f1a580fde90 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5dae2700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400b3b0 tx=0x7f1a54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5400e040 con 0x7f1a580fde90 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a54013420 con 0x7f1a580fde90 2024-03-14T13:27:43.460 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a580fcbb0 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a5401c5d0 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.461+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5811a1b0 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.462+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a58065e30 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.463+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1a54013590 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.463+0000 7f1a4e7fc700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 0x7f1a4406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.463+0000 7f1a5d2e1700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 0x7f1a4406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.463+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f1a5408cbc0 con 0x7f1a580fde90 2024-03-14T13:27:43.461 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.464+0000 7f1a5d2e1700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 0x7f1a4406ec70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a48007fd0 tx=0x7f1a48009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:43.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.468+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1a54056e40 con 0x7f1a580fde90 2024-03-14T13:27:43.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.743+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1a5804bf00 con 0x7f1a580fde90 2024-03-14T13:27:43.735 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.744+0000 7f1a4e7fc700 1 -- 172.21.15.111:0/1418751224 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f1a5405a460 con 0x7f1a580fde90 2024-03-14T13:27:43.737 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 msgr2=0x7f1a4406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 0x7f1a4406ec70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1a48007fd0 tx=0x7f1a48009040 comp rx=0 tx=0).stop 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 msgr2=0x7f1a580fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400b3b0 tx=0x7f1a54009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 shutdown_connections 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f1a4406c7c0 0x7f1a4406ec70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a580fde90 0x7f1a580fbd50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 --2- 172.21.15.111:0/1418751224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a580fe830 0x7f1a580fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 >> 172.21.15.111:0/1418751224 conn(0x7f1a58072bc0 msgr2=0x7f1a58103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 shutdown_connections 2024-03-14T13:27:43.738 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:43.746+0000 7f1a5fd46700 1 -- 172.21.15.111:0/1418751224 wait complete. 2024-03-14T13:27:43.747 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-03-14T13:27:43.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:43 smithi137 ceph-mon[108145]: pgmap v709: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:44.044 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:43 smithi111 ceph-mon[145314]: pgmap v709: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:44.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -e sha1=2f465319e3c83d48f999325459bcdfd0346449c0 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-14T13:27:44.719 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:27:44.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:44 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1418751224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:44.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:44 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1418751224' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:45.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:45 smithi137 ceph-mon[108145]: pgmap v710: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:46.049 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:45 smithi111 ceph-mon[145314]: pgmap v710: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:46.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.573+0000 7fde4681b700 1 -- 172.21.15.111:0/3540279593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 msgr2=0x7fde400fce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:46.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 --2- 172.21.15.111:0/3540279593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde400fce20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fde34004940 tx=0x7fde34009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:46.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 -- 172.21.15.111:0/3540279593 shutdown_connections 2024-03-14T13:27:46.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 --2- 172.21.15.111:0/3540279593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde400fce20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 --2- 172.21.15.111:0/3540279593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde40102e10 0x7fde401031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 -- 172.21.15.111:0/3540279593 >> 172.21.15.111:0/3540279593 conn(0x7fde400f8980 msgr2=0x7fde400fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 -- 172.21.15.111:0/3540279593 shutdown_connections 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 -- 172.21.15.111:0/3540279593 wait complete. 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 Processor -- start 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 -- start start 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.574+0000 7fde4681b700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde40102e10 0x7fde40113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde4681b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde4681b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde401146e0 con 0x7fde401037b0 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde4681b700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4010da40 con 0x7fde40102e10 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34720/0 (socket says 172.21.15.111:34720) 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 -- 172.21.15.111:0/467308681 learned_addr learned my addr 172.21.15.111:0/467308681 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 -- 172.21.15.111:0/467308681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde40102e10 msgr2=0x7fde40113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:46.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde40102e10 0x7fde40113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 -- 172.21.15.111:0/467308681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34005040 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3f7fe700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fde34009380 tx=0x7fde34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde3400e040 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde4010dcc0 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde34013420 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde3401c630 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.575+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde4010e280 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.576+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde400fe560 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.577+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fde340139e0 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.577+0000 7fde3cff9700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 0x7fde2806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.577+0000 7fde3ffff700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 0x7fde2806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.577+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fde3408cc60 con 0x7fde401037b0 2024-03-14T13:27:46.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.578+0000 7fde3ffff700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 0x7fde2806ec70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fde30006fd0 tx=0x7fde30006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:46.575 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.582+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fde3405a970 con 0x7fde401037b0 2024-03-14T13:27:46.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.856+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fde4004bed0 con 0x7fde401037b0 2024-03-14T13:27:46.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.857+0000 7fde3cff9700 1 -- 172.21.15.111:0/467308681 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7fde3405a500 con 0x7fde401037b0 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 msgr2=0x7fde2806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 0x7fde2806ec70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fde30006fd0 tx=0x7fde30006db0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 msgr2=0x7fde40114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fde34009380 tx=0x7fde34009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 shutdown_connections 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fde2806c7c0 0x7fde2806ec70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde40102e10 0x7fde40113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 --2- 172.21.15.111:0/467308681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fde401037b0 0x7fde40114000 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 >> 172.21.15.111:0/467308681 conn(0x7fde400f8980 msgr2=0x7fde400f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 shutdown_connections 2024-03-14T13:27:46.851 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:46.859+0000 7fde4681b700 1 -- 172.21.15.111:0/467308681 wait complete. 2024-03-14T13:27:46.861 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 18.2.2-825-g2f465319 (2f465319e3c83d48f999325459bcdfd0346449c0) reef (stable)" 2024-03-14T13:27:47.535 DEBUG:teuthology.parallel:result is None 2024-03-14T13:27:47.535 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-14T13:27:47.547 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T13:27:47.548 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 591fecbc-e201-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-14T13:27:47.820 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/mon.smithi111/config 2024-03-14T13:27:47.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:47 smithi111 ceph-mon[145314]: pgmap v711: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:47.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:47 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/467308681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:47.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:47 smithi137 ceph-mon[108145]: pgmap v711: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:47.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:47 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/467308681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 -- 172.21.15.111:0/601628138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 msgr2=0x7f41b80fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/601628138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b80fd5a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f41ac004940 tx=0x7f41ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 -- 172.21.15.111:0/601628138 shutdown_connections 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/601628138 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 0x7f41b80fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/601628138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b80fd5a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 -- 172.21.15.111:0/601628138 >> 172.21.15.111:0/601628138 conn(0x7f41b80f89b0 msgr2=0x7f41b80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 -- 172.21.15.111:0/601628138 shutdown_connections 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.541+0000 7f41bdc3f700 1 -- 172.21.15.111:0/601628138 wait complete. 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 Processor -- start 2024-03-14T13:27:49.536 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 -- start start 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 0x7f41b8113db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8114400 con 0x7f41b80fd1d0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41bdc3f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b810da00 con 0x7f41b80fe9d0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34750/0 (socket says 172.21.15.111:34750) 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.542+0000 7f41b77fe700 1 -- 172.21.15.111:0/3219152489 learned_addr learned my addr 172.21.15.111:0/3219152489 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41b6ffd700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 0x7f41b8113db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41b77fe700 1 -- 172.21.15.111:0/3219152489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 msgr2=0x7f41b8113db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41b77fe700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 0x7f41b8113db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41b77fe700 1 -- 172.21.15.111:0/3219152489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac005040 con 0x7f41b80fd1d0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41b77fe700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009180 tx=0x7f41ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac00e040 con 0x7f41b80fd1d0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41ac004290 con 0x7f41b80fd1d0 2024-03-14T13:27:49.537 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b810dc80 con 0x7f41b80fd1d0 2024-03-14T13:27:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41ac01c3f0 con 0x7f41b80fd1d0 2024-03-14T13:27:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.543+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b810e240 con 0x7f41b80fd1d0 2024-03-14T13:27:49.538 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.544+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b80fe570 con 0x7f41b80fd1d0 2024-03-14T13:27:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.544+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f41ac013790 con 0x7f41b80fd1d0 2024-03-14T13:27:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.545+0000 7f419ffff700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 0x7f41a006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.545+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f41ac08cd30 con 0x7f41b80fd1d0 2024-03-14T13:27:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.545+0000 7f41b6ffd700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 0x7f41a006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:49.539 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.545+0000 7f41b6ffd700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 0x7f41a006ec70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f41b810ebc0 tx=0x7f41a8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:49.541 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.549+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f41ac05aa40 con 0x7f41b80fd1d0 2024-03-14T13:27:49.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.785+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f41b80fe570 con 0x7f41b80fd1d0 2024-03-14T13:27:49.777 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.786+0000 7f419ffff700 1 -- 172.21.15.111:0/3219152489 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 42 v42) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f41ac013420 con 0x7f41b80fd1d0 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:e42 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:legacy client fscid: 1 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:Filesystem 'cephfs' (1) 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:fs_name cephfs 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 42 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-03-14T12:58:14.784520+0000 2024-03-14T13:27:49.778 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-03-14T13:25:57.565132+0000 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:tableserver 0 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:root 0 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:session_timeout 60 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:session_autoclose 300 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:max_file_size 1099511627776 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:max_xattr_size 65536 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:required_client_features {} 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure 0 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:last_failure_osd_epoch 82 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:max_mds 2 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:in 0,1 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:up {0=34388,1=34420} 2024-03-14T13:27:49.779 INFO:teuthology.orchestra.run.smithi111.stdout:failed 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:damaged 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:stopped 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:data_pools [3] 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:metadata_pool 2 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:inline_data disabled 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:balancer 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:bal_rank_mask -1 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:standby_count_wanted 1 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.zmslha{0:34388} state up:active seq 244 join_fscid=1 addr [v2:172.21.15.111:6826/4083640002,v1:172.21.15.111:6827/4083640002] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.ryucox{0:44305} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.137:6824/3411682065,v1:172.21.15.137:6825/3411682065] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi111.yzwmcx{1:34420} state up:active seq 239 join_fscid=1 addr [v2:172.21.15.111:6828/2220960815,v1:172.21.15.111:6829/2220960815] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.cephfs.smithi137.iiqzcp{1:44335} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.137:6826/2145348878,v1:172.21.15.137:6827/2145348878] compat {c=[1],r=[1],i=[7ff]}] 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:49.780 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:49.781 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 msgr2=0x7f41a006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:49.781 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 0x7f41a006ec70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f41b810ebc0 tx=0x7f41a8009040 comp rx=0 tx=0).stop 2024-03-14T13:27:49.781 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 msgr2=0x7f41b8113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009180 tx=0x7f41ac009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 shutdown_connections 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f41a006c7c0 0x7f41a006ec70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41b80fd1d0 0x7f41b8113870 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 --2- 172.21.15.111:0/3219152489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41b80fe9d0 0x7f41b8113db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 >> 172.21.15.111:0/3219152489 conn(0x7f41b80f89b0 msgr2=0x7f41b80fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 shutdown_connections 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:49.788+0000 7f41bdc3f700 1 -- 172.21.15.111:0/3219152489 wait complete. 2024-03-14T13:27:49.782 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 42 2024-03-14T13:27:49.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:49 smithi137 ceph-mon[108145]: pgmap v712: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:50.122 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:49 smithi111 ceph-mon[145314]: pgmap v712: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:50.465 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-03-14T13:27:50.480 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T13:27:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 -- 172.21.15.111:0/1285327629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 msgr2=0x7f5e480fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 --2- 172.21.15.111:0/1285327629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e480fbc80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5e30004940 tx=0x7f5e30009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 -- 172.21.15.111:0/1285327629 shutdown_connections 2024-03-14T13:27:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 --2- 172.21.15.111:0/1285327629 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e480fbc80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 --2- 172.21.15.111:0/1285327629 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 0x7f5e48102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 -- 172.21.15.111:0/1285327629 >> 172.21.15.111:0/1285327629 conn(0x7f5e48071c80 msgr2=0x7f5e48072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 -- 172.21.15.111:0/1285327629 shutdown_connections 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.613+0000 7f5e4df45700 1 -- 172.21.15.111:0/1285327629 wait complete. 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 Processor -- start 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 -- start start 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 0x7f5e4810e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4810f370 con 0x7f5e48102610 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e4df45700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4810f4b0 con 0x7f5e48101c70 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34154/0 (socket says 172.21.15.111:34154) 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.614+0000 7f5e3ffff700 1 -- 172.21.15.111:0/3561061811 learned_addr learned my addr 172.21.15.111:0/3561061811 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e477fe700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 0x7f5e4810e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e3ffff700 1 -- 172.21.15.111:0/3561061811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 msgr2=0x7f5e4810e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e3ffff700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 0x7f5e4810e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e3ffff700 1 -- 172.21.15.111:0/3561061811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30005040 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e3ffff700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5e30000c00 tx=0x7f5e30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3000e070 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e300093b0 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4810f730 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e3001ccb0 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.615+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e48117c50 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.616+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48104140 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.616+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5e3001c490 con 0x7f5e48102610 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.617+0000 7f5e44ff9700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 0x7f5e3406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.617+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f5e3008cde0 con 0x7f5e48102610 2024-03-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.617+0000 7f5e477fe700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 0x7f5e3406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.617+0000 7f5e477fe700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 0x7f5e3406ec70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009b80 tx=0x7f5e3800b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:50.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.621+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e30057120 con 0x7f5e48102610 2024-03-14T13:27:50.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.818+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e48108c20 con 0x7f5e48102610 2024-03-14T13:27:50.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.819+0000 7f5e44ff9700 1 -- 172.21.15.111:0/3561061811 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f5e3005a740 con 0x7f5e48102610 2024-03-14T13:27:50.814 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:50.815 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T13:10:40.893255+0000","last_up_change":"2024-03-14T13:09:42.188192+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-14T12:58:13.229267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-14T12:58:14.001600+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6803","nonce":2253644659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6805","nonce":2253644659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6809","nonce":2253644659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6807","nonce":2253644659}]},"public_addr":"172.21.15.111:6803/2253644659","cluster_addr":"172.21.15.111:6805/2253644659","heartbeat_back_addr":"172.21.15.111:6809/2253644659","heartbeat_front_addr":"172.21.15.111:6807/2253644659","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6811","nonce":2339106126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6813","nonce":2339106126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6817","nonce":2339106126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6815","nonce":2339106126}]},"public_addr":"172.21.15.111:6811/2339106126","cluster_addr":"172.21.15.111:6813/2339106126","heartbeat_back_addr":"172.21.15.111:6817/2339106126","heartbeat_front_addr":"172.21.15.111:6815/2339106126","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6819","nonce":3541679195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6821","nonce":3541679195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6825","nonce":3541679195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6823","nonce":3541679195}]},"public_addr":"172.21.15.111:6819/3541679195","cluster_addr":"172.21.15.111:6821/3541679195","heartbeat_back_addr":"172.21.15.111:6825/3541679195","heartbeat_front_addr":"172.21.15.111:6823/3541679195","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6801","nonce":137240476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6803","nonce":137240476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6807","nonce":137240476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6805","nonce":137240476}]},"public_addr":"172.21.15.137:6801/137240476","cluster_addr":"172.21.15.137:6803/137240476","heartbeat_back_addr":"172.21.15.137:6807/137240476","heartbeat_front_addr":"172.21.15.137:6805/137240476","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6809","nonce":2798642674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6811","nonce":2798642674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6815","nonce":2798642674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6813","nonce":2798642674}]},"public_addr":"172.21.15.137:6809/2798642674","cluster_addr":"172.21.15.137:6811/2798642674","heartbeat_back_addr":"172.21.15.137:6815/2798642674","heartbeat_front_addr":"172.21.15.137:6813/2798642674","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6817","nonce":4240986345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6819","nonce":4240986345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6823","nonce":4240986345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6821","nonce":4240986345}]},"public_addr":"172.21.15.137:6817/4240986345","cluster_addr":"172.21.15.137:6819/4240986345","heartbeat_back_addr":"172.21.15.137:6823/4240986345","heartbeat_front_addr":"172.21.15.137:6821/4240986345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6824/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3552305380":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/1484389211":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.137:0/2592190224":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/3277050730":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/2394623028":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/661526538":"2024-03-15T13:00:23.200618+0000","172.21.15.137:0/1152636518":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/3904007449":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/139754802":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/1611930537":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/4199526358":"2024-03-15T13:00:23.200618+0000","172.21.15.137:6825/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:0/1118840918":"2024-03-15T13:05:24.191564+0000","172.21.15.137:6828/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.137:6829/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/2868306717":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3646747677":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6827/1337358162":"2024-03-15T13:09:44.383031+0000","172.21.15.111:6800/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6800/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6826/1337358162":"2024-03-15T13:09:44.383031+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T13:27:50.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 msgr2=0x7f5e3406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.815 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 0x7f5e3406ec70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e38009b80 tx=0x7f5e3800b040 comp rx=0 tx=0).stop 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 msgr2=0x7f5e4810ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5e30000c00 tx=0x7f5e30009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 shutdown_connections 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f5e3406c7c0 0x7f5e3406ec70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e48101c70 0x7f5e4810e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 --2- 172.21.15.111:0/3561061811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5e48102610 0x7f5e4810ee30 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 >> 172.21.15.111:0/3561061811 conn(0x7f5e48071c80 msgr2=0x7f5e480faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 shutdown_connections 2024-03-14T13:27:50.816 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.821+0000 7f5e4df45700 1 -- 172.21.15.111:0/3561061811 wait complete. 2024-03-14T13:27:50.823 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:50 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3219152489' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:27:50.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T12:55:33.503741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-14T12:58:13.229267+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-14T12:58:14.001600+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T13:27:50.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 -- 172.21.15.111:0/863103880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14101be0 msgr2=0x7f0d14101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/863103880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14101be0 0x7f0d14101fb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004940 tx=0x7f0d08009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 -- 172.21.15.111:0/863103880 shutdown_connections 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/863103880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d14102580 0x7f0d140fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/863103880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14101be0 0x7f0d14101fb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.954+0000 7f0d18b6c700 1 -- 172.21.15.111:0/863103880 >> 172.21.15.111:0/863103880 conn(0x7f0d14072ad0 msgr2=0x7f0d14072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 -- 172.21.15.111:0/863103880 shutdown_connections 2024-03-14T13:27:50.948 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 -- 172.21.15.111:0/863103880 wait complete. 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 Processor -- start 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 -- start start 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d14101be0 0x7f0d1410e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1410f3f0 con 0x7f0d14102580 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d18b6c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1410f530 con 0x7f0d14101be0 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d11d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34170/0 (socket says 172.21.15.111:34170) 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.955+0000 7f0d11d9b700 1 -- 172.21.15.111:0/901195905 learned_addr learned my addr 172.21.15.111:0/901195905 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d11d9b700 1 -- 172.21.15.111:0/901195905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d14101be0 msgr2=0x7f0d1410e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d14101be0 0x7f0d1410e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d11d9b700 1 -- 172.21.15.111:0/901195905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08005040 con 0x7f0d14102580 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d11d9b700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c800 tx=0x7f0d04004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:50.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d0400a760 con 0x7f0d14102580 2024-03-14T13:27:50.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d0400ad70 con 0x7f0d14102580 2024-03-14T13:27:50.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d141179e0 con 0x7f0d14102580 2024-03-14T13:27:50.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d040113a0 con 0x7f0d14102580 2024-03-14T13:27:50.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.956+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d14117db0 con 0x7f0d14102580 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.957+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d140fd330 con 0x7f0d14102580 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.957+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0d04012400 con 0x7f0d14102580 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.958+0000 7f0d02ffd700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 0x7f0cfc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.958+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0d0408b9a0 con 0x7f0d14102580 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.958+0000 7f0d1259c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 0x7f0cfc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:50.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.958+0000 7f0d1259c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 0x7f0cfc06ec70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004f90 tx=0x7f0d08009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:50.954 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:50.962+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0d04055ce0 con 0x7f0d14102580 2024-03-14T13:27:50.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:50 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3219152489' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-14T13:27:51.147 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.156+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d14064970 con 0x7f0d14102580 2024-03-14T13:27:51.148 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.156+0000 7f0d02ffd700 1 -- 172.21.15.111:0/901195905 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d04059300 con 0x7f0d14102580 2024-03-14T13:27:51.148 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-03-14T13:27:51.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 msgr2=0x7f0cfc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 0x7f0cfc06ec70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0d08004f90 tx=0x7f0d08009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 msgr2=0x7f0d1410eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.150 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c800 tx=0x7f0d04004470 comp rx=0 tx=0).stop 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 shutdown_connections 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0cfc06c7c0 0x7f0cfc06ec70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d14101be0 0x7f0d1410e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 --2- 172.21.15.111:0/901195905 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0d14102580 0x7f0d1410eeb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.158+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 >> 172.21.15.111:0/901195905 conn(0x7f0d14072ad0 msgr2=0x7f0d14104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.159+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 shutdown_connections 2024-03-14T13:27:51.151 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.159+0000 7f0d18b6c700 1 -- 172.21.15.111:0/901195905 wait complete. 2024-03-14T13:27:51.161 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-14T13:27:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- 172.21.15.111:0/3184117219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20102ce0 msgr2=0x7feb201030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.285 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 --2- 172.21.15.111:0/3184117219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20102ce0 0x7feb201030b0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7feb10004940 tx=0x7feb10009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- 172.21.15.111:0/3184117219 shutdown_connections 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 --2- 172.21.15.111:0/3184117219 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb20103680 0x7feb20064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 --2- 172.21.15.111:0/3184117219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20102ce0 0x7feb201030b0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- 172.21.15.111:0/3184117219 >> 172.21.15.111:0/3184117219 conn(0x7feb20072ad0 msgr2=0x7feb20072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- 172.21.15.111:0/3184117219 shutdown_connections 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- 172.21.15.111:0/3184117219 wait complete. 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 Processor -- start 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- start start 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb20102ce0 0x7feb200fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200fdc20 con 0x7feb20103680 2024-03-14T13:27:51.286 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.292+0000 7feb280db700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200fc6c0 con 0x7feb20102ce0 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34186/0 (socket says 172.21.15.111:34186) 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 -- 172.21.15.111:0/1793510372 learned_addr learned my addr 172.21.15.111:0/1793510372 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 -- 172.21.15.111:0/1793510372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb20102ce0 msgr2=0x7feb200fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb20102ce0 0x7feb200fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 -- 172.21.15.111:0/1793510372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10005040 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb25676700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00b790 tx=0x7feb1c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1c00f6f0 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb200fc9a0 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb1c00fd00 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb1c0103a0 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.293+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb20119fe0 con 0x7feb20103680 2024-03-14T13:27:51.287 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.294+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb200fe430 con 0x7feb20103680 2024-03-14T13:27:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.295+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7feb1c00f850 con 0x7feb20103680 2024-03-14T13:27:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.295+0000 7feb167fc700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 0x7feb0c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.295+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7feb1c08b920 con 0x7feb20103680 2024-03-14T13:27:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.295+0000 7feb25e77700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 0x7feb0c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.288 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.295+0000 7feb25e77700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 0x7feb0c06ec70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feb10004f90 tx=0x7feb10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.299+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7feb1c055b20 con 0x7feb20103680 2024-03-14T13:27:51.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.495+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7feb20064970 con 0x7feb20103680 2024-03-14T13:27:51.487 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.496+0000 7feb167fc700 1 -- 172.21.15.111:0/1793510372 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7feb1c059140 con 0x7feb20103680 2024-03-14T13:27:51.488 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 msgr2=0x7feb0c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 0x7feb0c06ec70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feb10004f90 tx=0x7feb10009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 msgr2=0x7feb200fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00b790 tx=0x7feb1c009220 comp rx=0 tx=0).stop 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 shutdown_connections 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7feb0c06c7c0 0x7feb0c06ec70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb20102ce0 0x7feb200fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 --2- 172.21.15.111:0/1793510372 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feb20103680 0x7feb200fc180 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 >> 172.21.15.111:0/1793510372 conn(0x7feb20072ad0 msgr2=0x7feb20105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.490 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 shutdown_connections 2024-03-14T13:27:51.491 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.498+0000 7feb280db700 1 -- 172.21.15.111:0/1793510372 wait complete. 2024-03-14T13:27:51.502 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-14T13:27:51.622 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 -- 172.21.15.111:0/3346669143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374101be0 msgr2=0x7f6374101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 --2- 172.21.15.111:0/3346669143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374101be0 0x7f6374101fb0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6364004940 tx=0x7f6364009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 -- 172.21.15.111:0/3346669143 shutdown_connections 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 --2- 172.21.15.111:0/3346669143 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374102580 0x7f63740fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 --2- 172.21.15.111:0/3346669143 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374101be0 0x7f6374101fb0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 -- 172.21.15.111:0/3346669143 >> 172.21.15.111:0/3346669143 conn(0x7f6374072ad0 msgr2=0x7f6374072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 -- 172.21.15.111:0/3346669143 shutdown_connections 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.630+0000 7f6379171700 1 -- 172.21.15.111:0/3346669143 wait complete. 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.631+0000 7f6379171700 1 Processor -- start 2024-03-14T13:27:51.623 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.631+0000 7f6379171700 1 -- start start 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.631+0000 7f6379171700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374101be0 0x7f63741126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.631+0000 7f6379171700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f6379171700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6374113290 con 0x7f6374102580 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f6379171700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637410c850 con 0x7f6374101be0 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34208/0 (socket says 172.21.15.111:34208) 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 -- 172.21.15.111:0/3886676464 learned_addr learned my addr 172.21.15.111:0/3886676464 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f6372d9d700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374101be0 0x7f63741126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 -- 172.21.15.111:0/3886676464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374101be0 msgr2=0x7f63741126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374101be0 0x7f63741126c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.626 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 -- 172.21.15.111:0/3886676464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364005040 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f637259c700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6368002a50 tx=0x7f6368004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6368007cc0 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636800f040 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637410cae0 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636800d920 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.632+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637410d020 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.633+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63740fd330 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.634+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6368009680 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.634+0000 7f635b7fe700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 0x7f635c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.634+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f636808b810 con 0x7f6374102580 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.634+0000 7f6372d9d700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 0x7f635c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.627 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.634+0000 7f6372d9d700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 0x7f635c06ec70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6364004890 tx=0x7f6364009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.630 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.638+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f63680594a0 con 0x7f6374102580 2024-03-14T13:27:51.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.835+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f637410dd10 con 0x7f6374102580 2024-03-14T13:27:51.827 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.835+0000 7f635b7fe700 1 -- 172.21.15.111:0/3886676464 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6368059030 con 0x7f6374102580 2024-03-14T13:27:51.827 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:27:51.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.837+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 msgr2=0x7f635c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.837+0000 7f6379171700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 0x7f635c06ec70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6364004890 tx=0x7f6364009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 msgr2=0x7f6374112c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6368002a50 tx=0x7f6368004600 comp rx=0 tx=0).stop 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 shutdown_connections 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f635c06c7c0 0x7f635c06ec70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6374101be0 0x7f63741126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 --2- 172.21.15.111:0/3886676464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6374102580 0x7f6374112c40 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 >> 172.21.15.111:0/3886676464 conn(0x7f6374072ad0 msgr2=0x7f6374104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 shutdown_connections 2024-03-14T13:27:51.830 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.838+0000 7f6379171700 1 -- 172.21.15.111:0/3886676464 wait complete. 2024-03-14T13:27:51.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:51 smithi111 ceph-mon[145314]: pgmap v713: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:51.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:51 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3561061811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:27:51.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:51 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/901195905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:27:51.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:51 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1793510372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:27:51.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-14T13:27:51.960 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 -- 172.21.15.111:0/1932211999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 msgr2=0x7fe1d00fbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1932211999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d00fbc00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0004940 tx=0x7fe1c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 -- 172.21.15.111:0/1932211999 shutdown_connections 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1932211999 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d00fbc00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1932211999 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 0x7fe1d0101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 -- 172.21.15.111:0/1932211999 >> 172.21.15.111:0/1932211999 conn(0x7fe1d00f7700 msgr2=0x7fe1d00f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 -- 172.21.15.111:0/1932211999 shutdown_connections 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.967+0000 7fe1d571d700 1 -- 172.21.15.111:0/1932211999 wait complete. 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 Processor -- start 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 -- start start 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 0x7fe1d01126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d0113270 con 0x7fe1d0102590 2024-03-14T13:27:51.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1d571d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d010c830 con 0x7fe1d0101bf0 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34216/0 (socket says 172.21.15.111:34216) 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 -- 172.21.15.111:0/1164566595 learned_addr learned my addr 172.21.15.111:0/1164566595 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ceffd700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 0x7fe1d01126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 -- 172.21.15.111:0/1164566595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 msgr2=0x7fe1d01126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 0x7fe1d01126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 -- 172.21.15.111:0/1164566595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c0005040 con 0x7fe1d0102590 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.968+0000 7fe1ce7fc700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009380 tx=0x7fe1c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.969+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c000e040 con 0x7fe1d0102590 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.969+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1c0013420 con 0x7fe1d0102590 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.969+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d010ca60 con 0x7fe1d0102590 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.969+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c001c630 con 0x7fe1d0102590 2024-03-14T13:27:51.962 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.969+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d010cfa0 con 0x7fe1d0102590 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.970+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d00fd340 con 0x7fe1d0102590 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.970+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe1c00139b0 con 0x7fe1d0102590 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.970+0000 7fe1b77fe700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 0x7fe1b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.971+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe1c008cbe0 con 0x7fe1d0102590 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.971+0000 7fe1ceffd700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 0x7fe1b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:51.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.971+0000 7fe1ceffd700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 0x7fe1b806ec70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0102d90 tx=0x7fe1c400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:51.966 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:51.975+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe1c005a8f0 con 0x7fe1d0102590 2024-03-14T13:27:51.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:51 smithi137 ceph-mon[108145]: pgmap v713: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:51.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:51 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3561061811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:27:51.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:51 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/901195905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:27:51.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:51 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1793510372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:27:52.203 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.212+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1d00fd340 con 0x7fe1d0102590 2024-03-14T13:27:52.204 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.212+0000 7fe1b77fe700 1 -- 172.21.15.111:0/1164566595 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4873 (secure 0 0 0) 0x7fe1c005a480 con 0x7fe1d0102590 2024-03-14T13:27:52.204 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:52.204 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:57.565132+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:active","state_seq":244,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:active","state_seq":239,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44305":{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_44335":{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:52.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 msgr2=0x7fe1b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 0x7fe1b806ec70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0102d90 tx=0x7fe1c400b040 comp rx=0 tx=0).stop 2024-03-14T13:27:52.206 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 msgr2=0x7fe1d0112c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009380 tx=0x7fe1c0009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 shutdown_connections 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe1b806c7c0 0x7fe1b806ec70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1d0101bf0 0x7fe1d01126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 --2- 172.21.15.111:0/1164566595 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d0102590 0x7fe1d0112c20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.214+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 >> 172.21.15.111:0/1164566595 conn(0x7fe1d00f7700 msgr2=0x7fe1d00f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.215+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 shutdown_connections 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.215+0000 7fe1d571d700 1 -- 172.21.15.111:0/1164566595 wait complete. 2024-03-14T13:27:52.207 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 42 2024-03-14T13:27:52.218 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-03-14T13:27:52.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-03-14T13:27:52.338 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 -- 172.21.15.111:0/3509017463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381025b0 msgr2=0x7ff3380fbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 --2- 172.21.15.111:0/3509017463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381025b0 0x7ff3380fbc20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff32c004940 tx=0x7ff32c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 -- 172.21.15.111:0/3509017463 shutdown_connections 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 --2- 172.21.15.111:0/3509017463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff3381025b0 0x7ff3380fbc20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 --2- 172.21.15.111:0/3509017463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff338101c10 0x7ff338101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 -- 172.21.15.111:0/3509017463 >> 172.21.15.111:0/3509017463 conn(0x7ff3380f7720 msgr2=0x7ff3380f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 -- 172.21.15.111:0/3509017463 shutdown_connections 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.346+0000 7ff33e9be700 1 -- 172.21.15.111:0/3509017463 wait complete. 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 Processor -- start 2024-03-14T13:27:52.339 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 -- start start 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3381025b0 0x7ff338112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338113270 con 0x7ff338101c10 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff33e9be700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33810c830 con 0x7ff3381025b0 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.347+0000 7ff337fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34226/0 (socket says 172.21.15.111:34226) 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 -- 172.21.15.111:0/2447355215 learned_addr learned my addr 172.21.15.111:0/2447355215 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 -- 172.21.15.111:0/2447355215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3381025b0 msgr2=0x7ff338112c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3381025b0 0x7ff338112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 -- 172.21.15.111:0/2447355215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c005040 con 0x7ff338101c10 2024-03-14T13:27:52.341 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff337fff700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff32800fce0 tx=0x7ff3280046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32800fe90 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3280090d0 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33810cac0 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3280126a0 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.348+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33810d000 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.349+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3380fd360 con 0x7ff338101c10 2024-03-14T13:27:52.342 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.349+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff328009240 con 0x7ff338101c10 2024-03-14T13:27:52.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.350+0000 7ff334ff9700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 0x7ff32006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.350+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff32808e240 con 0x7ff338101c10 2024-03-14T13:27:52.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.350+0000 7ff3377fe700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 0x7ff32006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:52.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.350+0000 7ff3377fe700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 0x7ff32006ec70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff32c004f90 tx=0x7ff32c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:52.345 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.354+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff32805bed0 con 0x7ff338101c10 2024-03-14T13:27:52.581 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.590+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7ff33810d220 con 0x7ff338101c10 2024-03-14T13:27:52.582 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.590+0000 7ff334ff9700 1 -- 172.21.15.111:0/2447355215 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v42) v1 ==== 107+0+4867 (secure 0 0 0) 0x7ff32805ba60 con 0x7ff338101c10 2024-03-14T13:27:52.583 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:52.583 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:01:14.634340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi111.yzwmcx","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6829/1527723924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1527723924},{"type":"v1","addr":"172.21.15.111:6829","nonce":1527723924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/3644044473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3644044473},{"type":"v1","addr":"172.21.15.137:6827","nonce":3644044473}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:52.584 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 msgr2=0x7ff32006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 0x7ff32006ec70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff32c004f90 tx=0x7ff32c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 msgr2=0x7ff3381126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff32800fce0 tx=0x7ff3280046b0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 shutdown_connections 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff32006c7c0 0x7ff32006ec70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff338101c10 0x7ff3381126a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 --2- 172.21.15.111:0/2447355215 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3381025b0 0x7ff338112c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 >> 172.21.15.111:0/2447355215 conn(0x7ff3380f7720 msgr2=0x7ff3380f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 shutdown_connections 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.593+0000 7ff33e9be700 1 -- 172.21.15.111:0/2447355215 wait complete. 2024-03-14T13:27:52.585 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 19 2024-03-14T13:27:52.597 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-03-14T13:27:52.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 -- 172.21.15.111:0/1011606715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c064e40 msgr2=0x7fdb5c065210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/1011606715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c064e40 0x7fdb5c065210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c004940 tx=0x7fdb4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 -- 172.21.15.111:0/1011606715 shutdown_connections 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/1011606715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c065750 0x7fdb5c1091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/1011606715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c064e40 0x7fdb5c065210 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 -- 172.21.15.111:0/1011606715 >> 172.21.15.111:0/1011606715 conn(0x7fdb5c072ad0 msgr2=0x7fdb5c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 -- 172.21.15.111:0/1011606715 shutdown_connections 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.725+0000 7fdb60f3c700 1 -- 172.21.15.111:0/1011606715 wait complete. 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 Processor -- start 2024-03-14T13:27:52.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 -- start start 2024-03-14T13:27:52.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c064e40 0x7fdb5c115970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c116500 con 0x7fdb5c065750 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb60f3c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c10fb00 con 0x7fdb5c064e40 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb59d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb59d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34240/0 (socket says 172.21.15.111:34240) 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.726+0000 7fdb59d9b700 1 -- 172.21.15.111:0/3654042604 learned_addr learned my addr 172.21.15.111:0/3654042604 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb5a59c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c064e40 0x7fdb5c115970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb59d9b700 1 -- 172.21.15.111:0/3654042604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c064e40 msgr2=0x7fdb5c115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb59d9b700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c064e40 0x7fdb5c115970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb59d9b700 1 -- 172.21.15.111:0/3654042604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c005040 con 0x7fdb5c065750 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb59d9b700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000c8c0 tx=0x7fdb500044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb5000a760 con 0x7fdb5c065750 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb5000ad70 con 0x7fdb5c065750 2024-03-14T13:27:52.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb500113a0 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c10fde0 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.727+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c1103a0 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.728+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0fc220 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.728+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fdb50012400 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.729+0000 7fdb4affd700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 0x7fdb4406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.729+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fdb5008b9a0 con 0x7fdb5c065750 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.729+0000 7fdb5a59c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 0x7fdb4406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:52.722 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.729+0000 7fdb5a59c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 0x7fdb4406ec70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009fd0 tx=0x7fdb4c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:52.725 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.733+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdb50059770 con 0x7fdb5c065750 2024-03-14T13:27:52.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:52 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3886676464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:27:52.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:52 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1164566595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T13:27:52.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:52 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2447355215' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-14T13:27:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.969+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7fdb5c10be80 con 0x7fdb5c065750 2024-03-14T13:27:52.961 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.970+0000 7fdb4affd700 1 -- 172.21.15.111:0/3654042604 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v42) v1 ==== 107+0+4868 (secure 0 0 0) 0x7fdb50059300 con 0x7fdb5c065750 2024-03-14T13:27:52.962 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:52.962 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:03:17.747236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[1],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi111.yzwmcx","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6829/1527723924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1527723924},{"type":"v1","addr":"172.21.15.111:6829","nonce":1527723924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/3644044473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3644044473},{"type":"v1","addr":"172.21.15.137:6827","nonce":3644044473}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 msgr2=0x7fdb4406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 0x7fdb4406ec70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009fd0 tx=0x7fdb4c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 msgr2=0x7fdb5c115eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000c8c0 tx=0x7fdb500044c0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 shutdown_connections 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fdb4406c7c0 0x7fdb4406ec70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb5c064e40 0x7fdb5c115970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.964 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 --2- 172.21.15.111:0/3654042604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb5c065750 0x7fdb5c115eb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:52.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 >> 172.21.15.111:0/3654042604 conn(0x7fdb5c072ad0 msgr2=0x7fdb5c0f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:52.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 shutdown_connections 2024-03-14T13:27:52.965 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:52.972+0000 7fdb60f3c700 1 -- 172.21.15.111:0/3654042604 wait complete. 2024-03-14T13:27:52.965 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 20 2024-03-14T13:27:52.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-03-14T13:27:52.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:52 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3886676464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:27:52.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:52 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1164566595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-14T13:27:52.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:52 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2447355215' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-03-14T13:27:53.099 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 -- 172.21.15.111:0/471267423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 msgr2=0x7f1690101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 --2- 172.21.15.111:0/471267423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f1690101fb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f168c004940 tx=0x7f168c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 -- 172.21.15.111:0/471267423 shutdown_connections 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 --2- 172.21.15.111:0/471267423 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 0x7f16900fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 --2- 172.21.15.111:0/471267423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f1690101fb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 -- 172.21.15.111:0/471267423 >> 172.21.15.111:0/471267423 conn(0x7f1690072ad0 msgr2=0x7f1690072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 -- 172.21.15.111:0/471267423 shutdown_connections 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.105+0000 7f1697ddc700 1 -- 172.21.15.111:0/471267423 wait complete. 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 Processor -- start 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 -- start start 2024-03-14T13:27:53.100 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 0x7f169010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169010f310 con 0x7f1690101be0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1697ddc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169010f450 con 0x7f1690102580 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34264/0 (socket says 172.21.15.111:34264) 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 -- 172.21.15.111:0/3616962625 learned_addr learned my addr 172.21.15.111:0/3616962625 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695377700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 0x7f169010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 -- 172.21.15.111:0/3616962625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 msgr2=0x7f169010edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 0x7f169010edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.106+0000 7f1695b78700 1 -- 172.21.15.111:0/3616962625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c005040 con 0x7f1690101be0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f1695b78700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f168c009180 tx=0x7f168c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168c00e040 con 0x7f1690101be0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169010f6d0 con 0x7f1690101be0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f168c004290 con 0x7f1690101be0 2024-03-14T13:27:53.101 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f168c01c3f0 con 0x7f1690101be0 2024-03-14T13:27:53.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.107+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690117c50 con 0x7f1690101be0 2024-03-14T13:27:53.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.108+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900fd330 con 0x7f1690101be0 2024-03-14T13:27:53.102 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.108+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f168c013790 con 0x7f1690101be0 2024-03-14T13:27:53.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.108+0000 7f16867fc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 0x7f167c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.109+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f168c08cd30 con 0x7f1690101be0 2024-03-14T13:27:53.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.109+0000 7f1695377700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 0x7f167c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.109+0000 7f1695377700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 0x7f167c06ec70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1680009fd0 tx=0x7f1680009f60 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.105 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.113+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f168c056fb0 con 0x7f1690101be0 2024-03-14T13:27:53.343 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.352+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f16900fd330 con 0x7f1690101be0 2024-03-14T13:27:53.344 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.353+0000 7f16867fc700 1 -- 172.21.15.111:0/3616962625 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v42) v1 ==== 107+0+4867 (secure 0 0 0) 0x7f168c013420 con 0x7f1690101be0 2024-03-14T13:27:53.344 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:53.344 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:04:49.764363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi111.yzwmcx","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6829/1527723924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1527723924},{"type":"v1","addr":"172.21.15.111:6829","nonce":1527723924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[0],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/3644044473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3644044473},{"type":"v1","addr":"172.21.15.137:6827","nonce":3644044473}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 msgr2=0x7f167c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 0x7f167c06ec70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1680009fd0 tx=0x7f1680009f60 comp rx=0 tx=0).stop 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 msgr2=0x7f169010e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f168c009180 tx=0x7f168c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 shutdown_connections 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f167c06c7c0 0x7f167c06ec70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1690101be0 0x7f169010e890 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 --2- 172.21.15.111:0/3616962625 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1690102580 0x7f169010edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 >> 172.21.15.111:0/3616962625 conn(0x7f1690072ad0 msgr2=0x7f1690104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 shutdown_connections 2024-03-14T13:27:53.347 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.355+0000 7f1697ddc700 1 -- 172.21.15.111:0/3616962625 wait complete. 2024-03-14T13:27:53.348 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 21 2024-03-14T13:27:53.359 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-03-14T13:27:53.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- 172.21.15.111:0/692987338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84102580 msgr2=0x7fcd840fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/692987338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84102580 0x7fcd840fbbf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcd70004940 tx=0x7fcd70009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:53.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- 172.21.15.111:0/692987338 shutdown_connections 2024-03-14T13:27:53.479 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/692987338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84102580 0x7fcd840fbbf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/692987338 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84101be0 0x7fcd84101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- 172.21.15.111:0/692987338 >> 172.21.15.111:0/692987338 conn(0x7fcd84072ad0 msgr2=0x7fcd84072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- 172.21.15.111:0/692987338 shutdown_connections 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- 172.21.15.111:0/692987338 wait complete. 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 Processor -- start 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.486+0000 7fcd8ae11700 1 -- start start 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84102580 0x7fcd84112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd841131d0 con 0x7fcd84101be0 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8410c790 con 0x7fcd84102580 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34276/0 (socket says 172.21.15.111:34276) 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 -- 172.21.15.111:0/2510691496 learned_addr learned my addr 172.21.15.111:0/2510691496 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd83fff700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84102580 0x7fcd84112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.480 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 -- 172.21.15.111:0/2510691496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84102580 msgr2=0x7fcd84112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84102580 0x7fcd84112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 -- 172.21.15.111:0/2510691496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd70005040 con 0x7fcd84101be0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd88bad700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800eaf0 tx=0x7fcd7800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd78007cc0 con 0x7fcd84101be0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd7800f040 con 0x7fcd84101be0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8410c9c0 con 0x7fcd84101be0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd7800caa0 con 0x7fcd84101be0 2024-03-14T13:27:53.481 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.487+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8410cf00 con 0x7fcd84101be0 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.488+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd840fd330 con 0x7fcd84101be0 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.489+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcd7800cc00 con 0x7fcd84101be0 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.489+0000 7fcd817fa700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 0x7fcd7406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.489+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcd7808b800 con 0x7fcd84101be0 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.489+0000 7fcd83fff700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 0x7fcd7406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.482 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.490+0000 7fcd83fff700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 0x7fcd7406ec70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcd8410dea0 tx=0x7fcd70009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.485 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.494+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcd78059490 con 0x7fcd84101be0 2024-03-14T13:27:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.732+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7fcd8410dc50 con 0x7fcd84101be0 2024-03-14T13:27:53.724 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.733+0000 7fcd817fa700 1 -- 172.21.15.111:0/2510691496 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v42) v1 ==== 107+0+4866 (secure 0 0 0) 0x7fcd78059020 con 0x7fcd84101be0 2024-03-14T13:27:53.724 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:53.725 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:05:42.299211+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14494":{"gid":14494,"name":"cephfs.smithi111.yzwmcx","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.111:6829/1527723924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1527723924},{"type":"v1","addr":"172.21.15.111:6829","nonce":1527723924}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24273":{"gid":24273,"name":"cephfs.smithi137.iiqzcp","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.137:6827/3644044473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3644044473},{"type":"v1","addr":"172.21.15.137:6827","nonce":3644044473}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 msgr2=0x7fcd7406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 0x7fcd7406ec70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcd8410dea0 tx=0x7fcd70009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 msgr2=0x7fcd84112600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcd7800eaf0 tx=0x7fcd7800b040 comp rx=0 tx=0).stop 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 shutdown_connections 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fcd7406c7c0 0x7fcd7406ec70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd84101be0 0x7fcd84112600 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 --2- 172.21.15.111:0/2510691496 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd84102580 0x7fcd84112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.727 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 >> 172.21.15.111:0/2510691496 conn(0x7fcd84072ad0 msgr2=0x7fcd840ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 shutdown_connections 2024-03-14T13:27:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.735+0000 7fcd8ae11700 1 -- 172.21.15.111:0/2510691496 wait complete. 2024-03-14T13:27:53.728 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 22 2024-03-14T13:27:53.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 -- 172.21.15.111:0/764125012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 msgr2=0x7fd370101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 --2- 172.21.15.111:0/764125012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd370101fb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd360004940 tx=0x7fd360009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 -- 172.21.15.111:0/764125012 shutdown_connections 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 --2- 172.21.15.111:0/764125012 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 0x7fd3700fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 --2- 172.21.15.111:0/764125012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd370101fb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 -- 172.21.15.111:0/764125012 >> 172.21.15.111:0/764125012 conn(0x7fd370072ad0 msgr2=0x7fd370072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 -- 172.21.15.111:0/764125012 shutdown_connections 2024-03-14T13:27:53.857 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.865+0000 7fd375b06700 1 -- 172.21.15.111:0/764125012 wait complete. 2024-03-14T13:27:53.858 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 Processor -- start 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 -- start start 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 0x7fd37010ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37010f3b0 con 0x7fd370101be0 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd375b06700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37010f4f0 con 0x7fd370102580 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.860 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34302/0 (socket says 172.21.15.111:34302) 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36f7fe700 1 -- 172.21.15.111:0/758294259 learned_addr learned my addr 172.21.15.111:0/758294259 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36effd700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 0x7fd37010ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36f7fe700 1 -- 172.21.15.111:0/758294259 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 msgr2=0x7fd37010ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.866+0000 7fd36f7fe700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 0x7fd37010ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd36f7fe700 1 -- 172.21.15.111:0/758294259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360005040 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd36f7fe700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd36000b3b0 tx=0x7fd360009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36000e040 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd360013420 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37010f770 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd36001c610 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.867+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd370117cf0 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.868+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3700fd330 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.868+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd360013a00 con 0x7fd370101be0 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.869+0000 7fd357fff700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 0x7fd35806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:53.861 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.869+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd36008cc10 con 0x7fd370101be0 2024-03-14T13:27:53.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.869+0000 7fd36effd700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 0x7fd35806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:53.862 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.869+0000 7fd36effd700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 0x7fd35806ec70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd3700655d0 tx=0x7fd36400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:53.864 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:53.873+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd360056e90 con 0x7fd370101be0 2024-03-14T13:27:53.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:53 smithi137 ceph-mon[108145]: pgmap v714: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:53.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:53 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3654042604' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-14T13:27:53.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:53 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3616962625' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-14T13:27:54.103 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.112+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7fd3700653b0 con 0x7fd370101be0 2024-03-14T13:27:54.104 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.112+0000 7fd357fff700 1 -- 172.21.15.111:0/758294259 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v42) v1 ==== 107+0+3289 (secure 0 0 0) 0x7fd36005a4b0 con 0x7fd370101be0 2024-03-14T13:27:54.104 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:54.104 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.221504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14482,"mds_1":24261},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14482":{"gid":14482,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.111:6827/1337358162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1337358162},{"type":"v1","addr":"172.21.15.111:6827","nonce":1337358162}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24261":{"gid":24261,"name":"cephfs.smithi137.ryucox","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.137:6825/3856946079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3856946079},{"type":"v1","addr":"172.21.15.137:6825","nonce":3856946079}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:54.106 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 msgr2=0x7fd35806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.106 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 0x7fd35806ec70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd3700655d0 tx=0x7fd36400b040 comp rx=0 tx=0).stop 2024-03-14T13:27:54.106 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 msgr2=0x7fd37010e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd36000b3b0 tx=0x7fd360009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 shutdown_connections 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd35806c7c0 0x7fd35806ec70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd370101be0 0x7fd37010e930 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 --2- 172.21.15.111:0/758294259 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd370102580 0x7fd37010ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 >> 172.21.15.111:0/758294259 conn(0x7fd370072ad0 msgr2=0x7fd370104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 shutdown_connections 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.114+0000 7fd375b06700 1 -- 172.21.15.111:0/758294259 wait complete. 2024-03-14T13:27:54.107 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 23 2024-03-14T13:27:54.117 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:53 smithi111 ceph-mon[145314]: pgmap v714: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:27:54.117 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:53 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3654042604' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-03-14T13:27:54.117 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:53 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3616962625' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-03-14T13:27:54.118 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 23 2024-03-14T13:27:54.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-03-14T13:27:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 -- 172.21.15.111:0/1870976619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 msgr2=0x7fe54c0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1870976619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c0fbbf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe548004940 tx=0x7fe548009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 -- 172.21.15.111:0/1870976619 shutdown_connections 2024-03-14T13:27:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1870976619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c0fbbf0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.239 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1870976619 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 -- 172.21.15.111:0/1870976619 >> 172.21.15.111:0/1870976619 conn(0x7fe54c0f7710 msgr2=0x7fe54c0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 -- 172.21.15.111:0/1870976619 shutdown_connections 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.245+0000 7fe553dbb700 1 -- 172.21.15.111:0/1870976619 wait complete. 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 Processor -- start 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 -- start start 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c10e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10f380 con 0x7fe54c102580 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c10f4c0 con 0x7fe54c101be0 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34306/0 (socket says 172.21.15.111:34306) 2024-03-14T13:27:54.240 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 -- 172.21.15.111:0/1518722741 learned_addr learned my addr 172.21.15.111:0/1518722741 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551b57700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c10e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 -- 172.21.15.111:0/1518722741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 msgr2=0x7fe54c10e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c10e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 -- 172.21.15.111:0/1518722741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548005040 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551b57700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c10e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe551356700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe548009380 tx=0x7fe548009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54800e040 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe548013420 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c10f740 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.246+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe54801c630 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.247+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c117cc0 con 0x7fe54c102580 2024-03-14T13:27:54.241 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.247+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c0fd330 con 0x7fe54c102580 2024-03-14T13:27:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.248+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe5480139e0 con 0x7fe54c102580 2024-03-14T13:27:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.248+0000 7fe5427fc700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 0x7fe53806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.249+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe54808cc40 con 0x7fe54c102580 2024-03-14T13:27:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.249+0000 7fe551b57700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 0x7fe53806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.242 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.249+0000 7fe551b57700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 0x7fe53806ec70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0041d0 tx=0x7fe53c009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:54.244 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.253+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe548092020 con 0x7fe54c102580 2024-03-14T13:27:54.498 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.504+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7fe54c108b70 con 0x7fe54c102580 2024-03-14T13:27:54.499 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.505+0000 7fe5427fc700 1 -- 172.21.15.111:0/1518722741 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v42) v1 ==== 107+0+3266 (secure 0 0 0) 0x7fe548056ec0 con 0x7fe54c102580 2024-03-14T13:27:54.499 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:54.499 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/1531215732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1531215732},{"type":"v1","addr":"172.21.15.111:6829","nonce":1531215732}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 msgr2=0x7fe53806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 0x7fe53806ec70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0041d0 tx=0x7fe53c009040 comp rx=0 tx=0).stop 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 msgr2=0x7fe54c10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe548009380 tx=0x7fe548009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 shutdown_connections 2024-03-14T13:27:54.501 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fe53806c7c0 0x7fe53806ec70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54c101be0 0x7fe54c10e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 --2- 172.21.15.111:0/1518722741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe54c102580 0x7fe54c10ee40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 >> 172.21.15.111:0/1518722741 conn(0x7fe54c0f7710 msgr2=0x7fe54c0f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 shutdown_connections 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.507+0000 7fe553dbb700 1 -- 172.21.15.111:0/1518722741 wait complete. 2024-03-14T13:27:54.502 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 24 2024-03-14T13:27:54.511 DEBUG:tasks.fs:max_mds not reduced in epoch 24 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:54.511 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 24 2024-03-14T13:27:54.511 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-03-14T13:27:54.638 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 -- 172.21.15.111:0/483623810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 msgr2=0x7f01d8064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 --2- 172.21.15.111:0/483623810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d8064e40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f01c8004940 tx=0x7f01c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 -- 172.21.15.111:0/483623810 shutdown_connections 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 --2- 172.21.15.111:0/483623810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d8064e40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 --2- 172.21.15.111:0/483623810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 0x7f01d81030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 -- 172.21.15.111:0/483623810 >> 172.21.15.111:0/483623810 conn(0x7f01d8072ad0 msgr2=0x7f01d8072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 -- 172.21.15.111:0/483623810 shutdown_connections 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.645+0000 7f01e0216700 1 -- 172.21.15.111:0/483623810 wait complete. 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 Processor -- start 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 -- start start 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 0x7f01d80fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.639 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80fdb90 con 0x7f01d8103680 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01e0216700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80fc6c0 con 0x7f01d8102ce0 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01dd7b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01dd7b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34322/0 (socket says 172.21.15.111:34322) 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.646+0000 7f01dd7b1700 1 -- 172.21.15.111:0/2620330158 learned_addr learned my addr 172.21.15.111:0/2620330158 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01ddfb2700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 0x7f01d80fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01dd7b1700 1 -- 172.21.15.111:0/2620330158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 msgr2=0x7f01d80fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01dd7b1700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 0x7f01d80fbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01dd7b1700 1 -- 172.21.15.111:0/2620330158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c8005040 con 0x7f01d8103680 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01dd7b1700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f01c8008d50 tx=0x7f01c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c800e030 con 0x7f01d8103680 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d80fc940 con 0x7f01d8103680 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d8119e50 con 0x7f01d8103680 2024-03-14T13:27:54.640 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f01c8013de0 con 0x7f01d8103680 2024-03-14T13:27:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.647+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01c801cd90 con 0x7f01d8103680 2024-03-14T13:27:54.641 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.648+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f01c8025400 con 0x7f01d8103680 2024-03-14T13:27:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.649+0000 7f01ce7fc700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 0x7f01c406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.649+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f01c808d360 con 0x7f01d8103680 2024-03-14T13:27:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.649+0000 7f01ddfb2700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 0x7f01c406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.649+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d80fe430 con 0x7f01d8103680 2024-03-14T13:27:54.642 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.649+0000 7f01ddfb2700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 0x7f01c406ec70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01d4004600 tx=0x7f01d400b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:54.645 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.654+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f01c80575a0 con 0x7f01d8103680 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2510691496' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: pgmap v715: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/758294259' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1518722741' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-14T13:27:54.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:27:54.899 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.908+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7f01d80fe430 con 0x7f01d8103680 2024-03-14T13:27:54.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.909+0000 7f01ce7fc700 1 -- 172.21.15.111:0/2620330158 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f01c805abc0 con 0x7f01d8103680 2024-03-14T13:27:54.900 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:54.900 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/1531215732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1531215732},{"type":"v1","addr":"172.21.15.111:6829","nonce":1531215732}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34384,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/3595552801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":3595552801},{"type":"v1","addr":"172.21.15.111:6827","nonce":3595552801}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44287,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/1043472821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1043472821},{"type":"v1","addr":"172.21.15.137:6825","nonce":1043472821}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:54.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 msgr2=0x7f01c406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 0x7f01c406ec70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f01d4004600 tx=0x7f01d400b040 comp rx=0 tx=0).stop 2024-03-14T13:27:54.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 msgr2=0x7f01d80fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:54.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f01c8008d50 tx=0x7f01c8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:54.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 shutdown_connections 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f01c406c7c0 0x7f01c406ec70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01d8102ce0 0x7f01d80fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 --2- 172.21.15.111:0/2620330158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01d8103680 0x7f01d80fc180 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.911+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 >> 172.21.15.111:0/2620330158 conn(0x7f01d8072ad0 msgr2=0x7f01d81008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.912+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 shutdown_connections 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:54.912+0000 7f01e0216700 1 -- 172.21.15.111:0/2620330158 wait complete. 2024-03-14T13:27:54.904 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 25 2024-03-14T13:27:54.916 DEBUG:tasks.fs:max_mds not reduced in epoch 25 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:54.916 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 25 2024-03-14T13:27:54.917 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-03-14T13:27:54.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2510691496' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-03-14T13:27:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: pgmap v715: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:27:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/758294259' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-03-14T13:27:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:27:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1518722741' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-03-14T13:27:54.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:27:55.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 -- 172.21.15.111:0/3412580340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc102580 msgr2=0x7f68cc0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/3412580340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc102580 0x7f68cc0fbbf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f68b4004940 tx=0x7f68b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 -- 172.21.15.111:0/3412580340 shutdown_connections 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/3412580340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc102580 0x7f68cc0fbbf0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/3412580340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc101be0 0x7f68cc101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 -- 172.21.15.111:0/3412580340 >> 172.21.15.111:0/3412580340 conn(0x7f68cc072ad0 msgr2=0x7f68cc072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 -- 172.21.15.111:0/3412580340 shutdown_connections 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.047+0000 7f68d1eb5700 1 -- 172.21.15.111:0/3412580340 wait complete. 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 Processor -- start 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 -- start start 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc102580 0x7f68cc10edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.042 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc10f310 con 0x7f68cc101be0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68d1eb5700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc10f450 con 0x7f68cc102580 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34338/0 (socket says 172.21.15.111:34338) 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 -- 172.21.15.111:0/22500258 learned_addr learned my addr 172.21.15.111:0/22500258 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68caffd700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc102580 0x7f68cc10edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 -- 172.21.15.111:0/22500258 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc102580 msgr2=0x7f68cc10edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc102580 0x7f68cc10edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 -- 172.21.15.111:0/22500258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4005040 con 0x7f68cc101be0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.048+0000 7f68cb7fe700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00eb40 tx=0x7f68bc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.049+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68bc007cc0 con 0x7f68cc101be0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.049+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68bc00f040 con 0x7f68cc101be0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.049+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc10f6d0 con 0x7f68cc101be0 2024-03-14T13:27:55.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.049+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68bc00c990 con 0x7f68cc101be0 2024-03-14T13:27:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.049+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc117c00 con 0x7f68cc101be0 2024-03-14T13:27:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.050+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc0fd330 con 0x7f68cc101be0 2024-03-14T13:27:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.050+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f68bc00cc00 con 0x7f68cc101be0 2024-03-14T13:27:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.050+0000 7f68c3fff700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 0x7f68b806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.044 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.051+0000 7f68caffd700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 0x7f68b806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.051+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f68bc08b670 con 0x7f68cc101be0 2024-03-14T13:27:55.045 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.051+0000 7f68caffd700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 0x7f68b806ec70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0655b0 tx=0x7f68b4009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.046 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.055+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f68bc055920 con 0x7f68cc101be0 2024-03-14T13:27:55.290 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.299+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f68cc064970 con 0x7f68cc101be0 2024-03-14T13:27:55.291 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.299+0000 7f68c3fff700 1 -- 172.21.15.111:0/22500258 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v42) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f68cc064970 con 0x7f68cc101be0 2024-03-14T13:27:55.291 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:55.291 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/1531215732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1531215732},{"type":"v1","addr":"172.21.15.111:6829","nonce":1531215732}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44287,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/1043472821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1043472821},{"type":"v1","addr":"172.21.15.137:6825","nonce":1043472821}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:55.293 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 msgr2=0x7f68b806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 0x7f68b806ec70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0655b0 tx=0x7f68b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 msgr2=0x7f68cc10e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00eb40 tx=0x7f68bc00b040 comp rx=0 tx=0).stop 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 shutdown_connections 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f68b806c7c0 0x7f68b806ec70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68cc101be0 0x7f68cc10e890 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 --2- 172.21.15.111:0/22500258 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc102580 0x7f68cc10edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 >> 172.21.15.111:0/22500258 conn(0x7f68cc072ad0 msgr2=0x7f68cc104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 shutdown_connections 2024-03-14T13:27:55.294 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.302+0000 7f68d1eb5700 1 -- 172.21.15.111:0/22500258 wait complete. 2024-03-14T13:27:55.295 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 26 2024-03-14T13:27:55.306 DEBUG:tasks.fs:max_mds not reduced in epoch 26 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:55.306 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 26 2024-03-14T13:27:55.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-03-14T13:27:55.427 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 -- 172.21.15.111:0/4079668821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed8101c10 msgr2=0x7f6ed8101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4079668821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed8101c10 0x7f6ed8101fe0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8004940 tx=0x7f6ec8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 -- 172.21.15.111:0/4079668821 shutdown_connections 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4079668821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed81025b0 0x7f6ed80fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4079668821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed8101c10 0x7f6ed8101fe0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 -- 172.21.15.111:0/4079668821 >> 172.21.15.111:0/4079668821 conn(0x7f6ed80f7720 msgr2=0x7f6ed80f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 -- 172.21.15.111:0/4079668821 shutdown_connections 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.433+0000 7f6eddd28700 1 -- 172.21.15.111:0/4079668821 wait complete. 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 Processor -- start 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 -- start start 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed8101c10 0x7f6ed81126b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed8113280 con 0x7f6ed81025b0 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6eddd28700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed810c840 con 0x7f6ed8101c10 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6ed6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6ed6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34356/0 (socket says 172.21.15.111:34356) 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6ed6ffd700 1 -- 172.21.15.111:0/4023251507 learned_addr learned my addr 172.21.15.111:0/4023251507 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.434+0000 7f6ed77fe700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed8101c10 0x7f6ed81126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ed6ffd700 1 -- 172.21.15.111:0/4023251507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed8101c10 msgr2=0x7f6ed81126b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ed6ffd700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed8101c10 0x7f6ed81126b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ed6ffd700 1 -- 172.21.15.111:0/4023251507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8005040 con 0x7f6ed81025b0 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ed6ffd700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc002a50 tx=0x7f6ecc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ecc007cc0 con 0x7f6ed81025b0 2024-03-14T13:27:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ecc00f040 con 0x7f6ed81025b0 2024-03-14T13:27:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed810cad0 con 0x7f6ed81025b0 2024-03-14T13:27:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ecc00d920 con 0x7f6ed81025b0 2024-03-14T13:27:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.435+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed810d010 con 0x7f6ed81025b0 2024-03-14T13:27:55.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.436+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed80fd360 con 0x7f6ed81025b0 2024-03-14T13:27:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.437+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6ecc009680 con 0x7f6ed81025b0 2024-03-14T13:27:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.437+0000 7f6ebffff700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 0x7f6ec006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.438+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6ecc08bef0 con 0x7f6ed81025b0 2024-03-14T13:27:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.438+0000 7f6ed77fe700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 0x7f6ec006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.438+0000 7f6ed77fe700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 0x7f6ec006ec70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800b3b0 tx=0x7f6ec8009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.432 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.441+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6ecc0b9960 con 0x7f6ed81025b0 2024-03-14T13:27:55.670 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.679+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f6ed80640e0 con 0x7f6ed81025b0 2024-03-14T13:27:55.671 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.679+0000 7f6ebffff700 1 -- 172.21.15.111:0/4023251507 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7f6ecc0598f0 con 0x7f6ed81025b0 2024-03-14T13:27:55.671 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:55.671 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34376,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/1531215732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1531215732},{"type":"v1","addr":"172.21.15.111:6829","nonce":1531215732}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44287,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/1043472821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1043472821},{"type":"v1","addr":"172.21.15.137:6825","nonce":1043472821}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:55.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 msgr2=0x7f6ec006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 0x7f6ec006ec70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800b3b0 tx=0x7f6ec8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:55.673 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 msgr2=0x7f6ed8112c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc002a50 tx=0x7f6ecc004600 comp rx=0 tx=0).stop 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 shutdown_connections 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6ec006c7c0 0x7f6ec006ec70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ed8101c10 0x7f6ed81126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 --2- 172.21.15.111:0/4023251507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ed81025b0 0x7f6ed8112c30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.681+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 >> 172.21.15.111:0/4023251507 conn(0x7f6ed80f7720 msgr2=0x7f6ed80f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.682+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 shutdown_connections 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.682+0000 7f6eddd28700 1 -- 172.21.15.111:0/4023251507 wait complete. 2024-03-14T13:27:55.674 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 27 2024-03-14T13:27:55.685 DEBUG:tasks.fs:max_mds not reduced in epoch 27 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:55.685 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 27 2024-03-14T13:27:55.685 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- 172.21.15.111:0/1582942150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c102640 msgr2=0x7f640c0fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 --2- 172.21.15.111:0/1582942150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c102640 0x7f640c0fc450 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6408004940 tx=0x7f6408009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- 172.21.15.111:0/1582942150 shutdown_connections 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 --2- 172.21.15.111:0/1582942150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c102640 0x7f640c0fc450 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 --2- 172.21.15.111:0/1582942150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c101ca0 0x7f640c102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.806 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- 172.21.15.111:0/1582942150 >> 172.21.15.111:0/1582942150 conn(0x7f640c072230 msgr2=0x7f640c072630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- 172.21.15.111:0/1582942150 shutdown_connections 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- 172.21.15.111:0/1582942150 wait complete. 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 Processor -- start 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 -- start start 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.813+0000 7f64145c9700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c102640 0x7f640c10ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f64145c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c10f3d0 con 0x7f640c101ca0 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f64145c9700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c10f510 con 0x7f640c102640 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34374/0 (socket says 172.21.15.111:34374) 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 -- 172.21.15.111:0/3625755784 learned_addr learned my addr 172.21.15.111:0/3625755784 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6411b64700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c102640 0x7f640c10ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 -- 172.21.15.111:0/3625755784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c102640 msgr2=0x7f640c10ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c102640 0x7f640c10ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:55.807 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 -- 172.21.15.111:0/3625755784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6408005040 con 0x7f640c101ca0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6412365700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f63fc002a50 tx=0x7f63fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63fc007cc0 con 0x7f640c101ca0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c1179c0 con 0x7f640c101ca0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63fc00f040 con 0x7f640c101ca0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63fc009540 con 0x7f640c101ca0 2024-03-14T13:27:55.808 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.814+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c117d10 con 0x7f640c101ca0 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.815+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c0fefc0 con 0x7f640c101ca0 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.816+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f63fc0096a0 con 0x7f640c101ca0 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.816+0000 7f6402ffd700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 0x7f63f806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.817+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f63fc08bf40 con 0x7f640c101ca0 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.817+0000 7f6411b64700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 0x7f63f806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:55.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.817+0000 7f6411b64700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 0x7f63f806ec70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f640800b3b0 tx=0x7f6408009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:55.812 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:55.820+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f63fc056140 con 0x7f640c101ca0 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2620330158' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/22500258' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-14T13:27:55.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:55 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4023251507' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-14T13:27:56.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.057+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7f640c0fd460 con 0x7f640c101ca0 2024-03-14T13:27:56.049 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.058+0000 7f6402ffd700 1 -- 172.21.15.111:0/3625755784 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v42) v1 ==== 107+0+4048 (secure 0 0 0) 0x7f63fc059760 con 0x7f640c101ca0 2024-03-14T13:27:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:56.050 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44287,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/1043472821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1043472821},{"type":"v1","addr":"172.21.15.137:6825","nonce":1043472821}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 msgr2=0x7f63f806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 0x7f63f806ec70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f640800b3b0 tx=0x7f6408009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 msgr2=0x7f640c10e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f63fc002a50 tx=0x7f63fc0046b0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 shutdown_connections 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f63f806c7c0 0x7f63f806ec70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640c101ca0 0x7f640c10e950 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.052 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 --2- 172.21.15.111:0/3625755784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f640c102640 0x7f640c10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 >> 172.21.15.111:0/3625755784 conn(0x7f640c072230 msgr2=0x7f640c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 shutdown_connections 2024-03-14T13:27:56.053 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.060+0000 7f64145c9700 1 -- 172.21.15.111:0/3625755784 wait complete. 2024-03-14T13:27:56.053 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 28 2024-03-14T13:27:56.062 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2620330158' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-03-14T13:27:56.062 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:27:56.062 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:27:56.063 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:27:56.063 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/22500258' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-03-14T13:27:56.063 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:55 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4023251507' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-03-14T13:27:56.064 DEBUG:tasks.fs:max_mds not reduced in epoch 28 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:56.065 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 28 2024-03-14T13:27:56.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-03-14T13:27:56.185 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 -- 172.21.15.111:0/4190170328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c101be0 msgr2=0x7faa1c101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 --2- 172.21.15.111:0/4190170328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c101be0 0x7faa1c101fb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faa10004940 tx=0x7faa10009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 -- 172.21.15.111:0/4190170328 shutdown_connections 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 --2- 172.21.15.111:0/4190170328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c102580 0x7faa1c0fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 --2- 172.21.15.111:0/4190170328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c101be0 0x7faa1c101fb0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 -- 172.21.15.111:0/4190170328 >> 172.21.15.111:0/4190170328 conn(0x7faa1c072ad0 msgr2=0x7faa1c072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 -- 172.21.15.111:0/4190170328 shutdown_connections 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 -- 172.21.15.111:0/4190170328 wait complete. 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.192+0000 7faa21c0d700 1 Processor -- start 2024-03-14T13:27:56.186 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa21c0d700 1 -- start start 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa21c0d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c101be0 0x7faa1c10e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa21c0d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa21c0d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c10f3f0 con 0x7faa1c102580 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa21c0d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c10f530 con 0x7faa1c101be0 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34386/0 (socket says 172.21.15.111:34386) 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 -- 172.21.15.111:0/1017246029 learned_addr learned my addr 172.21.15.111:0/1017246029 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1b7fe700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c101be0 0x7faa1c10e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 -- 172.21.15.111:0/1017246029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c101be0 msgr2=0x7faa1c10e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c101be0 0x7faa1c10e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 -- 172.21.15.111:0/1017246029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10005040 con 0x7faa1c102580 2024-03-14T13:27:56.187 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa1affd700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c800 tx=0x7faa0c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.193+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa0c0106f0 con 0x7faa1c102580 2024-03-14T13:27:56.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.194+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c1179e0 con 0x7faa1c102580 2024-03-14T13:27:56.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.194+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa0c010d00 con 0x7faa1c102580 2024-03-14T13:27:56.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.194+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa0c0113a0 con 0x7faa1c102580 2024-03-14T13:27:56.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.194+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c117db0 con 0x7faa1c102580 2024-03-14T13:27:56.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.195+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0fd330 con 0x7faa1c102580 2024-03-14T13:27:56.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.195+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faa0c012400 con 0x7faa1c102580 2024-03-14T13:27:56.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.195+0000 7faa03fff700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 0x7faa0406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.196+0000 7faa1b7fe700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 0x7faa0406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.196+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa0c08ba10 con 0x7faa1c102580 2024-03-14T13:27:56.190 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.196+0000 7faa1b7fe700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 0x7faa0406ec70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faa10004f90 tx=0x7faa10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.200+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faa0c055c90 con 0x7faa1c102580 2024-03-14T13:27:56.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.439+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7faa1c064970 con 0x7faa1c102580 2024-03-14T13:27:56.431 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.439+0000 7faa03fff700 1 -- 172.21.15.111:0/1017246029 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7faa0c0592b0 con 0x7faa1c102580 2024-03-14T13:27:56.431 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:56.431 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44287,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/1043472821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":1043472821},{"type":"v1","addr":"172.21.15.137:6825","nonce":1043472821}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 msgr2=0x7faa0406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 0x7faa0406ec70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faa10004f90 tx=0x7faa10009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 msgr2=0x7faa1c10eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c800 tx=0x7faa0c004470 comp rx=0 tx=0).stop 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 shutdown_connections 2024-03-14T13:27:56.433 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa0406c7c0 0x7faa0406ec70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa1c101be0 0x7faa1c10e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 --2- 172.21.15.111:0/1017246029 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa1c102580 0x7faa1c10eeb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.441+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 >> 172.21.15.111:0/1017246029 conn(0x7faa1c072ad0 msgr2=0x7faa1c104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.442+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 shutdown_connections 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.442+0000 7faa21c0d700 1 -- 172.21.15.111:0/1017246029 wait complete. 2024-03-14T13:27:56.434 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 29 2024-03-14T13:27:56.445 DEBUG:tasks.fs:max_mds not reduced in epoch 29 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:56.445 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 29 2024-03-14T13:27:56.445 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-03-14T13:27:56.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 -- 172.21.15.111:0/1416330805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410102ce0 msgr2=0x7fd4101030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 --2- 172.21.15.111:0/1416330805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410102ce0 0x7fd4101030b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd404004940 tx=0x7fd404009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 -- 172.21.15.111:0/1416330805 shutdown_connections 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 --2- 172.21.15.111:0/1416330805 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd410103680 0x7fd4100fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 --2- 172.21.15.111:0/1416330805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410102ce0 0x7fd4101030b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 -- 172.21.15.111:0/1416330805 >> 172.21.15.111:0/1416330805 conn(0x7fd4100f8890 msgr2=0x7fd4100faca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 -- 172.21.15.111:0/1416330805 shutdown_connections 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.572+0000 7fd41725d700 1 -- 172.21.15.111:0/1416330805 wait complete. 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 Processor -- start 2024-03-14T13:27:56.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 -- start start 2024-03-14T13:27:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd410102ce0 0x7fd410113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.567 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101142e0 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.573+0000 7fd41725d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41010d8e0 con 0x7fd410102ce0 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34404/0 (socket says 172.21.15.111:34404) 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 -- 172.21.15.111:0/1483315407 learned_addr learned my addr 172.21.15.111:0/1483315407 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 -- 172.21.15.111:0/1483315407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd410102ce0 msgr2=0x7fd410113750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd410102ce0 0x7fd410113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 -- 172.21.15.111:0/1483315407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd404005040 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40ffff700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc002a50 tx=0x7fd3fc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3fc007cc0 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3fc00f040 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41010dbc0 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3fc00d990 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.574+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41010e180 con 0x7fd410103680 2024-03-14T13:27:56.568 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.575+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4100fe430 con 0x7fd410103680 2024-03-14T13:27:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.576+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd3fc009710 con 0x7fd410103680 2024-03-14T13:27:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.576+0000 7fd40d7fa700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.576+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd3fc08b890 con 0x7fd410103680 2024-03-14T13:27:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.576+0000 7fd414ff9700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.576+0000 7fd414ff9700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd404004f90 tx=0x7fd404009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.572 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.581+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd3fc059520 con 0x7fd410103680 2024-03-14T13:27:56.809 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.818+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7fd410109ca0 con 0x7fd410103680 2024-03-14T13:27:56.810 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.819+0000 7fd40d7fa700 1 -- 172.21.15.111:0/1483315407 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v42) v1 ==== 107+0+4048 (secure 0 0 0) 0x7fd3fc0590b0 con 0x7fd410103680 2024-03-14T13:27:56.811 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:56.811 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 msgr2=0x7fd40006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd404004f90 tx=0x7fd404009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 msgr2=0x7fd410113c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.813 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc002a50 tx=0x7fd3fc004600 comp rx=0 tx=0).stop 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 shutdown_connections 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd40006c7c0 0x7fd40006ec70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd410102ce0 0x7fd410113750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 --2- 172.21.15.111:0/1483315407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd410103680 0x7fd410113c90 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 >> 172.21.15.111:0/1483315407 conn(0x7fd4100f8890 msgr2=0x7fd4100f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 shutdown_connections 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.821+0000 7fd41725d700 1 -- 172.21.15.111:0/1483315407 wait complete. 2024-03-14T13:27:56.814 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 30 2024-03-14T13:27:56.825 DEBUG:tasks.fs:max_mds not reduced in epoch 30 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:56.825 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 30 2024-03-14T13:27:56.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 -- 172.21.15.111:0/1418373464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 msgr2=0x7fd1f0101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/1418373464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0101f90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4004940 tx=0x7fd1e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 -- 172.21.15.111:0/1418373464 shutdown_connections 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/1418373464 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 0x7fd1f00fbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/1418373464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0101f90 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.951+0000 7fd1f59aa700 1 -- 172.21.15.111:0/1418373464 >> 172.21.15.111:0/1418373464 conn(0x7fd1f00f7730 msgr2=0x7fd1f00f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 -- 172.21.15.111:0/1418373464 shutdown_connections 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 -- 172.21.15.111:0/1418373464 wait complete. 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 Processor -- start 2024-03-14T13:27:56.945 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 -- start start 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 0x7fd1f0112c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f0113250 con 0x7fd1f0101bc0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1f59aa700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f010c810 con 0x7fd1f0102560 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.952+0000 7fd1eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34434/0 (socket says 172.21.15.111:34434) 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 -- 172.21.15.111:0/3656202886 learned_addr learned my addr 172.21.15.111:0/3656202886 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1ee7fc700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 0x7fd1f0112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 -- 172.21.15.111:0/3656202886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 msgr2=0x7fd1f0112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 0x7fd1f0112c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 -- 172.21.15.111:0/3656202886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e4005040 con 0x7fd1f0101bc0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1eeffd700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4009360 tx=0x7fd1e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e400e040 con 0x7fd1f0101bc0 2024-03-14T13:27:56.946 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1e4013420 con 0x7fd1f0101bc0 2024-03-14T13:27:56.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f010ca40 con 0x7fd1f0101bc0 2024-03-14T13:27:56.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1e401c5f0 con 0x7fd1f0101bc0 2024-03-14T13:27:56.947 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.953+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f010cf80 con 0x7fd1f0101bc0 2024-03-14T13:27:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.954+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f00fd310 con 0x7fd1f0101bc0 2024-03-14T13:27:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.955+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd1e4013590 con 0x7fd1f0101bc0 2024-03-14T13:27:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.955+0000 7fd1d77fe700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 0x7fd1d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:56.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.955+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd1e408cd00 con 0x7fd1f0101bc0 2024-03-14T13:27:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.955+0000 7fd1ee7fc700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 0x7fd1d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:56.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.955+0000 7fd1ee7fc700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 0x7fd1d806ec70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0007fd0 tx=0x7fd1e0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:56.951 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:56.960+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd1e405aa10 con 0x7fd1f0101bc0 2024-03-14T13:27:56.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:56 smithi137 ceph-mon[108145]: pgmap v716: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:56.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:56 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3625755784' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-14T13:27:56.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:56 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1017246029' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-14T13:27:57.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:56 smithi111 ceph-mon[145314]: pgmap v716: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:27:57.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:56 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3625755784' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-03-14T13:27:57.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:56 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1017246029' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-03-14T13:27:57.188 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.197+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fd1f010d1a0 con 0x7fd1f0101bc0 2024-03-14T13:27:57.189 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.197+0000 7fd1d77fe700 1 -- 172.21.15.111:0/3656202886 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v42) v1 ==== 107+0+4830 (secure 0 0 0) 0x7fd1e405a5a0 con 0x7fd1f0101bc0 2024-03-14T13:27:57.189 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:57.189 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44279,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/513022872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":513022872},{"type":"v1","addr":"172.21.15.137:6827","nonce":513022872}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":24},{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.199+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 msgr2=0x7fd1d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.199+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 0x7fd1d806ec70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0007fd0 tx=0x7fd1e0009040 comp rx=0 tx=0).stop 2024-03-14T13:27:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.199+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 msgr2=0x7fd1f0112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.199+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4009360 tx=0x7fd1e4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:57.191 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 shutdown_connections 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd1d806c7c0 0x7fd1d806ec70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1f0101bc0 0x7fd1f0112680 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 --2- 172.21.15.111:0/3656202886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1f0102560 0x7fd1f0112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 >> 172.21.15.111:0/3656202886 conn(0x7fd1f00f7730 msgr2=0x7fd1f00f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 shutdown_connections 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.200+0000 7fd1f59aa700 1 -- 172.21.15.111:0/3656202886 wait complete. 2024-03-14T13:27:57.192 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 31 2024-03-14T13:27:57.203 DEBUG:tasks.fs:max_mds not reduced in epoch 31 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:57.203 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 31 2024-03-14T13:27:57.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 -- 172.21.15.111:0/1279583638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70101be0 msgr2=0x7f6f70101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 --2- 172.21.15.111:0/1279583638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70101be0 0x7f6f70101fb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6f60004940 tx=0x7f6f60009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 -- 172.21.15.111:0/1279583638 shutdown_connections 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 --2- 172.21.15.111:0/1279583638 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70102580 0x7f6f700fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 --2- 172.21.15.111:0/1279583638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70101be0 0x7f6f70101fb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 -- 172.21.15.111:0/1279583638 >> 172.21.15.111:0/1279583638 conn(0x7f6f70072ad0 msgr2=0x7f6f70072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 -- 172.21.15.111:0/1279583638 shutdown_connections 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.331+0000 7f6f76876700 1 -- 172.21.15.111:0/1279583638 wait complete. 2024-03-14T13:27:57.325 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 Processor -- start 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 -- start start 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 0x7f6f701137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70114350 con 0x7f6f70102580 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f76876700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7010d9a0 con 0x7f6f70101be0 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34448/0 (socket says 172.21.15.111:34448) 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/1572823325 learned_addr learned my addr 172.21.15.111:0/1572823325 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6ffff700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 0x7f6f701137c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/1572823325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 msgr2=0x7f6f701137c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 0x7f6f701137c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/1572823325 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60005040 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6ffff700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 0x7f6f701137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.332+0000 7f6f6f7fe700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6f64002a80 tx=0x7f6f64004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.333+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f64007cc0 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.333+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7010dc80 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.333+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f6400f040 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.333+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f6400da40 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.333+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7010e240 con 0x7f6f70102580 2024-03-14T13:27:57.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.334+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f700fd330 con 0x7f6f70102580 2024-03-14T13:27:57.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.334+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6f6400dbe0 con 0x7f6f70102580 2024-03-14T13:27:57.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.334+0000 7f6f6cff9700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 0x7f6f5806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.334+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6f6408b1f0 con 0x7f6f70102580 2024-03-14T13:27:57.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.335+0000 7f6f6ffff700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 0x7f6f5806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.328 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.335+0000 7f6f6ffff700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 0x7f6f5806ec70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6f60004560 tx=0x7f6f60009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:57.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.340+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6f64058e80 con 0x7f6f70102580 2024-03-14T13:27:57.569 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.577+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7f6f70064970 con 0x7f6f70102580 2024-03-14T13:27:57.570 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.578+0000 7f6f6cff9700 1 -- 172.21.15.111:0/1572823325 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v42) v1 ==== 107+0+4051 (secure 0 0 0) 0x7f6f64058a10 con 0x7f6f70102580 2024-03-14T13:27:57.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:57.571 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 msgr2=0x7f6f5806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 0x7f6f5806ec70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6f60004560 tx=0x7f6f60009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 msgr2=0x7f6f70113d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6f64002a80 tx=0x7f6f64004630 comp rx=0 tx=0).stop 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 shutdown_connections 2024-03-14T13:27:57.573 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f5806c7c0 0x7f6f5806ec70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f70101be0 0x7f6f701137c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 --2- 172.21.15.111:0/1572823325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f70102580 0x7f6f70113d00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 >> 172.21.15.111:0/1572823325 conn(0x7f6f70072ad0 msgr2=0x7f6f70064dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 shutdown_connections 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.580+0000 7f6f76876700 1 -- 172.21.15.111:0/1572823325 wait complete. 2024-03-14T13:27:57.574 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 32 2024-03-14T13:27:57.585 DEBUG:tasks.fs:max_mds not reduced in epoch 32 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:57.585 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 32 2024-03-14T13:27:57.585 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-03-14T13:27:57.706 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- 172.21.15.111:0/1199174915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 msgr2=0x7f19c40fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/1199174915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c40fbbf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f19b4004940 tx=0x7f19b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- 172.21.15.111:0/1199174915 shutdown_connections 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/1199174915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c40fbbf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/1199174915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 0x7f19c4101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- 172.21.15.111:0/1199174915 >> 172.21.15.111:0/1199174915 conn(0x7f19c4072ad0 msgr2=0x7f19c4072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- 172.21.15.111:0/1199174915 shutdown_connections 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- 172.21.15.111:0/1199174915 wait complete. 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 Processor -- start 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.713+0000 7f19c8f7e700 1 -- start start 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 0x7f19c41126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410c830 con 0x7f19c4102580 2024-03-14T13:27:57.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c410c9a0 con 0x7f19c4101be0 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34474/0 (socket says 172.21.15.111:34474) 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 -- 172.21.15.111:0/560861023 learned_addr learned my addr 172.21.15.111:0/560861023 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c259c700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 0x7f19c41126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 -- 172.21.15.111:0/560861023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 msgr2=0x7f19c41126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 0x7f19c41126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 -- 172.21.15.111:0/560861023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4005040 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c1d9b700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009380 tx=0x7f19b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b400e040 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b4013420 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c410cc20 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19b401c630 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.714+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c410d160 con 0x7f19c4102580 2024-03-14T13:27:57.708 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.715+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c40fd330 con 0x7f19c4102580 2024-03-14T13:27:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.716+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f19b40139e0 con 0x7f19c4102580 2024-03-14T13:27:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.716+0000 7f19b2ffd700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 0x7f19ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.716+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f19b408cc60 con 0x7f19c4102580 2024-03-14T13:27:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.716+0000 7f19c259c700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 0x7f19ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:57.709 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.717+0000 7f19c259c700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 0x7f19ac06ec70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006fd0 tx=0x7f19b8008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:57.712 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.722+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f19b405a970 con 0x7f19c4102580 2024-03-14T13:27:57.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:57 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1483315407' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-14T13:27:57.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:57 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3656202886' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-14T13:27:57.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:57 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1572823325' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-14T13:27:57.949 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.959+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f19c4108bf0 con 0x7f19c4102580 2024-03-14T13:27:57.950 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.959+0000 7f19b2ffd700 1 -- 172.21.15.111:0/560861023 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v42) v1 ==== 107+0+4833 (secure 0 0 0) 0x7f19b405a500 con 0x7f19c4102580 2024-03-14T13:27:57.950 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:57.950 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":24,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:09:44.383471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 msgr2=0x7f19ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 0x7f19ac06ec70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006fd0 tx=0x7f19b8008040 comp rx=0 tx=0).stop 2024-03-14T13:27:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 msgr2=0x7f19c4112c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009380 tx=0x7f19b4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 shutdown_connections 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f19ac06c7c0 0x7f19ac06ec70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19c4101be0 0x7f19c41126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 --2- 172.21.15.111:0/560861023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19c4102580 0x7f19c4112c20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 >> 172.21.15.111:0/560861023 conn(0x7f19c4072ad0 msgr2=0x7f19c4104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.961+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 shutdown_connections 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:57.962+0000 7f19c8f7e700 1 -- 172.21.15.111:0/560861023 wait complete. 2024-03-14T13:27:57.953 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 33 2024-03-14T13:27:57.964 DEBUG:tasks.fs:max_mds not reduced in epoch 33 as fs was failed for carrying out rapid multi-rank mds upgrade 2024-03-14T13:27:57.964 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 33 2024-03-14T13:27:57.964 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-03-14T13:27:57.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:57 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1483315407' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-03-14T13:27:57.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:57 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3656202886' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-03-14T13:27:57.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:57 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1572823325' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-03-14T13:27:58.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 -- 172.21.15.111:0/2993937060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 msgr2=0x7f026c0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 --2- 172.21.15.111:0/2993937060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c0fbea0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f025c004940 tx=0x7f025c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 -- 172.21.15.111:0/2993937060 shutdown_connections 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 --2- 172.21.15.111:0/2993937060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 0x7f026c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 --2- 172.21.15.111:0/2993937060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c0fbea0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 -- 172.21.15.111:0/2993937060 >> 172.21.15.111:0/2993937060 conn(0x7f026c0f7720 msgr2=0x7f026c0f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 -- 172.21.15.111:0/2993937060 shutdown_connections 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 -- 172.21.15.111:0/2993937060 wait complete. 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.092+0000 7f0270d1a700 1 Processor -- start 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 -- start start 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 0x7f026c115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.089 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c116430 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c10fa30 con 0x7f026c0fc3e0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34490/0 (socket says 172.21.15.111:34490) 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 -- 172.21.15.111:0/3406198033 learned_addr learned my addr 172.21.15.111:0/3406198033 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0269d9b700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 0x7f026c115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 -- 172.21.15.111:0/3406198033 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 msgr2=0x7f026c115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 0x7f026c115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 -- 172.21.15.111:0/3406198033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c005040 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f026a59c700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f025c009390 tx=0x7f025c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f025c00e040 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c10fd10 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f025c013420 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f025c01c630 con 0x7f026c0fbad0 2024-03-14T13:27:58.090 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.093+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c1102d0 con 0x7f026c0fbad0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.095+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c066470 con 0x7f026c0fbad0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.095+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f025c013a20 con 0x7f026c0fbad0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.095+0000 7f025affd700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 0x7f025406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.095+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f025c08cd00 con 0x7f026c0fbad0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.095+0000 7f0269d9b700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 0x7f025406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.096+0000 7f0269d9b700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 0x7f025406ec70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0260007fd0 tx=0x7f0260009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.091 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.100+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f025c05aa10 con 0x7f026c0fbad0 2024-03-14T13:27:58.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.336+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7f026c10bde0 con 0x7f026c0fbad0 2024-03-14T13:27:58.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.337+0000 7f025affd700 1 -- 172.21.15.111:0/3406198033 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v42) v1 ==== 107+0+4832 (secure 0 0 0) 0x7f025c05a5a0 con 0x7f026c0fbad0 2024-03-14T13:27:58.327 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:58.327 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29},{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:46.677283+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{},"failed":[0,1],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:58.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 msgr2=0x7f025406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.329 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 0x7f025406ec70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0260007fd0 tx=0x7f0260009040 comp rx=0 tx=0).stop 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 msgr2=0x7f026c1158a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f025c009390 tx=0x7f025c009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 shutdown_connections 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f025406c7c0 0x7f025406ec70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f026c0fbad0 0x7f026c1158a0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 --2- 172.21.15.111:0/3406198033 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f026c0fc3e0 0x7f026c115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 >> 172.21.15.111:0/3406198033 conn(0x7f026c0f7720 msgr2=0x7f026c0f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 shutdown_connections 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.339+0000 7f0270d1a700 1 -- 172.21.15.111:0/3406198033 wait complete. 2024-03-14T13:27:58.330 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 34 2024-03-14T13:27:58.341 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 34 2024-03-14T13:27:58.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 35 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- 172.21.15.111:0/3400081531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 msgr2=0x7f66440fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3400081531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f66440fbbf0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6634004940 tx=0x7f6634009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- 172.21.15.111:0/3400081531 shutdown_connections 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3400081531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f66440fbbf0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3400081531 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 0x7f6644101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- 172.21.15.111:0/3400081531 >> 172.21.15.111:0/3400081531 conn(0x7f6644072ad0 msgr2=0x7f6644072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:58.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- 172.21.15.111:0/3400081531 shutdown_connections 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- 172.21.15.111:0/3400081531 wait complete. 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 Processor -- start 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 -- start start 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.470+0000 7f664c2b8700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 0x7f66441126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664c2b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664c2b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644113230 con 0x7f6644102580 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664c2b8700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664410c7f0 con 0x7f6644101be0 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34510/0 (socket says 172.21.15.111:34510) 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 -- 172.21.15.111:0/3502442109 learned_addr learned my addr 172.21.15.111:0/3502442109 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664a054700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 0x7f66441126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 -- 172.21.15.111:0/3502442109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 msgr2=0x7f66441126a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 0x7f66441126a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 -- 172.21.15.111:0/3502442109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634005040 con 0x7f6644102580 2024-03-14T13:27:58.464 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f6649853700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6634009390 tx=0x7f6634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663400e040 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6634004290 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664410ca70 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6634013e70 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.471+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664410cfb0 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.472+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644112330 con 0x7f6644102580 2024-03-14T13:27:58.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.473+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6634013690 con 0x7f6644102580 2024-03-14T13:27:58.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.473+0000 7f663affd700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 0x7f663006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.473+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f663408cc70 con 0x7f6644102580 2024-03-14T13:27:58.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.473+0000 7f664a054700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 0x7f663006ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.473+0000 7f664a054700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 0x7f663006ec10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6640004f40 tx=0x7f664000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.468 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.477+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f663405aa30 con 0x7f6644102580 2024-03-14T13:27:58.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.716+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 35, "format": "json"} v 0) v1 -- 0x7f66440fd330 con 0x7f6644102580 2024-03-14T13:27:58.707 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.717+0000 7f663affd700 1 -- 172.21.15.111:0/3502442109 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 35, "format": "json"}]=0 dumped fsmap epoch 35 v42) v1 ==== 107+0+4855 (secure 0 0 0) 0x7f6634013420 con 0x7f6644102580 2024-03-14T13:27:58.708 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:58.708 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:46.682594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 msgr2=0x7f663006ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 0x7f663006ec10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6640004f40 tx=0x7f664000b040 comp rx=0 tx=0).stop 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 msgr2=0x7f6644112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6634009390 tx=0x7f6634009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 shutdown_connections 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f663006c760 0x7f663006ec10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.710 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6644101be0 0x7f66441126a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 --2- 172.21.15.111:0/3502442109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6644102580 0x7f6644112be0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 >> 172.21.15.111:0/3502442109 conn(0x7f6644072ad0 msgr2=0x7f6644104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:58.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 shutdown_connections 2024-03-14T13:27:58.711 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.719+0000 7f664c2b8700 1 -- 172.21.15.111:0/3502442109 wait complete. 2024-03-14T13:27:58.711 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 35 2024-03-14T13:27:58.722 DEBUG:tasks.fs:allow_standby_replay disabled in epoch 35 2024-03-14T13:27:58.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 36 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 -- 172.21.15.111:0/2165456373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0102580 msgr2=0x7f21e00fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 --2- 172.21.15.111:0/2165456373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0102580 0x7f21e00fbbf0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f21d0004940 tx=0x7f21d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 -- 172.21.15.111:0/2165456373 shutdown_connections 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 --2- 172.21.15.111:0/2165456373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0102580 0x7f21e00fbbf0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 --2- 172.21.15.111:0/2165456373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0101be0 0x7f21e0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 -- 172.21.15.111:0/2165456373 >> 172.21.15.111:0/2165456373 conn(0x7f21e0072ad0 msgr2=0x7f21e0072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:58.843 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 -- 172.21.15.111:0/2165456373 shutdown_connections 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.850+0000 7f21e4810700 1 -- 172.21.15.111:0/2165456373 wait complete. 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 Processor -- start 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 -- start start 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0102580 0x7f21e010ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e010f370 con 0x7f21e0101be0 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21e4810700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e010f4e0 con 0x7f21e0102580 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34516/0 (socket says 172.21.15.111:34516) 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 -- 172.21.15.111:0/3954545058 learned_addr learned my addr 172.21.15.111:0/3954545058 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21ddd9b700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0102580 0x7f21e010ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 -- 172.21.15.111:0/3954545058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0102580 msgr2=0x7f21e010ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0102580 0x7f21e010ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 -- 172.21.15.111:0/3954545058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0005040 con 0x7f21e0101be0 2024-03-14T13:27:58.844 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.851+0000 7f21de59c700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c8c0 tx=0x7f21d40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.852+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d400a980 con 0x7f21e0101be0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.852+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d400ce60 con 0x7f21e0101be0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.852+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e0117960 con 0x7f21e0101be0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.852+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21d40113a0 con 0x7f21e0101be0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.852+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e0117d00 con 0x7f21e0101be0 2024-03-14T13:27:58.845 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.853+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e00fd330 con 0x7f21e0101be0 2024-03-14T13:27:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.853+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f21d4012400 con 0x7f21e0101be0 2024-03-14T13:27:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.853+0000 7f21ceffd700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 0x7f21c806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.853+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f21d408bb90 con 0x7f21e0101be0 2024-03-14T13:27:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.853+0000 7f21ddd9b700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 0x7f21c806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:58.846 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.854+0000 7f21ddd9b700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 0x7f21c806ec70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f21d000b3b0 tx=0x7f21d0009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:58.848 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:58.858+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f21d4055e10 con 0x7f21e0101be0 2024-03-14T13:27:58.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:58 smithi137 ceph-mon[108145]: pgmap v717: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:58.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:58 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/560861023' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-14T13:27:58.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:58 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3406198033' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-14T13:27:59.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.094+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 36, "format": "json"} v 0) v1 -- 0x7f21e0065380 con 0x7f21e0101be0 2024-03-14T13:27:59.085 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.095+0000 7f21ceffd700 1 -- 172.21.15.111:0/3954545058 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 36, "format": "json"}]=0 dumped fsmap epoch 36 v42) v1 ==== 107+0+4854 (secure 0 0 0) 0x7f21d4059430 con 0x7f21e0101be0 2024-03-14T13:27:59.085 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:59.086 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:47.687899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 msgr2=0x7f21c806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 0x7f21c806ec70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f21d000b3b0 tx=0x7f21d0009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 msgr2=0x7f21e010e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c8c0 tx=0x7f21d40044c0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 shutdown_connections 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f21c806c7c0 0x7f21c806ec70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21e0101be0 0x7f21e010e8f0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 --2- 172.21.15.111:0/3954545058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21e0102580 0x7f21e010ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 >> 172.21.15.111:0/3954545058 conn(0x7f21e0072ad0 msgr2=0x7f21e0104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 shutdown_connections 2024-03-14T13:27:59.088 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.097+0000 7f21e4810700 1 -- 172.21.15.111:0/3954545058 wait complete. 2024-03-14T13:27:59.089 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 36 2024-03-14T13:27:59.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:58 smithi111 ceph-mon[145314]: pgmap v717: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:27:59.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:58 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/560861023' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-03-14T13:27:59.098 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:58 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3406198033' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-03-14T13:27:59.100 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 37 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 -- 172.21.15.111:0/469886604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 msgr2=0x7fd4b4101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 --2- 172.21.15.111:0/469886604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4101fd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8004940 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 -- 172.21.15.111:0/469886604 shutdown_connections 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 --2- 172.21.15.111:0/469886604 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b40fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 --2- 172.21.15.111:0/469886604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4101fd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 -- 172.21.15.111:0/469886604 >> 172.21.15.111:0/469886604 conn(0x7fd4b40f7750 msgr2=0x7fd4b40f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 -- 172.21.15.111:0/469886604 shutdown_connections 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.227+0000 7fd4bb011700 1 -- 172.21.15.111:0/469886604 wait complete. 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 Processor -- start 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 -- start start 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b4113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b4114300 con 0x7fd4b4101c00 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4bb011700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b410d900 con 0x7fd4b41025a0 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4b8dad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4b8dad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34532/0 (socket says 172.21.15.111:34532) 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.228+0000 7fd4b8dad700 1 -- 172.21.15.111:0/1023211881 learned_addr learned my addr 172.21.15.111:0/1023211881 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:59.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b3fff700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b4113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b8dad700 1 -- 172.21.15.111:0/1023211881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 msgr2=0x7fd4b4113cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b8dad700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b4113cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b8dad700 1 -- 172.21.15.111:0/1023211881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8005040 con 0x7fd4b4101c00 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b3fff700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b4113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b8dad700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8000c00 tx=0x7fd4a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a800e040 con 0x7fd4b4101c00 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4a8013420 con 0x7fd4b4101c00 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b410db80 con 0x7fd4b4101c00 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a801c630 con 0x7fd4b4101c00 2024-03-14T13:27:59.222 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.229+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b410e140 con 0x7fd4b4101c00 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.230+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b40fd350 con 0x7fd4b4101c00 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.230+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd4a8013590 con 0x7fd4b4101c00 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.231+0000 7fd4b17fa700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 0x7fd4a406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.231+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd4a808ccb0 con 0x7fd4b4101c00 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.231+0000 7fd4b3fff700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 0x7fd4a406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.231+0000 7fd4b3fff700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 0x7fd4a406ec70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009fd0 tx=0x7fd4a000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.225 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.235+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd4a805aa70 con 0x7fd4b4101c00 2024-03-14T13:27:59.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.481+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 37, "format": "json"} v 0) v1 -- 0x7fd4b4108bb0 con 0x7fd4b4101c00 2024-03-14T13:27:59.472 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.482+0000 7fd4b17fa700 1 -- 172.21.15.111:0/1023211881 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 37, "format": "json"}]=0 dumped fsmap epoch 37 v42) v1 ==== 107+0+4857 (secure 0 0 0) 0x7fd4a805a600 con 0x7fd4b4101c00 2024-03-14T13:27:59.472 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:59.473 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:53.549967+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:replay","state_seq":1,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:resolve","state_seq":236,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:59.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 msgr2=0x7fd4a406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 0x7fd4a406ec70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0009fd0 tx=0x7fd4a000b040 comp rx=0 tx=0).stop 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 msgr2=0x7fd4b4113770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8000c00 tx=0x7fd4a8009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 shutdown_connections 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fd4a406c7c0 0x7fd4a406ec70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd4b4101c00 0x7fd4b4113770 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 --2- 172.21.15.111:0/1023211881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd4b41025a0 0x7fd4b4113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 >> 172.21.15.111:0/1023211881 conn(0x7fd4b40f7750 msgr2=0x7fd4b4065bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 shutdown_connections 2024-03-14T13:27:59.475 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.484+0000 7fd4bb011700 1 -- 172.21.15.111:0/1023211881 wait complete. 2024-03-14T13:27:59.476 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 37 2024-03-14T13:27:59.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 38 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 -- 172.21.15.111:0/2500421237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150101be0 msgr2=0x7f9150101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 --2- 172.21.15.111:0/2500421237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150101be0 0x7f9150101fb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9138004940 tx=0x7f9138009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 -- 172.21.15.111:0/2500421237 shutdown_connections 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 --2- 172.21.15.111:0/2500421237 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150102580 0x7f91500fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 --2- 172.21.15.111:0/2500421237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150101be0 0x7f9150101fb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 -- 172.21.15.111:0/2500421237 >> 172.21.15.111:0/2500421237 conn(0x7f9150072ad0 msgr2=0x7f9150072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 -- 172.21.15.111:0/2500421237 shutdown_connections 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.615+0000 7f9155b91700 1 -- 172.21.15.111:0/2500421237 wait complete. 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 Processor -- start 2024-03-14T13:27:59.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 -- start start 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150101be0 0x7f915010e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915010f3d0 con 0x7f9150102580 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f9155b91700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915010f510 con 0x7f9150101be0 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34548/0 (socket says 172.21.15.111:34548) 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 -- 172.21.15.111:0/2351559006 learned_addr learned my addr 172.21.15.111:0/2351559006 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914f7fe700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150101be0 0x7f915010e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 -- 172.21.15.111:0/2351559006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150101be0 msgr2=0x7f915010e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150101be0 0x7f915010e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.616+0000 7f914effd700 1 -- 172.21.15.111:0/2351559006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138005040 con 0x7f9150102580 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f914effd700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9140002a50 tx=0x7f9140004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9140007cc0 con 0x7f9150102580 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91501179c0 con 0x7f9150102580 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f914000f040 con 0x7f9150102580 2024-03-14T13:27:59.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914000d920 con 0x7f9150102580 2024-03-14T13:27:59.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.617+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9150117d90 con 0x7f9150102580 2024-03-14T13:27:59.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.618+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91500fd330 con 0x7f9150102580 2024-03-14T13:27:59.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.618+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9140009680 con 0x7f9150102580 2024-03-14T13:27:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.618+0000 7f9147fff700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 0x7f913c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.619+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f914008b7f0 con 0x7f9150102580 2024-03-14T13:27:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.619+0000 7f914f7fe700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 0x7f913c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.619+0000 7f914f7fe700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 0x7f913c06ec70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9138004f90 tx=0x7f9138009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.614 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.623+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f91400559f0 con 0x7f9150102580 2024-03-14T13:27:59.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.861+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 38, "format": "json"} v 0) v1 -- 0x7f9150108b90 con 0x7f9150102580 2024-03-14T13:27:59.852 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.862+0000 7f9147fff700 1 -- 172.21.15.111:0/2351559006 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 38, "format": "json"}]=0 dumped fsmap epoch 38 v42) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f9140059010 con 0x7f9150102580 2024-03-14T13:27:59.853 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:27:59.853 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:54.551279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:resolve","state_seq":241,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:resolve","state_seq":236,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 msgr2=0x7f913c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 0x7f913c06ec70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9138004f90 tx=0x7f9138009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 msgr2=0x7f915010ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9140002a50 tx=0x7f9140004600 comp rx=0 tx=0).stop 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 shutdown_connections 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f913c06c7c0 0x7f913c06ec70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9150101be0 0x7f915010e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 --2- 172.21.15.111:0/2351559006 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9150102580 0x7f915010ee90 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 >> 172.21.15.111:0/2351559006 conn(0x7f9150072ad0 msgr2=0x7f9150104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.855 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 shutdown_connections 2024-03-14T13:27:59.856 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.864+0000 7f9155b91700 1 -- 172.21.15.111:0/2351559006 wait complete. 2024-03-14T13:27:59.856 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 38 2024-03-14T13:27:59.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:59 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3502442109' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-14T13:27:59.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:59 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3954545058' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-14T13:27:59.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:27:59 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1023211881' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-14T13:27:59.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 39 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 -- 172.21.15.111:0/915070966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 msgr2=0x7f9ff4064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/915070966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff4064e40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4004940 tx=0x7f9fe4009e30 comp rx=0 tx=0).stop 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 -- 172.21.15.111:0/915070966 shutdown_connections 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/915070966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff4064e40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/915070966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 0x7f9ff41030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 -- 172.21.15.111:0/915070966 >> 172.21.15.111:0/915070966 conn(0x7f9ff4072ad0 msgr2=0x7f9ff4072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 -- 172.21.15.111:0/915070966 shutdown_connections 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 -- 172.21.15.111:0/915070966 wait complete. 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.994+0000 7f9ff92dc700 1 Processor -- start 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 -- start start 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 0x7f9ff40fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.987 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40fdb90 con 0x7f9ff4103680 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40fc6c0 con 0x7f9ff4102ce0 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:34570/0 (socket says 172.21.15.111:34570) 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 -- 172.21.15.111:0/1635674866 learned_addr learned my addr 172.21.15.111:0/1635674866 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff2ffd700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 0x7f9ff40fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 -- 172.21.15.111:0/1635674866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 msgr2=0x7f9ff40fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 0x7f9ff40fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 -- 172.21.15.111:0/1635674866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4005040 con 0x7f9ff4103680 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff27fc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009390 tx=0x7f9fe4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fe400e040 con 0x7f9ff4103680 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff40fc940 con 0x7f9ff4103680 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fe4013420 con 0x7f9ff4103680 2024-03-14T13:27:59.988 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9fe401c630 con 0x7f9ff4103680 2024-03-14T13:27:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.995+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff4119ee0 con 0x7f9ff4103680 2024-03-14T13:27:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.996+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40fe430 con 0x7f9ff4103680 2024-03-14T13:27:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.997+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9fe40139e0 con 0x7f9ff4103680 2024-03-14T13:27:59.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.997+0000 7f9fdb7fe700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 0x7f9fdc06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:27:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.997+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9fe408cd40 con 0x7f9ff4103680 2024-03-14T13:27:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.997+0000 7f9ff2ffd700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 0x7f9fdc06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:27:59.990 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:27:59.998+0000 7f9ff2ffd700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 0x7f9fdc06ec70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007fd0 tx=0x7f9fe8009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:27:59.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.002+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9fe4090030 con 0x7f9ff4103680 2024-03-14T13:27:59.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:59 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3502442109' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 35, "format": "json"}]: dispatch 2024-03-14T13:27:59.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:59 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3954545058' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 36, "format": "json"}]: dispatch 2024-03-14T13:27:59.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:27:59 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1023211881' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 37, "format": "json"}]: dispatch 2024-03-14T13:28:00.229 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.239+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 39, "format": "json"} v 0) v1 -- 0x7f9ff4064970 con 0x7f9ff4103680 2024-03-14T13:28:00.230 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.240+0000 7f9fdb7fe700 1 -- 172.21.15.111:0/1635674866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 39, "format": "json"}]=0 dumped fsmap epoch 39 v42) v1 ==== 107+0+4864 (secure 0 0 0) 0x7f9fe4056fc0 con 0x7f9ff4103680 2024-03-14T13:28:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:28:00.230 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:55.553081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:reconnect","state_seq":242,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:reconnect","state_seq":237,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:28:00.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 msgr2=0x7f9fdc06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 0x7f9fdc06ec70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8007fd0 tx=0x7f9fe8009040 comp rx=0 tx=0).stop 2024-03-14T13:28:00.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 msgr2=0x7f9ff40fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.233 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009390 tx=0x7f9fe4009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 shutdown_connections 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f9fdc06c7c0 0x7f9fdc06ec70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ff4102ce0 0x7f9ff40fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 --2- 172.21.15.111:0/1635674866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ff4103680 0x7f9ff40fc180 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 >> 172.21.15.111:0/1635674866 conn(0x7f9ff4072ad0 msgr2=0x7f9ff4105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 shutdown_connections 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.242+0000 7f9ff92dc700 1 -- 172.21.15.111:0/1635674866 wait complete. 2024-03-14T13:28:00.234 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 39 2024-03-14T13:28:00.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 40 2024-03-14T13:28:00.363 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- 172.21.15.111:0/918185525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640102580 msgr2=0x7ff6400fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 --2- 172.21.15.111:0/918185525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640102580 0x7ff6400fbbf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff628004940 tx=0x7ff628009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- 172.21.15.111:0/918185525 shutdown_connections 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 --2- 172.21.15.111:0/918185525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640102580 0x7ff6400fbbf0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 --2- 172.21.15.111:0/918185525 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640101be0 0x7ff640101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- 172.21.15.111:0/918185525 >> 172.21.15.111:0/918185525 conn(0x7ff640072ad0 msgr2=0x7ff640072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- 172.21.15.111:0/918185525 shutdown_connections 2024-03-14T13:28:00.364 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- 172.21.15.111:0/918185525 wait complete. 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 Processor -- start 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.373+0000 7ff645b0d700 1 -- start start 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640102580 0x7ff640112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff640113170 con 0x7ff640101be0 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64010c770 con 0x7ff640102580 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45052/0 (socket says 172.21.15.111:45052) 2024-03-14T13:28:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 -- 172.21.15.111:0/350476812 learned_addr learned my addr 172.21.15.111:0/350476812 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff637fff700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640102580 0x7ff640112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 -- 172.21.15.111:0/350476812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640102580 msgr2=0x7ff640112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640102580 0x7ff640112b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 -- 172.21.15.111:0/350476812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628005040 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63f7fe700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff63000eb40 tx=0x7ff63000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630007cc0 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff63000f040 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630009570 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64010c9f0 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.374+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64010cfb0 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.375+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6400fd330 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.376+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff6300096d0 con 0x7ff640101be0 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.376+0000 7ff63cff9700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 0x7ff62c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.376+0000 7ff637fff700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 0x7ff62c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.376+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff63008b670 con 0x7ff640101be0 2024-03-14T13:28:00.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.377+0000 7ff637fff700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 0x7ff62c06ec70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff64010de80 tx=0x7ff628009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:00.371 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.381+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff630059430 con 0x7ff640101be0 2024-03-14T13:28:00.607 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.617+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 40, "format": "json"} v 0) v1 -- 0x7ff640064970 con 0x7ff640101be0 2024-03-14T13:28:00.608 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.617+0000 7ff63cff9700 1 -- 172.21.15.111:0/350476812 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 40, "format": "json"}]=0 dumped fsmap epoch 40 v42) v1 ==== 107+0+4858 (secure 0 0 0) 0x7ff630058fc0 con 0x7ff640101be0 2024-03-14T13:28:00.608 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:28:00.608 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:56.554917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:rejoin","state_seq":243,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:rejoin","state_seq":238,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:28:00.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 msgr2=0x7ff62c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 0x7ff62c06ec70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff64010de80 tx=0x7ff628009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:00.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 msgr2=0x7ff6401125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff63000eb40 tx=0x7ff63000b040 comp rx=0 tx=0).stop 2024-03-14T13:28:00.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 shutdown_connections 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff62c06c7c0 0x7ff62c06ec70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff640101be0 0x7ff6401125e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 --2- 172.21.15.111:0/350476812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff640102580 0x7ff640112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 >> 172.21.15.111:0/350476812 conn(0x7ff640072ad0 msgr2=0x7ff640104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 shutdown_connections 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.619+0000 7ff645b0d700 1 -- 172.21.15.111:0/350476812 wait complete. 2024-03-14T13:28:00.611 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 40 2024-03-14T13:28:00.622 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 41 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 -- 172.21.15.111:0/1793189948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68102ce0 msgr2=0x7f7c681030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/1793189948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68102ce0 0x7f7c681030b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c004940 tx=0x7f7c5c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 -- 172.21.15.111:0/1793189948 shutdown_connections 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/1793189948 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68103680 0x7f7c68064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/1793189948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68102ce0 0x7f7c681030b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 -- 172.21.15.111:0/1793189948 >> 172.21.15.111:0/1793189948 conn(0x7f7c68072ad0 msgr2=0x7f7c68072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 -- 172.21.15.111:0/1793189948 shutdown_connections 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.753+0000 7f7c6ed93700 1 -- 172.21.15.111:0/1793189948 wait complete. 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 Processor -- start 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 -- start start 2024-03-14T13:28:00.746 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68102ce0 0x7f7c681137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68114340 con 0x7f7c68103680 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6ed93700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6810d940 con 0x7f7c68102ce0 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c67fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45070/0 (socket says 172.21.15.111:45070) 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c67fff700 1 -- 172.21.15.111:0/909586047 learned_addr learned my addr 172.21.15.111:0/909586047 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.754+0000 7f7c6cb2f700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68102ce0 0x7f7c681137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c67fff700 1 -- 172.21.15.111:0/909586047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68102ce0 msgr2=0x7f7c681137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c67fff700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68102ce0 0x7f7c681137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c67fff700 1 -- 172.21.15.111:0/909586047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c005040 con 0x7f7c68103680 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c67fff700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7c54002a50 tx=0x7f7c54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c54007cc0 con 0x7f7c68103680 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6810dc20 con 0x7f7c68103680 2024-03-14T13:28:00.747 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c5400f040 con 0x7f7c68103680 2024-03-14T13:28:00.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c5400d920 con 0x7f7c68103680 2024-03-14T13:28:00.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.755+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6810e1e0 con 0x7f7c68103680 2024-03-14T13:28:00.748 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.756+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c680fe430 con 0x7f7c68103680 2024-03-14T13:28:00.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.756+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7c54009680 con 0x7f7c68103680 2024-03-14T13:28:00.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.757+0000 7f7c657fa700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 0x7f7c5806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:00.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.757+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c5408b890 con 0x7f7c68103680 2024-03-14T13:28:00.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.757+0000 7f7c6cb2f700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 0x7f7c5806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:00.749 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.757+0000 7f7c6cb2f700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 0x7f7c5806ec70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00b840 tx=0x7f7c5c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:00.751 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.761+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7c540594a0 con 0x7f7c68103680 2024-03-14T13:28:00.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:00 smithi111 ceph-mon[145314]: pgmap v718: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:00.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:00 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2351559006' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-14T13:28:00.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:00 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1635674866' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-14T13:28:00.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:00 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/350476812' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-14T13:28:00.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.999+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 41, "format": "json"} v 0) v1 -- 0x7f7c68109ca0 con 0x7f7c68103680 2024-03-14T13:28:00.989 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:00.999+0000 7f7c657fa700 1 -- 172.21.15.111:0/909586047 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 41, "format": "json"}]=0 dumped fsmap epoch 41 v42) v1 ==== 107+0+4858 (secure 0 0 0) 0x7f7c54059030 con 0x7f7c68103680 2024-03-14T13:28:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:28:00.990 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":44305,"name":"cephfs.smithi137.ryucox","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6825/3411682065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3411682065},{"type":"v1","addr":"172.21.15.137:6825","nonce":3411682065}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31},{"gid":44335,"name":"cephfs.smithi137.iiqzcp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6827/2145348878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":2145348878},{"type":"v1","addr":"172.21.15.137:6827","nonce":2145348878}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-14T12:58:14.784520+0000","modified":"2024-03-14T13:25:57.560079+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":34388,"mds_1":34420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34388":{"gid":34388,"name":"cephfs.smithi111.zmslha","rank":0,"incarnation":35,"state":"up:active","state_seq":244,"addr":"172.21.15.111:6827/4083640002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4083640002},{"type":"v1","addr":"172.21.15.111:6827","nonce":4083640002}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_34420":{"gid":34420,"name":"cephfs.smithi111.yzwmcx","rank":1,"incarnation":35,"state":"up:active","state_seq":239,"addr":"172.21.15.111:6829/2220960815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":2220960815},{"type":"v1","addr":"172.21.15.111:6829","nonce":2220960815}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 msgr2=0x7f7c5806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 0x7f7c5806ec70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00b840 tx=0x7f7c5c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 msgr2=0x7f7c68113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7c54002a50 tx=0x7f7c54004600 comp rx=0 tx=0).stop 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 shutdown_connections 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f7c5806c7c0 0x7f7c5806ec70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c68102ce0 0x7f7c681137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.992 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 --2- 172.21.15.111:0/909586047 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c68103680 0x7f7c68113cf0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:00.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 >> 172.21.15.111:0/909586047 conn(0x7f7c68072ad0 msgr2=0x7f7c68105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:00.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.001+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 shutdown_connections 2024-03-14T13:28:00.993 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.002+0000 7f7c6ed93700 1 -- 172.21.15.111:0/909586047 wait complete. 2024-03-14T13:28:00.993 INFO:teuthology.orchestra.run.smithi111.stderr:dumped fsmap epoch 41 2024-03-14T13:28:00.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:00 smithi137 ceph-mon[108145]: pgmap v718: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:00.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:00 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2351559006' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 38, "format": "json"}]: dispatch 2024-03-14T13:28:00.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:00 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1635674866' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 39, "format": "json"}]: dispatch 2024-03-14T13:28:00.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:00 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/350476812' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 40, "format": "json"}]: dispatch 2024-03-14T13:28:01.004 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-14T13:28:01.016 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-14T13:28:01.017 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T13:28:01.017 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T13:28:01.037 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T13:28:01.037 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T13:28:01.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T13:28:01.255 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- 172.21.15.111:0/2607527639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c064e40 msgr2=0x7f0a9c0652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/2607527639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c064e40 0x7f0a9c0652b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c004940 tx=0x7f0a8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:01.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- 172.21.15.111:0/2607527639 shutdown_connections 2024-03-14T13:28:01.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/2607527639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c064e40 0x7f0a9c0652b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/2607527639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c102d80 0x7f0a9c103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.257 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- 172.21.15.111:0/2607527639 >> 172.21.15.111:0/2607527639 conn(0x7f0a9c071c80 msgr2=0x7f0a9c072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- 172.21.15.111:0/2607527639 shutdown_connections 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- 172.21.15.111:0/2607527639 wait complete. 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 Processor -- start 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.263+0000 7f0aa08dc700 1 -- start start 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 0x7f0a9c1137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c114380 con 0x7f0a9c102d80 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c10d980 con 0x7f0a9c064e40 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45092/0 (socket says 172.21.15.111:45092) 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 -- 172.21.15.111:0/1619646587 learned_addr learned my addr 172.21.15.111:0/1619646587 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a9a59c700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 0x7f0a9c1137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 -- 172.21.15.111:0/1619646587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 msgr2=0x7f0a9c1137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.258 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 0x7f0a9c1137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 -- 172.21.15.111:0/1619646587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c005040 con 0x7f0a9c102d80 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a9a59c700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 0x7f0a9c1137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a99d9b700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c00 tx=0x7f0a8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8c00e040 con 0x7f0a9c102d80 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a8c013420 con 0x7f0a9c102d80 2024-03-14T13:28:01.259 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c10dc00 con 0x7f0a9c102d80 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a8c01c630 con 0x7f0a9c102d80 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.264+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c10e140 con 0x7f0a9c102d80 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.265+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c0fe4e0 con 0x7f0a9c102d80 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.266+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0a8c013590 con 0x7f0a9c102d80 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.266+0000 7f0a8affd700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 0x7f0a8406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.260 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.266+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f0a8c08cd30 con 0x7f0a9c102d80 2024-03-14T13:28:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.266+0000 7f0a9a59c700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 0x7f0a8406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.266+0000 7f0a9a59c700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 0x7f0a8406ec70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fd0 tx=0x7f0a9000b040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.261 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.270+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0a8c05aaf0 con 0x7f0a9c102d80 2024-03-14T13:28:01.462 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.471+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a9c0fe4e0 con 0x7f0a9c102d80 2024-03-14T13:28:01.463 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.472+0000 7f0a8affd700 1 -- 172.21.15.111:0/1619646587 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f0a8c05a680 con 0x7f0a9c102d80 2024-03-14T13:28:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:28:01.463 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T13:10:40.893255+0000","last_up_change":"2024-03-14T13:09:42.188192+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-14T12:58:13.229267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-14T12:58:14.001600+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6803","nonce":2253644659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6805","nonce":2253644659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6809","nonce":2253644659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6807","nonce":2253644659}]},"public_addr":"172.21.15.111:6803/2253644659","cluster_addr":"172.21.15.111:6805/2253644659","heartbeat_back_addr":"172.21.15.111:6809/2253644659","heartbeat_front_addr":"172.21.15.111:6807/2253644659","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6811","nonce":2339106126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6813","nonce":2339106126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6817","nonce":2339106126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6815","nonce":2339106126}]},"public_addr":"172.21.15.111:6811/2339106126","cluster_addr":"172.21.15.111:6813/2339106126","heartbeat_back_addr":"172.21.15.111:6817/2339106126","heartbeat_front_addr":"172.21.15.111:6815/2339106126","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6819","nonce":3541679195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6821","nonce":3541679195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6825","nonce":3541679195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6823","nonce":3541679195}]},"public_addr":"172.21.15.111:6819/3541679195","cluster_addr":"172.21.15.111:6821/3541679195","heartbeat_back_addr":"172.21.15.111:6825/3541679195","heartbeat_front_addr":"172.21.15.111:6823/3541679195","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6801","nonce":137240476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6803","nonce":137240476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6807","nonce":137240476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6805","nonce":137240476}]},"public_addr":"172.21.15.137:6801/137240476","cluster_addr":"172.21.15.137:6803/137240476","heartbeat_back_addr":"172.21.15.137:6807/137240476","heartbeat_front_addr":"172.21.15.137:6805/137240476","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6809","nonce":2798642674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6811","nonce":2798642674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6815","nonce":2798642674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6813","nonce":2798642674}]},"public_addr":"172.21.15.137:6809/2798642674","cluster_addr":"172.21.15.137:6811/2798642674","heartbeat_back_addr":"172.21.15.137:6815/2798642674","heartbeat_front_addr":"172.21.15.137:6813/2798642674","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6817","nonce":4240986345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6819","nonce":4240986345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6823","nonce":4240986345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6821","nonce":4240986345}]},"public_addr":"172.21.15.137:6817/4240986345","cluster_addr":"172.21.15.137:6819/4240986345","heartbeat_back_addr":"172.21.15.137:6823/4240986345","heartbeat_front_addr":"172.21.15.137:6821/4240986345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6824/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3552305380":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/1484389211":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.137:0/2592190224":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/3277050730":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/2394623028":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/661526538":"2024-03-15T13:00:23.200618+0000","172.21.15.137:0/1152636518":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/3904007449":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/139754802":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/1611930537":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/4199526358":"2024-03-15T13:00:23.200618+0000","172.21.15.137:6825/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:0/1118840918":"2024-03-15T13:05:24.191564+0000","172.21.15.137:6828/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.137:6829/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/2868306717":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3646747677":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6827/1337358162":"2024-03-15T13:09:44.383031+0000","172.21.15.111:6800/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6800/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6826/1337358162":"2024-03-15T13:09:44.383031+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T13:28:01.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.474+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 msgr2=0x7f0a8406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.465 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.474+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 0x7f0a8406ec70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009fd0 tx=0x7f0a9000b040 comp rx=0 tx=0).stop 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.474+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 msgr2=0x7f0a9c113d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.474+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c00 tx=0x7f0a8c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 shutdown_connections 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f0a8406c7c0 0x7f0a8406ec70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a9c064e40 0x7f0a9c1137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 --2- 172.21.15.111:0/1619646587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a9c102d80 0x7f0a9c113d30 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 >> 172.21.15.111:0/1619646587 conn(0x7f0a9c071c80 msgr2=0x7f0a9c0fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 shutdown_connections 2024-03-14T13:28:01.466 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.475+0000 7f0aa08dc700 1 -- 172.21.15.111:0/1619646587 wait complete. 2024-03-14T13:28:01.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T12:55:33.503741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-14T12:58:13.229267+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-14T12:58:14.001600+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T13:28:01.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T13:28:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.607+0000 7ff4f65bf700 1 -- 172.21.15.111:0/99534737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 msgr2=0x7ff4f00fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.607+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/99534737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f00fbbe0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0004940 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:01.600 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- 172.21.15.111:0/99534737 shutdown_connections 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/99534737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f00fbbe0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/99534737 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 0x7ff4f0101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- 172.21.15.111:0/99534737 >> 172.21.15.111:0/99534737 conn(0x7ff4f00f7740 msgr2=0x7ff4f00f9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- 172.21.15.111:0/99534737 shutdown_connections 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- 172.21.15.111:0/99534737 wait complete. 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 Processor -- start 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- start start 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 0x7ff4f010e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f010f370 con 0x7ff4f0102570 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.608+0000 7ff4f65bf700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f010f4b0 con 0x7ff4f0101bd0 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.601 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45098/0 (socket says 172.21.15.111:45098) 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 -- 172.21.15.111:0/1420417412 learned_addr learned my addr 172.21.15.111:0/1420417412 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4effff700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 0x7ff4f010e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 -- 172.21.15.111:0/1420417412 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 msgr2=0x7ff4f010e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 0x7ff4f010e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 -- 172.21.15.111:0/1420417412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e0005040 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ef7fe700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009380 tx=0x7ff4e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e000e040 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f010f730 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4e000bc20 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4e001c3f0 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.609+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f0117d60 con 0x7ff4f0102570 2024-03-14T13:28:01.602 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.610+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f00fd320 con 0x7ff4f0102570 2024-03-14T13:28:01.603 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.611+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff4e000bd90 con 0x7ff4f0102570 2024-03-14T13:28:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.611+0000 7ff4ecff9700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 0x7ff4d806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.611+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff4e008cdf0 con 0x7ff4f0102570 2024-03-14T13:28:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.611+0000 7ff4effff700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 0x7ff4d806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.604 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.611+0000 7ff4effff700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 0x7ff4d806ec70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.606 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.615+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff4e0057070 con 0x7ff4f0102570 2024-03-14T13:28:01.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.809+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4f0108be0 con 0x7ff4f0102570 2024-03-14T13:28:01.800 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.809+0000 7ff4ecff9700 1 -- 172.21.15.111:0/1420417412 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff4e005a690 con 0x7ff4f0102570 2024-03-14T13:28:01.800 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-03-14T13:28:01.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 msgr2=0x7ff4d806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 0x7ff4d806ec70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).stop 2024-03-14T13:28:01.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 msgr2=0x7ff4f010ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff4e0009380 tx=0x7ff4e0009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:01.802 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 shutdown_connections 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7ff4d806c7c0 0x7ff4d806ec70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4f0101bd0 0x7ff4f010e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 --2- 172.21.15.111:0/1420417412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4f0102570 0x7ff4f010ee30 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 >> 172.21.15.111:0/1420417412 conn(0x7ff4f00f7740 msgr2=0x7ff4f00f8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.811+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 shutdown_connections 2024-03-14T13:28:01.803 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.812+0000 7ff4f65bf700 1 -- 172.21.15.111:0/1420417412 wait complete. 2024-03-14T13:28:01.812 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:01 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/909586047' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-14T13:28:01.813 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:01 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1619646587' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:28:01.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 -- 172.21.15.111:0/245280778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850101be0 msgr2=0x7f4850101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 --2- 172.21.15.111:0/245280778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850101be0 0x7f4850101fb0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4840004940 tx=0x7f4840009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 -- 172.21.15.111:0/245280778 shutdown_connections 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 --2- 172.21.15.111:0/245280778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850102580 0x7f48500fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 --2- 172.21.15.111:0/245280778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850101be0 0x7f4850101fb0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 -- 172.21.15.111:0/245280778 >> 172.21.15.111:0/245280778 conn(0x7f4850072ad0 msgr2=0x7f4850072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 -- 172.21.15.111:0/245280778 shutdown_connections 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 -- 172.21.15.111:0/245280778 wait complete. 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.944+0000 7f48572d2700 1 Processor -- start 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 -- start start 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850101be0 0x7f485010e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.937 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485010f3f0 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485010f530 con 0x7f4850101be0 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45110/0 (socket says 172.21.15.111:45110) 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 -- 172.21.15.111:0/1379541406 learned_addr learned my addr 172.21.15.111:0/1379541406 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485506e700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850101be0 0x7f485010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 -- 172.21.15.111:0/1379541406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850101be0 msgr2=0x7f485010e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850101be0 0x7f485010e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 -- 172.21.15.111:0/1379541406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4840005040 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f485486d700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f484c002a50 tx=0x7f484c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484c007cc0 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f484c00f040 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48501179e0 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.945+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484c00d990 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.946+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4850117db0 con 0x7f4850102580 2024-03-14T13:28:01.938 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.946+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48500fd330 con 0x7f4850102580 2024-03-14T13:28:01.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.947+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f484c009710 con 0x7f4850102580 2024-03-14T13:28:01.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.947+0000 7f4845ffb700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 0x7f483c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:01.939 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.947+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f484c08b890 con 0x7f4850102580 2024-03-14T13:28:01.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.947+0000 7f485506e700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 0x7f483c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:01.940 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.948+0000 7f485506e700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 0x7f483c06ec70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4840004f90 tx=0x7f4840009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:01.942 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:01.952+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f484c055a90 con 0x7f4850102580 2024-03-14T13:28:01.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:01 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/909586047' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 41, "format": "json"}]: dispatch 2024-03-14T13:28:01.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:01 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1619646587' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:28:02.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.147+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4850064970 con 0x7f4850102580 2024-03-14T13:28:02.138 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.148+0000 7f4845ffb700 1 -- 172.21.15.111:0/1379541406 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f484c0590b0 con 0x7f4850102580 2024-03-14T13:28:02.138 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:28:02.140 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 msgr2=0x7f483c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 0x7f483c06ec70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4840004f90 tx=0x7f4840009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 msgr2=0x7f485010eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f484c002a50 tx=0x7f484c004600 comp rx=0 tx=0).stop 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 shutdown_connections 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f483c06c7c0 0x7f483c06ec70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4850101be0 0x7f485010e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 --2- 172.21.15.111:0/1379541406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4850102580 0x7f485010eeb0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 >> 172.21.15.111:0/1379541406 conn(0x7f4850072ad0 msgr2=0x7f4850104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 shutdown_connections 2024-03-14T13:28:02.141 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.150+0000 7f48572d2700 1 -- 172.21.15.111:0/1379541406 wait complete. 2024-03-14T13:28:02.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- 172.21.15.111:0/2187596752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258102ce0 msgr2=0x7f32581030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 --2- 172.21.15.111:0/2187596752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258102ce0 0x7f32581030b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f324c004940 tx=0x7f324c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- 172.21.15.111:0/2187596752 shutdown_connections 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 --2- 172.21.15.111:0/2187596752 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258103680 0x7f3258064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 --2- 172.21.15.111:0/2187596752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258102ce0 0x7f32581030b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- 172.21.15.111:0/2187596752 >> 172.21.15.111:0/2187596752 conn(0x7f3258072ad0 msgr2=0x7f3258072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:02.275 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- 172.21.15.111:0/2187596752 shutdown_connections 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- 172.21.15.111:0/2187596752 wait complete. 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 Processor -- start 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.282+0000 7f325c8f4700 1 -- start start 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 0x7f32581137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258114340 con 0x7f3258103680 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325810d940 con 0x7f3258102ce0 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45122/0 (socket says 172.21.15.111:45122) 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 -- 172.21.15.111:0/726989673 learned_addr learned my addr 172.21.15.111:0/726989673 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325659c700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 0x7f32581137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 -- 172.21.15.111:0/726989673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 msgr2=0x7f32581137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 0x7f32581137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 -- 172.21.15.111:0/726989673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c005040 con 0x7f3258103680 2024-03-14T13:28:02.276 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325659c700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 0x7f32581137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3255d9b700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3248002a50 tx=0x7f3248004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3248007cc0 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f324800f040 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324800d9a0 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325810dc20 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.283+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325810e1e0 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.284+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32580fe430 con 0x7f3258103680 2024-03-14T13:28:02.277 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.285+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f324800dc10 con 0x7f3258103680 2024-03-14T13:28:02.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.285+0000 7f3246ffd700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 0x7f324006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.285+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f324808b7f0 con 0x7f3258103680 2024-03-14T13:28:02.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.285+0000 7f325659c700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 0x7f324006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.278 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.286+0000 7f325659c700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 0x7f324006ec70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f324c00b840 tx=0x7f324c009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:02.280 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.290+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3248059500 con 0x7f3258103680 2024-03-14T13:28:02.473 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.483+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f3258109ca0 con 0x7f3258103680 2024-03-14T13:28:02.474 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.483+0000 7f3246ffd700 1 -- 172.21.15.111:0/726989673 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7f3248059090 con 0x7f3258103680 2024-03-14T13:28:02.474 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 msgr2=0x7f324006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 0x7f324006ec70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f324c00b840 tx=0x7f324c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 msgr2=0x7f3258113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3248002a50 tx=0x7f3248004600 comp rx=0 tx=0).stop 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 shutdown_connections 2024-03-14T13:28:02.476 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f324006c7c0 0x7f324006ec70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3258102ce0 0x7f32581137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 --2- 172.21.15.111:0/726989673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3258103680 0x7f3258113cf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 >> 172.21.15.111:0/726989673 conn(0x7f3258072ad0 msgr2=0x7f3258105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 shutdown_connections 2024-03-14T13:28:02.477 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.485+0000 7f325c8f4700 1 -- 172.21.15.111:0/726989673 wait complete. 2024-03-14T13:28:02.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-14T13:28:02.609 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 -- 172.21.15.111:0/737400723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34102580 msgr2=0x7faa340fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/737400723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34102580 0x7faa340fbbf0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7faa20004940 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 -- 172.21.15.111:0/737400723 shutdown_connections 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/737400723 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34102580 0x7faa340fbbf0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/737400723 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34101be0 0x7faa34101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 -- 172.21.15.111:0/737400723 >> 172.21.15.111:0/737400723 conn(0x7faa34072ad0 msgr2=0x7faa34072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 -- 172.21.15.111:0/737400723 shutdown_connections 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 -- 172.21.15.111:0/737400723 wait complete. 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.617+0000 7faa3b1b3700 1 Processor -- start 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa3b1b3700 1 -- start start 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa3b1b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.610 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa3b1b3700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34102580 0x7faa34112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa3b1b3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa34113230 con 0x7faa34101be0 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa3b1b3700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3410c7f0 con 0x7faa34102580 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45142/0 (socket says 172.21.15.111:45142) 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 -- 172.21.15.111:0/3339735686 learned_addr learned my addr 172.21.15.111:0/3339735686 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa33fff700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34102580 0x7faa34112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 -- 172.21.15.111:0/3339735686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34102580 msgr2=0x7faa34112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34102580 0x7faa34112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 -- 172.21.15.111:0/3339735686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa20005040 con 0x7faa34101be0 2024-03-14T13:28:02.611 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa38f4f700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c830 tx=0x7faa28004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.618+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa280106f0 con 0x7faa34101be0 2024-03-14T13:28:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.619+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa28010d00 con 0x7faa34101be0 2024-03-14T13:28:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.619+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa28012e90 con 0x7faa34101be0 2024-03-14T13:28:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.619+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3410cad0 con 0x7faa34101be0 2024-03-14T13:28:02.612 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.619+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3410cfe0 con 0x7faa34101be0 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.620+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa34112350 con 0x7faa34101be0 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.620+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7faa28010850 con 0x7faa34101be0 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.620+0000 7faa317fa700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 0x7faa2406ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.620+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7faa2808bac0 con 0x7faa34101be0 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.620+0000 7faa33fff700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 0x7faa2406ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:02.613 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.621+0000 7faa33fff700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 0x7faa2406ec70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa20004890 tx=0x7faa20009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:02.615 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.625+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7faa280597d0 con 0x7faa34101be0 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.833+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faa340fd330 con 0x7faa34101be0 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.833+0000 7faa317fa700 1 -- 172.21.15.111:0/3339735686 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2341 (secure 0 0 0) 0x7faa28059360 con 0x7faa34101be0 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6824/3856946079 2024-03-15T13:09:44.383047+0000 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/4041156471 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3552305380 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/337686092 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/1542503054 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/4041156471 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3123411794 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1484389211 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3030479424 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/337686092 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/2592190224 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/3277050730 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/975462484 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/2394623028 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/661526538 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/1152636518 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3904007449 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3170679579 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/119405983 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/139754802 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/1611930537 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1344961412 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1979966098 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/718360470 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/4199526358 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6825/3856946079 2024-03-15T13:09:44.383047+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1118840918 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6828/1920623081 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6829/1920623081 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/119405983 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/2868306717 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3646747677 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6827/1337358162 2024-03-15T13:09:44.383031+0000 2024-03-14T13:28:02.826 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/3203466385 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/1542503054 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3756036511 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3365856044 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/3203466385 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6826/1337358162 2024-03-15T13:09:44.383031+0000 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 msgr2=0x7faa2406ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 0x7faa2406ec70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa20004890 tx=0x7faa20009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 msgr2=0x7faa341126a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c830 tx=0x7faa28004470 comp rx=0 tx=0).stop 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 shutdown_connections 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7faa2406c7c0 0x7faa2406ec70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa34101be0 0x7faa341126a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 --2- 172.21.15.111:0/3339735686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa34102580 0x7faa34112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:02.828 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 >> 172.21.15.111:0/3339735686 conn(0x7faa34072ad0 msgr2=0x7faa34104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:02.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 shutdown_connections 2024-03-14T13:28:02.829 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:02.836+0000 7faa3b1b3700 1 -- 172.21.15.111:0/3339735686 wait complete. 2024-03-14T13:28:02.829 INFO:teuthology.orchestra.run.smithi111.stderr:listed 39 entries 2024-03-14T13:28:02.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:02 smithi111 ceph-mon[145314]: pgmap v719: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:02.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:02 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1420417412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:28:02.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:02 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1379541406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:28:02.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:02 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/726989673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:28:02.838 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-03-14T13:28:02.838 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-14T13:28:02.859 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-14T13:28:02.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:02 smithi137 ceph-mon[108145]: pgmap v719: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:02.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:02 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1420417412' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:28:02.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:02 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1379541406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:28:02.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:02 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/726989673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:28:03.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- 172.21.15.111:0/2313099183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 msgr2=0x7f34ac106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.020 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 --2- 172.21.15.111:0/2313099183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac106cd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f3498004940 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- 172.21.15.111:0/2313099183 shutdown_connections 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 --2- 172.21.15.111:0/2313099183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac106cd0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 --2- 172.21.15.111:0/2313099183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 0x7f34ac0fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- 172.21.15.111:0/2313099183 >> 172.21.15.111:0/2313099183 conn(0x7f34ac0f77a0 msgr2=0x7f34ac0f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- 172.21.15.111:0/2313099183 shutdown_connections 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- 172.21.15.111:0/2313099183 wait complete. 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 Processor -- start 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.028+0000 7f34b2b77700 1 -- start start 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 0x7f34ac113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac114310 con 0x7f34ac0fe690 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac10d910 con 0x7f34ac0fdcf0 2024-03-14T13:28:03.021 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45160/0 (socket says 172.21.15.111:45160) 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 -- 172.21.15.111:0/4141577866 learned_addr learned my addr 172.21.15.111:0/4141577866 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b0913700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 0x7f34ac113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 -- 172.21.15.111:0/4141577866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 msgr2=0x7f34ac113780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 0x7f34ac113780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 -- 172.21.15.111:0/4141577866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005040 con 0x7f34ac0fe690 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34abfff700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f349800b3b0 tx=0x7f3498009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f349800e040 con 0x7f34ac0fe690 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac10db90 con 0x7f34ac0fe690 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3498013420 con 0x7f34ac0fe690 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f349801c630 con 0x7f34ac0fe690 2024-03-14T13:28:03.022 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.029+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac10e0d0 con 0x7f34ac0fe690 2024-03-14T13:28:03.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.031+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac066a90 con 0x7f34ac0fe690 2024-03-14T13:28:03.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.031+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3498013590 con 0x7f34ac0fe690 2024-03-14T13:28:03.023 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.031+0000 7f34a97fa700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 0x7f349c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.032+0000 7f34b0913700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 0x7f349c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.032+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f349808cd40 con 0x7f34ac0fe690 2024-03-14T13:28:03.024 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.032+0000 7f34b0913700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 0x7f349c06ec70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f34a0007fd0 tx=0x7f34a0009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.026 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.036+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f349805a950 con 0x7f34ac0fe690 2024-03-14T13:28:03.220 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.229+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34ac064dc0 con 0x7f34ac0fe690 2024-03-14T13:28:03.221 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.231+0000 7f34a97fa700 1 -- 172.21.15.111:0/4141577866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f349805a4e0 con 0x7f34ac0fe690 2024-03-14T13:28:03.221 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-03-14T13:28:03.222 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":86,"fsid":"591fecbc-e201-11ee-95c9-87774f69a715","created":"2024-03-14T12:50:39.965050+0000","modified":"2024-03-14T13:10:40.893255+0000","last_up_change":"2024-03-14T13:09:42.188192+0000","last_in_change":"2024-03-14T12:56:31.266856+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-14T12:55:33.503741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-14T12:58:13.229267+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-14T12:58:14.001600+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"726491ca-c4bb-4580-8256-3f3d0890e6d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":50,"up_from":54,"up_thru":79,"down_at":51,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6803","nonce":2253644659}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6805","nonce":2253644659}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6809","nonce":2253644659}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2253644659},{"type":"v1","addr":"172.21.15.111:6807","nonce":2253644659}]},"public_addr":"172.21.15.111:6803/2253644659","cluster_addr":"172.21.15.111:6805/2253644659","heartbeat_back_addr":"172.21.15.111:6809/2253644659","heartbeat_front_addr":"172.21.15.111:6807/2253644659","state":["exists","up"]},{"osd":1,"uuid":"83c5fd1b-e903-43f7-b441-c4427d6ffb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":55,"up_from":59,"up_thru":79,"down_at":56,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6811","nonce":2339106126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6813","nonce":2339106126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6817","nonce":2339106126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":2339106126},{"type":"v1","addr":"172.21.15.111:6815","nonce":2339106126}]},"public_addr":"172.21.15.111:6811/2339106126","cluster_addr":"172.21.15.111:6813/2339106126","heartbeat_back_addr":"172.21.15.111:6817/2339106126","heartbeat_front_addr":"172.21.15.111:6815/2339106126","state":["exists","up"]},{"osd":2,"uuid":"d0fe3aa1-1501-4a28-b5ab-2b680418a5ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":60,"up_from":64,"up_thru":79,"down_at":61,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6819","nonce":3541679195}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6821","nonce":3541679195}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6825","nonce":3541679195}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3541679195},{"type":"v1","addr":"172.21.15.111:6823","nonce":3541679195}]},"public_addr":"172.21.15.111:6819/3541679195","cluster_addr":"172.21.15.111:6821/3541679195","heartbeat_back_addr":"172.21.15.111:6825/3541679195","heartbeat_front_addr":"172.21.15.111:6823/3541679195","state":["exists","up"]},{"osd":3,"uuid":"f6c3f447-d2db-47da-9099-57ca0eff4075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":27,"last_clean_end":65,"up_from":69,"up_thru":79,"down_at":66,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6801","nonce":137240476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6803","nonce":137240476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6807","nonce":137240476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":137240476},{"type":"v1","addr":"172.21.15.137:6805","nonce":137240476}]},"public_addr":"172.21.15.137:6801/137240476","cluster_addr":"172.21.15.137:6803/137240476","heartbeat_back_addr":"172.21.15.137:6807/137240476","heartbeat_front_addr":"172.21.15.137:6805/137240476","state":["exists","up"]},{"osd":4,"uuid":"c1318352-47b7-4cba-8ea1-0b06d8e9843c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":35,"last_clean_end":70,"up_from":74,"up_thru":79,"down_at":71,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6809","nonce":2798642674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6811","nonce":2798642674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6815","nonce":2798642674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":2798642674},{"type":"v1","addr":"172.21.15.137:6813","nonce":2798642674}]},"public_addr":"172.21.15.137:6809/2798642674","cluster_addr":"172.21.15.137:6811/2798642674","heartbeat_back_addr":"172.21.15.137:6815/2798642674","heartbeat_front_addr":"172.21.15.137:6813/2798642674","state":["exists","up"]},{"osd":5,"uuid":"1323bb35-1d44-4189-8bbf-c192740aaa58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":75,"up_from":79,"up_thru":79,"down_at":76,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6817","nonce":4240986345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6819","nonce":4240986345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6823","nonce":4240986345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":4240986345},{"type":"v1","addr":"172.21.15.137:6821","nonce":4240986345}]},"public_addr":"172.21.15.137:6817/4240986345","cluster_addr":"172.21.15.137:6819/4240986345","heartbeat_back_addr":"172.21.15.137:6823/4240986345","heartbeat_front_addr":"172.21.15.137:6821/4240986345","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:54:31.850752+0000","dead_epoch":50},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:01.077965+0000","dead_epoch":55},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:30.671088+0000","dead_epoch":60},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:55:55.807786+0000","dead_epoch":65},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:22.140861+0000","dead_epoch":70},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-14T12:56:48.749134+0000","dead_epoch":75}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6824/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:6801/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3552305380":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6801/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.111:6801/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:6800/4041156471":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/3123411794":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/1484389211":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3030479424":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6800/337686092":"2024-03-15T12:51:06.184731+0000","172.21.15.137:0/2592190224":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/3277050730":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/975462484":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/2394623028":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/661526538":"2024-03-15T13:00:23.200618+0000","172.21.15.137:0/1152636518":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/3904007449":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3170679579":"2024-03-15T12:51:38.987669+0000","172.21.15.111:6801/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.137:0/139754802":"2024-03-15T13:01:36.643440+0000","172.21.15.137:0/1611930537":"2024-03-15T13:01:36.643440+0000","172.21.15.111:0/1344961412":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/1979966098":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/718360470":"2024-03-15T12:53:13.176472+0000","172.21.15.111:0/4199526358":"2024-03-15T13:00:23.200618+0000","172.21.15.137:6825/3856946079":"2024-03-15T13:09:44.383047+0000","172.21.15.111:0/1118840918":"2024-03-15T13:05:24.191564+0000","172.21.15.137:6828/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.137:6829/1920623081":"2024-03-15T13:01:36.643440+0000","172.21.15.111:6800/119405983":"2024-03-15T12:51:38.987669+0000","172.21.15.111:0/2868306717":"2024-03-15T13:05:24.191564+0000","172.21.15.111:0/3646747677":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6827/1337358162":"2024-03-15T13:09:44.383031+0000","172.21.15.111:6800/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6800/1542503054":"2024-03-15T13:00:23.200618+0000","172.21.15.111:0/3756036511":"2024-03-15T12:51:06.184731+0000","172.21.15.111:0/3365856044":"2024-03-15T12:53:13.176472+0000","172.21.15.111:6801/3203466385":"2024-03-15T13:05:24.191564+0000","172.21.15.111:6826/1337358162":"2024-03-15T13:09:44.383031+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-14T13:28:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 msgr2=0x7f349c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.223 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 0x7f349c06ec70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f34a0007fd0 tx=0x7f34a0009040 comp rx=0 tx=0).stop 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 msgr2=0x7f34ac113cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f349800b3b0 tx=0x7f3498009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 shutdown_connections 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f349c06c7c0 0x7f349c06ec70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac0fdcf0 0x7f34ac113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 --2- 172.21.15.111:0/4141577866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34ac0fe690 0x7f34ac113cc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 >> 172.21.15.111:0/4141577866 conn(0x7f34ac0f77a0 msgr2=0x7f34ac0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 shutdown_connections 2024-03-14T13:28:03.224 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.233+0000 7f34b2b77700 1 -- 172.21.15.111:0/4141577866 wait complete. 2024-03-14T13:28:03.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-14T12:55:33.503741+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-14T12:58:13.229267+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-14T12:58:14.001600+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-14T13:28:03.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 -- 172.21.15.111:0/572217374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 msgr2=0x7f8220064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 --2- 172.21.15.111:0/572217374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220064e40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8214004940 tx=0x7f8214009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 -- 172.21.15.111:0/572217374 shutdown_connections 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 --2- 172.21.15.111:0/572217374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220064e40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 --2- 172.21.15.111:0/572217374 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 0x7f82201030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.366+0000 7f82249dd700 1 -- 172.21.15.111:0/572217374 >> 172.21.15.111:0/572217374 conn(0x7f8220072ad0 msgr2=0x7f8220072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 -- 172.21.15.111:0/572217374 shutdown_connections 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 -- 172.21.15.111:0/572217374 wait complete. 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 Processor -- start 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 -- start start 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 0x7f8220113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.359 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220114320 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f82249dd700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822010d920 con 0x7f8220102ce0 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f821dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f821dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45178/0 (socket says 172.21.15.111:45178) 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.367+0000 7f821dd9b700 1 -- 172.21.15.111:0/3582843088 learned_addr learned my addr 172.21.15.111:0/3582843088 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f821e59c700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 0x7f8220113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f821dd9b700 1 -- 172.21.15.111:0/3582843088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 msgr2=0x7f8220113790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f821dd9b700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 0x7f8220113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f821dd9b700 1 -- 172.21.15.111:0/3582843088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214005040 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f821dd9b700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82140093b0 tx=0x7f8214009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821400e040 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822010dba0 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8214013420 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f821401c630 con 0x7f8220103680 2024-03-14T13:28:03.360 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.368+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822010e160 con 0x7f8220103680 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.369+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82200fe430 con 0x7f8220103680 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.369+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f82140139e0 con 0x7f8220103680 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.370+0000 7f820effd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 0x7f820806ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.370+0000 7f821e59c700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 0x7f820806ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.370+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f821408ccc0 con 0x7f8220103680 2024-03-14T13:28:03.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.370+0000 7f821e59c700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 0x7f820806ec70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82100041d0 tx=0x7f8210009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.365 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.374+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f821405a9d0 con 0x7f8220103680 2024-03-14T13:28:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.571+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8220064970 con 0x7f8220103680 2024-03-14T13:28:03.562 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.572+0000 7f820effd700 1 -- 172.21.15.111:0/3582843088 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f821405a560 con 0x7f8220103680 2024-03-14T13:28:03.562 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-03-14T13:28:03.564 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 msgr2=0x7f820806ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 0x7f820806ec70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82100041d0 tx=0x7f8210009040 comp rx=0 tx=0).stop 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 msgr2=0x7f8220113cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82140093b0 tx=0x7f8214009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 shutdown_connections 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f820806c7c0 0x7f820806ec70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8220102ce0 0x7f8220113790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 --2- 172.21.15.111:0/3582843088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8220103680 0x7f8220113cd0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 >> 172.21.15.111:0/3582843088 conn(0x7f8220072ad0 msgr2=0x7f8220105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 shutdown_connections 2024-03-14T13:28:03.565 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.574+0000 7f82249dd700 1 -- 172.21.15.111:0/3582843088 wait complete. 2024-03-14T13:28:03.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-14T13:28:03.699 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1576382346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 msgr2=0x7fc2c00fbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1576382346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c00fbc00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc2b0004940 tx=0x7fc2b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1576382346 shutdown_connections 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1576382346 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c00fbc00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1576382346 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 0x7fc2c0101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1576382346 >> 172.21.15.111:0/1576382346 conn(0x7fc2c00f7720 msgr2=0x7fc2c00f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1576382346 shutdown_connections 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.707+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1576382346 wait complete. 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 Processor -- start 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 -- start start 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 0x7fc2c010e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.700 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c010f3c0 con 0x7fc2c0102590 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c7d6f700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c010f500 con 0x7fc2c0101bf0 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45194/0 (socket says 172.21.15.111:45194) 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 -- 172.21.15.111:0/1875650530 learned_addr learned my addr 172.21.15.111:0/1875650530 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 -- 172.21.15.111:0/1875650530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 msgr2=0x7fc2c010e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c5b0b700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 0x7fc2c010e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 0x7fc2c010e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.708+0000 7fc2c530a700 1 -- 172.21.15.111:0/1875650530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0005040 con 0x7fc2c0102590 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2c530a700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00093b0 tx=0x7fc2b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2b000e040 con 0x7fc2c0102590 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2b0013420 con 0x7fc2c0102590 2024-03-14T13:28:03.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c010f780 con 0x7fc2c0102590 2024-03-14T13:28:03.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2b001c630 con 0x7fc2c0102590 2024-03-14T13:28:03.702 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.709+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c0117d00 con 0x7fc2c0102590 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.710+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c00fd340 con 0x7fc2c0102590 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.710+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc2b00139e0 con 0x7fc2c0102590 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.710+0000 7fc2b67fc700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 0x7fc2ac06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.711+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fc2b008cce0 con 0x7fc2c0102590 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.711+0000 7fc2c5b0b700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 0x7fc2ac06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:03.703 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.711+0000 7fc2c5b0b700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 0x7fc2ac06ec70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc004c40 tx=0x7fc2bc009040 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:03.705 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.715+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc2b0091070 con 0x7fc2c0102590 2024-03-14T13:28:03.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:03 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3339735686' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T13:28:03.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:03 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/4141577866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:28:03.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:03 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/3582843088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:28:03.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.909+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc2c0108b80 con 0x7fc2c0102590 2024-03-14T13:28:03.900 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.910+0000 7fc2b67fc700 1 -- 172.21.15.111:0/1875650530 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc2b0057010 con 0x7fc2c0102590 2024-03-14T13:28:03.900 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:28:03.902 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 msgr2=0x7fc2ac06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 0x7fc2ac06ec70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc004c40 tx=0x7fc2bc009040 comp rx=0 tx=0).stop 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 msgr2=0x7fc2c010ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00093b0 tx=0x7fc2b0009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 shutdown_connections 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fc2ac06c7c0 0x7fc2ac06ec70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c0101bf0 0x7fc2c010e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 --2- 172.21.15.111:0/1875650530 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2c0102590 0x7fc2c010ee80 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 >> 172.21.15.111:0/1875650530 conn(0x7fc2c00f7720 msgr2=0x7fc2c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 shutdown_connections 2024-03-14T13:28:03.903 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:03.912+0000 7fc2c7d6f700 1 -- 172.21.15.111:0/1875650530 wait complete. 2024-03-14T13:28:03.915 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-14T13:28:03.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:03 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3339735686' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T13:28:03.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:03 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/4141577866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-14T13:28:03.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:03 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/3582843088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 -- 172.21.15.111:0/2563352469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20102ce0 msgr2=0x7fab201030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 --2- 172.21.15.111:0/2563352469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20102ce0 0x7fab201030b0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fab10004940 tx=0x7fab10009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 -- 172.21.15.111:0/2563352469 shutdown_connections 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 --2- 172.21.15.111:0/2563352469 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20103680 0x7fab20064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 --2- 172.21.15.111:0/2563352469 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20102ce0 0x7fab201030b0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.036 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 -- 172.21.15.111:0/2563352469 >> 172.21.15.111:0/2563352469 conn(0x7fab20072ad0 msgr2=0x7fab20072ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:04.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 -- 172.21.15.111:0/2563352469 shutdown_connections 2024-03-14T13:28:04.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.045+0000 7fab2771c700 1 -- 172.21.15.111:0/2563352469 wait complete. 2024-03-14T13:28:04.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 Processor -- start 2024-03-14T13:28:04.037 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 -- start start 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 0x7fab201137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab20114340 con 0x7fab20103680 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab2771c700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2010d940 con 0x7fab20102ce0 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45210/0 (socket says 172.21.15.111:45210) 2024-03-14T13:28:04.039 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 -- 172.21.15.111:0/1489113451 learned_addr learned my addr 172.21.15.111:0/1489113451 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab254b8700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 0x7fab201137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 -- 172.21.15.111:0/1489113451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 msgr2=0x7fab201137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 0x7fab201137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab24cb7700 1 -- 172.21.15.111:0/1489113451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab10005040 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.046+0000 7fab254b8700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 0x7fab201137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab24cb7700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00c800 tx=0x7fab1c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1c00a760 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab1c00ad70 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2010dc20 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1c0113a0 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.047+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2010e1e0 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.048+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab200fe430 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.048+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fab1c012400 con 0x7fab20103680 2024-03-14T13:28:04.040 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.048+0000 7fab15ffb700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 0x7fab0c06ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.049+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fab1c08bad0 con 0x7fab20103680 2024-03-14T13:28:04.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.049+0000 7fab254b8700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 0x7fab0c06ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:04.041 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.049+0000 7fab254b8700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 0x7fab0c06ec70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fab1000b3b0 tx=0x7fab10009e30 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:04.043 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.053+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fab1c0597a0 con 0x7fab20103680 2024-03-14T13:28:04.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.245+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fab20109ca0 con 0x7fab20103680 2024-03-14T13:28:04.235 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.245+0000 7fab15ffb700 1 -- 172.21.15.111:0/1489113451 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v86) v1 ==== 107+0+11 (secure 0 0 0) 0x7fab1c059330 con 0x7fab20103680 2024-03-14T13:28:04.236 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 32 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 msgr2=0x7fab0c06ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 0x7fab0c06ec70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fab1000b3b0 tx=0x7fab10009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 msgr2=0x7fab20113cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00c800 tx=0x7fab1c004470 comp rx=0 tx=0).stop 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 shutdown_connections 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7fab0c06c7c0 0x7fab0c06ec70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fab20102ce0 0x7fab201137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 --2- 172.21.15.111:0/1489113451 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab20103680 0x7fab20113cf0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 >> 172.21.15.111:0/1489113451 conn(0x7fab20072ad0 msgr2=0x7fab20105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 shutdown_connections 2024-03-14T13:28:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.247+0000 7fab2771c700 1 -- 172.21.15.111:0/1489113451 wait complete. 2024-03-14T13:28:04.250 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-14T13:28:04.369 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2850607389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 msgr2=0x7f6f88101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2850607389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88101fe0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c004940 tx=0x7f6f7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2850607389 shutdown_connections 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2850607389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f881025b0 0x7f6f880fbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2850607389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88101fe0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2850607389 >> 172.21.15.111:0/2850607389 conn(0x7f6f880f7720 msgr2=0x7f6f880f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2850607389 shutdown_connections 2024-03-14T13:28:04.370 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.379+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2850607389 wait complete. 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 Processor -- start 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 -- start start 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f881025b0 0x7f6f88112c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8810c810 con 0x7f6f88101c10 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f8d48a700 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8810c980 con 0x7f6f881025b0 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:45222/0 (socket says 172.21.15.111:45222) 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 -- 172.21.15.111:0/2802957675 learned_addr learned my addr 172.21.15.111:0/2802957675 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 -- 172.21.15.111:0/2802957675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f881025b0 msgr2=0x7f6f88112c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f881025b0 0x7f6f88112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 -- 172.21.15.111:0/2802957675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c005040 con 0x7f6f88101c10 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.380+0000 7f6f86ffd700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009180 tx=0x7f6f7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:04.373 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7c00e040 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8810cc00 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f7c004290 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f7c01c3f0 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8810d140 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.381+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f880fd360 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.382+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 39) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6f7c013790 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.382+0000 7f6f6f7fe700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 0x7f6f7006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.382+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6f7c08cd30 con 0x7f6f88101c10 2024-03-14T13:28:04.374 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.382+0000 7f6f867fc700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 0x7f6f7006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-14T13:28:04.375 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.383+0000 7f6f867fc700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 0x7f6f7006ec70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f78006fd0 tx=0x7f6f78006db0 comp rx=0 tx=0).ready entity=mgr.34104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-14T13:28:04.377 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.387+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6f7c05aa40 con 0x7f6f88101c10 2024-03-14T13:28:04.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.596+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6f880fd360 con 0x7f6f88101c10 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.596+0000 7f6f6f7fe700 1 -- 172.21.15.111:0/2802957675 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 39 entries v86) v1 ==== 81+0+2341 (secure 0 0 0) 0x7f6f7c013420 con 0x7f6f88101c10 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6824/3856946079 2024-03-15T13:09:44.383047+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/4041156471 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3552305380 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/337686092 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/1542503054 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/4041156471 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3123411794 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1484389211 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3030479424 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/337686092 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/2592190224 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/3277050730 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/975462484 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/2394623028 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/661526538 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/1152636518 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3904007449 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3170679579 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/119405983 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/139754802 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:0/1611930537 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1344961412 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1979966098 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/718360470 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/4199526358 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6825/3856946079 2024-03-15T13:09:44.383047+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/1118840918 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6828/1920623081 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.137:6829/1920623081 2024-03-15T13:01:36.643440+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/119405983 2024-03-15T12:51:38.987669+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/2868306717 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3646747677 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6827/1337358162 2024-03-15T13:09:44.383031+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/3203466385 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6800/1542503054 2024-03-15T13:00:23.200618+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3756036511 2024-03-15T12:51:06.184731+0000 2024-03-14T13:28:04.589 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:0/3365856044 2024-03-15T12:53:13.176472+0000 2024-03-14T13:28:04.590 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6801/3203466385 2024-03-15T13:05:24.191564+0000 2024-03-14T13:28:04.590 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111:6826/1337358162 2024-03-15T13:09:44.383031+0000 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 msgr2=0x7f6f7006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 0x7f6f7006ec70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f78006fd0 tx=0x7f6f78006db0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 msgr2=0x7f6f88112680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009180 tx=0x7f6f7c009e30 comp rx=0 tx=0).stop 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 shutdown_connections 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:6800/977525747,v1:172.21.15.111:6801/977525747] conn(0x7f6f7006c7c0 0x7f6f7006ec70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f88101c10 0x7f6f88112680 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 --2- 172.21.15.111:0/2802957675 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f881025b0 0x7f6f88112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 >> 172.21.15.111:0/2802957675 conn(0x7f6f880f7720 msgr2=0x7f6f880f8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 shutdown_connections 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:2024-03-14T13:28:04.599+0000 7f6f8d48a700 1 -- 172.21.15.111:0/2802957675 wait complete. 2024-03-14T13:28:04.591 INFO:teuthology.orchestra.run.smithi111.stderr:listed 39 entries 2024-03-14T13:28:04.601 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi111.front.sepia.ceph.com... 2024-03-14T13:28:04.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-14T13:28:04.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-14T13:28:04.644 INFO:teuthology.orchestra.run:waiting for 300 2024-03-14T13:28:04.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:04 smithi111 ceph-mon[145314]: pgmap v720: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:04.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:04 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1875650530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:28:04.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:04 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/1489113451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:28:04.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:04 smithi111 ceph-mon[145314]: from='client.? 172.21.15.111:0/2802957675' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T13:28:04.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:04 smithi137 ceph-mon[108145]: pgmap v720: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:04.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:04 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1875650530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-14T13:28:04.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:04 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/1489113451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-14T13:28:04.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:04 smithi137 ceph-mon[108145]: from='client.? 172.21.15.111:0/2802957675' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-14T13:28:07.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:06 smithi111 ceph-mon[145314]: pgmap v721: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:07.246 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:06 smithi137 ceph-mon[108145]: pgmap v721: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:09.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:08 smithi111 ceph-mon[145314]: pgmap v722: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:09.246 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:08 smithi137 ceph-mon[108145]: pgmap v722: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:10.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:10.246 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:11.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:11 smithi137 ceph-mon[108145]: pgmap v723: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:12.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:11 smithi111 ceph-mon[145314]: pgmap v723: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:13.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:13 smithi137 ceph-mon[108145]: pgmap v724: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:14.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:13 smithi111 ceph-mon[145314]: pgmap v724: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:15.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:15 smithi137 ceph-mon[108145]: pgmap v725: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:16.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:15 smithi111 ceph-mon[145314]: pgmap v725: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:17.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:17 smithi111 ceph-mon[145314]: pgmap v726: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:17.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:17 smithi137 ceph-mon[108145]: pgmap v726: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:19.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:19 smithi137 ceph-mon[108145]: pgmap v727: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:20.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:19 smithi111 ceph-mon[145314]: pgmap v727: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:21.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:21 smithi137 ceph-mon[108145]: pgmap v728: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:22.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:21 smithi111 ceph-mon[145314]: pgmap v728: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:23.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:23 smithi137 ceph-mon[108145]: pgmap v729: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:24.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:23 smithi111 ceph-mon[145314]: pgmap v729: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:24.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:25.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:25.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:25 smithi137 ceph-mon[108145]: pgmap v730: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:26.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:25 smithi111 ceph-mon[145314]: pgmap v730: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:27.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:27 smithi111 ceph-mon[145314]: pgmap v731: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:27.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:27 smithi137 ceph-mon[108145]: pgmap v731: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:29.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:29 smithi137 ceph-mon[108145]: pgmap v732: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:30.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:29 smithi111 ceph-mon[145314]: pgmap v732: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:31.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:31 smithi137 ceph-mon[108145]: pgmap v733: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:32.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:31 smithi111 ceph-mon[145314]: pgmap v733: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:32.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:32 smithi137 ceph-mon[108145]: pgmap v734: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:33.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:32 smithi111 ceph-mon[145314]: pgmap v734: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:35.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:35 smithi137 ceph-mon[108145]: pgmap v735: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:36.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:35 smithi111 ceph-mon[145314]: pgmap v735: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:37.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:37 smithi111 ceph-mon[145314]: pgmap v736: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:37.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:37 smithi137 ceph-mon[108145]: pgmap v736: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:39.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:39 smithi137 ceph-mon[108145]: pgmap v737: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:39.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:40.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:39 smithi111 ceph-mon[145314]: pgmap v737: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:40.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:41.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:41 smithi137 ceph-mon[108145]: pgmap v738: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:42.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:41 smithi111 ceph-mon[145314]: pgmap v738: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:43.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:43 smithi137 ceph-mon[108145]: pgmap v739: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:44.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:43 smithi111 ceph-mon[145314]: pgmap v739: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:45.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:45 smithi137 ceph-mon[108145]: pgmap v740: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:46.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:45 smithi111 ceph-mon[145314]: pgmap v740: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:47.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:47 smithi111 ceph-mon[145314]: pgmap v741: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:47.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:47 smithi137 ceph-mon[108145]: pgmap v741: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:49.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:49 smithi137 ceph-mon[108145]: pgmap v742: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:50.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:49 smithi111 ceph-mon[145314]: pgmap v742: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:51.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:51 smithi137 ceph-mon[108145]: pgmap v743: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:52.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:51 smithi111 ceph-mon[145314]: pgmap v743: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:53.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:53 smithi137 ceph-mon[108145]: pgmap v744: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:54.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:53 smithi111 ceph-mon[145314]: pgmap v744: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:28:54.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:55.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:28:55.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:55 smithi137 ceph-mon[108145]: pgmap v745: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:55.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:28:55.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:28:55.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:28:55.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:55 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:28:56.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:55 smithi111 ceph-mon[145314]: pgmap v745: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:28:56.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:28:56.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:28:56.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:28:56.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:55 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:28:56.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:56 smithi137 ceph-mon[108145]: pgmap v746: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:57.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:56 smithi111 ceph-mon[145314]: pgmap v746: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:28:59.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:28:58 smithi111 ceph-mon[145314]: pgmap v747: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:28:59.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:28:58 smithi137 ceph-mon[108145]: pgmap v747: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:01.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:01 smithi137 ceph-mon[108145]: pgmap v748: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:02.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:01 smithi111 ceph-mon[145314]: pgmap v748: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:03.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:03 smithi137 ceph-mon[108145]: pgmap v749: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:04.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:03 smithi111 ceph-mon[145314]: pgmap v749: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:05.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:05 smithi137 ceph-mon[108145]: pgmap v750: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:06.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:05 smithi111 ceph-mon[145314]: pgmap v750: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:07.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:07 smithi111 ceph-mon[145314]: pgmap v751: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:07.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:07 smithi137 ceph-mon[108145]: pgmap v751: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:09.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:09 smithi137 ceph-mon[108145]: pgmap v752: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:09.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:10.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:09 smithi111 ceph-mon[145314]: pgmap v752: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:10.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:11.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:11 smithi137 ceph-mon[108145]: pgmap v753: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:12.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:11 smithi111 ceph-mon[145314]: pgmap v753: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:13.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:13 smithi137 ceph-mon[108145]: pgmap v754: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:14.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:13 smithi111 ceph-mon[145314]: pgmap v754: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:15.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:15 smithi137 ceph-mon[108145]: pgmap v755: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:16.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:15 smithi111 ceph-mon[145314]: pgmap v755: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:17.880 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:17 smithi111 ceph-mon[145314]: pgmap v756: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:17.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:17 smithi137 ceph-mon[108145]: pgmap v756: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:19.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:19 smithi137 ceph-mon[108145]: pgmap v757: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:20.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:19 smithi111 ceph-mon[145314]: pgmap v757: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:20.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:20 smithi137 ceph-mon[108145]: pgmap v758: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:21.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:20 smithi111 ceph-mon[145314]: pgmap v758: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:23.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:22 smithi111 ceph-mon[145314]: pgmap v759: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:23.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:22 smithi137 ceph-mon[108145]: pgmap v759: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:25.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:24 smithi111 ceph-mon[145314]: pgmap v760: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:25.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:25.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:24 smithi137 ceph-mon[108145]: pgmap v760: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:25.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:27.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:26 smithi111 ceph-mon[145314]: pgmap v761: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:27.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:26 smithi137 ceph-mon[108145]: pgmap v761: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:29.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:28 smithi111 ceph-mon[145314]: pgmap v762: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:29.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:28 smithi137 ceph-mon[108145]: pgmap v762: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:31.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:31 smithi137 ceph-mon[108145]: pgmap v763: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:32.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:31 smithi111 ceph-mon[145314]: pgmap v763: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:34.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:33 smithi111 ceph-mon[145314]: pgmap v764: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:35.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:33 smithi137 ceph-mon[108145]: pgmap v764: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:35.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:35 smithi137 ceph-mon[108145]: pgmap v765: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:36.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:35 smithi111 ceph-mon[145314]: pgmap v765: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:37.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:37 smithi111 ceph-mon[145314]: pgmap v766: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:37.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:37 smithi137 ceph-mon[108145]: pgmap v766: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:39.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:39 smithi137 ceph-mon[108145]: pgmap v767: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:39.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:40.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:39 smithi111 ceph-mon[145314]: pgmap v767: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:40.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:41.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:41 smithi137 ceph-mon[108145]: pgmap v768: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:42.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:41 smithi111 ceph-mon[145314]: pgmap v768: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:43.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:43 smithi137 ceph-mon[108145]: pgmap v769: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:44.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:43 smithi111 ceph-mon[145314]: pgmap v769: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:45.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:45 smithi137 ceph-mon[108145]: pgmap v770: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:46.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:45 smithi111 ceph-mon[145314]: pgmap v770: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:47.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:47 smithi111 ceph-mon[145314]: pgmap v771: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:47.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:47 smithi137 ceph-mon[108145]: pgmap v771: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:49.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:49 smithi137 ceph-mon[108145]: pgmap v772: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:50.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:49 smithi111 ceph-mon[145314]: pgmap v772: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:51.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:50 smithi111 ceph-mon[145314]: pgmap v773: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:51.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:50 smithi137 ceph-mon[108145]: pgmap v773: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:53.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:52 smithi111 ceph-mon[145314]: pgmap v774: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:53.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:52 smithi137 ceph-mon[108145]: pgmap v774: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:29:55.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:54 smithi111 ceph-mon[145314]: pgmap v775: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:55.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:55.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:54 smithi137 ceph-mon[108145]: pgmap v775: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:29:55.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:29:57.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:56 smithi111 ceph-mon[145314]: pgmap v776: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:57.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:29:57.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:29:57.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:29:57.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:29:57.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:56 smithi137 ceph-mon[108145]: pgmap v776: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:29:57.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:29:57.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:29:57.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:29:57.245 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:29:59.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:29:58 smithi111 ceph-mon[145314]: pgmap v777: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:29:59.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:29:58 smithi137 ceph-mon[108145]: pgmap v777: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:00.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:00 smithi137 ceph-mon[108145]: overall HEALTH_OK 2024-03-14T13:30:01.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:00 smithi111 ceph-mon[145314]: overall HEALTH_OK 2024-03-14T13:30:01.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:01 smithi137 ceph-mon[108145]: pgmap v778: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:02.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:01 smithi111 ceph-mon[145314]: pgmap v778: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:03.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:03 smithi137 ceph-mon[108145]: pgmap v779: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:04.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:03 smithi111 ceph-mon[145314]: pgmap v779: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:05.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:05 smithi137 ceph-mon[108145]: pgmap v780: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:06.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:05 smithi111 ceph-mon[145314]: pgmap v780: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:07.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:07 smithi111 ceph-mon[145314]: pgmap v781: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:07.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:07 smithi137 ceph-mon[108145]: pgmap v781: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:09.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:09 smithi137 ceph-mon[108145]: pgmap v782: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:09.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:10.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:09 smithi111 ceph-mon[145314]: pgmap v782: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:10.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:11.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:11 smithi137 ceph-mon[108145]: pgmap v783: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:12.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:11 smithi111 ceph-mon[145314]: pgmap v783: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:13.995 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:13 smithi137 ceph-mon[108145]: pgmap v784: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:14.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:13 smithi111 ceph-mon[145314]: pgmap v784: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:15.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:15 smithi137 ceph-mon[108145]: pgmap v785: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:16.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:15 smithi111 ceph-mon[145314]: pgmap v785: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:17.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:17 smithi111 ceph-mon[145314]: pgmap v786: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:17.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:17 smithi137 ceph-mon[108145]: pgmap v786: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:18.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:18 smithi137 ceph-mon[108145]: pgmap v787: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:19.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:18 smithi111 ceph-mon[145314]: pgmap v787: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:21.996 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:21 smithi137 ceph-mon[108145]: pgmap v788: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:22.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:21 smithi111 ceph-mon[145314]: pgmap v788: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:23.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:23 smithi137 ceph-mon[108145]: pgmap v789: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:24.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:23 smithi111 ceph-mon[145314]: pgmap v789: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:24.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:25.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:25.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:25 smithi137 ceph-mon[108145]: pgmap v790: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:26.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:25 smithi111 ceph-mon[145314]: pgmap v790: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:27.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:27 smithi111 ceph-mon[145314]: pgmap v791: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:27.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:27 smithi137 ceph-mon[108145]: pgmap v791: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:29.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:29 smithi137 ceph-mon[108145]: pgmap v792: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:30.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:29 smithi111 ceph-mon[145314]: pgmap v792: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:31.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:31 smithi137 ceph-mon[108145]: pgmap v793: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:32.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:31 smithi111 ceph-mon[145314]: pgmap v793: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:33.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:33 smithi137 ceph-mon[108145]: pgmap v794: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:34.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:33 smithi111 ceph-mon[145314]: pgmap v794: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:30:35.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:35 smithi137 ceph-mon[108145]: pgmap v795: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:36.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:35 smithi111 ceph-mon[145314]: pgmap v795: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:37.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:36 smithi111 ceph-mon[145314]: pgmap v796: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:37.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:36 smithi137 ceph-mon[108145]: pgmap v796: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:39.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:38 smithi111 ceph-mon[145314]: pgmap v797: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:39.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:38 smithi137 ceph-mon[108145]: pgmap v797: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:40.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:40.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:41.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:41 smithi137 ceph-mon[108145]: pgmap v798: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:42.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:41 smithi111 ceph-mon[145314]: pgmap v798: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:43.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:43 smithi137 ceph-mon[108145]: pgmap v799: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-14T13:30:44.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:43 smithi111 ceph-mon[145314]: pgmap v799: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-14T13:30:45.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:45 smithi137 ceph-mon[108145]: pgmap v800: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-14T13:30:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:45 smithi111 ceph-mon[145314]: pgmap v800: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 3 op/s 2024-03-14T13:30:47.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:47 smithi111 ceph-mon[145314]: pgmap v801: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:30:47.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:47 smithi137 ceph-mon[108145]: pgmap v801: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:30:49.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:49 smithi137 ceph-mon[108145]: pgmap v802: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:30:50.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:49 smithi111 ceph-mon[145314]: pgmap v802: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-14T13:30:51.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:51 smithi137 ceph-mon[108145]: pgmap v803: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:30:52.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:51 smithi111 ceph-mon[145314]: pgmap v803: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 3 op/s 2024-03-14T13:30:53.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:52 smithi111 ceph-mon[145314]: pgmap v804: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-14T13:30:53.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:52 smithi137 ceph-mon[108145]: pgmap v804: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4 op/s 2024-03-14T13:30:55.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:54 smithi111 ceph-mon[145314]: pgmap v805: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:55.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:55.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:54 smithi137 ceph-mon[108145]: pgmap v805: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:30:55.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:30:57.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:56 smithi111 ceph-mon[145314]: pgmap v806: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:57.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:56 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:30:57.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:56 smithi137 ceph-mon[108145]: pgmap v806: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:30:57.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:56 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:30:57.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:30:57.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:30:57.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:30:58.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:30:58.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:30:58.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:30:59.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:30:58 smithi111 ceph-mon[145314]: pgmap v807: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:30:59.244 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:30:58 smithi137 ceph-mon[108145]: pgmap v807: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:01.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:01 smithi137 ceph-mon[108145]: pgmap v808: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:02.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:01 smithi111 ceph-mon[145314]: pgmap v808: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:03.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:03 smithi137 ceph-mon[108145]: pgmap v809: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:04.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:03 smithi111 ceph-mon[145314]: pgmap v809: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:05.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:05 smithi137 ceph-mon[108145]: pgmap v810: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:06.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:05 smithi111 ceph-mon[145314]: pgmap v810: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:07.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:07 smithi111 ceph-mon[145314]: pgmap v811: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:07.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:07 smithi137 ceph-mon[108145]: pgmap v811: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:09.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:09 smithi137 ceph-mon[108145]: pgmap v812: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:09.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:10.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:09 smithi111 ceph-mon[145314]: pgmap v812: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:10.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:11.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:11 smithi137 ceph-mon[108145]: pgmap v813: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:12.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:11 smithi111 ceph-mon[145314]: pgmap v813: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:13.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:13 smithi137 ceph-mon[108145]: pgmap v814: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:14.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:13 smithi111 ceph-mon[145314]: pgmap v814: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:15.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:15 smithi137 ceph-mon[108145]: pgmap v815: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:16.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:15 smithi111 ceph-mon[145314]: pgmap v815: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:17.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:17 smithi111 ceph-mon[145314]: pgmap v816: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:17.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:17 smithi137 ceph-mon[108145]: pgmap v816: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:19.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:19 smithi137 ceph-mon[108145]: pgmap v817: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:20.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:19 smithi111 ceph-mon[145314]: pgmap v817: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:21.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:21 smithi137 ceph-mon[108145]: pgmap v818: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:22.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:21 smithi111 ceph-mon[145314]: pgmap v818: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:23.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:23 smithi137 ceph-mon[108145]: pgmap v819: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:24.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:23 smithi111 ceph-mon[145314]: pgmap v819: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:24.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:25.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:25.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:25 smithi137 ceph-mon[108145]: pgmap v820: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:26.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:25 smithi111 ceph-mon[145314]: pgmap v820: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:26.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:26 smithi137 ceph-mon[108145]: pgmap v821: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:27.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:26 smithi111 ceph-mon[145314]: pgmap v821: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:29.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:28 smithi111 ceph-mon[145314]: pgmap v822: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:29.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:28 smithi137 ceph-mon[108145]: pgmap v822: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:31.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:31 smithi137 ceph-mon[108145]: pgmap v823: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:32.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:31 smithi111 ceph-mon[145314]: pgmap v823: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:33.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:33 smithi137 ceph-mon[108145]: pgmap v824: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:34.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:33 smithi111 ceph-mon[145314]: pgmap v824: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:35.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:35 smithi137 ceph-mon[108145]: pgmap v825: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:36.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:35 smithi111 ceph-mon[145314]: pgmap v825: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:37.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:37 smithi111 ceph-mon[145314]: pgmap v826: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:37.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:37 smithi137 ceph-mon[108145]: pgmap v826: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:39.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:39 smithi137 ceph-mon[108145]: pgmap v827: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:39.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:40.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:39 smithi111 ceph-mon[145314]: pgmap v827: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:40.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:41.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:40 smithi111 ceph-mon[145314]: pgmap v828: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:41.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:40 smithi137 ceph-mon[108145]: pgmap v828: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:43.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:42 smithi111 ceph-mon[145314]: pgmap v829: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:43.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:42 smithi137 ceph-mon[108145]: pgmap v829: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:45.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:45 smithi137 ceph-mon[108145]: pgmap v830: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:46.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:45 smithi111 ceph-mon[145314]: pgmap v830: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:47.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:47 smithi111 ceph-mon[145314]: pgmap v831: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:47.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:47 smithi137 ceph-mon[108145]: pgmap v831: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:49.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:49 smithi137 ceph-mon[108145]: pgmap v832: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:50.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:49 smithi111 ceph-mon[145314]: pgmap v832: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:51.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:51 smithi137 ceph-mon[108145]: pgmap v833: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:52.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:51 smithi111 ceph-mon[145314]: pgmap v833: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:53.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:53 smithi137 ceph-mon[108145]: pgmap v834: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:54.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:53 smithi111 ceph-mon[145314]: pgmap v834: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:31:54.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:55.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:31:55.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:55 smithi137 ceph-mon[108145]: pgmap v835: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:56.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:55 smithi111 ceph-mon[145314]: pgmap v835: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:31:57.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:56 smithi111 ceph-mon[145314]: pgmap v836: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:57.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:56 smithi137 ceph-mon[108145]: pgmap v836: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:31:57.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:31:57.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:31:57.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:31:57.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:31:58.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:31:58.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:31:58.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:31:58.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:31:59.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:31:58 smithi111 ceph-mon[145314]: pgmap v837: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:31:59.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:31:58 smithi137 ceph-mon[108145]: pgmap v837: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:01.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:01 smithi137 ceph-mon[108145]: pgmap v838: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:02.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:01 smithi111 ceph-mon[145314]: pgmap v838: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:03.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:03 smithi137 ceph-mon[108145]: pgmap v839: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:04.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:03 smithi111 ceph-mon[145314]: pgmap v839: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:05.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:05 smithi137 ceph-mon[108145]: pgmap v840: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:06.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:05 smithi111 ceph-mon[145314]: pgmap v840: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:07.878 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:07 smithi111 ceph-mon[145314]: pgmap v841: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:07.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:07 smithi137 ceph-mon[108145]: pgmap v841: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:09.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:09 smithi137 ceph-mon[108145]: pgmap v842: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:09.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:09 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:10.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:09 smithi111 ceph-mon[145314]: pgmap v842: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:10.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:09 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:11.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:11 smithi137 ceph-mon[108145]: pgmap v843: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:12.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:11 smithi111 ceph-mon[145314]: pgmap v843: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:13.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:12 smithi111 ceph-mon[145314]: pgmap v844: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:13.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:12 smithi137 ceph-mon[108145]: pgmap v844: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:15.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:15 smithi137 ceph-mon[108145]: pgmap v845: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:16.129 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:15 smithi111 ceph-mon[145314]: pgmap v845: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:17.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:17 smithi111 ceph-mon[145314]: pgmap v846: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:17.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:17 smithi137 ceph-mon[108145]: pgmap v846: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:19.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:19 smithi137 ceph-mon[108145]: pgmap v847: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:20.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:19 smithi111 ceph-mon[145314]: pgmap v847: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:21.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:21 smithi137 ceph-mon[108145]: pgmap v848: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:22.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:21 smithi111 ceph-mon[145314]: pgmap v848: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:23.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:23 smithi137 ceph-mon[108145]: pgmap v849: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:24.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:23 smithi111 ceph-mon[145314]: pgmap v849: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:24.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:24 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:25.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:24 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:25.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:25 smithi137 ceph-mon[108145]: pgmap v850: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:26.130 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:25 smithi111 ceph-mon[145314]: pgmap v850: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:26.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:26 smithi137 ceph-mon[108145]: pgmap v851: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:27.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:26 smithi111 ceph-mon[145314]: pgmap v851: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:29.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:28 smithi111 ceph-mon[145314]: pgmap v852: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:29.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:28 smithi137 ceph-mon[108145]: pgmap v852: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:31.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:31 smithi137 ceph-mon[108145]: pgmap v853: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:32.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:31 smithi111 ceph-mon[145314]: pgmap v853: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:33.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:33 smithi137 ceph-mon[108145]: pgmap v854: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:34.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:33 smithi111 ceph-mon[145314]: pgmap v854: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:35.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:35 smithi137 ceph-mon[108145]: pgmap v855: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:36.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:35 smithi111 ceph-mon[145314]: pgmap v855: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:37.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:37 smithi111 ceph-mon[145314]: pgmap v856: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:37.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:37 smithi137 ceph-mon[108145]: pgmap v856: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:39.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:39 smithi137 ceph-mon[108145]: pgmap v857: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:39.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:39 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:40.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:39 smithi111 ceph-mon[145314]: pgmap v857: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:40.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:39 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:41.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:41 smithi137 ceph-mon[108145]: pgmap v858: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:42.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:41 smithi111 ceph-mon[145314]: pgmap v858: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:42.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:42 smithi137 ceph-mon[108145]: pgmap v859: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:43.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:42 smithi111 ceph-mon[145314]: pgmap v859: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:45.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:45 smithi137 ceph-mon[108145]: pgmap v860: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:46.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:45 smithi111 ceph-mon[145314]: pgmap v860: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:47.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:47 smithi111 ceph-mon[145314]: pgmap v861: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:47.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:47 smithi137 ceph-mon[108145]: pgmap v861: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:49.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:49 smithi137 ceph-mon[108145]: pgmap v862: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:50.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:49 smithi111 ceph-mon[145314]: pgmap v862: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:51.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:51 smithi137 ceph-mon[108145]: pgmap v863: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:52.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:51 smithi111 ceph-mon[145314]: pgmap v863: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:53.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:52 smithi111 ceph-mon[145314]: pgmap v864: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:53.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:52 smithi137 ceph-mon[108145]: pgmap v864: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:32:55.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:54 smithi111 ceph-mon[145314]: pgmap v865: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:55.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:54 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:55.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:54 smithi137 ceph-mon[108145]: pgmap v865: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-14T13:32:55.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:54 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-14T13:32:57.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:56 smithi111 ceph-mon[145314]: pgmap v866: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:57.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:56 smithi137 ceph-mon[108145]: pgmap v866: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:32:58.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:57 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:32:58.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:57 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-14T13:32:59.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:58 smithi111 ceph-mon[145314]: pgmap v867: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:59.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:32:59.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:32:59.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:32:58 smithi111 ceph-mon[145314]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:32:59.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:58 smithi137 ceph-mon[108145]: pgmap v867: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-14T13:32:59.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-14T13:32:59.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-14T13:32:59.243 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:32:58 smithi137 ceph-mon[108145]: from='mgr.34104 172.21.15.111:0/1693371532' entity='mgr.smithi111.atysez' 2024-03-14T13:33:01.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:01 smithi137 ceph-mon[108145]: pgmap v868: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:33:02.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:01 smithi111 ceph-mon[145314]: pgmap v868: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-14T13:33:03.711 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-03-14T13:33:03.712 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-14T13:33:03.714 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-14T13:33:03.727 INFO:tasks.cephadm:Teardown begin 2024-03-14T13:33:03.727 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/cephadm.py", line 1750, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-14T13:33:03.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T13:33:03.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T13:33:03.804 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-14T13:33:03.804 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-14T13:33:03.826 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-14T13:33:03.865 INFO:tasks.cephadm:Stopping all daemons... 2024-03-14T13:33:03.865 INFO:tasks.cephadm.mon.smithi111:Stopping mon.smithi111... 2024-03-14T13:33:03.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111 2024-03-14T13:33:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:03 smithi137 ceph-mon[108145]: pgmap v869: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:33:04.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:03 smithi111 ceph-mon[145314]: pgmap v869: 65 pgs: 65 active+clean; 255 MiB data, 979 MiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-14T13:33:04.127 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:03 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:04.520 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:04 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:33:04.146+0000 7eff57e95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:04.520 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:04 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111[145291]: 2024-03-14T13:33:04.146+0000 7eff57e95700 -1 mon.smithi111@0(leader) e3 *** Got Signal Terminated *** 2024-03-14T13:33:04.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Mar 14 13:33:04 smithi111 bash[202795]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi111 2024-03-14T13:33:05.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi111.service' 2024-03-14T13:33:05.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:05.145 INFO:tasks.cephadm.mon.smithi111:Stopped mon.smithi111 2024-03-14T13:33:05.146 INFO:tasks.cephadm.mon.smithi137:Stopping mon.smithi137... 2024-03-14T13:33:05.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi137 2024-03-14T13:33:05.467 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:05 smithi137 systemd[1]: Stopping Ceph mon.smithi137 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:05.742 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:05 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137[108122]: 2024-03-14T13:33:05.478+0000 7f0038e82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi137 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:05.742 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:05 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137[108122]: 2024-03-14T13:33:05.478+0000 7f0038e82700 -1 mon.smithi137@1(peon) e3 *** Got Signal Terminated *** 2024-03-14T13:33:06.242 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Mar 14 13:33:06 smithi137 bash[152452]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-mon-smithi137 2024-03-14T13:33:06.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@mon.smithi137.service' 2024-03-14T13:33:06.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:06.643 INFO:tasks.cephadm.mon.smithi137:Stopped mon.smithi137 2024-03-14T13:33:06.643 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-14T13:33:06.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.0 2024-03-14T13:33:06.947 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:06 smithi111 systemd[1]: Stopping Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:07.377 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:06 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:33:06.960+0000 7f7560ae5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:07.377 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:06 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:33:06.960+0000 7f7560ae5700 -1 osd.0 86 *** Got signal Terminated *** 2024-03-14T13:33:07.377 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:06 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0[159804]: 2024-03-14T13:33:06.960+0000 7f7560ae5700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:13.127 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:12 smithi111 bash[202955]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-0 2024-03-14T13:33:14.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.0.service' 2024-03-14T13:33:14.626 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:14 smithi111 systemd[1]: ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.0.service: Succeeded. 2024-03-14T13:33:14.626 INFO:journalctl@ceph.osd.0.smithi111.stdout:Mar 14 13:33:14 smithi111 systemd[1]: Stopped Ceph osd.0 for 591fecbc-e201-11ee-95c9-87774f69a715. 2024-03-14T13:33:14.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:14.659 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-14T13:33:14.659 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-14T13:33:14.659 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.1 2024-03-14T13:33:14.967 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:33:14 smithi111 systemd[1]: Stopping Ceph osd.1 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:15.377 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:33:14 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:33:14.979+0000 7f4fa339a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:15.377 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:33:14 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:33:14.979+0000 7f4fa339a700 -1 osd.1 86 *** Got signal Terminated *** 2024-03-14T13:33:15.377 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:33:14 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1[166481]: 2024-03-14T13:33:14.979+0000 7f4fa339a700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:21.127 INFO:journalctl@ceph.osd.1.smithi111.stdout:Mar 14 13:33:20 smithi111 bash[203232]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-1 2024-03-14T13:33:22.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.1.service' 2024-03-14T13:33:22.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:22.661 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-14T13:33:22.661 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-14T13:33:22.661 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.2 2024-03-14T13:33:22.877 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:33:22 smithi111 systemd[1]: Stopping Ceph osd.2 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:23.377 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:33:23 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:33:23.095+0000 7f8f0109b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:23.377 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:33:23 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:33:23.095+0000 7f8f0109b700 -1 osd.2 86 *** Got signal Terminated *** 2024-03-14T13:33:23.377 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:33:23 smithi111 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2[174052]: 2024-03-14T13:33:23.095+0000 7f8f0109b700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:29.377 INFO:journalctl@ceph.osd.2.smithi111.stdout:Mar 14 13:33:29 smithi111 bash[203497]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-2 2024-03-14T13:33:30.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.2.service' 2024-03-14T13:33:30.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:30.948 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-14T13:33:30.948 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-14T13:33:30.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.3 2024-03-14T13:33:31.243 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:33:31 smithi137 systemd[1]: Stopping Ceph osd.3 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:31.742 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:33:31 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:33:31.267+0000 7f4b955b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:31.742 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:33:31 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:33:31.267+0000 7f4b955b8700 -1 osd.3 86 *** Got signal Terminated *** 2024-03-14T13:33:31.743 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:33:31 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3[121143]: 2024-03-14T13:33:31.267+0000 7f4b955b8700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:35.303 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:35 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:35.019+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:35.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:35 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:35.317+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:36.306 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:36 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:36.021+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:36.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:36 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:36.319+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:37.294 INFO:journalctl@ceph.osd.3.smithi137.stdout:Mar 14 13:33:37 smithi137 bash[152608]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-3 2024-03-14T13:33:37.294 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:37 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:37.014+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:37.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:37 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:37.307+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:38.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:38 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:38.044+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:38.718 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:38 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:38.343+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:39.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.3.service' 2024-03-14T13:33:39.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:39.053 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-14T13:33:39.053 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-14T13:33:39.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.4 2024-03-14T13:33:39.306 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:39 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:39.049+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:39.306 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:39 smithi137 systemd[1]: Stopping Ceph osd.4 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:39.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:39 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:39.319+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:39.743 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:39 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:39.361+0000 7fc91fed4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:39.743 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:39 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:39.361+0000 7fc91fed4700 -1 osd.4 86 *** Got signal Terminated *** 2024-03-14T13:33:39.743 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:39 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:39.361+0000 7fc91fed4700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:40.334 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:40 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:40.011+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:40.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:40 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:40.347+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:41.283 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:41 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:41.020+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:41.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:41 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:41.298+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:42.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:41 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:41.993+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:42.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:41 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:41.993+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:17.753983+0000 front 2024-03-14T13:33:17.754009+0000 (oldest deadline 2024-03-14T13:33:41.853928+0000) 2024-03-14T13:33:42.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:42 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:42.262+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:43.294 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:43 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:43.033+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:43.294 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:43 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:43.033+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:17.753983+0000 front 2024-03-14T13:33:17.754009+0000 (oldest deadline 2024-03-14T13:33:41.853928+0000) 2024-03-14T13:33:43.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:43 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:43.308+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:44.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:43 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:43.991+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.753338+0000 front 2024-03-14T13:33:10.753275+0000 (oldest deadline 2024-03-14T13:33:34.253193+0000) 2024-03-14T13:33:44.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:43 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4[128029]: 2024-03-14T13:33:43.991+0000 7fc91a4c6700 -1 osd.4 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:17.753983+0000 front 2024-03-14T13:33:17.754009+0000 (oldest deadline 2024-03-14T13:33:41.853928+0000) 2024-03-14T13:33:44.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:44 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:44.259+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:44.742 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:44 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:44.259+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:45.242 INFO:journalctl@ceph.osd.4.smithi137.stdout:Mar 14 13:33:45 smithi137 bash[152873]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-4 2024-03-14T13:33:45.681 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:45 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:45.268+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:45.681 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:45 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:45.268+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:46.392 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:46 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:46.282+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:46.392 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:46 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:46.282+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:46.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.4.service' 2024-03-14T13:33:46.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:46.719 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-14T13:33:46.719 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-14T13:33:46.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.5 2024-03-14T13:33:46.992 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:46 smithi137 systemd[1]: Stopping Ceph osd.5 for 591fecbc-e201-11ee-95c9-87774f69a715... 2024-03-14T13:33:47.255 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:47 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:47.015+0000 7f06402fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-14T13:33:47.255 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:47 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:47.015+0000 7f06402fe700 -1 osd.5 86 *** Got signal Terminated *** 2024-03-14T13:33:47.255 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:47 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:47.015+0000 7f06402fe700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-14T13:33:47.255 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:47 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:47.269+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:47.255 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:47 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:47.269+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:48.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:48 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:48.235+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:48.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:48 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:48.235+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:49.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:49 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:49.189+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:49.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:49 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:49.189+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:50.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:50 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:50.166+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:50.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:50 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:50.166+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:51.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:51 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:51.151+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-03-14T13:33:10.365467+0000 front 2024-03-14T13:33:10.365400+0000 (oldest deadline 2024-03-14T13:33:35.065404+0000) 2024-03-14T13:33:51.492 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:51 smithi137 ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5[134141]: 2024-03-14T13:33:51.151+0000 7f063b0f1700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-03-14T13:33:19.666883+0000 front 2024-03-14T13:33:19.666925+0000 (oldest deadline 2024-03-14T13:33:43.766647+0000) 2024-03-14T13:33:52.992 INFO:journalctl@ceph.osd.5.smithi137.stdout:Mar 14 13:33:52 smithi137 bash[153136]: ceph-591fecbc-e201-11ee-95c9-87774f69a715-osd-5 2024-03-14T13:33:54.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-591fecbc-e201-11ee-95c9-87774f69a715@osd.5.service' 2024-03-14T13:33:54.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-14T13:33:54.468 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-14T13:33:54.468 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 591fecbc-e201-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-14T13:33:57.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi111.stderr:ceph-fuse[100002]: fuse finished with error 0 and tester_r 0 2024-03-14T13:34:15.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 591fecbc-e201-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-14T13:34:27.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T13:34:27.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-14T13:34:27.175 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-14T13:34:27.177 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi111/crash 2024-03-14T13:34:27.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash -- . 2024-03-14T13:34:27.214 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-14T13:34:27.214 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-03-14T13:34:27.216 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi137/crash 2024-03-14T13:34:27.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash -- . 2024-03-14T13:34:27.243 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/591fecbc-e201-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-14T13:34:27.243 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2024-03-14T13:34:27.244 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-14T13:34:27.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/591fecbc-e201-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-14T13:34:27.288 INFO:teuthology.orchestra.run.smithi111.stderr:grep: /var/log/ceph/591fecbc-e201-11ee-95c9-87774f69a715/ceph.log: No such file or directory 2024-03-14T13:34:27.290 INFO:tasks.cephadm:Compressing logs... 2024-03-14T13:34:27.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T13:34:27.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T13:34:27.358 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-14T13:34:27.359 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-14T13:34:45.574 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/ceph-client.1.log: file size changed while zipping 2024-03-14T13:36:07.348 INFO:tasks.cephadm:Archiving logs... 2024-03-14T13:36:07.348 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi111/log 2024-03-14T13:36:07.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-14T13:36:19.487 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi137/log 2024-03-14T13:36:19.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-14T13:36:31.050 INFO:tasks.cephadm:Removing cluster... 2024-03-14T13:36:31.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 591fecbc-e201-11ee-95c9-87774f69a715 --force 2024-03-14T13:36:31.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 591fecbc-e201-11ee-95c9-87774f69a715 --force 2024-03-14T13:36:32.121 INFO:tasks.cephadm:Removing cephadm ... 2024-03-14T13:36:32.121 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-14T13:36:32.141 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-14T13:36:32.159 INFO:tasks.cephadm:Teardown complete 2024-03-14T13:36:32.159 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-14T13:36:32.175 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-14T13:36:32.175 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-14T13:36:32.176 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-14T13:36:32.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-14T13:36:32.276 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-14T13:36:32.288 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-14T13:36:32.288 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T13:36:32.291 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-14T13:36:32.306 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-03-14T13:36:32.307 INFO:teuthology.orchestra.run.smithi137.stderr:bash: ntpq: command not found 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +191us[ +173us] +/- 100ms 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1032us[-1050us] +/- 64ms 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +1093us[+1076us] +/- 66ms 2024-03-14T13:36:32.311 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +177us[ +168us] +/- 100ms 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1038us[-1048us] +/- 64ms 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +1079us[+1069us] +/- 66ms 2024-03-14T13:36:32.313 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-14T13:36:32.314 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-14T13:36:32.325 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-14T13:36:32.325 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-14T13:36:32.363 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-14T13:36:32.442 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-14T13:36:32.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T13:36:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T13:36:32.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-14T13:36:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T13:36:32.576 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-14T13:36:32.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-14T13:36:32.613 INFO:teuthology.task.internal:Duration was 3451.599083 seconds 2024-03-14T13:36:32.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-14T13:36:32.646 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-14T13:36:32.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-14T13:36:32.649 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-14T13:36:32.693 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T13:36:32.695 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-14T13:36:32.887 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-14T13:36:32.887 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-03-14T13:36:32.887 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-14T13:36:32.909 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-03-14T13:36:32.910 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-14T13:36:32.933 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-14T13:36:32.933 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T13:36:32.955 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-14T13:36:33.029 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-14T13:36:33.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-14T13:36:33.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-14T13:36:33.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-14T13:36:33.175 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-14T13:36:33.175 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-14T13:36:33.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-14T13:36:33.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-14T13:36:33.241 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-14T13:36:33.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-14T13:36:33.268 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-03-14T13:36:33.296 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-03-14T13:36:33.314 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-14T13:36:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T13:36:33.353 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-14T13:36:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T13:36:33.371 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-14T13:36:33.384 INFO:teuthology.task.internal:Transferring archived files... 2024-03-14T13:36:33.385 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi111 2024-03-14T13:36:33.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-14T13:36:33.438 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807/remote/smithi137 2024-03-14T13:36:33.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-14T13:36:33.480 INFO:teuthology.task.internal:Removing archive directory... 2024-03-14T13:36:33.481 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-14T13:36:33.483 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-14T13:36:33.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-14T13:36:33.546 INFO:teuthology.task.internal:Not uploading archives. 2024-03-14T13:36:33.546 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-14T13:36:33.584 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-14T13:36:33.585 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-14T13:36:33.587 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-14T13:36:33.603 INFO:teuthology.orchestra.run.smithi111.stdout: 525357 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 14 13:36 /home/ubuntu/cephtest 2024-03-14T13:36:33.603 INFO:teuthology.orchestra.run.smithi111.stdout: 527611 4 d--------- 2 ubuntu ubuntu 4096 Mar 14 12:58 /home/ubuntu/cephtest/mnt.0 2024-03-14T13:36:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-14T13:36:33.604 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-03-14T13:36:33.604 INFO:teuthology.orchestra.run.smithi111.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-14T13:36:33.604 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_2f465319e3c83d48f999325459bcdfd0346449c0/qa/tasks/cephfs/fuse_mount.py", line 400, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-14T13:36:33.605 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-14T13:36:33.624 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-14T13:36:33.657 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-03-14T13:36:33.657 INFO:teuthology.nuke:Checking targets against current locks 2024-03-14T13:36:33.684 DEBUG:teuthology.nuke:shortname: smithi111 2024-03-14T13:36:33.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T13:36:33.710 DEBUG:teuthology.nuke:shortname: smithi137 2024-03-14T13:36:33.711 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-14T13:36:33.714 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 12:30:11.944335', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCiGeaTlEBDf3fT4y2Tk4oW9BNNEBI8QQBc7u0GkFK/HgXXXMYiW2qApndruZ1euBW5ZTI8V5MAjiFnTvyj9wFw='} 2024-03-14T13:36:33.739 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-13_19:31:43-fs-wip-yuri-testing-2024-03-12-1240-reef-distro-default-smithi/7598807', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-14 12:30:11.942737', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDwDu9CRd7MFONPpr3DW2WyFZg0XARx+QOWfcPT3N/MZZCnEq1K3ArSnz/UQqirSlx11E4o71e2gumdqoWLGqZQ='} 2024-03-14T13:36:33.771 INFO:teuthology.orchestra.console.smithi111:Power off 2024-03-14T13:36:33.771 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-14T13:36:33.794 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-03-14T13:36:33.795 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:33.810 INFO:teuthology.orchestra.console.smithi137:Power off 2024-03-14T13:36:33.810 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-14T13:36:33.824 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-14T13:36:33.833 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-03-14T13:36:33.834 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:33.859 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-14T13:36:37.826 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:37.859 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:37.936 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-03-14T13:36:37.969 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-03-14T13:36:41.938 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:41.970 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-14T13:36:42.049 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-03-14T13:36:42.079 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-03-14T13:36:42.150 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-03-14T13:36:42.180 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-03-14T13:36:42.285 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3451.5990829467773 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-14T13:36:42.285 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-14T13:36:42.378 INFO:teuthology.run:FAIL